Patched but still vulnerable: Is `apt update && upgrade` enough?

You SSH into a box, run sudo apt update && sudo apt upgrade -y, see “0 upgraded, 0 newly installed”, and close the terminal. The system is patched. The CVE is closed. You move on.

That system is, in many cases, still vulnerable.

apt is a package manager. Its job is to put new files on disk and update package metadata. That is all it does. It does not unload kernel modules from memory. It does not relink shared libraries that are already mapped into running processes. It does not reload microcode from EEPROM into the CPU. It does not rebuild your container images. Every one of those things is your responsibility, and every one of them is a place where “I patched it” silently turns into “I did not, in fact, patch it.”

This article walks through every category of patch that apt upgrade does not finish, how to detect what’s still vulnerable, and how to automate the gap closure.

What apt upgrade Actually Does

When you run apt upgrade, dpkg replaces files on disk and runs the package’s postinst script. The postinst is allowed to restart services it owns — and most well-maintained packages do. nginx, postgres, sshd, openssh-server: their postinst scripts will restart the daemon as part of the upgrade.

That sounds complete, until you realize what postinst cannot do:

Every row in the right column is a place where the old, vulnerable code is still resident in memory and still serving requests, even though the on-disk version is fixed.

Category 1: The Kernel — The Most Common Hidden Vulnerability

Kernel CVEs are the textbook example. When apt installs linux-image-6.8.0-58-generic, it puts the new kernel image into /boot/vmlinuz-6.8.0-58-generic and updates grub. The kernel that is currently scheduling your processes, however, is whatever was loaded by the bootloader the last time the machine started — possibly a kernel from six months ago with a remotely exploitable nf_tables UAF.

Detection

Ubuntu and Debian set a flag file the moment any package’s postinst decides a reboot is needed:

1
2
3
4
5

# The flag itself
ls -la /var/run/reboot-required

# Which packages triggered it
cat /var/run/reboot-required.pkgs

Compare the running kernel to the installed kernel:

1
2
3
4
5
6
7
8
9

# Currently running
uname -r
# 6.8.0-52-generic

# Latest installed
dpkg -l 'linux-image-*' | awk '/^ii/ {print $2}' | sort -V | tail -n1
# linux-image-6.8.0-58-generic

# If they don't match — you have a kernel waiting on a reboot.

Check the boot history to see how long you’ve been running an old kernel:

1
2
3

journalctl --list-boots
last -x reboot | head -5
uptime -p

An uptime of 200 days on a server that auto-installs kernels every week means you have ~25 kernels worth of CVEs you haven’t actually applied.

Live Patching: The Only Real Workaround

If you cannot reboot — and many production fleets cannot, casually — live patching is the only honest answer. There are four serious options:

Enabling Canonical Livepatch on Ubuntu:

1
2
3
4
5
6
7
8
9

# Enable Ubuntu Pro (free for personal use, requires registration)
sudo pro attach <token>

# Enable livepatch
sudo pro enable livepatch

# Verify
sudo pro status
canonical-livepatch status --verbose

After this, kernel CVEs deemed live-patchable are applied to the running kernel without a reboot, usually within hours of upstream release.

Category 2: Shared Libraries — The Silent Killer

This is the category that bites the most people, because nothing flags it. Imagine the following sequence:

1. nginx starts. The dynamic linker loads /usr/lib/x86_64-linux-gnu/libssl.so.3 from disk into nginx’s address space.
2. A CVE drops in OpenSSL. You run apt upgrade. dpkg replaces the file on disk.
3. nginx is still running. Its memory image still contains the old, vulnerable libssl code, mapped from the now-deleted inode. New requests are served by the vulnerable code path.
4. /var/run/reboot-required is not set, because dpkg has no way to know nginx loaded the library.

You will pass any “is libssl patched?” check that looks at the on-disk file. You will fail any actual exploitation test, because the running process still holds the old version.

Detection

The Linux kernel keeps deleted-but-still-mapped files visible via /proc/<pid>/maps and lsof. They show up as (deleted) or with the DEL marker:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Every running process that has a mapped file marked as deleted
sudo lsof +c 0 2>/dev/null | grep -E 'DEL.*lib' | awk '{print $1, $2, $NF}' | sort -u

# Same idea, more focused — list only the processes (not every mapping)
sudo ss -tulnp 2>/dev/null | awk '/users:/ {print $NF}'

# Per-process deep check
for pid in $(pgrep -x nginx); do
  echo "=== nginx PID $pid ==="
  sudo cat /proc/$pid/maps | grep -E '\(deleted\)' | awk '{print $NF}' | sort -u
done

But the right tool already exists: needrestart. On Ubuntu 22.04+ it’s installed by default and runs automatically after every apt upgrade. If it isn’t, install it:

1

sudo apt install needrestart

Using needrestart

1
2
3
4
5
6
7
8

# Show every service/daemon running with stale code, plus the kernel status
sudo needrestart

# Kernel-only check (-k), batch mode (-b)
sudo needrestart -k -b

# Restart services automatically without prompting (use with care!)
sudo needrestart -r a

Sample output after a libssl upgrade with no manual restart:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

Restarting services...
 systemctl restart packagekit.service
 systemctl restart polkit.service

Service restarts being deferred:
 systemctl restart nginx.service
 systemctl restart postgresql@16-main.service
 systemctl restart ssh.service

No containers need to be restarted.

The “deferred” services are exactly the ones you need to restart manually — needrestart flagged them, but its default policy is to ask before touching network-facing services. If you want it fully automatic, edit /etc/needrestart/needrestart.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# /etc/needrestart/needrestart.conf

# Restart mode: (l)ist only, (i)nteractive, or (a)utomatic
$nrconf{restart} = 'a';

# Also check the kernel
$nrconf{kernelhints} = -1;

# Don't ask about microcode either
$nrconf{ucodehints} = 1;

# Blacklist: services that should NEVER be auto-restarted (DBs, brokers, anything with state)
$nrconf{blacklist_rc} = [
    qr(^postgresql),
    qr(^mysql),
    qr(^mariadb),
    qr(^redis),
    qr(^kafka),
];

The blacklist is where you should be careful: stateful services need a coordinated restart strategy, not a systemctl restart from a cron job at 3am.

Category 3: Microcode — The Patch You Forget Exists

CPU microcode patches ship as Debian packages: intel-microcode and amd64-microcode. They patch silicon-level bugs (Spectre, Meltdown, Downfall, Reptar, the entire MDS/L1TF/SRBDS family, and more recently INCEPTION, RetBleed, ZenBleed). When apt upgrade installs a new microcode package, it writes the new blobs to /lib/firmware/intel-ucode/ or /lib/firmware/amd-ucode/. The CPU does not load them until next boot.

Worse: some CPUs only accept microcode updates very early in boot (loaded by the bootloader via the initrd’s microcode CPIO header), so a warm reboot may not be enough — you may need a power cycle to load a brand-new revision.

Detection

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Currently loaded microcode revision
grep microcode /proc/cpuinfo | uniq
# microcode       : 0x129

# Check what kernel sees about microcode loading
dmesg | grep -iE 'microcode|ucode' | head -20

# Compare to what's installed on disk
dpkg -l intel-microcode amd64-microcode 2>/dev/null
ls -la /lib/firmware/intel-ucode/ /lib/firmware/amd-ucode/ 2>/dev/null

If /proc/cpuinfo shows a revision lower than the latest in /lib/firmware, your CPU is still running the old microcode. None of the speculation-side mitigations the kernel reports as “active” actually do what they say without the corresponding microcode loaded.

1
2

# Side-channel mitigation status (each file = one CVE class)
grep . /sys/devices/system/cpu/vulnerabilities/*

If you see lines like Mitigation: Microcode for one issue and Vulnerable: No microcode for another, that’s a direct signal that a microcode update is sitting on disk, unused.

Category 4: systemd, dbus, and Other PID-1-adjacent Services

systemd is PID 1. You cannot systemctl restart systemd. When apt upgrade ships a new systemd binary, it can’t restart itself — it can only re-execute itself in place:

1
2
3
4
5

# Re-exec systemd (loads the new binary, keeps state)
sudo systemctl daemon-reexec

# Reload unit files (much weaker — just re-reads the config)
sudo systemctl daemon-reload

needrestart will tell you when systemd needs a re-exec. If you ignore it, systemd-managed services will continue running, but the PID-1 process itself is the old, possibly vulnerable binary, and any systemctl command may behave unexpectedly because the systemctl client is talking to a stale daemon.

The same goes for dbus, polkit, and udev. These have direct security implications — polkit in particular has been responsible for several local-root CVEs (pwnkit, etc.), and a missed polkit restart after upgrade leaves the vulnerability exploitable.

Category 5: Containers — Host Patching Does Nothing

This is the biggest blind spot in modern infrastructure. You patched the host. The host’s libssl is fresh. But your containers are still running images built six months ago, and container userspace is fully isolated from the host’s userspace. Every container has its own copy of glibc, openssl, libcurl, libxml2, etc.

1
2
3
4
5
6
7

# What the host has
dpkg -l libssl3 2>/dev/null

# What the container has — completely different
docker exec mynginx dpkg -l libssl3 2>/dev/null
# OR for Alpine-based images
docker exec mynginx apk info -v openssl 2>/dev/null

If the host shows 3.0.13-0ubuntu3.5 and the container shows 3.0.2-0ubuntu1.18, the container is vulnerable to every OpenSSL CVE published in the last 18 months — regardless of what apt upgrade did on the host.

The Only Real Fix: Rebuild the Image

1
2
3
4
5
6
7
8

# Pull the latest base image
docker pull nginx:1.27-alpine

# Rebuild your image — this re-runs apt-get/apk update layers
docker compose build --pull --no-cache

# Roll out
docker compose up -d

In a CI/CD pipeline, this should be automated:

1. Scheduled job (daily or on base-image webhook) runs docker pull for every base image you depend on.
2. If any base image changed, trigger a CI build of every downstream image.
3. Run a vulnerability scanner (trivy image, grype, docker scout) against the freshly built image.
4. If the scan passes the policy, push to the registry and trigger a rolling update.

A static nginx:1.21 tag pinned in your Dockerfile for stability is also a static collection of every CVE in nginx 1.21’s dependencies. Pin to a digest, but track the tag’s evolution and rebuild when the upstream digest changes.

I’ve covered the operational side of this in detail in Vulnerability Management: Where to Start and Why Automation Is Non-Negotiable.

Putting It All Together: The Post-Upgrade Verification Script

Here’s a single script you can drop into /usr/local/sbin/post-upgrade-check.sh and run after every upgrade (or via cron):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

#!/usr/bin/env bash
# post-upgrade-check.sh — flag everything `apt upgrade` left undone
set -euo pipefail

red()   { printf '\033[31m%s\033[0m\n' "$*"; }
green() { printf '\033[32m%s\033[0m\n' "$*"; }
yellow(){ printf '\033[33m%s\033[0m\n' "$*"; }

echo "=== Reboot flag ==="
if [[ -f /var/run/reboot-required ]]; then
  red "REBOOT REQUIRED"
  [[ -f /var/run/reboot-required.pkgs ]] && cat /var/run/reboot-required.pkgs
else
  green "No reboot flag set (but check the rest of this report anyway)"
fi

echo
echo "=== Kernel mismatch ==="
running="$(uname -r)"
installed="$(dpkg -l 'linux-image-[0-9]*' 2>/dev/null | awk '/^ii/ {print $2}' | \
             sed 's/linux-image-//' | sort -V | tail -n1)"
if [[ "$running" != "$installed" ]]; then
  red "Running: $running"
  red "Installed: $installed"
  red "→ Reboot to load the patched kernel"
else
  green "Running kernel matches latest installed: $running"
fi

echo
echo "=== Microcode ==="
loaded="$(grep -m1 microcode /proc/cpuinfo | awk '{print $3}')"
echo "Loaded microcode revision: $loaded"
if dmesg 2>/dev/null | grep -qi 'microcode updated early'; then
  green "Microcode loaded early at boot"
else
  yellow "No 'microcode updated early' message — verify intel-microcode/amd64-microcode is current"
fi

echo
echo "=== Speculation mitigations ==="
for f in /sys/devices/system/cpu/vulnerabilities/*; do
  status="$(cat "$f")"
  case "$status" in
    *Vulnerable*)         red    "$(basename "$f"): $status" ;;
    *Mitigation*)         green  "$(basename "$f"): $status" ;;
    *Not\ affected*)      green  "$(basename "$f"): $status" ;;
    *)                    yellow "$(basename "$f"): $status" ;;
  esac
done

echo
echo "=== Processes with deleted libraries ==="
mapfile -t stale < <(sudo lsof +c 0 2>/dev/null | awk '/DEL.*lib/ {print $1"/"$2}' | sort -u)
if (( ${#stale[@]} > 0 )); then
  red "Processes still mapping deleted libraries (need restart):"
  printf '  %s\n' "${stale[@]}"
else
  green "No processes are holding deleted libraries"
fi

echo
echo "=== needrestart ==="
if command -v needrestart >/dev/null 2>&1; then
  sudo needrestart -b -k -l 2>&1 || true
else
  yellow "needrestart not installed: sudo apt install needrestart"
fi

echo
echo "=== Containers vs host libssl ==="
if command -v docker >/dev/null 2>&1; then
  host_ssl="$(dpkg -l libssl3 2>/dev/null | awk '/^ii/ {print $3}')"
  echo "Host libssl3: ${host_ssl:-not installed}"
  for c in $(docker ps --format '{{.Names}}'); do
    cssl="$(docker exec "$c" sh -c 'dpkg -l libssl3 2>/dev/null | awk "/^ii/ {print \$3}" || apk info -v openssl 2>/dev/null' 2>/dev/null || true)"
    [[ -n "$cssl" ]] && echo "  $c: $cssl"
  done
fi

Run it after every apt upgrade (or wire it into the upgrade itself):

1

sudo apt update && sudo apt upgrade -y && sudo /usr/local/sbin/post-upgrade-check.sh

The output is intentionally noisy. Every red line is a real vulnerability that survived your patch run.

Automating the Whole Thing

Combining unattended-upgrades, needrestart, and a controlled reboot strategy gets you most of the way to “actually patched, not just apt-upgraded.” The idea is:

1. unattended-upgrades installs security updates daily.
2. needrestart runs automatically after every apt transaction and restarts safe services.
3. Reboot windows are scheduled and coordinated, not opportunistic — handled by Ansible (or the orchestrator of your choice) so a load balancer can drain a node first.
4. Live patching covers the gap between reboot windows for critical kernel CVEs.

Ansible: Coordinated Reboot After Patching

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

- name: Patch and reboot if required, in a controlled way
  hosts: web_servers
  serial: 1                       # one host at a time, rolling
  become: true
  tasks:
    - name: Drain from load balancer
      community.general.haproxy:
        state: disabled
        host: "{{ inventory_hostname }}"
        backend: web_backend
      delegate_to: "{{ groups['load_balancers'][0] }}"

    - name: Run apt update + upgrade
      ansible.builtin.apt:
        update_cache: true
        upgrade: dist
        autoremove: true
      register: apt_result

    - name: Run needrestart and report stale services
      ansible.builtin.command: needrestart -b -r l
      register: nr_out
      changed_when: false
      failed_when: false

    - name: Reboot if /var/run/reboot-required exists
      ansible.builtin.reboot:
        reboot_timeout: 600
        post_reboot_delay: 30
      when: ansible_facts['os_family'] == 'Debian'
      register: reboot_result
      ignore_errors: false
      vars:
        ansible_become: true
      check_mode: false
      # Only reboot if the flag file exists
      when_stat_check: true
      when: lookup('ansible.builtin.file', '/var/run/reboot-required', errors='ignore') is not none

    - name: Verify the system is up
      ansible.builtin.wait_for_connection:
        timeout: 300

    - name: Re-add to load balancer
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: web_backend
      delegate_to: "{{ groups['load_balancers'][0] }}"

The serial: 1 is the part most people skip. Without it, every host in the group reboots in parallel. With a load balancer in front, you want exactly one host out of rotation at a time.

If you want this to also apply CIS hardening or set up unattended-upgrades itself, the Ubuntu Hardening article covers that side.

What “Patched” Should Actually Mean

A system is patched when all of the following are true:

If any of those are false, the package version on disk is irrelevant. The vulnerable code is still in memory, still serving requests, still exploitable.

apt upgrade is the first step of patching. Treating it as the last step is one of the most common — and most consequential — gaps in real-world Linux operations.

Further Reading

• needrestart(1) — Debian manpage
• Ubuntu Livepatch service documentation
• TuxCare KernelCare technical overview
• Linux kernel speculation mitigations sysfs interface
• Ubuntu Hardening: Automation is one of the key — the same patching strategy as part of a full CIS baseline
• Vulnerability Management: Where to Start and Why Automation Is Non-Negotiable — the program-level view of all this