SAGE: CVE-driven patching for Kubernetes nodes, one node at a time

SAGE dashboard overview: 5 of 5 nodes in a clean state, no serious CVEs open, no node in quarantine — The SAGE dashboard on the test cluster after remediation: 5 nodes out of 5 clean, no serious CVE open, nothing in quarantine.

SAGE (SSVC-Aware Guarded Engine) is a patch management system for the nodes of a Kubernetes cluster. For every vulnerable package on every node it asks one question, "does this CVE really affect us?", then decides what to do and does it, one node at a time.

Code, operations guide, architecture and every design decision are on GitHub: github.com/nicologiuliani6/sage.

The problem

Before writing any code, I built a test cluster on an OpenStack cloud: 1 master and 3 workers on Rocky Linux 9.4, provisioned with Terraform and installed with kubespray. A freshly installed cluster, nothing exotic on it. Running dnf updateinfo on the nodes reported 308 security advisories across 147 packages.

That number is the whole problem. Scanners flag a CVE as soon as a vulnerable package is installed, but many vulnerabilities only matter under conditions that may not exist on the node: a kernel module that has to be loaded, a service that has to be listening, a component that has to actually be in use. You are left with two bad options:

Update everything, blindly. Reboots, drained nodes and regressions, for CVEs that might not apply at all.
Ignore what looks irrelevant. Fine until one of those "irrelevant" CVEs turns out to be exploitable on your hardware.

SAGE takes a third option: decide case by case, with evidence, and keep the evidence.

What it decides

Action	When
`update`	the CVE applies, or there is any doubt
`not_affected`	the CVE does not apply, verified on the node: no update, a VEX document is written instead
`mitigate`	what makes it exploitable can be switched off, for example by blocking a kernel module
`remove`	the package has been unused for days and nothing needs it
`skip`	the node is not in any configured group, so it is not touched

How it works

Deterministic collection. A Kubernetes Job connects to every node over SSH with Ansible and collects facts: installed packages, security advisories, loaded kernel modules, listening services, running processes.
Public threat data. Each CVE is enriched with the CISA KEV catalog (is it being exploited in the wild?), the FIRST EPSS score (how likely is exploitation?) and the official CVE record with CISA's SSVC assessment.
The LLM as an extractor, nothing more. A language model reads only the public description of the CVE and the package name, never data from the nodes, and lists the conditions the vulnerability needs. It must pick them from a closed catalog (kernel_module, network_service, component_in_use); answers are validated against a schema and cached.
Verification on the real node. Every condition is checked against the collected facts, with consistency checks. A condition that cannot be verified counts as true. The model can never declare a node safe on its own word.
Decision by policy, not by the model. An OPA policy written in Rego picks the action. Two rules are fixed and cannot be configured away: when in doubt, update, and a CVE in the KEV catalog is never "not affected".
Remediation, one node at a time. Groups go in order (workers first, masters last), a canary node goes first, and a node is never patched by a Job running on itself. Each node goes through cordon, drain, updating only the advisories that are needed, reboot if required, health checks, uncordon. If anything fails, the node is rolled back and put in quarantine, and SAGE stops until a person looks at it.
Report. A plan per node and an OpenVEX document for every CVE that was declared not affected, with the evidence behind it.

An always-on controller (a Deployment holding a Lease, so only one copy is ever active) loops over the nodes and hands the heavy work to Jobs. If the LLM is unreachable, SAGE falls back through a chain of endpoints, and with no model at all it keeps working on the deterministic rules alone.

No data leaves the cluster

I prototyped the extraction step with a hosted model, then moved it inside the infrastructure: a dedicated node in the cluster serves an 8B model with Ollama, on CPU only. For this job, reading a CVE description and naming the conditions it needs, a small local model turned out to do very well. The prompt only ever contains public CVE data, and now even that never leaves the cluster.

On the test cluster

At the first check, all four nodes were critical, with 8 CVEs from the KEV catalog still open and the update waves ready: workers with a canary first, then the master.

First check: four critical nodes and 8 open KEV CVEs, with the update waves waiting — First check: four critical nodes, 8 actively exploited CVEs (KEV) to fix, waves waiting with the canary first.

Then SAGE worked through the nodes, one at a time.

Remediation in progress: three nodes already clean, the canary still being processed — Remediation in progress: three nodes already clean, one still being patched.

Log of a worker remediation: uncordon, system pod checks and an Ansible recap with no failures — The end of a worker's remediation: uncordon, check that the system pods are back, Ansible recap with zero failures.

After more than 400 cycles, all five nodes, including the one running the LLM, are clean: no KEV vulnerability open, no node in quarantine, no failed remediation.

One card per node: three workers, the master and the LLM node, all clean — One card per node, with every decision and its evidence: three workers, the master and the LLM node.

Design rules

When in doubt, update. It is a guarantee of the policy, not a setting.
No hardcoding. Groups, SSH user, LLM endpoints, protected packages and modules all come from configuration; a missing required field stops startup with a clear error.
SAGE does not depend on what it updates. One node at a time, never on itself, and it keeps working without the LLM.
Traceability. Every decision has its evidence, every configuration change goes into a history with who, when and why.
Free and open source components only, pinned by hash, SHA or digest.
Every design choice is written down as an architecture decision record.

It runs on RHEL-family nodes (dnf), SUSE (zypper), Debian and Ubuntu (apt), even mixed in the same cluster, and installs from a single executable on a master that asks a few questions and deploys everything with a Helm chart. A read-only web dashboard shows nodes, CVEs, cycles, health, logs and history.

What's next

Two or three LLM nodes behind a single endpoint for high availability, still with all data inside, and GPUs if they become available.
Deterministic applicability at the version level (Red Hat CSAF VEX) and at the kernel configuration level.
A proper evaluation: updates and reboots avoided compared with dnf-automatic, agreement with an expert's decisions, stability of the model's answers.
Testing on a cluster with 3 masters, then at scale.