devdocs - case 4795 part 2 design: Ubuntu priorities from daily OVAL files

Kernel notice USN-8959-1 (2,894 CVEs) could not be assessed by per-CVE
ubuntu.com lookups, and the deferral held back its alert. Plan: alert at
once when the known priorities already reach the threshold; read all
priorities from Canonical's daily OVAL files; OSV.dev as the per-CVE
fallback. Decisions recorded (John, 2026-11-04).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

devdocs/design/4795-advisory-priorities-from-oval.md

@@ -0,0 +1,140 @@
+# Case 4795 part 2 — Ubuntu CVE priorities from Canonical's daily OVAL files: design note
+
+Status: approved by John 2026-11-04 (decisions below). Nothing built yet. Release: Sockeye 8.2.3.
+
+Follows the monitor's design note, `4738-security-advisory-monitor.md`. Part 1 of 4795 (timeouts,
+lookup budget, deferral) is in 8.2.2.
+
+## What went wrong in production (2026-11-03 and 04)
+
+- Kernel notice **USN-8959-1** (published 2026-11-03, 26.04 `linux` 7.0.0-42 and 24.04 `linux-hwe-7.0`)
+  lists **2,894 CVEs**. The monitor asks ubuntu.com for each CVE's priority, one request each.
+- Each 3-hour pass got through 6 to 47 of them before the lookups timed out (10 s each) and the
+  pass's budget stopped them. A single request answers in about a second, so ubuntu.com is
+  probably throttling the four parallel requests. At that rate the notice needs about two weeks.
+- Part 1 did its job: the job queue was not held up and other advisories were still recorded. The
+  Wednesday digest arrived correctly with 3 medium advisories.
+- But the deferred notice is recorded nowhere, so it sends no alert. The CVEs read so far already
+  included 1 critical and 16 high; the full list has **9 critical and 224 high**. The alert that
+  should have gone out on 2026-11-03 was held back.
+- After 24 hours, the source counted as failing and the ops problem email went out. It will repeat
+  every 24 hours until the notice clears, and every monthly kernel notice will do the same.
+
+The design cannot finish a kernel notice. More time or retries won't fix that; it needs another
+source for priorities.
+
+## What the alternatives offer (checked 2026-11-04 from ayanova-2 and the workstation)
+
+| Source | Result |
+|---|---|
+| ubuntu.com bulk search `cves.json?priority=high&package=linux` | 504 after 30 s, twice. Unusable |
+| OSV.dev `api.osv.dev/v1/vulns/<id>` | Has the USN with all 2,894 CVEs; each CVE has `severity` type `Ubuntu` = Ubuntu's priority. Fast (0.3 s) and reliable, but a kernel CVE record is about **280 KB** (it lists every kernel binary for every release): about 800 MB per kernel notice. The bulk `Ubuntu/all.zip` is 809 MB; the per-release zips are stale (24.04: 2024-10) or missing (26.04) |
+| **Canonical's OVAL files** on `security-metadata.canonical.com/oval/` | One file per release with **every CVE's priority**, regenerated daily (about 00:50 UTC). `com.ubuntu.resolute.cve.oval.xml.bz2` 4.9 MB (91 MB unpacked, 19,567 CVEs), `com.ubuntu.noble.cve.oval.xml.bz2` 11 MB (240 MB unpacked, 36,516 CVEs). A separate host from the ubuntu.com website; downloads took 2 s |
+
+The OVAL data checked out against what we have:
+
+- All 2,894 CVEs of USN-8959-1 are in the files.
+- Against the 2,119 priorities cached on production, 2,112 agree, 5 are not in the files (older
+  releases only), and 2 differ. One of the 2 (CVE-2025-40098) was changed on ubuntu.com the day
+  before, so the file was right and our 30-day cache was stale.
+- 18,912 CVEs appear in both releases, and none has a different priority in each.
+
+OSV.dev's Ubuntu data comes from Canonical's own repository (`github.com/canonical/ubuntu-security-notices`),
+so it is the same data as the OVAL files.
+
+Where the priority sits: each `<definition class="vulnerability">` has
+`metadata/advisory/cve` with attribute `priority` (critical, high, medium, low, negligible) and the
+CVE id as text, for example
+`<cve href="https://ubuntu.com/security/CVE-2002-2439" priority="low" public="20191023">CVE-2002-2439</cve>`.
+
+## Plan
+
+### 1. Alert at once when the known CVEs already reach the threshold (small fix, own commit)
+
+A deferred notice can only get worse as more priorities arrive. So when the lookups stop part way:
+
+- If what is known already makes it an alert (a high or critical priority, or a CVE in KEV), record
+  it and send the alert now. The body adds a line: "Priorities not read yet for N of M CVEs".
+- Only a notice with nothing above medium so far is deferred, as today.
+
+In code: in `RunAsync`, defer only when `budget.SkippedSome && outcome != AdvisoryOutcome.Alerted`.
+Unknown priorities already count as medium in `SecurityAdvisoryPolicy.Evaluate`, so a known
+critical is what makes the outcome `Alerted`. With part 2 in place this path should rarely run;
+it stays as the safety net.
+
+### 2. Priorities from the daily OVAL files
+
+- **New step at the start of each pass**, before the notice sources: for each Ubuntu codename in
+  the active assets (`noble`, `resolute`), download
+  `https://security-metadata.canonical.com/oval/com.ubuntu.<codename>.cve.oval.xml.bz2` if the last
+  successful read is more than 20 hours old.
+  - Conditional GET with `If-Modified-Since`, so an unchanged file costs one small request.
+  - Download to a temporary file (60 s timeout, refused above 100 MB), then parse it, then delete
+    it. A broken download changes nothing.
+  - Its health is tracked like the other sources: a row `ubuntu-oval` in `asecuritysource`, so
+    failures reach the existing failing-source report.
+- **Parsing streams:** `Org.BouncyCastle.Utilities.Bzip2.CBZip2InputStream` (public; BouncyCastle
+  2.7.0 is already a dependency, **no new package**) into an `XmlReader` with DTDs prohibited and no
+  resolver. Read each `cve` element's `priority` attribute and text.
+  - Ids must match `CVE-\d{4}-\d+`, and priorities must be one of the five known values; anything
+    else is skipped.
+  - Measured on the workstation with the real files: 26.04 in 2.3 s, 24.04 in 4.7 s, 50 MB peak
+    for the whole process. ayanova-2 (1 CPU) will be slower; I'll measure it at the deploy. That
+    is once a day; on the other passes it is one small request per file.
+- **Into the existing CVE cache `asecuritycve`** (no schema change):
+  - Insert the missing rows and update the rows whose priority changed, in batches, with one
+    `INSERT … ON CONFLICT (id) DO UPDATE … WHERE priority IS DISTINCT FROM excluded.priority`
+    per batch.
+  - Unchanged rows are not rewritten, so a daily refresh doesn't rewrite 37,000 rows.
+  - The first run fills about 37,000 rows (a few MB).
+- **Lookups use the cache, and fall back only for what the file lacks:**
+  - While the OVAL read for that release has succeeded within the last 48 hours, cached
+    priorities count as current, with no 30-day staleness check.
+  - Only CVEs not in the cache are looked up one by one: CVEs published since the morning's file,
+    a handful a day.
+  - The 30-day staleness rule stays for when the OVAL read is failing.
+- **Failing-source report:** a daily source needs a longer grace before it is reported: 48 hours
+  for `ubuntu-oval`, still 24 for the others.
+- **Dry run:** unchanged. It uses the same cache, so it no longer takes minutes.
+
+### 3. Not in this part
+
+- Part 3 (publish and fix dates for Business case 4786). The OVAL files carry each CVE's
+  `public_date`, which makes part 3 cheaper, but it needs a schema change and its own decision.
+- Using OSV.dev for PostgreSQL or Debian advisories: a separate decision, linked to 4786.
+- 4796 (separate job loops) is still the structural fix for anything slow on the job loop.
+
+## Decided (John, 2026-11-04)
+
+1. **Fallback for CVEs not yet in the file: OSV.dev** instead of ubuntu.com. OSV.dev is more
+   reliable (one of two ubuntu.com requests failed again during these checks), and about 280 KB
+   for a kernel CVE is fine for a handful a day. At build time, first check how soon a new Ubuntu
+   CVE appears on OSV.dev; if it lags by more than a day, keep ubuntu.com and say so.
+2. **The held USN-8959-1: let the alert go out.** Once 8.2.3 is deployed, the next pass assesses it
+   in full and sends the alert (9 critical, 224 high), although Thursday's updates will already
+   have installed that kernel on ayanova-2. It is the simplest course, and it proves the fix.
+
+## Tests
+
+- **Regression test for plan 1, failing on the current code:**
+  - A notice whose CVEs include one cached as critical, plus others the budget skips. Today it is
+    deferred with no alert; after the fix it alerts and says how many priorities are missing.
+  - A notice with only medium known and some skipped is still deferred.
+- **Parser test:** a small `.bz2` fixture cut from the real 26.04 file (a few definitions,
+  including one with an unexpected priority value and one with a malformed id). It checks the
+  values read and the values skipped.
+- **Cache test:**
+  - With a fresh OVAL read, an old cached row is not looked up again.
+  - With the OVAL read failing, the 30-day rule applies again.
+  - A CVE missing from the file goes to the fallback.
+- The existing `SecurityAdvisoryTests` fixtures keep covering the matching and severity rules.
+
+## Commits (all case 4795, in Sockeye)
+
+1. `devdocs - case 4795 part 2 design: Ubuntu priorities from daily OVAL files` (this note)
+2. `case 4795 - alert at once when the known CVE priorities already reach the alert threshold`
+3. `case 4795 - Ubuntu CVE priorities from Canonical's daily OVAL files; per-CVE lookups only for what they lack`
+4. `case 4795 - per-CVE fallback from OSV.dev instead of ubuntu.com`
+
+Then Sockeye 8.2.3 by the usual runbook. AyaNova has no advisory monitor, so there is nothing to
+carry over.