case 4690 - report browser: launch chrome-headless-shell; writable config/cache folders for the browser; Ubuntu 24.04 droplet results

Full Chrome 155 would not start under a Linux service user with no writable
home (chrome_crashpad_handler: --database is required). chrome-headless-shell
starts, and measured on a 1 GB droplet is 3-4x faster and smaller per render.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

devdocs/modernization/NET10-PORT.md

@@ -91,6 +91,7 @@ them. Items touching those three areas carry more weight here than they did in S
 - [x] **PuppeteerSharp 17 → 25.x.** Reporting is AyaNova's main Puppeteer surface (`biz/ReportBiz.cs`). Done: 25.15.0 (Sockeye: 25.12.0); compiles with no API changes; launch arguments fixed (below). Downloads Chrome for Testing 155 (was 124). The download location and failure handling are separate section 3 items
   - [x] `BrowserFetcher.DefaultChromiumRevision`: n/a, already `DownloadAsync()` with no argument (`ReportBiz.cs:800`, case 4586)
   - [x] `IJSHandle.ExecutionContext`: n/a, the only use is commented out (`ReportBiz.cs:858-860`). Evaluate results are `JsonElement?`: check any remaining `Evaluate*` call sites
+  - [x] **Launch `chrome-headless-shell`, not full Chrome** (found on the Ubuntu 24.04 droplet). Full Chrome 155 would not start under the `www-data` service: `chrome_crashpad_handler: --database is required`, because Chrome now insists on writing config and crash-reporter files under the user's home (`/var/www`, not writable); Chrome 124 in 8.2.4 did not. Measured on the 1 GB droplet printing a report-sized page, 5 runs each: full Chrome (with writable folders) 4.1–4.9 s and about 1.6 GB summed RSS per render, headless shell 1.0–1.5 s and about 620 MB, PDFs within 0.2% in size. A browser starts for every render, so the shell is chosen on merit (John's decision). Also `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` are pointed at `report-browser/browser-home` (temp folder when no data path) so a full Chrome given in `AYANOVA_REPORT_RENDER_BROWSER_PATH` starts too
   - The default download is Chrome for Testing, into `Chrome/` and `ChromeHeadlessShell/` beside the app. AyaNova 8.2.0 already moved from Chromium to Chrome to follow Puppeteer's default, so the browser choice is settled
   - [x] Launch arguments must be separate array elements. Fixed: defaults are `new[] { "--no-sandbox" }` (`--headless` dropped, `LaunchOptions.Headless` already adds it, as in Sockeye); the customer setting is split on whitespace. A customer value containing a quoted argument with spaces would now be split; none is known. **Browser default locale:** the old `--headless` argument ran Chrome's old headless mode, which takes its default language from the OS (en-CA → Chrome's en-GB, day-first); current Chrome headless always defaults to en-US. Report scripts that format without a locale (stock sample report 80 calls `toLocaleDateString()` bare) switched from 22/10/2026 to 10/22/2026. Fixed by adding `--lang=<server CultureInfo.CurrentCulture>` unless the customer setting already has a `--lang`; skipped when the server culture is invariant. Stock helpers in `ay-report.js` pass the client's language explicitly and were never affected. **Both paths were wrong:** the defaults are one element `"--headless --no-sandbox "` (`ReportBiz.cs:812`, `:829`), and `AYANOVA_REPORT_RENDER_BROWSER_PARAMS` is passed whole as one element (`:819-822`)
   - AppArmor restricts unprivileged user namespaces on Ubuntu 24.04+, which affects the Chrome sandbox
@@ -127,8 +128,8 @@ suite is waiting on (case 4653, case 4654, the contract issue).
 
 ## 4. Test under realistic conditions
 
-- [ ] **Database role.** Run under a non-superuser role **named like the database**, with a password containing spaces and `@ : / ? # %`. Boot, on-demand backup, scheduled backup, restore. Sockeye ran as `postgres` for years and never exercised either bug in section 3
+- [x] **Database role.** Done on the Ubuntu 24.04 droplet (2026-10-23): role `"AyaNova"`, no superuser, no CREATEDB, password `p @:/?#% x`; boot, on-demand backup (818 entries), 22/22 reports and the integration suite all pass. Scheduled backup uses the same code path as on-demand. Run under a non-superuser role **named like the database**, with a password containing spaces and `@ : / ? # %`. Boot, on-demand backup, scheduled backup, restore. Sockeye ran as `postgres` for years and never exercised either bug in section 3
-- [ ] **Restore with a dedicated role.** `createdb -O <role>`, then `pg_restore --no-owner --no-privileges --role=<role>`. Without `--role`, a superuser restore leaves every table owned by `postgres`, and the integrity check, which counts through `information_schema`, fails. Compare with the restore instructions in the customer manual (`docs/`) and note any change needed
+- [x] **Restore with a dedicated role.** Done on the droplet: `createdb --owner=AyaNova`, `pg_restore --no-owner --no-privileges --role=AyaNova`; every table owned by the role, integrity check and licence pass at boot. A plain superuser restore (`dropdb`/`createdb`/`pg_restore` as `postgres`) of an 8.2.4 dump into 8.3.0 also works. Manual comparison still to do. `createdb -O <role>`, then `pg_restore --no-owner --no-privileges --role=<role>`. Without `--role`, a superuser restore leaves every table owned by `postgres`, and the integrity check, which counts through `information_schema`, fails. Compare with the restore instructions in the customer manual (`docs/`) and note any change needed
 - [ ] **Reports, all advanced reports**, compared with the baseline PDFs. On Windows and on Linux. On Linux also: `ldd` on the downloaded browser binary for missing libraries (install exactly what is reported; several Ubuntu 24.04+ names carry a `t64` suffix); the sandbox under the service account; no orphaned `chrome` processes after a cancelled render
 - [ ] Integration suite: same result as the baseline
 - [ ] Licence verification gate: green
@@ -138,7 +139,7 @@ suite is waiting on (case 4653, case 4654, the contract issue).
 ## 5. Release
 
 - [ ] **Every host that runs a framework-dependent build needs the .NET 10 runtime** before it gets the release:
-  - Hosted subscription droplets (Ubuntu 24.04). How `aspnetcore-runtime-10.0` is obtained on 24.04 needs checking **[unverified]**: to be settled on a dedicated DigitalOcean test droplet (section 4). On 26.04 it is in Ubuntu's own archive
+  - Hosted subscription droplets (Ubuntu 24.04). **Confirmed on the test droplet:** `apt-get install -y aspnetcore-runtime-10.0` gets 10.0.13 from Ubuntu's own archive (`noble-updates/main`, `noble-security/main`), no PPA. Installing it replaces `dotnet-host-8.0` with `dotnet-host-10.0` (the shared `dotnet` command); the 8.0 runtime stays installed and unused. Tested upgrade 8.2.4 → 8.3.0 in place: stop service, install runtime, the manual's file swap, start; boots and keeps the licence. 26.04: to be confirmed on the same droplet rebuilt
   - **Subscription server setup script** (`sockeye-client/src/views/biz-subscription-server.vue`, around lines 1350-1423; filled in per server and copied to the clipboard from the "Script" menu, used for every new subscription droplet): `apt-get install -y aspnetcore-runtime-8.0` (line 1377) → the .NET 10 runtime from whichever source the test droplet confirms; remove the `mkdir`/`chmod g+rwx` of `/var/ayanova/Chrome` and `ChromeHeadlessShell` (lines 1396-1397, 1402-1403), the browser now downloads to `/var/ayanova/data/report-browser`. Verify by running the updated script on the test droplet. Part of this port (John, 2026-10-22)
   - Self-hosted Linux customers (server and desktop zips are framework-dependent)
   - Windows LAN: `lan.iss` prompts for the ASP.NET Core Hosting Bundle. Update its text and link to .NET 10
@@ -184,6 +185,7 @@ Found during the port, out of scope. One line each.
 - **Race, affects production after every restart:** `FormFieldReference.FormFieldReferenceList` (`biz/FormFieldReference.cs:57-59`) builds a static dictionary lazily with no lock, assigning it before filling it. A concurrent first request sees it half-built and throws `FormFieldReferenceList: <type> is not valid`. Seen twice on .NET 10, each time on the first form-field use after boot: `VendorCrud.CRUD`, then `FormCustom.FormCustomUpdate`. Fix: build the dictionary fully before publishing it (`Lazy<T>` or a static initialiser). Pre-existing, not a port change. John's decision: its own case, fixed with the other cases going into this release, not under the port
 - `raven-launcher` `Program.cs:98` calls `PGProcess.StandardInput.Close()` but standard input was never redirected, so it throws `InvalidOperationException` at shutdown: the "Postgres shut down" step never runs and the launcher exits with an error when the server stops. Postgres was started detached by `pg_ctl start`, so it probably stays running. Pre-existing; candidate for its own case
 - Sockeye 8.1.0 has the NLog 6 archive naming problem fixed here for AyaNova: archives are `log-sockeye_NN.txt`, old `-0` to `-8` archives are never deleted, and its log viewer still sorts by name (it also filters on `ayanova.txt`, not `sockeye.txt`, so the current file is listed twice). Same fix applies: `ArchiveSuffixFormat = "-{0}"` and the date sort
+- **SuperUser can be deleted (serious, pre-existing, own case; John to decide if it goes in this release):** `UserBiz.ValidateCanDelete` (`biz/UserBiz.cs:1055`) only refuses a delete when the user has related records; nothing protects user id 1, although updates to it are blocked (`UserBiz.cs:946`). On the droplet, whose sample data left SuperUser with no related records, the integration test `UserWithActivityShouldNotBeDeleteable` (`DELETE User/1`) succeeded and removed the account; `AYANOVA_SET_SUPERUSER_PW` cannot bring it back (it only resets an existing user 1). Same code in 8.2.4; likely also why `superuser` could not log in on the reused local database at the first baseline attempt. Any user with delete rights on Users can do this
 - `RunProgram.Run` (and its `RunWindows`/`RunLinuxShell`/`RunOSX` helpers) has no callers since the backup moved to `RunNoShell`. It runs through `cmd.exe /C` or `/bin/sh -c` and logs its full command line on error: delete it rather than reuse it
 - **Own case after the port (John):** the subscription server setup script is hard-coded in `biz-subscription-server.vue`, so every change needs a new Sockeye client build. Make it editable without a rebuild
 - **Improvement, own case after the port (John to create):** bring back logging of JavaScript error details from report page console errors, as Sockeye's port did. AyaNova logs only `ERROR: {message text}`; the per-argument detail (`arg instanceof Error ? arg.message : arg`) is commented out at `biz/ReportBiz.cs:873-876` since case 4586, because `IJSHandle.ExecutionContext` was removed. Sockeye 8.1.0 (`server/biz/ReportBiz.cs:575`) evaluates on the handle instead: `args.Message.Args.Select(arg => arg.EvaluateFunctionAsync("(arg) => arg instanceof Error ? arg.message : arg"))`, which also needs the console handler to be `async` again
@@ -215,3 +217,4 @@ Found during the port, out of scope. One line each.
 | 2026-10-22 | Specific error for report browser download / launch failures | Download blocked (a file named `report-browser` in the data folder): job failed with `rendererror.exception` "…could not be downloaded to "…\report-browser" (check internet access and that the folder is writable): … Cannot create …", Error in the log, server still up. Launch with `AYANOVA_REPORT_RENDER_BROWSER_PATH` pointing nowhere: "…could not be started from "C:\no-such-folder\chrome.exe" with arguments '--no-sandbox --lang=en-CA' …: Browser was not found…", server still up. Normal: 22/22 reports, page counts = PuppeteerSharp 17; integration suite = baseline |
 | 2026-10-22 | Data-list date filters `>=`, `<`, `!=`; null check | `WorkOrderServiceDate` through `/data-list`, 590 work orders. Minute with 6 records off second 0, before: `>=` 142 (= `>`), `<` 448 (= `<=`), `!=` 590 (all). After: `>=` 148, `<` 442, `!=` 584; identities `>=` = `>`+K, `<` = `<=`−K, `!=` = all−K hold. Same identities hold for a minute with 5 records exactly on second 0. Integration suite = baseline |
 | 2026-10-22 | `raven-test-integration` → `net10.0` (`2c0cfc4`), no package changes | Builds with no warnings; suite runs as `net10.0` against the .NET 10 server: 96 passed, 2 known failures, 5 skipped = baseline. FluentAssertions kept at 5.10.3 (8+ is commercially licensed) |
+| 2026-10-23 | Ubuntu 24.04.5 test droplet (`net10port.onayanova.com`, 1 GB): 8.2.4 on .NET 8 set up by John like a customer (subscription trial licence, sample data, PostgreSQL 18.6 from PGDG) | `aspnetcore-runtime-10.0` 10.0.13 from `noble-updates/main`. In-place upgrade to 8.3.0: boots on .NET 10.0.13, schema 20, licence kept. Reports first **failed to launch** (full Chrome 155, crashpad needs a writable home; see section 1); fixed by launching `chrome-headless-shell` (measured 3–4× faster and smaller): 22/22 on Windows (page counts = PuppeteerSharp 17, 1.14 s avg vs 1.8 s) and on the droplet; `ldd` no missing libraries; cancelled render leaves no browser processes. Integration suite through an SSH tunnel: 28 failures, all from **SuperUser being deleted** by `UserWithActivityShouldNotBeDeleteable` (pre-existing bug, see Deferred); after restoring the pre-upgrade dump and excluding that test: 95 passed, 2 known failures, 5 skipped = baseline. On-demand backup 818 entries. Non-superuser role `"AyaNova"` named like the database with password `p @:/?#% x`, database restored with `--role`: boot, backup, 22/22 reports, suite = baseline |

server/AyaNova/biz/ReportBiz.cs

@@ -795,8 +795,11 @@ What could be the issue here?
 
                 //case 4586
                 //puppeteer has gone chrome dev by default and so does puppeteersharp which tests ok so I'm going to allow the default
-                //this means the files go into Chrome and ChromeHeadlessShell folders, under the data path (not beside the app, see ServerBootConfig)
+                //this means the files go into a ChromeHeadlessShell folder, under the data path (not beside the app, see ServerBootConfig)
-                var FetcherOptions = new BrowserFetcherOptions();
+                //chrome-headless-shell, not full Chrome: it is Chrome's headless-only build for automation and PDF printing.
+                //Measured on a 1GB Ubuntu droplet printing a report-sized page: about 1s and 620MB vs 4.5s and 1.6GB for full Chrome,
+                //and a browser is started for every report render. Full Chrome also failed to start under a service user with no writable home
+                var FetcherOptions = new BrowserFetcherOptions { Browser = SupportedBrowser.ChromeHeadlessShell };
                 if (!string.IsNullOrWhiteSpace(ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH))
                     FetcherOptions.Path = ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH;
                 try
@@ -804,6 +807,7 @@ What could be the issue here?
                     var InstalledBrowser = await new BrowserFetcher(FetcherOptions).DownloadAsync();
                     //launch exactly what was downloaded, otherwise Puppeteer looks in its default folder beside the app
                     lo.ExecutablePath = InstalledBrowser.GetExecutablePath();
+                    lo.HeadlessMode = HeadlessMode.Shell;
                     log.LogDebug($"Report browser: {lo.ExecutablePath}");
                 }
                 catch (Exception ex)
@@ -856,6 +860,27 @@ What could be the issue here?
             {
                 lo.Args = lo.Args.Append($"--lang={ServerLanguage}").ToArray();
             }
+            //Chrome keeps config and crash-reporter files under the user's home folder and fails to start when it isn't writable,
+            //as for a Linux service user like www-data (home /var/www). Point it at folders we own instead; harmless on Windows.
+            //Matters mostly for a full Chrome given in AYANOVA_REPORT_RENDER_BROWSER_PATH, chrome-headless-shell starts without them
+            var BrowserWorkFolder = Path.Combine(
+                string.IsNullOrWhiteSpace(ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH)
+                    ? ServerBootConfig.AYANOVA_TEMP_FILES_PATH
+                    : ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH,
+                "browser-home"
+            );
+            try
+            {
+                Directory.CreateDirectory(Path.Combine(BrowserWorkFolder, "config"));
+                Directory.CreateDirectory(Path.Combine(BrowserWorkFolder, "cache"));
+                lo.Env["XDG_CONFIG_HOME"] = Path.Combine(BrowserWorkFolder, "config");
+                lo.Env["XDG_CACHE_HOME"] = Path.Combine(BrowserWorkFolder, "cache");
+            }
+            catch (Exception ex)
+            {
+                //not fatal, the launch below reports any real problem
+                log.LogWarning(ex, $"Could not create report browser work folders under {BrowserWorkFolder}");
+            }
             log.LogDebug($"Report browser launch arguments: '{string.Join(" ", lo.Args)}'");
             System.Text.StringBuilder PageLog = new System.Text.StringBuilder();