From bf0555fb31abf2c59483ec1f22a246fb6e9fa4bd Mon Sep 17 00:00:00 2001 From: John Cardinal Date: Fri, 23 Oct 2026 08:12:21 -0700 Subject: [PATCH] case 4690 - report browser: launch chrome-headless-shell; writable config/cache folders for the browser; Ubuntu 24.04 droplet results Full Chrome 155 would not start under a Linux service user with no writable home (chrome_crashpad_handler: --database is required). chrome-headless-shell starts, and measured on a 1 GB droplet is 3-4x faster and smaller per render. Co-Authored-By: Claude Opus 5.5 --- devdocs/modernization/NET10-PORT.md | 9 ++++++--- server/AyaNova/biz/ReportBiz.cs | 29 +++++++++++++++++++++++++++-- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/devdocs/modernization/NET10-PORT.md b/devdocs/modernization/NET10-PORT.md index 4f734e4a..7dd12a2c 100644 --- a/devdocs/modernization/NET10-PORT.md +++ b/devdocs/modernization/NET10-PORT.md @@ -91,6 +91,7 @@ them. Items touching those three areas carry more weight here than they did in S - [x] **PuppeteerSharp 17 → 25.x.** Reporting is AyaNova's main Puppeteer surface (`biz/ReportBiz.cs`). Done: 25.15.0 (Sockeye: 25.12.0); compiles with no API changes; launch arguments fixed (below). Downloads Chrome for Testing 155 (was 124). The download location and failure handling are separate section 3 items - [x] `BrowserFetcher.DefaultChromiumRevision`: n/a, already `DownloadAsync()` with no argument (`ReportBiz.cs:800`, case 4586) - [x] `IJSHandle.ExecutionContext`: n/a, the only use is commented out (`ReportBiz.cs:858-860`). Evaluate results are `JsonElement?`: check any remaining `Evaluate*` call sites + - [x] **Launch `chrome-headless-shell`, not full Chrome** (found on the Ubuntu 24.04 droplet). Full Chrome 155 would not start under the `www-data` service: `chrome_crashpad_handler: --database is required`, because Chrome now insists on writing config and crash-reporter files under the user's home (`/var/www`, not writable); Chrome 124 in 8.2.4 did not. Measured on the 1 GB droplet printing a report-sized page, 5 runs each: full Chrome (with writable folders) 4.1–4.9 s and about 1.6 GB summed RSS per render, headless shell 1.0–1.5 s and about 620 MB, PDFs within 0.2% in size. A browser starts for every render, so the shell is chosen on merit (John's decision). Also `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` are pointed at `report-browser/browser-home` (temp folder when no data path) so a full Chrome given in `AYANOVA_REPORT_RENDER_BROWSER_PATH` starts too - The default download is Chrome for Testing, into `Chrome/` and `ChromeHeadlessShell/` beside the app. AyaNova 8.2.0 already moved from Chromium to Chrome to follow Puppeteer's default, so the browser choice is settled - [x] Launch arguments must be separate array elements. Fixed: defaults are `new[] { "--no-sandbox" }` (`--headless` dropped, `LaunchOptions.Headless` already adds it, as in Sockeye); the customer setting is split on whitespace. A customer value containing a quoted argument with spaces would now be split; none is known. **Browser default locale:** the old `--headless` argument ran Chrome's old headless mode, which takes its default language from the OS (en-CA → Chrome's en-GB, day-first); current Chrome headless always defaults to en-US. Report scripts that format without a locale (stock sample report 80 calls `toLocaleDateString()` bare) switched from 22/10/2026 to 10/22/2026. Fixed by adding `--lang=` unless the customer setting already has a `--lang`; skipped when the server culture is invariant. Stock helpers in `ay-report.js` pass the client's language explicitly and were never affected. **Both paths were wrong:** the defaults are one element `"--headless --no-sandbox "` (`ReportBiz.cs:812`, `:829`), and `AYANOVA_REPORT_RENDER_BROWSER_PARAMS` is passed whole as one element (`:819-822`) - AppArmor restricts unprivileged user namespaces on Ubuntu 24.04+, which affects the Chrome sandbox @@ -127,8 +128,8 @@ suite is waiting on (case 4653, case 4654, the contract issue). ## 4. Test under realistic conditions -- [ ] **Database role.** Run under a non-superuser role **named like the database**, with a password containing spaces and `@ : / ? # %`. Boot, on-demand backup, scheduled backup, restore. Sockeye ran as `postgres` for years and never exercised either bug in section 3 -- [ ] **Restore with a dedicated role.** `createdb -O `, then `pg_restore --no-owner --no-privileges --role=`. Without `--role`, a superuser restore leaves every table owned by `postgres`, and the integrity check, which counts through `information_schema`, fails. Compare with the restore instructions in the customer manual (`docs/`) and note any change needed +- [x] **Database role.** Done on the Ubuntu 24.04 droplet (2026-10-23): role `"AyaNova"`, no superuser, no CREATEDB, password `p @:/?#% x`; boot, on-demand backup (818 entries), 22/22 reports and the integration suite all pass. Scheduled backup uses the same code path as on-demand. Run under a non-superuser role **named like the database**, with a password containing spaces and `@ : / ? # %`. Boot, on-demand backup, scheduled backup, restore. Sockeye ran as `postgres` for years and never exercised either bug in section 3 +- [x] **Restore with a dedicated role.** Done on the droplet: `createdb --owner=AyaNova`, `pg_restore --no-owner --no-privileges --role=AyaNova`; every table owned by the role, integrity check and licence pass at boot. A plain superuser restore (`dropdb`/`createdb`/`pg_restore` as `postgres`) of an 8.2.4 dump into 8.3.0 also works. Manual comparison still to do. `createdb -O `, then `pg_restore --no-owner --no-privileges --role=`. Without `--role`, a superuser restore leaves every table owned by `postgres`, and the integrity check, which counts through `information_schema`, fails. Compare with the restore instructions in the customer manual (`docs/`) and note any change needed - [ ] **Reports, all advanced reports**, compared with the baseline PDFs. On Windows and on Linux. On Linux also: `ldd` on the downloaded browser binary for missing libraries (install exactly what is reported; several Ubuntu 24.04+ names carry a `t64` suffix); the sandbox under the service account; no orphaned `chrome` processes after a cancelled render - [ ] Integration suite: same result as the baseline - [ ] Licence verification gate: green @@ -138,7 +139,7 @@ suite is waiting on (case 4653, case 4654, the contract issue). ## 5. Release - [ ] **Every host that runs a framework-dependent build needs the .NET 10 runtime** before it gets the release: - - Hosted subscription droplets (Ubuntu 24.04). How `aspnetcore-runtime-10.0` is obtained on 24.04 needs checking **[unverified]**: to be settled on a dedicated DigitalOcean test droplet (section 4). On 26.04 it is in Ubuntu's own archive + - Hosted subscription droplets (Ubuntu 24.04). **Confirmed on the test droplet:** `apt-get install -y aspnetcore-runtime-10.0` gets 10.0.13 from Ubuntu's own archive (`noble-updates/main`, `noble-security/main`), no PPA. Installing it replaces `dotnet-host-8.0` with `dotnet-host-10.0` (the shared `dotnet` command); the 8.0 runtime stays installed and unused. Tested upgrade 8.2.4 → 8.3.0 in place: stop service, install runtime, the manual's file swap, start; boots and keeps the licence. 26.04: to be confirmed on the same droplet rebuilt - **Subscription server setup script** (`sockeye-client/src/views/biz-subscription-server.vue`, around lines 1350-1423; filled in per server and copied to the clipboard from the "Script" menu, used for every new subscription droplet): `apt-get install -y aspnetcore-runtime-8.0` (line 1377) → the .NET 10 runtime from whichever source the test droplet confirms; remove the `mkdir`/`chmod g+rwx` of `/var/ayanova/Chrome` and `ChromeHeadlessShell` (lines 1396-1397, 1402-1403), the browser now downloads to `/var/ayanova/data/report-browser`. Verify by running the updated script on the test droplet. Part of this port (John, 2026-10-22) - Self-hosted Linux customers (server and desktop zips are framework-dependent) - Windows LAN: `lan.iss` prompts for the ASP.NET Core Hosting Bundle. Update its text and link to .NET 10 @@ -184,6 +185,7 @@ Found during the port, out of scope. One line each. - **Race, affects production after every restart:** `FormFieldReference.FormFieldReferenceList` (`biz/FormFieldReference.cs:57-59`) builds a static dictionary lazily with no lock, assigning it before filling it. A concurrent first request sees it half-built and throws `FormFieldReferenceList: is not valid`. Seen twice on .NET 10, each time on the first form-field use after boot: `VendorCrud.CRUD`, then `FormCustom.FormCustomUpdate`. Fix: build the dictionary fully before publishing it (`Lazy` or a static initialiser). Pre-existing, not a port change. John's decision: its own case, fixed with the other cases going into this release, not under the port - `raven-launcher` `Program.cs:98` calls `PGProcess.StandardInput.Close()` but standard input was never redirected, so it throws `InvalidOperationException` at shutdown: the "Postgres shut down" step never runs and the launcher exits with an error when the server stops. Postgres was started detached by `pg_ctl start`, so it probably stays running. Pre-existing; candidate for its own case - Sockeye 8.1.0 has the NLog 6 archive naming problem fixed here for AyaNova: archives are `log-sockeye_NN.txt`, old `-0` to `-8` archives are never deleted, and its log viewer still sorts by name (it also filters on `ayanova.txt`, not `sockeye.txt`, so the current file is listed twice). Same fix applies: `ArchiveSuffixFormat = "-{0}"` and the date sort +- **SuperUser can be deleted (serious, pre-existing, own case; John to decide if it goes in this release):** `UserBiz.ValidateCanDelete` (`biz/UserBiz.cs:1055`) only refuses a delete when the user has related records; nothing protects user id 1, although updates to it are blocked (`UserBiz.cs:946`). On the droplet, whose sample data left SuperUser with no related records, the integration test `UserWithActivityShouldNotBeDeleteable` (`DELETE User/1`) succeeded and removed the account; `AYANOVA_SET_SUPERUSER_PW` cannot bring it back (it only resets an existing user 1). Same code in 8.2.4; likely also why `superuser` could not log in on the reused local database at the first baseline attempt. Any user with delete rights on Users can do this - `RunProgram.Run` (and its `RunWindows`/`RunLinuxShell`/`RunOSX` helpers) has no callers since the backup moved to `RunNoShell`. It runs through `cmd.exe /C` or `/bin/sh -c` and logs its full command line on error: delete it rather than reuse it - **Own case after the port (John):** the subscription server setup script is hard-coded in `biz-subscription-server.vue`, so every change needs a new Sockeye client build. Make it editable without a rebuild - **Improvement, own case after the port (John to create):** bring back logging of JavaScript error details from report page console errors, as Sockeye's port did. AyaNova logs only `ERROR: {message text}`; the per-argument detail (`arg instanceof Error ? arg.message : arg`) is commented out at `biz/ReportBiz.cs:873-876` since case 4586, because `IJSHandle.ExecutionContext` was removed. Sockeye 8.1.0 (`server/biz/ReportBiz.cs:575`) evaluates on the handle instead: `args.Message.Args.Select(arg => arg.EvaluateFunctionAsync("(arg) => arg instanceof Error ? arg.message : arg"))`, which also needs the console handler to be `async` again @@ -215,3 +217,4 @@ Found during the port, out of scope. One line each. | 2026-10-22 | Specific error for report browser download / launch failures | Download blocked (a file named `report-browser` in the data folder): job failed with `rendererror.exception` "…could not be downloaded to "…\report-browser" (check internet access and that the folder is writable): … Cannot create …", Error in the log, server still up. Launch with `AYANOVA_REPORT_RENDER_BROWSER_PATH` pointing nowhere: "…could not be started from "C:\no-such-folder\chrome.exe" with arguments '--no-sandbox --lang=en-CA' …: Browser was not found…", server still up. Normal: 22/22 reports, page counts = PuppeteerSharp 17; integration suite = baseline | | 2026-10-22 | Data-list date filters `>=`, `<`, `!=`; null check | `WorkOrderServiceDate` through `/data-list`, 590 work orders. Minute with 6 records off second 0, before: `>=` 142 (= `>`), `<` 448 (= `<=`), `!=` 590 (all). After: `>=` 148, `<` 442, `!=` 584; identities `>=` = `>`+K, `<` = `<=`−K, `!=` = all−K hold. Same identities hold for a minute with 5 records exactly on second 0. Integration suite = baseline | | 2026-10-22 | `raven-test-integration` → `net10.0` (`2c0cfc4`), no package changes | Builds with no warnings; suite runs as `net10.0` against the .NET 10 server: 96 passed, 2 known failures, 5 skipped = baseline. FluentAssertions kept at 5.10.3 (8+ is commercially licensed) | +| 2026-10-23 | Ubuntu 24.04.5 test droplet (`net10port.onayanova.com`, 1 GB): 8.2.4 on .NET 8 set up by John like a customer (subscription trial licence, sample data, PostgreSQL 18.6 from PGDG) | `aspnetcore-runtime-10.0` 10.0.13 from `noble-updates/main`. In-place upgrade to 8.3.0: boots on .NET 10.0.13, schema 20, licence kept. Reports first **failed to launch** (full Chrome 155, crashpad needs a writable home; see section 1); fixed by launching `chrome-headless-shell` (measured 3–4× faster and smaller): 22/22 on Windows (page counts = PuppeteerSharp 17, 1.14 s avg vs 1.8 s) and on the droplet; `ldd` no missing libraries; cancelled render leaves no browser processes. Integration suite through an SSH tunnel: 28 failures, all from **SuperUser being deleted** by `UserWithActivityShouldNotBeDeleteable` (pre-existing bug, see Deferred); after restoring the pre-upgrade dump and excluding that test: 95 passed, 2 known failures, 5 skipped = baseline. On-demand backup 818 entries. Non-superuser role `"AyaNova"` named like the database with password `p @:/?#% x`, database restored with `--role`: boot, backup, 22/22 reports, suite = baseline | diff --git a/server/AyaNova/biz/ReportBiz.cs b/server/AyaNova/biz/ReportBiz.cs index 1025ac87..326d726f 100644 --- a/server/AyaNova/biz/ReportBiz.cs +++ b/server/AyaNova/biz/ReportBiz.cs @@ -795,8 +795,11 @@ What could be the issue here? //case 4586 //puppeteer has gone chrome dev by default and so does puppeteersharp which tests ok so I'm going to allow the default - //this means the files go into Chrome and ChromeHeadlessShell folders, under the data path (not beside the app, see ServerBootConfig) - var FetcherOptions = new BrowserFetcherOptions(); + //this means the files go into a ChromeHeadlessShell folder, under the data path (not beside the app, see ServerBootConfig) + //chrome-headless-shell, not full Chrome: it is Chrome's headless-only build for automation and PDF printing. + //Measured on a 1GB Ubuntu droplet printing a report-sized page: about 1s and 620MB vs 4.5s and 1.6GB for full Chrome, + //and a browser is started for every report render. Full Chrome also failed to start under a service user with no writable home + var FetcherOptions = new BrowserFetcherOptions { Browser = SupportedBrowser.ChromeHeadlessShell }; if (!string.IsNullOrWhiteSpace(ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH)) FetcherOptions.Path = ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH; try @@ -804,6 +807,7 @@ What could be the issue here? var InstalledBrowser = await new BrowserFetcher(FetcherOptions).DownloadAsync(); //launch exactly what was downloaded, otherwise Puppeteer looks in its default folder beside the app lo.ExecutablePath = InstalledBrowser.GetExecutablePath(); + lo.HeadlessMode = HeadlessMode.Shell; log.LogDebug($"Report browser: {lo.ExecutablePath}"); } catch (Exception ex) @@ -856,6 +860,27 @@ What could be the issue here? { lo.Args = lo.Args.Append($"--lang={ServerLanguage}").ToArray(); } + //Chrome keeps config and crash-reporter files under the user's home folder and fails to start when it isn't writable, + //as for a Linux service user like www-data (home /var/www). Point it at folders we own instead; harmless on Windows. + //Matters mostly for a full Chrome given in AYANOVA_REPORT_RENDER_BROWSER_PATH, chrome-headless-shell starts without them + var BrowserWorkFolder = Path.Combine( + string.IsNullOrWhiteSpace(ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH) + ? ServerBootConfig.AYANOVA_TEMP_FILES_PATH + : ServerBootConfig.REPORT_RENDER_BROWSER_DOWNLOAD_PATH, + "browser-home" + ); + try + { + Directory.CreateDirectory(Path.Combine(BrowserWorkFolder, "config")); + Directory.CreateDirectory(Path.Combine(BrowserWorkFolder, "cache")); + lo.Env["XDG_CONFIG_HOME"] = Path.Combine(BrowserWorkFolder, "config"); + lo.Env["XDG_CACHE_HOME"] = Path.Combine(BrowserWorkFolder, "cache"); + } + catch (Exception ex) + { + //not fatal, the launch below reports any real problem + log.LogWarning(ex, $"Could not create report browser work folders under {BrowserWorkFolder}"); + } log.LogDebug($"Report browser launch arguments: '{string.Join(" ", lo.Args)}'"); System.Text.StringBuilder PageLog = new System.Text.StringBuilder();