diff --git a/CLAUDE.md b/CLAUDE.md index c8f6b991..b7ad1a1a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -97,5 +97,6 @@ dotnet publish server\AyaNova\AyaNova.csproj -c Release -r linux-x64 --no-self-c | `devdocs/modernization/PROJECT-OVERVIEW.md` | Products, deployment targets, business context | | `devdocs/modernization/CODEBASE-REFERENCE.md` | Stack, structure, configuration, API conventions | | `devdocs/modernization/TESTING.md` | Test suites and their state | -| `devdocs/build/BUILD.md` | Release build, packaging, version numbers | +| `devdocs/build/BUILD.md` | Release build, packaging | +| `devdocs/build/RELEASE-PROCEDURE.md` | Release steps from code-complete to announcement, tagging, subscription droplet upgrades and weekly updates, version bump | | `devdocs/build/DEV-VM-SETUP.md` | Deprecated build VM plan; its tool list still serves for setting up a build machine. Releases are built on the workstation | diff --git a/devdocs/build/BUILD.md b/devdocs/build/BUILD.md index e64c64bb..36c4002b 100644 --- a/devdocs/build/BUILD.md +++ b/devdocs/build/BUILD.md @@ -112,15 +112,8 @@ timestamp utility. ## Version numbers -Set in five places for every release: - -| File | Setting | At 8.2.4 | -|---|---|---| -| `raven\server\AyaNova\AyaNova.csproj` | `` | 8.2.4 | -| `raven\dist\install\windows\x64\standalone.iss` | `#define MyAppVersion` | 8.2.4 | -| `raven\dist\install\windows\x64\lan.iss` | `#define MyAppVersion` | 8.2.4 | -| `raven-client\ayanova\package.json` | `"version"` | 8.2.4 | -| `ravenqbi\install\qbi.iss` | `#define MyAppVersion` | 8.0.2 (own cycle) | +The full list of files to change, and when, is in `RELEASE-PROCEDURE.md`, step 12 (version bump). +QBI has its own cycle: `ravenqbi\install\qbi.iss`, `#define MyAppVersion` (8.0.2). ## Front-end build notes diff --git a/devdocs/build/RELEASE-PROCEDURE.md b/devdocs/build/RELEASE-PROCEDURE.md new file mode 100644 index 00000000..48dcfd2f --- /dev/null +++ b/devdocs/build/RELEASE-PROCEDURE.md @@ -0,0 +1,365 @@ +# AyaNova v8 release and server update procedure + +How a release goes from finished code to customers, and how the hosted subscription droplets are +updated every week. Replaces John's notes scattered across info cases (the Subversion-era +release notes and the subscription upgrade scripts of info case 3450). The build itself (tools, +`build-release.bat`, packages) is in `BUILD.md`; release-specific extras (for 8.3.0: the .NET 10 +runtime, the one-time droplet hardening) are in that release's checklist +(`devdocs/modernization/NET10-PORT.md`, section 5). + +Every command is a complete line for the shell named above it: PowerShell on the workstation, +bash on a droplet or the ayanova.com web server (logged in as root). + +## Release order at a glance + +1. Code complete: every case committed and pushed, local E2E passes +2. Docs: changelog, manual copyright date +3. Build with `build-release.bat` +4. Stage the installers and docs in the `next` folders on ayanova.com +5. Test on devtest: upgrade it from `next`, remote E2E, check each change +6. Tag the release in git +7. Publish: docs and downloads go live (not announced yet) +8. **Upgrade the subscription droplets** (Thursday, by time zone) +9. **Announce**: upload `latest-version.json` +10. Download every published file back and check its version +11. Tell subscription customers +12. Start the next version + +**Subscription droplets are upgraded before the release is announced** (John, 2026-10-26, from +8.3.0 on). Uploading `latest-version.json` is what tells every AyaNova installation that a new +version exists, so it is the very last publishing step. Between steps 7 and 9 the new docs and +downloads are already visible on the website; that gap is accepted. + +## 1. Code complete + +- All "next release" cases done, each committed with its case number and pushed, in every repo. +- Every repo clean and in step with its remote. All four must say `nothing to commit, working tree clean` and `up to date with 'origin/main'`: + +```powershell +git -C C:\data\code\raven status; git -C C:\data\code\raven-client status; git -C C:\data\code\raven-launcher status; git -C C:\data\code\raven-test-integration status +``` + +- Integration tests pass against the local server (`raven-test-integration`, see `devdocs/modernization/TESTING.md`). +- E2E passes against the local debug server (API on 7575, client dev server on 8080). In `raven-test\e2e`, `cypress.json` `expectedServerVersion` must be the new version: + +```powershell +cd C:\data\code\raven-test\e2e; .\local-run.bat +``` + +## 2. Docs + +**Often missed: check carefully against the previous release.** + +- `docs/8.0/ayanova/docs/changelog.md`: the new version's section is complete and reads well; remove "IN PROGRESS". +- `docs/8.0/ayanova/mkdocs.yml`: `copyright` line, year range and `REV-` date. +- `dist/latest-version.json`: the new version and its changelog anchor (for example `#ayanova-830`). Set at the version bump; check it here. + +## 3. Build + +Before the build, update the .NET SDK so the Windows standalone package and the launcher carry the +newest .NET runtime (they bundle the runtime of the SDK that builds them; see `BUILD.md`, Tools). +If winget finds nothing newer, nothing changes: + +```powershell +winget upgrade --id Microsoft.DotNet.SDK.10 -e --source winget +``` + +Then run the full build; it must end with "COMPLETED WITHOUT ERROR". Details in `BUILD.md`. + +```powershell +cd C:\data\code\raven; .\build-release.bat +``` + +Write down the commit each repo was built from (the tag in step 6 goes on exactly these): + +```powershell +git -C C:\data\code\raven log --oneline -1; git -C C:\data\code\raven-client log --oneline -1; git -C C:\data\code\raven-launcher log --oneline -1 +``` + +## 4. Stage in `next` + +On the ayanova.com web server (`ssh root@ayanova.com`), empty the previous release's `next` folders: + +```bash +rm -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/docs-next/* +``` + +Then, from PowerShell on the workstation, upload the installers and the built manual (each +command passes `scp` the full list of files and folders to copy): + +```powershell +scp (Get-ChildItem C:\data\code\raven\dist\installers -File).FullName root@ayanova.com:/var/www/html/ayanova.com/download/next/ +``` + +```powershell +scp -r (Get-ChildItem C:\data\code\raven\server\AyaNova\wwwroot\docs).FullName root@ayanova.com:/var/www/html/ayanova.com/docs-next/ +``` + +## 5. Test on devtest + +devtest.onayanova.com mirrors the customer droplets; it is the final confirmation before any +customer is touched (`devtestn10` is the scratch droplet for earlier, riskier tests). + +1. Bring its OS up to date as in the [weekly update](#weekly-os-update), reboot. +2. Any one-time steps this release needs (for 8.3.0: [the .NET 10 runtime and hardening](#one-time-steps-for-830)). +3. Upgrade it with the [next version upgrade commands](#upgrade-commands). +4. Remote E2E: + + ```powershell + cd C:\data\code\raven-test\e2e; .\remote-run.bat + ``` + +5. Check each item in the changelog on it, and that the new version number shows everywhere (login page, About form). + +If anything fails: fix it as a case, rebuild (step 3), restage (step 4) and test again. Nothing +is tagged or published until devtest passes. + +## 6. Tag the release in git + +A tag is a permanent name on one commit, so `8.3.0` always points at exactly the code that was +shipped, years later and whatever happens on `main` afterwards. An **annotated** tag (`-a`) also +records who tagged it, when, and a message. Tags are not sent by an ordinary `git push`, so each +one is pushed by name. Tag the repos whose code is in the release, at the commits noted in step 3 +(if nothing was committed since, that is the current `main`): + +```powershell +git -C C:\data\code\raven tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven push origin 8.3.0 +``` + +```powershell +git -C C:\data\code\raven-client tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-client push origin 8.3.0 +``` + +```powershell +git -C C:\data\code\raven-launcher tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-launcher push origin 8.3.0 +``` + +Check that the tags are on the server (each lists `refs/tags/8.3.0`): + +```powershell +git -C C:\data\code\raven ls-remote --tags origin; git -C C:\data\code\raven-client ls-remote --tags origin; git -C C:\data\code\raven-launcher ls-remote --tags origin +``` + +A pushed tag is never moved or deleted: if a problem turns up after release, the fix becomes the +next patch version (8.3.1) with its own tag. To see later what any release contained: +`git -C C:\data\code\raven show 8.3.0`, or compare two releases with +`git -C C:\data\code\raven log --oneline 8.2.4..8.3.0` (from 8.3.0 on; earlier releases were not tagged in git). + +## 7. Publish (not announced yet) + +On the ayanova.com web server. In the archive folder name (`YYYY-MM-DD-vX.Y.Z`), replace the +example date and version with the real release date and version: + +```bash +rm -r /var/www/html/ayanova.com/docs/* && cp -r /var/www/html/ayanova.com/docs-next/* /var/www/html/ayanova.com/docs +``` + +```bash +mkdir /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0 && cp -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0 +``` + +Remove the oldest release folder in `download/old`: only the three most recent releases are kept +(since December 2023; customers whose maintenance lapsed must purchase to upgrade, and the older +versions are kept only to diagnose a regression). List the folder, then remove the oldest; at +8.3.0 that is 8.2.2: + +```bash +ls /var/www/html/ayanova.com/download/old +``` + +```bash +rm -r /var/www/html/ayanova.com/download/old/2024-10-23-v8.2.2 +``` + +```bash +cp /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download && cp /var/www/html/ayanova.com/download/latest-version.json /var/www/html/ayanova.com/download/previous-latest-version.json +``` + +Do **not** upload the new `latest-version.json` yet. + +## 8. Upgrade the subscription droplets + +On the normal Thursday schedule and times (or out of band for an emergency), as in the +[weekly update](#weekly-os-update): snapshot, health check, OS update, reboot; then any +one-time steps for this release, then the [release upgrade commands](#upgrade-commands); then the +health check and the log entry. Mark each server as upgraded in its Sockeye subscription server record. + +## 9. Announce + +Only after the last subscription droplet is upgraded, from PowerShell on the workstation (overwrites +the old one; from this moment every AyaNova installation shows that the new version is available): + +```powershell +scp C:\data\code\raven\dist\latest-version.json root@ayanova.com:/var/www/html/ayanova.com/download/ +``` + +## 10. Check what was published + +Download every installer and package back from ayanova.com and check its version: Windows +installers by the .exe's ProductVersion, Linux zips by the `AyaNova.dll` inside. (Found +2026-10-25: the LAN installer on the website was still 8.2.3, so 8.2.4 had never been uploaded for LAN.) + +```powershell +$ProgressPreference = 'SilentlyContinue'; $d = "$env:TEMP\ayanova-published"; Remove-Item $d -Recurse -Force -ErrorAction SilentlyContinue; New-Item -ItemType Directory $d | Out-Null; foreach ($f in 'ayanova-windows-x64-single-setup.exe','ayanova-windows-x64-lan-setup.exe','ayanova-linux-x64-server.zip','ayanova-linux-x64-desktop.zip','ayanova-subscription-linux-x64-server.zip') { Invoke-WebRequest "https://www.ayanova.com/download/$f" -OutFile "$d\$f" -UseBasicParsing; if ($f -like '*.exe') { "$f $((Get-Item "$d\$f").VersionInfo.ProductVersion)" } else { Expand-Archive "$d\$f" "$d\$($f -replace '\.zip$','')" ; "$f $((Get-Item "$d\$($f -replace '\.zip$','')\AyaNova.dll").VersionInfo.ProductVersion)" } } +``` + +Every line must show the new version. + +## 11. Tell subscription customers + +Subscription customers are told after their server was upgraded, not before. Perpetual licence +customers see the new version in AyaNova itself; tell one only if they are waiting for a +specific change. Message: + +> Subject: Your AyaNova service was upgraded +> +> Your AyaNova service has been upgraded to the latest version. +> +> This release contains these features and improvements: +> https://ayanova.com/docs/changelog/#ayanova-830 +> +> Be sure to backup your data regularly: +> https://ayanova.com/docs/sub-service-intro/ + +## 12. Start the next version + +Version numbers follow [semantic versioning](https://semver.org/): MAJOR for incompatible API +changes, MINOR for new backward compatible functionality, PATCH for backward compatible fixes. + +The version bump is the **first commit** of the next release, `case NNNN - version bump`, one +commit per repo, made when work on the next release starts. Set the new version in: + +| Repo | File | What | +|---|---|---| +| raven | `server/AyaNova/AyaNova.csproj` | `` and `` | +| raven | `server/AyaNova/util/AyaNovaVersion.cs` | `VersionString` | +| raven | `dist/install/windows/x64/lan.iss` | `#define MyAppVersion` | +| raven | `dist/install/windows/x64/standalone.iss` | `#define MyAppVersion` | +| raven | `dist/latest-version.json` | `latestversion` and the `changelogurl` anchor | +| raven | `docs/8.0/ayanova/docs/changelog.md` | new version heading with "IN PROGRESS" under it | +| raven | `docs/8.0/ayanova/docs/ay-about.md` | the example version number | +| raven-client | `ayanova/package.json`, `ayanova/package-lock.json` | `"version"` (both places in the lock file) | +| raven-client | `ayanova/src/api/ayanova-version.js` | `version` | +| raven-launcher | `ayanova-version.cs` | `VersionString` | +| raven-test | `e2e/cypress.json` | `expectedServerVersion` | + +Searching each repo for the previous version number finds them all: + +```powershell +git -C C:\data\code\raven grep -n -F "8.3.0" -- server dist docs/8.0/ayanova/docs/ay-about.md; git -C C:\data\code\raven-client grep -n -F "8.3.0" -- ayanova/package.json ayanova/src/api; git -C C:\data\code\raven-launcher grep -n -F "8.3.0" +``` + +QBI (`ravenqbi`) has its own version cycle (`install\qbi.iss`). + +## Upgrade commands + +Run on a subscription droplet as root. They stop AyaNova, download the new zip, replace the +program files (keeping `config.json` and everything in `data`), and start AyaNova again. `curl -f` +makes a failed download stop the whole chain before anything is deleted; without it a missing +file would be saved as an error page, the program files deleted and the unzip would fail, leaving +the server down until the snapshot is restored. + +**Next version, for testing on devtest** (from the `next` folder): + +```bash +cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/next/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service +``` + +**Release version, for customer droplets:** + +```bash +cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service +``` + +Then `reboot`, and the health check: `https://.onayanova.com/health`. + +## One-time steps for 8.3.0 + +Every existing droplet, once, right before its 8.3.0 upgrade. Both are safe to run again. + +**.NET 10 runtime** (8.3.0 needs it; on Ubuntu 24.04 and 26.04 it comes from Ubuntu's own archive): + +```bash +apt-get install -y aspnetcore-runtime-10.0 +``` + +**Hardening** (case 4727; what the setup script now does for new droplets): AyaNova listens only on +127.0.0.1, the program folder becomes read only for the service account, `config.json` 640, the +old 8.2.x Chrome folders are removed, and the systemd security settings are added as a drop-in. +Tested 2026-10-26 on `devtestn10`. Save it on the droplet with `nano /root/harden-830.sh`, paste, +save, then run `bash /root/harden-830.sh`; it must end with `HARDENING DONE`. It takes effect when +the upgrade commands start AyaNova. + +```bash +#!/bin/bash +# AyaNova 8.3.0: one-time hardening of an existing subscription droplet (case 4727, F36 and F38). +# Run once per droplet as root, right before the 8.3.0 upgrade commands. Safe to run again. +set -e + +# safety copies, kept from the first run only +[ -e /root/config.json.before-8.3.0 ] || cp /var/ayanova/config.json /root/config.json.before-8.3.0 +[ -e /root/ayanova.service.before-8.3.0 ] || cp /etc/systemd/system/ayanova.service /root/ayanova.service.before-8.3.0 + +# F36: nginx is the only way in: listen on this server only, not on its public address +sed -i 's#http://[*]:7575#http://127.0.0.1:7575#' /var/ayanova/config.json +grep -q '"http://127.0.0.1:7575"' /var/ayanova/config.json + +# F38: program files read only for the service account; it writes to data only +chmod -R go-w /var/ayanova +chmod -R g+rw /var/ayanova/data +# config.json holds the database password: readable by root and the service group only +chown root:www-data /var/ayanova/config.json +chmod 640 /var/ayanova/config.json +# 8.2.x report browser folders; 8.3.0 downloads the browser into data/report-browser +rm -rf /var/ayanova/Chrome /var/ayanova/ChromeHeadlessShell + +# F38: systemd sandboxing, as a drop-in so the existing unit file is left as it is +mkdir -p /etc/systemd/system/ayanova.service.d +cat > /etc/systemd/system/ayanova.service.d/hardening.conf <<'EOF' +[Service] +NoNewPrivileges=true +ProtectSystem=strict +ReadWritePaths=/var/ayanova/data +PrivateTmp=true +ProtectHome=true +EOF +systemctl daemon-reload + +echo HARDENING DONE +``` + +After the upgrade, check that AyaNova listens only on 127.0.0.1 (`ss -tln | grep 7575` shows +`127.0.0.1:7575`) and that the health check passes. + +## Weekly OS update + +Every Thursday, Ubuntu updates on every subscription droplet. Customers have been told updates +happen on Thursday nights after 7 pm their time (downtime under a minute; none has asked for +another time). All droplets have phased updates turned off, so they get the same packages +devtest was tested with (`/etc/apt/apt.conf.d/52no-phasing`; the setup script adds it to new droplets). + +Order: devtest first (it mirrors the customer droplets), then customers in groups by time zone, +at John's 7 am (Singapore, Australia), 4 pm (US east coast and Europe, most servers) and 7 pm +(US west coast). + +Per droplet: + +1. In the DigitalOcean control panel: delete last week's snapshot, take a new one. +2. Confirm the server is healthy: `https://.onayanova.com/health`. +3. SSH in, remove last week's leftovers, update, reboot (always reboot: skipping it caused intermittent problems in the past): + + ```bash + apt autoremove + ``` + + ```bash + apt-get update && apt-get -V full-upgrade + ``` + + ```bash + reboot + ``` + +4. Copy the session's console output into the week's log file (kept until the next week's updates, to look back at if something comes up). +5. Mark the date and time of the update in the server's Sockeye subscription server record. +6. Refresh the server's AyaNova page and confirm the login page appears; open `/health` in place of `/login` and paste the result into the week's log as the last entry for that server.