raven/devdocs/build/RELEASE-PROCEDURE.md

# AyaNova v8 release and server update procedure

How a release goes from finished code to customers, and how the hosted subscription droplets are
updated every week. Replaces John's notes scattered across info cases (the Subversion-era
release notes and the subscription upgrade scripts of info case 3450). The build itself (tools,
`build-release.bat`, packages) is in `BUILD.md`; release-specific extras (for 8.3.0: the .NET 10
runtime, the one-time droplet hardening) are in that release's checklist
(`devdocs/modernization/NET10-PORT.md`, section 5).

Every command is a complete line for the shell named above it: PowerShell on the workstation,
bash on a droplet or the ayanova.com web server (logged in as root).

## Release order at a glance

1. Code complete: every case committed and pushed, local E2E passes
2. Docs: changelog, manual copyright date
3. Build with `build-release.bat`
4. Stage the installers and docs in the `next` folders on ayanova.com
5. Test on devtest: upgrade it from `next`, remote E2E, check each change
6. Tag the release in git
7. Publish: docs and downloads go live (not announced yet)
8. **Upgrade the subscription droplets** (Thursday, by time zone)
9. **Announce**: upload `latest-version.json`
10. Download every published file back and check its version
11. Tell subscription customers
12. Start the next version

**Subscription droplets are upgraded before the release is announced** (John, 2026-10-26, from
8.3.0 on). Uploading `latest-version.json` is what tells every AyaNova installation that a new
version exists, so it is the very last publishing step. Between steps 7 and 9 the new docs and
downloads are already visible on the website; that gap is accepted.

## 1. Code complete

- All "next release" cases done, each committed with its case number and pushed, in every repo.
- Every repo clean and in step with its remote. All four must say `nothing to commit, working tree clean` and `up to date with 'origin/main'`:

```powershell
git -C C:\data\code\raven status; git -C C:\data\code\raven-client status; git -C C:\data\code\raven-launcher status; git -C C:\data\code\raven-test-integration status
```

- Integration tests pass against the local server (`raven-test-integration`, see `devdocs/modernization/TESTING.md`).
- E2E passes against the local debug server (API on 7575, client dev server on 8080). In `raven-test\e2e`, `cypress.json` `expectedServerVersion` must be the new version:

```powershell
cd C:\data\code\raven-test\e2e; .\local-run.bat
```

## 2. Docs

**Often missed: check carefully against the previous release.**

- `docs/8.0/ayanova/docs/changelog.md`: the new version's section is complete and reads well; remove "IN PROGRESS".
- `docs/8.0/ayanova/mkdocs.yml`: `copyright` line, year range and `REV-` date.
- `dist/latest-version.json`: the new version and its changelog anchor (for example `#ayanova-830`). Set at the version bump; check it here.

## 3. Build

Before the build, update the .NET SDK so the Windows standalone package and the launcher carry the
newest .NET runtime (they bundle the runtime of the SDK that builds them; see `BUILD.md`, Tools).
If winget finds nothing newer, nothing changes:

```powershell
winget upgrade --id Microsoft.DotNet.SDK.10 -e --source winget
```

Then run the full build; it must end with "COMPLETED WITHOUT ERROR". Details in `BUILD.md`.

```powershell
cd C:\data\code\raven; .\build-release.bat
```

Write down the commit each repo was built from (the tag in step 6 goes on exactly these):

```powershell
git -C C:\data\code\raven log --oneline -1; git -C C:\data\code\raven-client log --oneline -1; git -C C:\data\code\raven-launcher log --oneline -1
```

## 4. Stage in `next`

On the ayanova.com web server (`ssh root@ayanova.com`), empty the previous release's `next` folders:

```bash
rm -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/docs-next/*
```

Then, from PowerShell on the workstation, upload the installers and the built manual (each
command passes `scp` the full list of files and folders to copy):

```powershell
scp (Get-ChildItem C:\data\code\raven\dist\installers -File).FullName root@ayanova.com:/var/www/html/ayanova.com/download/next/
```

```powershell
scp -r (Get-ChildItem C:\data\code\raven\server\AyaNova\wwwroot\docs).FullName root@ayanova.com:/var/www/html/ayanova.com/docs-next/
```

## 5. Test on devtest

devtest.onayanova.com mirrors the customer droplets; it is the final confirmation before any
customer is touched. For earlier, riskier tests, create a temporary droplet when needed and
destroy it when done, so it does not run up costs (for 8.3.0 that was `devtestn10`).

1. Bring its OS up to date as in the [weekly update](#weekly-os-update), reboot.
2. Any one-time steps this release needs (for 8.3.0: [the .NET 10 runtime and hardening](#one-time-steps-for-830)).
3. Upgrade it with the [next version upgrade commands](#upgrade-commands).
4. Remote E2E:

   ```powershell
   cd C:\data\code\raven-test\e2e; .\remote-run.bat
   ```

5. Check each item in the changelog on it, and that the new version number shows everywhere (login page, About form).

If anything fails: fix it as a case, rebuild (step 3), restage (step 4) and test again. Nothing
is tagged or published until devtest passes.

## 6. Tag the release in git

A tag is a permanent name on one commit, so `8.3.0` always points at exactly the code that was
shipped, years later and whatever happens on `main` afterwards. An **annotated** tag (`-a`) also
records who tagged it, when, and a message. Tags are not sent by an ordinary `git push`, so each
one is pushed by name. Tag the repos whose code is in the release, at the commits noted in step 3
(if nothing was committed since, that is the current `main`):

```powershell
git -C C:\data\code\raven tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven push origin 8.3.0
```

```powershell
git -C C:\data\code\raven-client tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-client push origin 8.3.0
```

```powershell
git -C C:\data\code\raven-launcher tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-launcher push origin 8.3.0
```

Check that the tags are on the server (each lists `refs/tags/8.3.0`):

```powershell
git -C C:\data\code\raven ls-remote --tags origin; git -C C:\data\code\raven-client ls-remote --tags origin; git -C C:\data\code\raven-launcher ls-remote --tags origin
```

A pushed tag is never moved or deleted: if a problem turns up after release, the fix becomes the
next patch version (8.3.1) with its own tag. To see later what any release contained:
`git -C C:\data\code\raven show 8.3.0`, or compare two releases with
`git -C C:\data\code\raven log --oneline 8.2.4..8.3.0` (from 8.3.0 on; earlier releases were not tagged in git).

## 7. Publish (not announced yet)

On the ayanova.com web server. In the archive folder name (`YYYY-MM-DD-vX.Y.Z`), replace the
example date and version with the real release date and version:

```bash
rm -r /var/www/html/ayanova.com/docs/* && cp -r /var/www/html/ayanova.com/docs-next/* /var/www/html/ayanova.com/docs
```

```bash
mkdir /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0 && cp -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0
```

Remove the oldest release folder in `download/old`: only the three most recent releases are kept
(since December 2023; customers whose maintenance lapsed must purchase to upgrade, and the older
versions are kept only to diagnose a regression). List the folder, then remove the oldest; at
8.3.0 that is 8.2.2:

```bash
ls /var/www/html/ayanova.com/download/old
```

```bash
rm -r /var/www/html/ayanova.com/download/old/2024-10-23-v8.2.2
```

```bash
cp /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download && cp /var/www/html/ayanova.com/download/latest-version.json /var/www/html/ayanova.com/download/previous-latest-version.json
```

Do **not** upload the new `latest-version.json` yet.

## 8. Upgrade the subscription droplets

On the normal Thursday schedule and times (or out of band for an emergency), as in the
[weekly update](#weekly-os-update): snapshot, health check, OS update, reboot; then any
one-time steps for this release, then the [release upgrade commands](#upgrade-commands); then the
health check and the log entry. Mark each server as upgraded in its Sockeye subscription server record.

## 9. Announce

Only after the last subscription droplet is upgraded, from PowerShell on the workstation (overwrites
the old one; from this moment every AyaNova installation shows that the new version is available):

```powershell
scp C:\data\code\raven\dist\latest-version.json root@ayanova.com:/var/www/html/ayanova.com/download/
```

## 10. Check what was published

Download every installer and package back from ayanova.com and check its version: Windows
installers by the .exe's ProductVersion, Linux zips by the `AyaNova.dll` inside. (Found
2026-10-25: the LAN installer on the website was still 8.2.3, so 8.2.4 had never been uploaded for LAN.)

```powershell
$ProgressPreference = 'SilentlyContinue'; $d = "$env:TEMP\ayanova-published"; Remove-Item $d -Recurse -Force -ErrorAction SilentlyContinue; New-Item -ItemType Directory $d | Out-Null; foreach ($f in 'ayanova-windows-x64-single-setup.exe','ayanova-windows-x64-lan-setup.exe','ayanova-linux-x64-server.zip','ayanova-linux-x64-desktop.zip','ayanova-subscription-linux-x64-server.zip') { Invoke-WebRequest "https://www.ayanova.com/download/$f" -OutFile "$d\$f" -UseBasicParsing; if ($f -like '*.exe') { "$f  $((Get-Item "$d\$f").VersionInfo.ProductVersion)" } else { Expand-Archive "$d\$f" "$d\$($f -replace '\.zip$','')" ; "$f  $((Get-Item "$d\$($f -replace '\.zip$','')\AyaNova.dll").VersionInfo.ProductVersion)" } }
```

Every line must show the new version.

## 11. Tell subscription customers

Subscription customers are told after their server was upgraded, not before. Perpetual licence
customers see the new version in AyaNova itself; tell one only if they are waiting for a
specific change. Message:

> Subject: Your AyaNova service was upgraded
>
> Your AyaNova service has been upgraded to the latest version.
>
> This release contains these features and improvements:
> https://ayanova.com/docs/changelog/#ayanova-830
>
> Be sure to backup your data regularly:
> https://ayanova.com/docs/sub-service-intro/

## 12. Start the next version

Version numbers follow [semantic versioning](https://semver.org/): MAJOR for incompatible API
changes, MINOR for new backward compatible functionality, PATCH for backward compatible fixes.

The version bump is the **first commit** of the next release, `case NNNN - version bump`, one
commit per repo, made when work on the next release starts. Set the new version in:

| Repo | File | What |
|---|---|---|
| raven | `server/AyaNova/AyaNova.csproj` | `<Version>` and `<FileVersion>` |
| raven | `server/AyaNova/util/AyaNovaVersion.cs` | `VersionString` |
| raven | `dist/install/windows/x64/lan.iss` | `#define MyAppVersion` |
| raven | `dist/install/windows/x64/standalone.iss` | `#define MyAppVersion` |
| raven | `dist/latest-version.json` | `latestversion` and the `changelogurl` anchor |
| raven | `docs/8.0/ayanova/docs/changelog.md` | new version heading with "IN PROGRESS" under it |
| raven | `docs/8.0/ayanova/docs/ay-about.md` | the example version number |
| raven-client | `ayanova/package.json`, `ayanova/package-lock.json` | `"version"` (both places in the lock file) |
| raven-client | `ayanova/src/api/ayanova-version.js` | `version` |
| raven-launcher | `ayanova-version.cs` | `VersionString` |
| raven-test | `e2e/cypress.json` | `expectedServerVersion` |

Searching each repo for the previous version number finds them all:

```powershell
git -C C:\data\code\raven grep -n -F "8.3.0" -- server dist docs/8.0/ayanova/docs/ay-about.md; git -C C:\data\code\raven-client grep -n -F "8.3.0" -- ayanova/package.json ayanova/src/api; git -C C:\data\code\raven-launcher grep -n -F "8.3.0"
```

QBI (`ravenqbi`) has its own version cycle (`install\qbi.iss`).

## Upgrade commands

Run on a subscription droplet as root. They stop AyaNova, download the new zip, replace the
program files (keeping `config.json` and everything in `data`), and start AyaNova again. `curl -f`
makes a failed download stop the whole chain before anything is deleted; without it a missing
file would be saved as an error page, the program files deleted and the unzip would fail, leaving
the server down until the snapshot is restored.

**Next version, for testing on devtest** (from the `next` folder):

```bash
cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/next/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service
```

**Release version, for customer droplets:**

```bash
cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service
```

Then `reboot`, and the health check: `https://<server>.onayanova.com/health`.

## One-time steps for 8.3.0

Every existing droplet, once, right before its 8.3.0 upgrade. Both are safe to run again.

**.NET 10 runtime** (8.3.0 needs it; on Ubuntu 24.04 and 26.04 it comes from Ubuntu's own archive):

```bash
apt-get install -y aspnetcore-runtime-10.0
```

**Hardening** (case 4727; what the setup script now does for new droplets): AyaNova listens only on
127.0.0.1, the program folder becomes read only for the service account, `config.json` 640, the
old 8.2.x Chrome folders are removed, and the systemd security settings are added as a drop-in.
Tested 2026-10-26 on `devtestn10`. Save it on the droplet with `nano /root/harden-830.sh`, paste,
save, then run `bash /root/harden-830.sh`; it must end with `HARDENING DONE`. It takes effect when
the upgrade commands start AyaNova.

```bash
#!/bin/bash
# AyaNova 8.3.0: one-time hardening of an existing subscription droplet (case 4727, F36 and F38).
# Run once per droplet as root, right before the 8.3.0 upgrade commands. Safe to run again.
set -e

# safety copies, kept from the first run only
[ -e /root/config.json.before-8.3.0 ] || cp /var/ayanova/config.json /root/config.json.before-8.3.0
[ -e /root/ayanova.service.before-8.3.0 ] || cp /etc/systemd/system/ayanova.service /root/ayanova.service.before-8.3.0

# F36: nginx is the only way in: listen on this server only, not on its public address
sed -i 's#http://[*]:7575#http://127.0.0.1:7575#' /var/ayanova/config.json
grep -q '"http://127.0.0.1:7575"' /var/ayanova/config.json

# F38: program files read only for the service account; it writes to data only
chmod -R go-w /var/ayanova
chmod -R g+rw /var/ayanova/data
# config.json holds the database password: readable by root and the service group only
chown root:www-data /var/ayanova/config.json
chmod 640 /var/ayanova/config.json
# 8.2.x report browser folders; 8.3.0 downloads the browser into data/report-browser
rm -rf /var/ayanova/Chrome /var/ayanova/ChromeHeadlessShell

# F38: systemd sandboxing, as a drop-in so the existing unit file is left as it is
mkdir -p /etc/systemd/system/ayanova.service.d
cat > /etc/systemd/system/ayanova.service.d/hardening.conf <<'EOF'
[Service]
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/ayanova/data
PrivateTmp=true
ProtectHome=true
EOF
systemctl daemon-reload

echo HARDENING DONE
```

After the upgrade, check that AyaNova listens only on 127.0.0.1 (`ss -tln | grep 7575` shows
`127.0.0.1:7575`) and that the health check passes.

## Weekly OS update

Every Thursday, Ubuntu updates on every subscription droplet. Customers have been told updates
happen on Thursday nights after 7 pm their time (downtime under a minute; none has asked for
another time). All droplets have phased updates turned off, so they get the same packages
devtest was tested with (`/etc/apt/apt.conf.d/52no-phasing`; the setup script adds it to new droplets).

Order: devtest first (it mirrors the customer droplets), then customers in groups by time zone,
at John's 7 am (Singapore, Australia), 4 pm (US east coast and Europe, most servers) and 7 pm
(US west coast).

Per droplet:

1. In the DigitalOcean control panel: delete last week's snapshot, take a new one.
2. Confirm the server is healthy: `https://<server>.onayanova.com/health`.
3. SSH in, remove last week's leftovers, update, reboot (always reboot: skipping it caused intermittent problems in the past):

   ```bash
   apt autoremove
   ```

   ```bash
   apt-get update && apt-get -V full-upgrade
   ```

   ```bash
   reboot
   ```

4. Copy the session's console output into the week's log file (kept until the next week's updates, to look back at if something comes up).
5. Mark the date and time of the update in the server's Sockeye subscription server record.
6. Refresh the server's AyaNova page and confirm the login page appears; open `/health` in place of `/login` and paste the result into the week's log as the last entry for that server.