raven/devdocs/build/RELEASE-PROCEDURE.md

AyaNova v8 release and server update procedure

How a release goes from finished code to customers, and how the hosted subscription droplets are updated every week. Replaces John's notes scattered across info cases (the Subversion-era release notes and the subscription upgrade scripts of info case 3450). The build itself (tools, build-release.bat, packages) is in BUILD.md; release-specific extras (for 8.3.0: the .NET 10 runtime, the one-time droplet hardening) are in that release's checklist (devdocs/modernization/NET10-PORT.md, section 5).

Every command is a complete line for the shell named above it: PowerShell on the workstation, bash on a droplet or the ayanova.com web server (logged in as root).

Release order at a glance

1. Code complete: every case committed and pushed, local E2E passes
2. Docs: changelog, manual copyright date
3. Build with build-release.bat
4. Stage the installers and docs in the next folders on ayanova.com
5. Test on devtest: upgrade it from next, remote E2E, check each change
6. Tag the release in git
7. Publish: docs and downloads go live (not announced yet)
8. Upgrade the subscription droplets (Thursday, by time zone)
9. Announce: upload latest-version.json
10. Download every published file back and check its version
11. Tell subscription customers
12. Start the next version

Subscription droplets are upgraded before the release is announced (John, 2026-10-26, from 8.3.0 on). Uploading latest-version.json is what tells every AyaNova installation that a new version exists, so it is the very last publishing step. Between steps 7 and 9 the new docs and downloads are already visible on the website; that gap is accepted.

1. Code complete

• All "next release" cases done, each committed with its case number and pushed, in every repo.
• Every repo clean and in step with its remote. All four must say nothing to commit, working tree clean and up to date with 'origin/main':

git -C C:\data\code\raven status; git -C C:\data\code\raven-client status; git -C C:\data\code\raven-launcher status; git -C C:\data\code\raven-test-integration status

• Integration tests pass against the local server (raven-test-integration, see devdocs/modernization/TESTING.md).
• E2E passes against the local debug server (API on 7575, client dev server on 8080). In raven-test\e2e, cypress.json expectedServerVersion must be the new version:

cd C:\data\code\raven-test\e2e; .\local-run.bat

2. Docs

Often missed: check carefully against the previous release.

• docs/8.0/ayanova/docs/changelog.md: the new version's section is complete and reads well; remove "IN PROGRESS".
• docs/8.0/ayanova/mkdocs.yml: copyright line, year range and REV- date.
• dist/latest-version.json: the new version and its changelog anchor (for example #ayanova-830). Set at the version bump; check it here.

3. Build

Before the build, update the .NET SDK so the Windows standalone package and the launcher carry the newest .NET runtime (they bundle the runtime of the SDK that builds them; see BUILD.md, Tools). If winget finds nothing newer, nothing changes:

winget upgrade --id Microsoft.DotNet.SDK.10 -e --source winget

Then run the full build; it must end with "COMPLETED WITHOUT ERROR". Details in BUILD.md.

cd C:\data\code\raven; .\build-release.bat

Write down the commit each repo was built from (the tag in step 6 goes on exactly these):

git -C C:\data\code\raven log --oneline -1; git -C C:\data\code\raven-client log --oneline -1; git -C C:\data\code\raven-launcher log --oneline -1

4. Stage in next

On the ayanova.com web server (ssh root@ayanova.com), empty the previous release's next folders:

rm -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/docs-next/*

Then, from PowerShell on the workstation, upload the installers and the built manual (each command passes scp the full list of files and folders to copy):

scp (Get-ChildItem C:\data\code\raven\dist\installers -File).FullName root@ayanova.com:/var/www/html/ayanova.com/download/next/

scp -r (Get-ChildItem C:\data\code\raven\server\AyaNova\wwwroot\docs).FullName root@ayanova.com:/var/www/html/ayanova.com/docs-next/

5. Test on devtest

devtest.onayanova.com mirrors the customer droplets; it is the final confirmation before any customer is touched. For earlier, riskier tests, create a temporary droplet when needed and destroy it when done, so it does not run up costs (for 8.3.0 that was devtestn10).

1. Bring its OS up to date as in the weekly update, reboot.

2. Any one-time steps this release needs (for 8.3.0: the .NET 10 runtime and hardening).

3. Upgrade it with the next version upgrade commands.

4. Remote E2E:

   cd C:\data\code\raven-test\e2e; .\remote-run.bat
   

5. Check each item in the changelog on it, and that the new version number shows everywhere (login page, About form).

If anything fails: fix it as a case, rebuild (step 3), restage (step 4) and test again. Nothing is tagged or published until devtest passes.

6. Tag the release in git

A tag is a permanent name on one commit, so 8.3.0 always points at exactly the code that was shipped, years later and whatever happens on main afterwards. An annotated tag (-a) also records who tagged it, when, and a message. Tags are not sent by an ordinary git push, so each one is pushed by name. Tag the repos whose code is in the release, at the commits noted in step 3 (if nothing was committed since, that is the current main):

git -C C:\data\code\raven tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven push origin 8.3.0

git -C C:\data\code\raven-client tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-client push origin 8.3.0

git -C C:\data\code\raven-launcher tag -a 8.3.0 -m "AyaNova 8.3.0"; git -C C:\data\code\raven-launcher push origin 8.3.0

Check that the tags are on the server (each lists refs/tags/8.3.0):

git -C C:\data\code\raven ls-remote --tags origin; git -C C:\data\code\raven-client ls-remote --tags origin; git -C C:\data\code\raven-launcher ls-remote --tags origin

A pushed tag is never moved or deleted: if a problem turns up after release, the fix becomes the next patch version (8.3.1) with its own tag. To see later what any release contained: git -C C:\data\code\raven show 8.3.0, or compare two releases with git -C C:\data\code\raven log --oneline 8.2.4..8.3.0 (from 8.3.0 on; earlier releases were not tagged in git).

7. Publish (not announced yet)

On the ayanova.com web server. In the archive folder name (YYYY-MM-DD-vX.Y.Z), replace the example date and version with the real release date and version:

rm -r /var/www/html/ayanova.com/docs/* && cp -r /var/www/html/ayanova.com/docs-next/* /var/www/html/ayanova.com/docs

mkdir /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0 && cp -r /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download/old/2026-11-05-v8.3.0

Remove the oldest release folder in download/old: only the three most recent releases are kept (since December 2023; customers whose maintenance lapsed must purchase to upgrade, and the older versions are kept only to diagnose a regression). List the folder, then remove the oldest; at 8.3.0 that is 8.2.2:

ls /var/www/html/ayanova.com/download/old

rm -r /var/www/html/ayanova.com/download/old/2024-10-23-v8.2.2

cp /var/www/html/ayanova.com/download/next/* /var/www/html/ayanova.com/download && cp /var/www/html/ayanova.com/download/latest-version.json /var/www/html/ayanova.com/download/previous-latest-version.json

Do not upload the new latest-version.json yet.

8. Upgrade the subscription droplets

On the normal Thursday schedule and times (or out of band for an emergency), as in the weekly update: snapshot, health check, OS update, reboot; then any one-time steps for this release, then the release upgrade commands; then the health check and the log entry. Mark each server as upgraded in its Sockeye subscription server record.

9. Announce

Only after the last subscription droplet is upgraded, from PowerShell on the workstation (overwrites the old one; from this moment every AyaNova installation shows that the new version is available):

scp C:\data\code\raven\dist\latest-version.json root@ayanova.com:/var/www/html/ayanova.com/download/

10. Check what was published

Download every installer and package back from ayanova.com and check its version: Windows installers by the .exe's ProductVersion, Linux zips by the AyaNova.dll inside. (Found 2026-10-25: the LAN installer on the website was still 8.2.3, so 8.2.4 had never been uploaded for LAN.)

$ProgressPreference = 'SilentlyContinue'; $d = "$env:TEMP\ayanova-published"; Remove-Item $d -Recurse -Force -ErrorAction SilentlyContinue; New-Item -ItemType Directory $d | Out-Null; foreach ($f in 'ayanova-windows-x64-single-setup.exe','ayanova-windows-x64-lan-setup.exe','ayanova-linux-x64-server.zip','ayanova-linux-x64-desktop.zip','ayanova-subscription-linux-x64-server.zip') { Invoke-WebRequest "https://www.ayanova.com/download/$f" -OutFile "$d\$f" -UseBasicParsing; if ($f -like '*.exe') { "$f  $((Get-Item "$d\$f").VersionInfo.ProductVersion)" } else { Expand-Archive "$d\$f" "$d\$($f -replace '\.zip$','')" ; "$f  $((Get-Item "$d\$($f -replace '\.zip$','')\AyaNova.dll").VersionInfo.ProductVersion)" } }

Every line must show the new version.

11. Tell subscription customers

Subscription customers are told after their server was upgraded, not before. Perpetual licence customers see the new version in AyaNova itself; tell one only if they are waiting for a specific change. Message:

Subject: Your AyaNova service was upgraded

Your AyaNova service has been upgraded to the latest version.

This release contains these features and improvements: https://ayanova.com/docs/changelog/#ayanova-830

Be sure to backup your data regularly: https://ayanova.com/docs/sub-service-intro/

12. Start the next version

Version numbers follow semantic versioning: MAJOR for incompatible API changes, MINOR for new backward compatible functionality, PATCH for backward compatible fixes.

The version bump is the first commit of the next release, case NNNN - version bump, one commit per repo, made when work on the next release starts. Set the new version in:

Repo	File	What
raven	server/AyaNova/AyaNova.csproj	<Version> and <FileVersion>
raven	server/AyaNova/util/AyaNovaVersion.cs	VersionString
raven	dist/install/windows/x64/lan.iss	#define MyAppVersion
raven	dist/install/windows/x64/standalone.iss	#define MyAppVersion
raven	dist/latest-version.json	latestversion and the changelogurl anchor
raven	docs/8.0/ayanova/docs/changelog.md	new version heading with "IN PROGRESS" under it
raven	docs/8.0/ayanova/docs/ay-about.md	the example version number
raven-client	ayanova/package.json, ayanova/package-lock.json	"version" (both places in the lock file)
raven-client	ayanova/src/api/ayanova-version.js	version
raven-launcher	ayanova-version.cs	VersionString
raven-test	e2e/cypress.json	expectedServerVersion

Searching each repo for the previous version number finds them all:

git -C C:\data\code\raven grep -n -F "8.3.0" -- server dist docs/8.0/ayanova/docs/ay-about.md; git -C C:\data\code\raven-client grep -n -F "8.3.0" -- ayanova/package.json ayanova/src/api; git -C C:\data\code\raven-launcher grep -n -F "8.3.0"

QBI (ravenqbi) has its own version cycle (install\qbi.iss).

Upgrade commands

Run on a subscription droplet as root. They stop AyaNova, download the new zip, replace the program files (keeping config.json and everything in data), and start AyaNova again. curl -f makes a failed download stop the whole chain before anything is deleted; without it a missing file would be saved as an error page, the program files deleted and the unzip would fail, leaving the server down until the snapshot is restored.

Next version, for testing on devtest (from the next folder):

cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/next/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service

Release version, for customer droplets:

cd /var/ayanova && systemctl stop ayanova.service && curl -fO https://www.ayanova.com/download/ayanova-subscription-linux-x64-server.zip && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-subscription-linux-x64-server.zip -x "config.json" && systemctl start ayanova.service

Then reboot, and the health check: https://<server>.onayanova.com/health.

One-time steps for 8.3.0

Every existing droplet, once, right before its 8.3.0 upgrade. Both are safe to run again.

.NET 10 runtime (8.3.0 needs it; on Ubuntu 24.04 and 26.04 it comes from Ubuntu's own archive):

apt-get install -y aspnetcore-runtime-10.0

Hardening (case 4727; what the setup script now does for new droplets): AyaNova listens only on 127.0.0.1, the program folder becomes read only for the service account, config.json 640, the old 8.2.x Chrome folders are removed, and the systemd security settings are added as a drop-in. Tested 2026-10-26 on devtestn10. Save it on the droplet with nano /root/harden-830.sh, paste, save, then run bash /root/harden-830.sh; it must end with HARDENING DONE. It takes effect when the upgrade commands start AyaNova.

#!/bin/bash
# AyaNova 8.3.0: one-time hardening of an existing subscription droplet (case 4727, F36 and F38).
# Run once per droplet as root, right before the 8.3.0 upgrade commands. Safe to run again.
set -e

# safety copies, kept from the first run only
[ -e /root/config.json.before-8.3.0 ] || cp /var/ayanova/config.json /root/config.json.before-8.3.0
[ -e /root/ayanova.service.before-8.3.0 ] || cp /etc/systemd/system/ayanova.service /root/ayanova.service.before-8.3.0

# F36: nginx is the only way in: listen on this server only, not on its public address
sed -i 's#http://[*]:7575#http://127.0.0.1:7575#' /var/ayanova/config.json
grep -q '"http://127.0.0.1:7575"' /var/ayanova/config.json

# F38: program files read only for the service account; it writes to data only
chmod -R go-w /var/ayanova
chmod -R g+rw /var/ayanova/data
# config.json holds the database password: readable by root and the service group only
chown root:www-data /var/ayanova/config.json
chmod 640 /var/ayanova/config.json
# 8.2.x report browser folders; 8.3.0 downloads the browser into data/report-browser
rm -rf /var/ayanova/Chrome /var/ayanova/ChromeHeadlessShell

# F38: systemd sandboxing, as a drop-in so the existing unit file is left as it is
mkdir -p /etc/systemd/system/ayanova.service.d
cat > /etc/systemd/system/ayanova.service.d/hardening.conf <<'EOF'
[Service]
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/ayanova/data
PrivateTmp=true
ProtectHome=true
EOF
systemctl daemon-reload

echo HARDENING DONE

After the upgrade, check that AyaNova listens only on 127.0.0.1 (ss -tln | grep 7575 shows 127.0.0.1:7575) and that the health check passes.

Weekly OS update

Every Thursday, Ubuntu updates on every subscription droplet. Customers have been told updates happen on Thursday nights after 7 pm their time (downtime under a minute; none has asked for another time). All droplets have phased updates turned off, so they get the same packages devtest was tested with (/etc/apt/apt.conf.d/52no-phasing; the setup script adds it to new droplets).

Order: devtest first (it mirrors the customer droplets), then customers in groups by time zone, at John's 7 am (Singapore, Australia), 4 pm (US east coast and Europe, most servers) and 7 pm (US west coast).

Per droplet:

1. In the DigitalOcean control panel: delete last week's snapshot, take a new one.

2. Confirm the server is healthy: https://<server>.onayanova.com/health.

3. SSH in, remove last week's leftovers, update, reboot (always reboot: skipping it caused intermittent problems in the past):

   apt autoremove
   

   apt-get update && apt-get -V full-upgrade
   

   reboot
   

4. Copy the session's console output into the week's log file (kept until the next week's updates, to look back at if something comes up).

5. Mark the date and time of the update in the server's Sockeye subscription server record.

6. Refresh the server's AyaNova page and confirm the login page appears; open /health in place of /login and paste the result into the week's log as the last entry for that server.