The deploy was the one pipeline with no failure notification. It either worked or
you found out by noticing the site was stale — the same shape as the off-site
backup before today.
alert.js now takes CHECKLIST_ENV_FILE and CHECKLIST_NODE_MODULES, because the
deploy job runs in a container that has the repo but no server/.env and no
node_modules; both live on the NAS at the mounted deploy directory.
What this still does not catch, and the reason it is written down in the workflow:
a runner that wedges. Today the NAS runner claimed run 7, executed none of its six
steps, and Gitea timed it out fifteen minutes later — every step was marked failed
without one having run, so no in-job step could have reported it. The container
said "Up 2 hours" throughout and admitted on restart that it had been holding the
job. A stuck runner looks exactly like a slow one, and catching it needs something
watching from outside the run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly Drive backup runs on the ingest Mac (the one whose account is
"Studio"), from a copy of this project made before the repo existed. Fixes made
anywhere else therefore never reach the machine that actually runs the backup at
04:15 — which is how the fix for a silent backup failure could itself sit unused.
adopt-repo.sh converts that folder into a clone without losing the things git
deliberately does not carry: server/.env, the Play upload keystore and its
properties, ios-signing.config and local.properties. It renames the old folder
rather than deleting it, restores those files into the clone, re-points the
LaunchAgent, and puts the old folder back if the clone fails. --dry-run prints
the plan and changes nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Failures went to REPORT_EMAIL, which is the media@ shared inbox the Sunday digest
already uses. An alert competing with a routine weekly mail in a shared inbox is
an alert that gets scrolled past, which is most of the way back to the logfile
nobody read.
ALERT_EMAIL is now a separate, comma-separated list for breakage only, falling
back to REPORT_EMAIL so a machine whose .env predates this still alerts somewhere
rather than silently not at all.
Verified by sending one test alert, which arrived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly copy to Drive had been dead since 15 September. DRIVE_DIR pointed at
Apps/Backups/Checklist Backups; the folder is Apps/Checklist Backups, with no
Backups level. The guard for a missing folder did its job and exited — into a
logfile nobody reads, so from the outside a broken backup and a healthy no-op
looked identical. Eight days were missing; this run copied them.
That is the second time the same failure has hidden in the same place. The
header comment already records the first: ten weeks stale in July, found in
September. Correcting the path alone would leave the mechanism that hid it
untouched, so:
- The Drive folder is probed rather than hard-coded, across every mounted
GoogleDrive-* account and both known folder layouts. The laptop mounts
christian@ where the Studio mounts media@, and the folder has moved once
already, so a literal path is the wrong thing to depend on.
- Every failure now emails as well as logs, via scripts/alert.js, which reuses
the server's own SMTP settings. A failed alert never masks the failure it was
reporting.
- A run that does not error but leaves the newest off-site copy more than
STALE_DAYS old now alerts too. That is the check that would have caught both
incidents; the error paths only fire when the script actually runs.
What this still cannot catch is the script never running at all — a sleeping
Mac or an unloaded LaunchAgent is silent by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project has lived only as a folder on the Mac Studio and, briefly, as a
transfer tarball. This is the first commit: the Express server and its web
front end, the Capacitor iOS and Android wrappers, and the release and backup
scripts.
The repo is the source, not a backup of the machine. Left out on purpose, by
the .gitignore files that were already in the tree: node_modules, build output,
mobile/www (regenerated from server/public on every build), and server/data,
which holds the live checklists and belongs on the NAS rather than in git.
Every credential is excluded too — server/.env, the Play upload keystore and
its properties file, the App Store Connect key and ios-signing.config. The one
secret that was not in an ignored file was the NAS SSH password, hard-coded in
build-deploy.sh and grepped back out of it by backup-to-drive.sh. Both now read
it from the login keychain (service checklist-nas, account Christian), falling
back to NAS_PASS in the environment, so rotating it is one keychain entry
rather than a history rewrite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
