Lets the email be read before it is sent, which matters for a message whose whole
job is to be clear at a glance a week from now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every alert here only fires on failure, which leaves silence ambiguous: either
all is well, or the thing meant to warn you is itself broken. That ambiguity is
exactly what let the off-site backup sit dead for ten weeks, and then for another
eight days after it was supposedly fixed.
weekly-health.sh sends one note a week whether or not anything is wrong. It checks
the four things worth knowing: the website answers, off-site backups are current,
the NAS is reachable, and whether the phone apps have fallen behind the website.
If everything is fine it says so in a sentence and lists what it checked. If
something is not, it leads with what needs doing.
It is deliberately the only scheduled mail. Everything else stays quiet unless it
needs something, so a message in the inbox always means either the weekly note or
a real problem.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deploy was the one pipeline with no failure notification. It either worked or
you found out by noticing the site was stale — the same shape as the off-site
backup before today.
alert.js now takes CHECKLIST_ENV_FILE and CHECKLIST_NODE_MODULES, because the
deploy job runs in a container that has the repo but no server/.env and no
node_modules; both live on the NAS at the mounted deploy directory.
What this still does not catch, and the reason it is written down in the workflow:
a runner that wedges. Today the NAS runner claimed run 7, executed none of its six
steps, and Gitea timed it out fifteen minutes later — every step was marked failed
without one having run, so no in-job step could have reported it. The container
said "Up 2 hours" throughout and admitted on restart that it had been holding the
job. A stuck runner looks exactly like a slow one, and catching it needs something
watching from outside the run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly Drive backup runs on the ingest Mac (the one whose account is
"Studio"), from a copy of this project made before the repo existed. Fixes made
anywhere else therefore never reach the machine that actually runs the backup at
04:15 — which is how the fix for a silent backup failure could itself sit unused.
adopt-repo.sh converts that folder into a clone without losing the things git
deliberately does not carry: server/.env, the Play upload keystore and its
properties, ios-signing.config and local.properties. It renames the old folder
rather than deleting it, restores those files into the clone, re-points the
LaunchAgent, and puts the old folder back if the clone fails. --dry-run prints
the plan and changes nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Failures went to REPORT_EMAIL, which is the media@ shared inbox the Sunday digest
already uses. An alert competing with a routine weekly mail in a shared inbox is
an alert that gets scrolled past, which is most of the way back to the logfile
nobody read.
ALERT_EMAIL is now a separate, comma-separated list for breakage only, falling
back to REPORT_EMAIL so a machine whose .env predates this still alerts somewhere
rather than silently not at all.
Verified by sending one test alert, which arrived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly copy to Drive had been dead since 15 September. DRIVE_DIR pointed at
Apps/Backups/Checklist Backups; the folder is Apps/Checklist Backups, with no
Backups level. The guard for a missing folder did its job and exited — into a
logfile nobody reads, so from the outside a broken backup and a healthy no-op
looked identical. Eight days were missing; this run copied them.
That is the second time the same failure has hidden in the same place. The
header comment already records the first: ten weeks stale in July, found in
September. Correcting the path alone would leave the mechanism that hid it
untouched, so:
- The Drive folder is probed rather than hard-coded, across every mounted
GoogleDrive-* account and both known folder layouts. The laptop mounts
christian@ where the Studio mounts media@, and the folder has moved once
already, so a literal path is the wrong thing to depend on.
- Every failure now emails as well as logs, via scripts/alert.js, which reuses
the server's own SMTP settings. A failed alert never masks the failure it was
reporting.
- A run that does not error but leaves the newest off-site copy more than
STALE_DAYS old now alerts too. That is the check that would have caught both
incidents; the error paths only fire when the script actually runs.
What this still cannot catch is the script never running at all — a sleeping
Mac or an unloaded LaunchAgent is silent by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project has lived only as a folder on the Mac Studio and, briefly, as a
transfer tarball. This is the first commit: the Express server and its web
front end, the Capacitor iOS and Android wrappers, and the release and backup
scripts.
The repo is the source, not a backup of the machine. Left out on purpose, by
the .gitignore files that were already in the tree: node_modules, build output,
mobile/www (regenerated from server/public on every build), and server/data,
which holds the live checklists and belongs on the NAS rather than in git.
Every credential is excluded too — server/.env, the Play upload keystore and
its properties file, the App Store Connect key and ios-signing.config. The one
secret that was not in an ignored file was the NAS SSH password, hard-coded in
build-deploy.sh and grepped back out of it by backup-to-drive.sh. Both now read
it from the login keychain (service checklist-nas, account Christian), falling
back to NAS_PASS in the environment, so rotating it is one keychain entry
rather than a history rewrite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
