The previous commit moved the fetch to scp, which was a regression: modern scp
speaks SFTP, this Synology does not enable that subsystem, and every file failed
with "subsystem request failed on channel 0". It failed safe — nothing was
copied and nothing was emailed — but nothing would have copied again either.
Caught by running it rather than by reading it.
With key auth there is no pty in the way, so streaming the file through a plain
ssh session is byte-exact. Verified against the NAS's own byte count: 390163 in,
390163 out, parses with 43 lists. scp -O would also work, but the legacy protocol
is being retired from OpenSSH.
Worth recording what this proves about the original fault: today's file was never
corrupt. cat was never the problem. The pty expect wrapped around it was.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each nightly backup is ~275 KB and was pulled with `cat` through an expect
pseudo-terminal, then recovered by slicing from the first { to the last } of the
transcript. A pty does line discipline and has finite buffers — it is not a file
transfer — so captures arrived truncated often enough to fail the JSON check most
nights between 29 September and 8 October, emailing each time. No backup was ever
lost; every file copied on a later attempt.
The ingest Mac now has a key authorised on the NAS, so there is no password prompt
to automate and the file comes over scp, byte for byte. expect is gone from this
script, and so is the NAS password: nothing here reads the keychain any more,
which also removes the reason it could not be run by hand over SSH.
The JSON check stays, but it only verifies now — there is no transcript to repair.
A backup that is present but unreadable is still worse than one that is obviously
missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly backup has emailed "1 backup file(s) could not be copied" most nights
since 29 September. Nothing was wrong with the backups: every file that failed
that way copied cleanly on a later night, and the only one still outstanding is
today's. The alert was reporting a transport hiccup as if it were data loss.
The fetch is the fragile part. Each file is ~275 KB and arrives through an expect
pseudo-terminal, which does line discipline and has finite buffers, so a capture
occasionally comes back truncated and fails to parse. It is now retried three
times within the same run, which should turn most of these into a non-event.
Alert severity now depends on the file's AGE rather than on this run. Today's
backup failing is a retry tomorrow and is logged, not mailed. A file still missing
after STALE_DAYS means the retries are not recovering and that does raise mail,
naming the files and their ages.
A nightly email that never needed action is how an alert stops being read, which
is the exact failure this script was written to prevent — it was training its
reader to ignore it.
The real cure is to stop pulling files through a pty at all: with key-based SSH to
the NAS this could be a plain scp and be byte-exact. That needs a key installed on
the NAS and is left for a deliberate change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deploy was the one pipeline with no failure notification. It either worked or
you found out by noticing the site was stale — the same shape as the off-site
backup before today.
alert.js now takes CHECKLIST_ENV_FILE and CHECKLIST_NODE_MODULES, because the
deploy job runs in a container that has the repo but no server/.env and no
node_modules; both live on the NAS at the mounted deploy directory.
What this still does not catch, and the reason it is written down in the workflow:
a runner that wedges. Today the NAS runner claimed run 7, executed none of its six
steps, and Gitea timed it out fifteen minutes later — every step was marked failed
without one having run, so no in-job step could have reported it. The container
said "Up 2 hours" throughout and admitted on restart that it had been holding the
job. A stuck runner looks exactly like a slow one, and catching it needs something
watching from outside the run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A push fires the web deploy and the app build at the same instant, and on commit
b03fb25 the second SSH session to Gitea came back with "Failed to update public
key" 32 seconds in. The pull succeeded by hand immediately afterwards, so it was
contention between the two runs rather than anything about the code — but the job
had already failed and emailed "App build FAILED".
The pull is now attempted three times with a pause between, and only gives up
after the third. It is the one step here that depends on another machine being
responsive at a moment we do not control, so it is the one worth retrying.
The point is not the build: a false failure alert is what teaches you to ignore
the alert, and an ignored alert is how the off-site backup sat dead for eight days
without anyone noticing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This directory feeds both targets by different routes, and the difference is the
thing people get wrong: the website serves it directly, while the apps carry a
copy frozen at build time. Someone editing here reasonably assumes a deploy is
enough, and for the website it is.
Written at the top of the file rather than in a README because this is where the
assumption gets made.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Run 4 built both apps successfully and then emailed to say the build had FAILED.
Two separate faults.
The notification step died on bash 3.2, which macOS still ships as /bin/bash. It
mis-parses a heredoc inside command substitution when the text contains an
apostrophe: the word "anyone's" was enough to produce "unexpected EOF while
looking for matching '". Reproduced on the ingest Mac before and after the fix.
The body is written to a file first, which avoids the construct entirely.
The failure alert was then sent because it keyed off failure() for the whole job,
so a bug in the success notification presented as a broken build. It is now scoped
to steps.build.outcome, so it reports only what it claims to report.
An alert that fires when nothing is wrong trains you to ignore it, which costs more
than the missing alert it replaced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alert steps passed their message body as a bare multi-line argument starting
at column 0. That ends the YAML block scalar, which makes the whole file invalid:
"a multiline key may not be an implicit key". Gitea still listed the workflow as
active and simply never created a run for it — no error anywhere, which is why two
pushes appeared to do nothing.
The bodies are built with indented heredocs now, so every line stays inside the
block scalar. Validated by parsing the file rather than by eye; a workflow that is
registered but silently never triggers looks exactly like one that works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also serves as the first real exercise of the build-apps workflow: a change under
mobile/src should rebuild both apps on the ingest Mac without touching the web
deploy, which only watches server/**.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The web app has deployed on push since earlier today, but the phone apps carry
their own frozen copy of the UI, so a change to server/public silently left them
behind until someone remembered to cut a release. Remembering was the weak part.
A second Actions runner now runs on the ingest Mac — in host mode, because an app
build needs Xcode, the Android SDK and the signing keys, none of which exist in
the NAS runner's Linux container. A push touching server/public, mobile/src or
the Capacitor config rebuilds and signs both apps, then emails to say the shipped
apps are behind, or that the build broke.
It deliberately stops before uploading. An upload needs a build number that has
never been used, Play refuses to reuse or lower a versionCode, and bumping it
means committing the new number back from the build machine on every run. That
needs write access from that machine and permanently changes the numbering
scheme — a one-way door, so shipping stays an explicit ./release.sh.
release.config.json is excluded from the trigger paths so that bumping a build
number by hand cannot set off a rebuild of what was just built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plist existed only in the transfer bundle, in a version that still pointed at
~/Documents — the path that macOS TCC silently refuses to let launchd execute from,
and the reason the nightly backup did nothing for eight days while logging nothing.
Captured from the ingest Mac as it now runs, with a README explaining why the
checkout must live outside ~/Documents and why the script cannot glob the Drive
folder. Both are easy to undo by accident and expensive to rediscover.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Under launchd, macOS allows a named path on ~/Library/CloudStorage to be stat'd
but blocks enumerating the directory. A glob therefore expands to nothing, and
the reporting block read that as zero: the first real LaunchAgent run after the
TCC fix logged "0 copies off-site" and emailed "no off-site copies at all"
while 57 files sat in the folder. The copy loop was unaffected, because it uses
filenames taken from the NAS rather than a glob.
Freshness is now derived by testing explicit filenames for the last 60 days,
which works in both contexts. Enumeration is still used for the total count, but
a count of zero is reported as unknown rather than alerted on — an unreadable
folder and an empty one are different things, and only one of them is an
emergency.
Preferred over granting Full Disk Access: a privacy grant is exactly what an OS
update silently reset here in the first place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly Drive backup runs on the ingest Mac (the one whose account is
"Studio"), from a copy of this project made before the repo existed. Fixes made
anywhere else therefore never reach the machine that actually runs the backup at
04:15 — which is how the fix for a silent backup failure could itself sit unused.
adopt-repo.sh converts that folder into a clone without losing the things git
deliberately does not carry: server/.env, the Play upload keystore and its
properties, ios-signing.config and local.properties. It renames the old folder
rather than deleting it, restores those files into the clone, re-points the
LaunchAgent, and puts the old folder back if the clone fails. --dry-run prints
the plan and changes nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Failures went to REPORT_EMAIL, which is the media@ shared inbox the Sunday digest
already uses. An alert competing with a routine weekly mail in a shared inbox is
an alert that gets scrolled past, which is most of the way back to the logfile
nobody read.
ALERT_EMAIL is now a separate, comma-separated list for breakage only, falling
back to REPORT_EMAIL so a machine whose .env predates this still alerts somewhere
rather than silently not at all.
Verified by sending one test alert, which arrived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nightly copy to Drive had been dead since 15 September. DRIVE_DIR pointed at
Apps/Backups/Checklist Backups; the folder is Apps/Checklist Backups, with no
Backups level. The guard for a missing folder did its job and exited — into a
logfile nobody reads, so from the outside a broken backup and a healthy no-op
looked identical. Eight days were missing; this run copied them.
That is the second time the same failure has hidden in the same place. The
header comment already records the first: ten weeks stale in July, found in
September. Correcting the path alone would leave the mechanism that hid it
untouched, so:
- The Drive folder is probed rather than hard-coded, across every mounted
GoogleDrive-* account and both known folder layouts. The laptop mounts
christian@ where the Studio mounts media@, and the folder has moved once
already, so a literal path is the wrong thing to depend on.
- Every failure now emails as well as logs, via scripts/alert.js, which reuses
the server's own SMTP settings. A failed alert never masks the failure it was
reporting.
- A run that does not error but leaves the newest off-site copy more than
STALE_DAYS old now alerts too. That is the check that would have caught both
incidents; the error paths only fire when the script actually runs.
What this still cannot catch is the script never running at all — a sleeping
Mac or an unloaded LaunchAgent is silent by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
act_runner already mounts /var/run/docker.sock into every job container, so
declaring it again in the workflow made the daemon refuse to create the
container at all: "Duplicate mount point: /var/run/docker.sock". The deploy
directory still has to be declared, and still at its own host path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now a server change reached the NAS only when someone ran ./release.sh --web
from a machine on the church network. That made the deploy depend on which laptop
was awake and where it was, which is the same fragility that let the nightly Drive
backup go stale unnoticed.
An act_runner now runs in Docker on the NAS itself and picks up this workflow, so
a push to main rebuilds the container regardless of where the push came from.
The workflow keeps the protections the old remote deploy script had, because they
are the ones that matter: server/data and server/.env are excluded from the copy
and then explicitly checked for afterwards, and the container is recreated with
'up -d --build' rather than 'down' first, so an interrupted deploy leaves the old
container serving instead of tearing the stack down. rsync runs without --delete,
matching the tarball deploy it replaces — a stale file is survivable, a wiped
data/ is not.
This covers the web app only. The iOS and Android apps bundle a copy of the UI at
build time, so they still need ./release.sh and a store release; server-side
changes reach them immediately, since they call this same server live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The setup notes described moving the project as a tarball and never mentioned
git, because until now there was no repo. They now lead with the clone URL and
what a push needs — the credential helper, a token, and the hostname rather
than 10.3.8.90, since the stored credential is keyed to the hostname and a
remote written against the bare IP prompts every time.
server/.gitignore only excluded data/*.json, which was enough for the checklist
file and nothing else. Running the server locally creates data/uploads, data/docs
and data/backups beside it, so an attached PDF or a backup would have been
committed on the next add. The whole directory is ignored now. That data lives
on the NAS; a stale copy of it restored over production is the accident worth
designing out.
Also records that the NAS password is a keychain item rather than a file, so a
future machine knows to store it instead of hunting for the line that used to
be in build-deploy.sh.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The project has lived only as a folder on the Mac Studio and, briefly, as a
transfer tarball. This is the first commit: the Express server and its web
front end, the Capacitor iOS and Android wrappers, and the release and backup
scripts.
The repo is the source, not a backup of the machine. Left out on purpose, by
the .gitignore files that were already in the tree: node_modules, build output,
mobile/www (regenerated from server/public on every build), and server/data,
which holds the live checklists and belongs on the NAS rather than in git.
Every credential is excluded too — server/.env, the Play upload keystore and
its properties file, the App Store Connect key and ios-signing.config. The one
secret that was not in an ignored file was the NAS SSH password, hard-coded in
build-deploy.sh and grepped back out of it by backup-to-drive.sh. Both now read
it from the login keychain (service checklist-nas, account Christian), falling
back to NAS_PASS in the environment, so rotating it is one keychain entry
rather than a history rewrite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
