christian/recurring-checklist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the backup alerts in plain language

Browse Source 
The alerts were written for whoever was debugging them, not for whoever reads
them on a phone on a Sunday. They led with the mechanism and left the reader to
work out whether anything was actually broken.

Each one now opens by saying what is and is not affected — the checklists and
the website are almost always fine, and an off-site copy problem is not an
emergency — then what happened, then what to do. The scary-sounding ones say
plainly that previous backups are untouched, because that is the first thing
worth knowing and it was never stated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Waters Church
2026-10-08 09:40:33 -04:00
co-authored by Claude Opus 5
parent 672accc6a1
commit 32f0d06536
2 changed files with 83 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-to-drive.sh
+44 -25
View File
@@ -69,26 +69,36 @@ SSH_KEY="$HOME/.ssh/id_ed25519_checklist_nas"
SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=20 -i $SSH_KEY" SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=20 -i $SSH_KEY"
if [ ! -f "$SSH_KEY" ]; then if [ ! -f "$SSH_KEY" ]; then
die "no SSH key for the NAS at $SSH_KEY" \ die "tonight's backup did not run — the key for the NAS is missing" \
"Generate one and authorise it on the NAS: "Nothing was copied off-site tonight. Previous backups are untouched and safe.
This machine signs in to the NAS with a key file, and that file is gone from
$SSH_KEY
To fix it, on this Mac:
ssh-keygen -t ed25519 -f $SSH_KEY -N '' ssh-keygen -t ed25519 -f $SSH_KEY -N ''
ssh-copy-id -i $SSH_KEY $NAS_USER@$NAS_HOST" ssh-copy-id -i $SSH_KEY $NAS_USER@$NAS_HOST"
fi fi
if ! ssh $SSH_OPTS "$NAS_USER@$NAS_HOST" true 2>/dev/null; then if ! ssh $SSH_OPTS "$NAS_USER@$NAS_HOST" true 2>/dev/null; then
die "cannot reach the NAS over SSH" \ die "tonight's backup did not run — could not reach the NAS" \
"Key auth to $NAS_USER@$NAS_HOST failed. Either the NAS is down, or the key at "Nothing was copied off-site tonight. Previous backups are untouched and safe.
$SSH_KEY is no longer in its authorized_keys."
Either the NAS is off or off the network, or this machine's key is no longer
accepted by it. Worth checking the NAS is up first; that is the usual cause."
fi fi
# Google Drive streams files on demand; if the app is signed out or not running the path # Google Drive streams files on demand; if the app is signed out or not running the path
# simply is not there. Writing to a path Drive is not watching would look like it worked and # simply is not there. Writing to a path Drive is not watching would look like it worked and
# sync nothing, so refuse rather than pretend. # sync nothing, so refuse rather than pretend.
if [ -z "$DRIVE_DIR" ]; then if [ -z "$DRIVE_DIR" ]; then
die "cannot find the Drive backup folder" \ die "tonight's backup did not run — the Google Drive folder is missing" \
"Looked under every ~/Library/CloudStorage/GoogleDrive-* account for "Nothing was copied off-site tonight. Previous backups are untouched and safe.
Shared drives/Waters Creative/Apps/Checklist Backups
and found none. Is Google Drive for Desktop running and signed in, and does that The folder it copies into could not be found:
folder still exist under that name?" Shared drives > Waters Creative > Apps > Checklist Backups
Usually this means Google Drive is not running or has signed out on this Mac.
It can also mean the folder was renamed or moved."
fi fi
say "Drive folder: $DRIVE_DIR" say "Drive folder: $DRIVE_DIR"
@@ -100,9 +110,12 @@ run_remote() { # $1 = remote command
REMOTE_LIST="$(run_remote "ls -1 $REMOTE_DIR/checklists-*.json 2>/dev/null | xargs -n1 basename" \ REMOTE_LIST="$(run_remote "ls -1 $REMOTE_DIR/checklists-*.json 2>/dev/null | xargs -n1 basename" \
| grep -E '^checklists-[0-9]{4}-[0-9]{2}-[0-9]{2}\.json$' | sort)" | grep -E '^checklists-[0-9]{4}-[0-9]{2}-[0-9]{2}\.json$' | sort)"
if [ -z "$REMOTE_LIST" ]; then if [ -z "$REMOTE_LIST" ]; then
die "no backups found on the NAS, or SSH failed" \ die "the NAS has no backups to copy" \
"Expected $REMOTE_DIR/checklists-<date>.json on $NAS_HOST. Either the NAS's own "Copies already in Google Drive are untouched and safe, but no new ones were found.
nightly job has stopped writing them, or this machine can no longer reach the NAS."
The NAS writes a backup of the checklists every night at 03:30, and none are
there. That is a problem on the NAS itself rather than with this copy job — the
checklist app may have stopped writing them."
fi fi
say "NAS has $(echo "$REMOTE_LIST" | wc -l | tr -d ' ') backup(s)" say "NAS has $(echo "$REMOTE_LIST" | wc -l | tr -d ' ') backup(s)"
@@ -204,12 +217,16 @@ say "✓ done — $TOTAL copies off-site, newest ${NEWEST:-none}"
# tomorrow is noise, and nightly noise is how an alert stops being read — which is the whole # tomorrow is noise, and nightly noise is how an alert stops being read — which is the whole
# failure mode this script exists to prevent. # failure mode this script exists to prevent.
if [ -n "${OVERDUE:-}" ]; then if [ -n "${OVERDUE:-}" ]; then
alert "backup file(s) still missing off-site after several days" \ alert "some backups are not reaching Google Drive" \
"These NAS backups have failed to copy for more than $STALE_DAYS days, so the "The website and the checklists themselves are fine. This is about the off-site
retries are not recovering on their own: copies only.
These have failed to copy for more than $STALE_DAYS days, so the automatic retries
are not fixing it:
$OVERDUE $OVERDUE
Log: $LOG" Everything else copied normally. Worth a look when convenient — it is not urgent
unless it keeps growing."
elif [ "${FAILED:-0}" -gt 0 ]; then elif [ "${FAILED:-0}" -gt 0 ]; then
say "($FAILED file(s) failed this run but are recent — they will be retried tomorrow, not alerting)" say "($FAILED file(s) failed this run but are recent — they will be retried tomorrow, not alerting)"
fi fi
@@ -218,15 +235,17 @@ fi
# failure path above assumes this script RAN; if the Mac is asleep or the LaunchAgent is # failure path above assumes this script RAN; if the Mac is asleep or the LaunchAgent is
# unloaded, nothing fires at all. # unloaded, nothing fires at all.
if [ -z "$NEWEST" ]; then if [ -z "$NEWEST" ]; then
alert "no recent off-site copy" \ alert "no recent backup in Google Drive" \
"No checklists-<date>.json for any of the last 60 days was found in "The checklists themselves are fine — this is about the off-site copies.
$DRIVE_DIR
Either copying has stopped, or this machine cannot read that folder." Nothing from the last 60 days is in the Drive folder. Either the copying has
stopped, or this Mac can no longer read that folder."
elif [ "$AGE_DAYS" -gt "$STALE_DAYS" ]; then elif [ "$AGE_DAYS" -gt "$STALE_DAYS" ]; then
alert "off-site copy is $AGE_DAYS days old" \ alert "backups have stopped reaching Google Drive" \
"Newest file off-site is $NEWEST ($AGE_DAYS days old); more than $STALE_DAYS days "The checklists themselves are fine — this is about the off-site copies.
means the chain from the NAS to Drive is not working, even though this run did
not error." The newest one in Drive is $AGE_DAYS days old ($NEWEST). Tonight's run did not
report an error, which means something is failing quietly rather than loudly."
else else
say "freshness OK — newest off-site copy is $AGE_DAYS day(s) old" say "freshness OK — newest off-site copy is $AGE_DAYS day(s) old"
fi fi
deploy/org.waterschurch.checklist-weekly-health.plist
+39
View File
@@ -0,0 +1,39 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>org.waterschurch.checklist-weekly-health</string>
<!-- The one piece of mail that arrives on a schedule. Monday morning, so a problem is
seen at the start of a week rather than discovered on a Sunday. -->
<key>ProgramArguments</key>
<array>
<string>/Users/Studio/checklist-app/scripts/weekly-health.sh</string>
</array>
<key>StartCalendarInterval</key>
<dict>
<key>Weekday</key><integer>1</integer>
<key>Hour</key><integer>8</integer>
<key>Minute</key><integer>30</integer>
</dict>
<key>RunAtLoad</key>
<false/>
<key>StandardOutPath</key>
<string>/Users/Studio/Library/Logs/checklist-weekly-health.out.log</string>
<key>StandardErrorPath</key>
<string>/Users/Studio/Library/Logs/checklist-weekly-health.err.log</string>
<!-- Same two traps as the backup agent: launchd's PATH is minimal and has no node, and
it cannot execute anything inside ~/Documents, which is why this lives in
~/checklist-app. See deploy/README.md. -->
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key>
<string>/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
</dict>
</dict>
</plist>