Make the website deploy report its own failures
Build apps / build (push) Successful in 1m30s
Deploy web to NAS / deploy (push) Successful in 1m14s

The deploy was the one pipeline with no failure notification. It either worked or
you found out by noticing the site was stale — the same shape as the off-site
backup before today.

alert.js now takes CHECKLIST_ENV_FILE and CHECKLIST_NODE_MODULES, because the
deploy job runs in a container that has the repo but no server/.env and no
node_modules; both live on the NAS at the mounted deploy directory.

What this still does not catch, and the reason it is written down in the workflow:
a runner that wedges. Today the NAS runner claimed run 7, executed none of its six
steps, and Gitea timed it out fifteen minutes later — every step was marked failed
without one having run, so no in-job step could have reported it. The container
said "Up 2 hours" throughout and admitted on restart that it had been holding the
job. A stuck runner looks exactly like a slow one, and catching it needs something
watching from outside the run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Waters Church
2026-09-24 16:11:22 -04:00
co-authored by Claude Opus 5
parent 17254696e7
commit 39c65cf9de
2 changed files with 32 additions and 3 deletions
+25
View File
@@ -88,3 +88,28 @@ jobs:
docker ps --filter name=recurring-checklist --format '{{.Names}} {{.Status}}'
docker logs --tail 30 recurring-checklist
exit 1
- name: Tell Christian the deploy broke
# The web deploy had no failure notification at all until now, which is the same
# silent-failure shape as the backup: it either worked or you found out by looking.
#
# NOT covered by this: a runner that wedges. On 2026-09-24 the NAS runner claimed
# this job, ran none of its six steps, and Gitea timed it out — no step ran, so no
# step could report. Catching that needs something watching from outside the run.
if: failure()
run: |
D=/volume2/docker/recurring-checklist
BODY_FILE="$(mktemp)"
cat > "$BODY_FILE" <<MSG
A push to main failed to deploy the website to the NAS.
The running container was left alone — the deploy recreates it in place and only
after the files are copied, so the previous version should still be serving. Worth
confirming: https://check.watersmedia.org
Full log:
http://chronicles-ab.waterschurch.org:3000/christian/recurring-checklist/actions
MSG
CHECKLIST_ENV_FILE="$D/.env" CHECKLIST_NODE_MODULES="$D/node_modules" \
node scripts/alert.js "Website deploy FAILED" "$(cat "$BODY_FILE")" || true
rm -f "$BODY_FILE"