Fetch backups over scp with a key, not cat through a pty
Each nightly backup is ~275 KB and was pulled with `cat` through an expect
pseudo-terminal, then recovered by slicing from the first { to the last } of the
transcript. A pty does line discipline and has finite buffers — it is not a file
transfer — so captures arrived truncated often enough to fail the JSON check most
nights between 29 September and 8 October, emailing each time. No backup was ever
lost; every file copied on a later attempt.
The ingest Mac now has a key authorised on the NAS, so there is no password prompt
to automate and the file comes over scp, byte for byte. expect is gone from this
script, and so is the NAS password: nothing here reads the keychain any more,
which also removes the reason it could not be run by hand over SSH.
The JSON check stays, but it only verifies now — there is no transcript to repair.
A backup that is present but unreadable is still worse than one that is obviously
missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Waters Church
co-authored by
Claude Opus 5
Claude Opus 5
parent
5db4aada74
commit
cf3c496c17
+34
-35
@@ -57,13 +57,27 @@ for acct in "$HOME/Library/CloudStorage"/GoogleDrive-*; do
|
||||
done
|
||||
done
|
||||
|
||||
# The SSH password comes from the login keychain — the same entry build-deploy.sh reads, so
|
||||
# there is still one copy of the credential to rotate, and it is not in the project at all.
|
||||
# Store it with: security add-generic-password -U -s checklist-nas -a Christian -w
|
||||
NAS_PASS="${NAS_PASS:-$(security find-generic-password -s checklist-nas -a Christian -w 2>/dev/null || true)}"
|
||||
if [ -z "$NAS_PASS" ]; then
|
||||
die "no NAS password in the keychain" \
|
||||
"Store it with: security add-generic-password -U -s checklist-nas -a Christian -w"
|
||||
# Key-based SSH. This used to pull each file with `cat` through an expect pseudo-terminal,
|
||||
# which is how a ~275 KB JSON arrived truncated often enough to email a false alarm most
|
||||
# nights between 29 September and 8 October 2026. A pty does line discipline and has finite
|
||||
# buffers; it is not a file transfer. With a key there is no password prompt to automate, so
|
||||
# the file can come over scp and be byte-exact.
|
||||
#
|
||||
# The key lives on this machine only and is authorised on the NAS for this user. There is no
|
||||
# password here any more, and nothing to read out of the keychain.
|
||||
SSH_KEY="$HOME/.ssh/id_ed25519_checklist_nas"
|
||||
SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=20 -i $SSH_KEY"
|
||||
|
||||
if [ ! -f "$SSH_KEY" ]; then
|
||||
die "no SSH key for the NAS at $SSH_KEY" \
|
||||
"Generate one and authorise it on the NAS:
|
||||
ssh-keygen -t ed25519 -f $SSH_KEY -N ''
|
||||
ssh-copy-id -i $SSH_KEY $NAS_USER@$NAS_HOST"
|
||||
fi
|
||||
if ! ssh $SSH_OPTS "$NAS_USER@$NAS_HOST" true 2>/dev/null; then
|
||||
die "cannot reach the NAS over SSH" \
|
||||
"Key auth to $NAS_USER@$NAS_HOST failed. Either the NAS is down, or the key at
|
||||
$SSH_KEY is no longer in its authorized_keys."
|
||||
fi
|
||||
|
||||
# Google Drive streams files on demand; if the app is signed out or not running the path
|
||||
@@ -78,17 +92,8 @@ folder still exist under that name?"
|
||||
fi
|
||||
say "Drive folder: $DRIVE_DIR"
|
||||
|
||||
run_remote() { # $1 = remote command; prints stdout, strips the pty's CRs
|
||||
local cmd="$1"
|
||||
expect <<EXP 2>/dev/null | tr -d '\r'
|
||||
set timeout 240
|
||||
log_user 1
|
||||
spawn bash -c "ssh -o StrictHostKeyChecking=no -o ConnectTimeout=20 -o PreferredAuthentications=password -o PubkeyAuthentication=no -o NumberOfPasswordPrompts=1 $NAS_USER@$NAS_HOST \"$cmd\""
|
||||
expect {
|
||||
-nocase "password:" { send "$NAS_PASS\r"; exp_continue }
|
||||
eof
|
||||
}
|
||||
EXP
|
||||
run_remote() { # $1 = remote command
|
||||
ssh $SSH_OPTS "$NAS_USER@$NAS_HOST" "$1"
|
||||
}
|
||||
|
||||
# 1. What does the NAS have?
|
||||
@@ -120,28 +125,22 @@ else
|
||||
[ -z "$f" ] && continue
|
||||
n=$((n+1)); [ $n -gt $MAX_PER_RUN ] && { say "· stopping at $MAX_PER_RUN this run"; break; }
|
||||
TMP="$(mktemp -t checklist-backup)"; OUT="$(mktemp -t checklist-backup-json)"
|
||||
# Retried, because the fetch is the fragile part. Each of these files is ~275 KB and it
|
||||
# comes back through an expect pseudo-terminal, which does line discipline and has finite
|
||||
# buffers — so a capture occasionally arrives truncated and fails to parse. That is a
|
||||
# transport hiccup, not a corrupt backup: every file that failed this way between
|
||||
# 2026-09-29 and 2026-10-07 copied cleanly on a later night. Retrying in the same run
|
||||
# turns a nightly email into a non-event.
|
||||
# Still retried, though the reason it used to be needed is gone. The truncation that
|
||||
# failed a file most nights between 29 September and 7 October 2026 came from pulling it
|
||||
# through a pty; scp cannot truncate silently. A network blip mid-copy is still possible,
|
||||
# and retrying costs a few seconds against an email nobody needed.
|
||||
ok=0
|
||||
for try in 1 2 3; do
|
||||
: > "$TMP"
|
||||
run_remote "cat $REMOTE_DIR/$f" > "$TMP"
|
||||
# expect echoes its own `spawn …` banner and the password prompt onto stdout before the
|
||||
# file's contents, so the capture is never pure JSON. Slice from the first { to the last }
|
||||
# and only then validate — a truncated or half-read file that *looks* like a backup is
|
||||
# worse than an obviously missing one.
|
||||
scp -q $SSH_OPTS "$NAS_USER@$NAS_HOST:$REMOTE_DIR/$f" "$TMP" 2>/dev/null || true
|
||||
# scp delivers the file byte for byte, so there is nothing to slice out of a transcript
|
||||
# any more. The check stays because a backup that is present but unreadable is worse
|
||||
# than one that is obviously missing — it just verifies rather than repairs now.
|
||||
if node -e '
|
||||
const fs=require("fs");
|
||||
const t=fs.readFileSync(process.argv[1],"utf8");
|
||||
const a=t.indexOf("{"), b=t.lastIndexOf("}");
|
||||
if(a<0||b<=a) process.exit(1);
|
||||
const j=JSON.parse(t.slice(a,b+1)); // throws => exit non-zero
|
||||
if(!Array.isArray(j.lists)) process.exit(1); // shape check, not just valid JSON
|
||||
fs.writeFileSync(process.argv[2], JSON.stringify(j));
|
||||
const j=JSON.parse(fs.readFileSync(process.argv[1],"utf8")); // throws => non-zero
|
||||
if(!Array.isArray(j.lists)) process.exit(1); // shape, not just syntax
|
||||
fs.copyFileSync(process.argv[1], process.argv[2]);
|
||||
' "$TMP" "$OUT" 2>/dev/null; then
|
||||
ok=1; break
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user