Audio failover: press FOH Backup by itself when Studio B dies; Companion button presses as fixes

A new fix kind "companion_press" presses any Companion button over its HTTP API (page, row, col).
A fix marked "failover" runs by itself within seconds of its check failing: inside the "failover"
windows (Sunday 06:30-13:30), even while ALARMS is muted, once per fault, never back the other way.
The hidden STB AUDIO key (inside AUDIO) watches Studio B's ping every second, Pro Tools over SSH
every 2 s and its Dante device; each check's fix is the FOH Backup press. The keys say ON FOH BACKUP
until STREAM STB on the Restart page switches back. Settings page: the press builder, the failover
tick and undo, the Failover calendar, help entry "A fallback button".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

README.md

@@ -19,7 +19,7 @@ There are two parts:
 Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more:
 
 - REC: HD1–6 and the Tascam.
-- AUDIO: X32, DiGiCo and Dante.
+- AUDIO: X32, DiGiCo, Dante and STB AUDIO (the Studio B audio source for the stream; no key of its own).
 - RF: all six Shure units (AD4Q ×2, ULXD4, PSM1000 ×3).
 - SHOW: MIDI, APPS, VISTA and SPOTIFY. SPOTIFY has no key of its own any more; it shows inside SHOW, and its
   force-play key is on the Restart page.
@@ -58,6 +58,7 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
   | DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) |
   | DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
   | DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) |
+  | STB AUDIO (inside AUDIO): STUDIO B NO PING / PRO TOOLS NOT RUNNING / STB TNX GONE | presses FOH Backup on Companion's Audio page (the stream audio comes from the FOH console). The same press happens by itself on Sundays, see Audio failover. |
   | INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH | restarts that one on NA Ingest |
   | TASCAM (always, even when it looks fine) | reconnects Companion to the recorder and looks again after 20 s. **HUNG - POWER CYCLE IT**: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). **NO NETWORK / CHECK POWER**: it does not answer at all. |
   | any button whose Companion connection is **disabled** | turns the connection back on (nothing else) |
@@ -79,6 +80,24 @@ During that window, with ALARMS on, the panel fixes a problem by itself once it
 - **Daily log:** everything auto-fix does is written as it happens to `~/Desktop/Auto-Fix Logs/<date> <weekday> Auto-Fix.log` on NA Ingest (e.g. `2026-10-11 Sunday Auto-Fix.log`): the problem, what it did, whether it worked, when the fault cleared, when it gave up or held back. Double-click it to open it in Console, which shows new lines live. One file per day; each is deleted a calendar month after its date (the 11 Oct file on 11 Nov).
 - Press ALARMS to mute the panel for maintenance: that also stops auto-fix. The window and the list are `"auto_fix"` in `config.json`.
 
+## Audio failover (Studio B -> FOH Backup)
+
+The stream's audio comes in on the Dante device STUDIO-A-Resi-AJA-SDI, channels 1/2. Companion's Audio page (page 4) picks the source: **Studio B** (row 0, col 6: STUDIO-B-RedNet-TNX 63/64 to the AJA, X32 snippet 1) or **FOH Backup** (row 0, col 5: FOH-DiGiCo-DMI-2 63/64 to the AJA, X32 snippet 8). If Studio B crashes on a Sunday, the panel presses FOH Backup for you.
+
+**STB AUDIO** (inside AUDIO, no key of its own) watches three things: Studio B answers ping (10.1.13.175, pinged every second, **STUDIO B NO PING** after 2 misses), Pro Tools is running on the Studio B Mac Studio (`pgrep -xq 'Pro Tools'` over SSH every 2 s, **PRO TOOLS NOT RUNNING**), and the Dante device STUDIO-B-RedNet-TNX is there (**STB TNX GONE**). Each check's fix is the FOH Backup press (`stream_foh_backup`, a `companion_press` fix: `POST /api/location/4/0/5/press` on Companion's HTTP API).
+
+- **When:** inside the `"failover"` windows (Sunday 06:30–13:30 today), even while ALARMS is muted. Once per fault: it re-arms once the fault has been clear for 10 s (a check that cannot ask, say an SSH hiccup, does not count as clear).
+- **How fast:** about 5–6 s from a crash to the press (2–3 s to notice, 1 s debounce, 2 s `after_seconds`). A vanished Dante device takes as long as Companion's Dante module needs to notice it.
+- **Unlike auto-fix:** it does not wait 20 s, does not stop for a network blip, and is not limited to 2 tries per 10 minutes. The one guard: when several Dante devices vanish at once (that is NA Ingest's view of Dante, not Studio B) it holds back, like auto-fix.
+- **What the keys show:** AUDIO (and the hidden STB AUDIO) carry an extra line **ON FOH BACKUP** while the fault lasts and for 15 minutes after (`note_seconds`), or until STREAM STB is held. Hold AUDIO right after and it says JUST TRIED.
+- **Log and alert:** every press goes to the daily auto-fix log as `FAILOVER:` lines, to the watchdog log, and to `alert_webhook` if set.
+- **Switching back is yours.** It never switches back by itself: when Studio B is up again, hold **STREAM STB** on the Restart page for 2 s (the fix's `"undo"`). That presses Studio B on the Audio page and clears ON FOH BACKUP.
+- **Never:** a press outside the window, a second press for the same fault, or a press back to Studio B.
+
+In `config.json`: the `"failover"` block (`windows`, `after_seconds` 2, `note_seconds` 900); `"failover": true` and `"undo"` on the fix (`stream_foh_backup` undoes with `stream_studio_b`); `"interval_seconds"` on ping checks (default 3) and remote checks (default 10). The remote poll runs each computer's checks in its own thread, so a dead Studio B never delays the checks on another Mac.
+
+**Another fallback button:** any Companion button can be a fix. On the settings page, Keys › Fixes › + Fix › "Press a Companion button" (page, row, column and what the button says), tick failover under more and pick its undo; Settings › Auto-fix › Failover calendar holds the window and `after_seconds`; Keys › + Key / + check… adds the checks that call it (a ping with interval seconds and failures, a program on a Mac, a Dante device) with fix = the press; Keys › Restart page › + Restart key adds the keys that press it by hand. The Help page "How do I add…" has an entry "A fallback button".
+
 ## Restart page (page 20)
 
 Tap **RESTART PAGE** on the status page (row 2). Each key restarts one thing **even when the status page says it is fine**; hold it 2 s. A key **flashes red (HOLD TO FIX)** while the thing it restarts is down, so you can see which one to hold. It shows RESTARTING, then DONE (or FAILED / what to do). The same thing will not run twice within a minute. **STATUS PAGE** (bottom left) goes back.
@@ -93,6 +112,10 @@ Tap **RESTART PAGE** on the status page (row 2). Each key restarts one thing **e
 | PRODDASH / CAPTIONS / BRIDGE | that service on NA Ingest |
 | TASCAM | reconnects Companion to the recorder (see TASCAM above) |
 | SPOTIFY | plays the Christian EDM 2026 playlist on NA Ingest (opens Spotify first if it is closed) |
+| STREAM STB | presses Studio B on Companion's Audio page: the stream audio back on Studio B (how you switch back after a failover, see Audio failover) |
+| STREAM FOH | presses FOH Backup on Companion's Audio page: the stream audio from the FOH console. Flashes HOLD TO FIX while Studio B is down; once the failover has pressed it, it shows ON FOH BACKUP (green) instead. |
+
+A pressing key says PRESSING, then DONE; a press that finds no button there says NO BUTTON THERE.
 
 Add or move keys in `"restart_page"` in `config.json`, then `python3 make_page.py` and `python3 push_page.py --which restart`. `push_page.py` refuses any page that holds real buttons unless it already is that page.
 
@@ -178,6 +201,7 @@ The **page number key** on every page follows the ALL button. It's green when ev
 | TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection. |
 | X32, ARENA, OBS | Companion connection |
 | DIGICO (inside AUDIO) | Companion's `DiGiCo_FOH` connection; the relay helper `org.waterschurch.digico-relay` on NA Ingest (`~/digico-bridge/relay.py`: Companion and the DiGiCo tablets share the console's one External Control slot through it), **DIGICO RELAY STOPPED** / **NOT LOADED**, hold AUDIO restarts it; the console itself answers ping at 10.1.60.100, **DIGICO NO PING**. Companion's OSC connection reads "good" even with the console off, so the ping is what catches that. |
+| STB AUDIO (inside AUDIO) | the Studio B audio source for the stream: Studio B answers ping (10.1.13.175, every second, **STUDIO B NO PING** after 2 misses); Pro Tools running on the Studio B Mac Studio (over SSH every 2 s, **PRO TOOLS NOT RUNNING**); the Dante device STUDIO-B-RedNet-TNX (**STB TNX GONE**). Hold 2 s presses FOH Backup; on Sundays that press happens by itself (see Audio failover). |
 | ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. |
 | LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
 | DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". |
@@ -200,7 +224,7 @@ With Editing on every screen grows inputs. Changes go into one shared draft on t
 
 **Keys** draws both pages as the deck; with Editing on a key is a chip you drag to move (or drop on the tray to hide), + New key / + New restart key add one, Delete this key removes one, the page numbers and "this campus has a Restart page" sit under the tabs. **Overview** carries the jobs people actually do: Change the playlist (paste the link; the key, the play fix and the key guide follow), Add or remove a HyperDeck (tick the decks in use; the HD keys and the REC group follow, "may have no disk" per deck), Add a MIDI link (the Mac, its session and peer from the live list or typed, which key), and under More jobs: Add a computer, Make a PC program startable. Site setup › Companion offers "Add a key for it" on every connection nothing watches. **Computers › Setup** is the one-time checklist per computer with a light and a [how] per line and the set-up scripts generated from that computer's record (`settings/scripts.py` fills the templates in `remote-access/`: the login key, this Mac's addresses, one scheduled task per program listed); Remove this computer drops the fixes and checks that used it and says so. **Site setup › Site** shows the running and staged code with **Install the staged folder** (install.sh runs as a one-shot launchd job with the config left alone), **Export site bundle** (a zip to drop in as `sites/<id>/`) and **New campus from a bundle** (this campus's name, addresses and code stay; every copied computer is flagged until touched).
 
-Fixes and the remote checks are built from fields (`settings/builders.py`): a builder (Start a Windows program, Restart a Mac program, Restart a helper, Reopen Ableton with its last set, Restart DVS...) and its fields make the command; the stored command must equal what the fields build or the config is refused. Every Mac value is shell-quoted, every Windows value is regex-checked. A Custom command (Advanced) is the one place a command is typed by hand; Review shows commands in red.
+Fixes and the remote checks are built from fields (`settings/builders.py`): a builder (Start a Windows program, Restart a Mac program, Restart a helper, Reopen Ableton with its last set, Restart DVS, Press a Companion button (page, row, column)...) and its fields make the command; the stored command must equal what the fields build or the config is refused. Every Mac value is shell-quoted, every Windows value is regex-checked. A Custom command (Advanced) is the one place a command is typed by hand; Review shows commands in red. A fix built with Press a Companion button can be ticked failover (under more) and given an undo; the Failover calendar and its `after_seconds` sit under Settings › Auto-fix, under the auto-fix window.
 
 The page reads what the watchdog writes next to its config (`snapshot.json` every 5 s, `started.json`, `live-now.json`, `fixes-running.json`) instead of asking the gear again. Its own look at the computers (`settings/probes.py`) runs fixed read-only commands over the same login key, fetches whole lists and matches them here, never while a fix runs.