Status Indicators — Companion STATUS page for NA Ingest

One Companion page (page 19, "19 Status") that is solid green when everything is fine and blinks red with the actual problem ("HD3 NO DISK", "LIGHT PC MIDI LOST") when something needs attention. The page number key on every other page follows the same colour, so you see a problem from any page.

There are two parts:

• status_panel.py — a watchdog that runs on NA Ingest as the LaunchAgent org.waterschurch.status-panel. Every second it reads Companion's connection statuses and module variables. It also checks what Companion can't see: network MIDI peers, Bonjour MIDI sessions, pings, launchd services, disk space, the Bible TV app, and whether the Resi cloud is reachable. It writes the result into one Companion custom variable.
• status-page.companionconfig — the page. Every button just draws what the watchdog wrote and blinks on your existing flash_tick trigger.

Layout (Stream Deck XL; MK.2 and Mini show the top-left corner)

	col 0	col 1	col 2	col 3	col 4	col 5	col 6	col 7
row 0	UP	ALL	REC	AUDIO	RF	SHOW	VIDEO	STREAM
row 1	page	HD1	HD2	HD3	HD4	HD5	HD6	TASCAM
row 2	DOWN	X32	DANTE	MIDI	APPS	VISTA	RESTART PAGE	READY
row 3	ALARMS	ATEM	OBS	RESI	NET	INGEST	LED	ARENA

Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more:

• REC: HD1–6 and the Tascam.
• AUDIO: X32, DiGiCo, Dante and STB AUDIO (the Studio B audio source for the stream; no key of its own).
• RF: all six Shure units (AD4Q ×2, ULXD4, PSM1000 ×3).
• SHOW: MIDI, APPS, VISTA and SPOTIFY. SPOTIFY has no key of its own any more; it shows inside SHOW, and its force-play key is on the Restart page.

READY (row 2, col 7) is the weekly worship checklist. Tap it to open the READY page; see below.

• VIDEO: ATEM, LED and Arena.
• STREAM: OBS and Resi.

Colours

Colour	Meaning
Green	OK. The second line shows something useful, like 28h of record time or 19/19 Dante devices.
Blinking red	Something is wrong.
Solid dark red	Acknowledged. It stays red until the problem clears, then goes green. A new failure blinks again.
Amber	Warning: low disk, battery at 2 bars, RF interference, a device that keeps dropping out ("WING GONE x3"), a network blip in the last 3 minutes ("BLIP 10:21").
Dim red, not blinking	A fault while alarms are muted.
Grey	Not in use: an optional HyperDeck with no disk, or a disabled connection.
Purple	The watchdog has stopped. The master shows WATCHDOG DOWN and Companion's own error count.

Buttons

• Press any button: acknowledge its alarms (stops the blinking). Press ALL to acknowledge everything.

• Hold any button for 2 seconds: fix what that button is failing on right now. A green button does nothing, the same fix won't re-run within 60 s, and every command comes from "fixes" in config.json. Hold a group button to fix all its members.

  Failing	Hold does
  APPS: PROPRES OFFLINE / NOT RUNNING	restarts ProPresenter on its Mac (the newest version installed)
  APPS: ABLETON / TRANSLATE NOT RUNNING	starts Ableton on the Tracks Mac / Church Translate on NA Translation
  APPS: BIBLE APP DOWN	restarts the Bible app on the Sanctuary TV PC (SSH + its "Start BibleApp" task from remote-access/windows-setup.ps1)
  VISTA: SILENT	starts Vista on the Lighting PC if it is not running. A running Vista is never closed.
  ARENA	starts Resolume Arena on the Lighting PC if it is not running, then reconnects Companion
  OBS	starts OBS on the LiveStream PC if it is not running, then reconnects Companion. If the PC is off, it is woken (Wake-on-LAN) and the button shows WAKING PC; hold again once someone has logged in.
  MIDI: LIGHT PC / FX LIGHTS / SUPERRACK MIDI LOST	reconnects that peer to the session it dropped out of, from the Mac's side (NA Ingest's own session, or NA Tracks' over SSH), like Connect in Audio MIDI Setup. Other peers are left alone and nothing is restarted. The address is where the peer was last seen, else "address" on the check. Not for a missing Bonjour advert ("... MIDI GONE"). Keep the Lighting PC's network on Public: its Bonjour firewall rules only allow Public, and on Private it vanishes from the MIDI directory and cannot rejoin a session (2026-10-08). Never restart rtpMIDI as a fix: it drops every session and does not rejoin them.
  MIDI: LIGHT PC MIDI GONE	starts rtpMIDI on the Lighting PC if it is not running (its service, then the tray app). Its service also restarts itself if it crashes (Windows service recovery, set by windows-setup.ps1). Audio MIDI Setup does not need to be open on the Macs: their sessions run in macOS's MIDI server (tested 2026-10-08: quitting it left NA Ingest's session connected).
  DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE	starts Dante Virtual Soundcard's engine on that Mac (remote-access/mac-dvs-helper.sh)
  DANTE: any device powered by its switch port (today the four Studio B boxes)	power-cycles the port it is on now, found by its MAC on the Netgear Dante switches
  DANTE: any other device GONE	restarts Companion's Dante module (hardware: if still gone, it is off or unplugged)
  STB AUDIO (inside AUDIO): STUDIO B NO PING / PRO TOOLS NOT RUNNING / STB TNX GONE	presses FOH Backup on Companion's Audio page (the stream audio comes from the FOH console). The same press happens by itself on Sundays, see Audio failover.
  INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH	restarts that one on NA Ingest
  TASCAM (always, even when it looks fine)	reconnects Companion to the recorder and looks again after 20 s. HUNG - POWER CYCLE IT: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). NO NETWORK / CHECK POWER: it does not answer at all.
  any button whose Companion connection is disabled	turns the connection back on (nothing else)
  HD1–6 (always, even when it looks fine)	switches Companion's connection to that deck off and on, which is what gets a restarted deck green, and checks again after 15 s. Never reboots the deck, and leaves a recording deck alone.
  anything else with a Companion connection	restarts that Companion connection

  The Dante map is automatic. Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in "dante_switches". It saves the map to dante-map.json in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service netgear-status-panel).

  It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key ~/.ssh/id_status_panel. That key was authorized on the Macs with remote-access/mac-authorize.sh.

• ALARMS: alarms are armed all the time (service_windows in config.json: every day, 00:00–23:59). Press ALARMS to mute the blinking and stop every auto-fix, for example during maintenance; press it again to turn both back on. Both come back by themselves at midnight. The key says both states (ALARMS OFF / AUTO-FIX OFF / TIL 12AM). Holds still fix while muted; the Sunday stream-audio failover still switches by itself. The settings page's Overview has the same switch.

Auto-fix on Sundays (06:30–13:30)

During that window, with ALARMS on, the panel fixes a problem by itself once it has lasted 20 s, but only with gentle fixes: Companion reconnects, HyperDeck off/on, the Tascam reconnect, MIDI reconnects, starting rtpMIDI, and starting Vista, Resolume, OBS, Ableton or Church Translate if they are closed, playing the Spotify playlist, plus DVS on one Mac that dropped out of Dante. The key shows AUTO-FIXING.

• At most 2 tries per problem, 90 s apart; then it is yours to hold.
• It waits while the network is having a blip, and does nothing to Dante when several devices vanish at once (that is NA Ingest's view or the Dante network, not the Macs).
• Never automatic: restarting a running ProPresenter, Bible app, ProdCom (or DVS on NA Ingest, which restarts ProdCom), captions or the bridge, power-cycling a switch port, or turning on a connection someone switched off. load_config refuses an auto fix that closes a program.
• Daily log: everything auto-fix does is written as it happens to ~/Desktop/Auto-Fix Logs/<date> <weekday> Auto-Fix.log on NA Ingest (e.g. 2026-10-11 Sunday Auto-Fix.log): the problem, what it did, whether it worked, when the fault cleared, when it gave up or held back. Double-click it to open it in Console, which shows new lines live. One file per day; each is deleted a calendar month after its date (the 11 Oct file on 11 Nov).
• Stopping it: press ALARMS (on the fly; nothing at all fixes itself until you press it again or midnight), or switch auto-fix off as a whole under Settings › Auto-fix (stays off until switched on). The window and the list are "auto_fix" in config.json.
• READY page fixes ticked under Settings › Auto-fix act on the checklist's gaps too, amber or red. A fix marked "anytime": true (only a READY fix may be; today the LTC write, ready_ltc) runs at any hour, not only in the auto-fix hours: once the weekend's set is on Tracks, every song's timecode is written into Planning Center for all four campuses by itself. ALARMS muted stops it like every other auto-fix; it catches up once they are back on.

Audio failover (Studio B -> FOH Backup)

The stream's audio comes in on the Dante device STUDIO-A-Resi-AJA-SDI, channels 1/2. Companion's Audio page (page 4) picks the source: Studio B (row 0, col 6: STUDIO-B-RedNet-TNX 63/64 to the AJA, X32 snippet 1) or FOH Backup (row 0, col 5: FOH-DiGiCo-DMI-2 63/64 to the AJA, X32 snippet 8). If Studio B crashes on a Sunday, the panel presses FOH Backup for you.

STB AUDIO (inside AUDIO, no key of its own) watches three things: Studio B answers ping (10.1.13.175, pinged every second, STUDIO B NO PING after 2 misses), Pro Tools is running on the Studio B Mac Studio (pgrep -xq 'Pro Tools' over SSH every 2 s, PRO TOOLS NOT RUNNING), and the Dante device STUDIO-B-RedNet-TNX is there (STB TNX GONE). Each check's fix is the FOH Backup press (stream_foh_backup, a companion_press fix: POST /api/location/4/0/5/press on Companion's HTTP API).

• When: inside the "failover" windows (Sunday 06:30–13:30 today), even while ALARMS is muted. Once per fault: it re-arms once the fault has been clear for 10 s (a check that cannot ask, say an SSH hiccup, does not count as clear).
• How fast: about 5–6 s from a crash to the press (2–3 s to notice, 1 s debounce, 2 s after_seconds). A vanished Dante device takes as long as Companion's Dante module needs to notice it.
• Unlike auto-fix: it does not wait 20 s, does not stop for a network blip, and is not limited to 2 tries per 10 minutes. The one guard: when several Dante devices vanish at once (that is NA Ingest's view of Dante, not Studio B) it holds back, like auto-fix.
• What the keys show: AUDIO (and the hidden STB AUDIO) carry an extra line ON FOH BACKUP while the fault lasts and for 15 minutes after (note_seconds). Hold AUDIO right after and it says JUST TRIED.
• Log and alert: every press goes to the daily auto-fix log as FAILOVER: lines, to the watchdog log, and to alert_webhook if set.
• Switching back is yours. It never switches back by itself: when Studio B is up again, press Studio B on Companion's Studio B page (page 7) or Audio page (page 4). The alert names that button (the fix's "undo").
• Never: a press outside the window, a second press for the same fault, or a press back to Studio B.

In config.json: the "failover" block (windows, after_seconds 2, note_seconds 900); "failover": true and "undo" on the fix (stream_foh_backup undoes with stream_studio_b); "interval_seconds" on ping checks (default 3) and remote checks (default 10). The remote poll runs each computer's checks in its own thread, so a dead Studio B never delays the checks on another Mac.

Another fallback button: any Companion button can be a fix. On the settings page, Keys › Fixes › + Fix › "Press a Companion button" (page, row, column and what the button says), tick failover under more and pick its undo; Settings › Auto switch has it all on one screen: which buttons switch by themselves (tick, page/row/column, the button to switch back with, Press it now), the hours (calendar), after_seconds and note_seconds, and what counts as down (how often each check looks, how many misses, how long a fault must last). Keys › + Key / + check… adds more checks that call it (a ping, a program on a Mac, a Dante device) with fix = the press. The Help page "How do I add…" has an entry "A fallback button".

Restart page (page 20)

Tap RESTART PAGE on the status page (row 2). Each key restarts one thing even when the status page says it is fine; hold it 2 s. A key flashes red (HOLD TO FIX) while the thing it restarts is down, so you can see which one to hold. It shows RESTARTING, then DONE (or FAILED / what to do). The same thing will not run twice within a minute. STATUS PAGE (bottom left) goes back.

Key	Restarts
DVS PROP / TRACKS / TRANSL / INGEST	Dante Virtual Soundcard's engine on that Mac (audio to and from it drops 5–10 s). DVS INGEST also restarts ProdCom afterwards, which otherwise keeps a dead input.
PROPRESENTER	ProPresenter (quits and reopens the newest version)
ABLETON	Ableton on the Tracks Mac, reopening the set it had open (from Live's own log; templates skipped). Unsaved changes are lost, and Live may ask to recover - click it on the Tracks Mac. APPS's hold, which starts a closed Ableton, opens that set too.
TRANSLATE	Church Translate on NA Translation
PRODCOM	ProdCom on NA Ingest (lobby captions blank for 5–15 s)
PRODDASH / CAPTIONS / BRIDGE	that service on NA Ingest
TASCAM	reconnects Companion to the recorder (see TASCAM above)
SPOTIFY	plays the Christian EDM 2026 playlist on NA Ingest (opens Spotify first if it is closed)

A pressing key says PRESSING, then DONE; a press that finds no button there says NO BUTTON THERE.

Add or move keys in "restart_page" in config.json, then python3 make_page.py and python3 push_page.py --which restart. push_page.py refuses any page that holds real buttons unless it already is that page.

READY page (page 24): the weekly checklist

Tap READY on the status page (row 2, col 7). Each key is one line of the worship weekly checklist, checked against this weekend's plan in Planning Center by readiness.py (it looks every 2 minutes, and never while a HyperDeck is recording). STATUS PAGE (bottom left) goes back.

	col 1	col 2	col 3	col 4	col 5
row 0 (Ableton)	TRACKS	FOH MIX	BOH MIX (tap)	LIGHTS	MIDI
row 1 (ProP / Resolume)	PROP	TIMECODE	BKGDS	LED CROSS
row 2 (backstage)	LINEUP TV
row 3 (outgoing)	TRACKS OUT	PROP OUT

The READY key at row 3 col 7 (and the one on the status page) shows how many keys still have something to do and the first of them.

Colours: amber means not done yet. From Saturday 12:00 until Sunday 13:30 ("due" in "ready"), what is still missing turns blinking red. A song that is not mixed stays amber, because it is only a guess. These keys are not part of ALL, so an unfinished checklist on a Tuesday never turns the page-number keys red.

BOH MIX is a tap key: only a person can judge it. Tap once when it is done; the key shows DONE THU 14:02. Tap again to take it back. It resets on Monday at midnight.

Hold any key for 2 s to fix what can be fixed, or to look again now if there is no fix. Hold READY to fix everything on the page at once.

Key	Checks	Hold does
TRACKS	this weekend's set exists (Ableton/Projects/NA/261018 NA Project), it has a song group for every PCO song, and Live last opened it ("LIVE HAS OLD SET")	looks again
FOH MIX	NOT MIXED (amber): every track of a song is still at 0 dB. TO MASTER: a stem goes to Master instead of out to the console	looks again
LIGHTS	the LIGHTING CUES track has a clip under every song	looks again
MIDI	the Tracks Mac's MIDI links (FX LIGHTS, TRK LIVEBOX, TRK STUDIO B) and Ableton running	reconnects the MIDI link that dropped
PROP	the ProPresenter playlist for the plan's date has every PCO item. MC n/m SLIDES: PCO lists more announcements than there are slides. MC SLIDES OLD?: the slide notes still list last week's announcements. BAPTISM: NO SAMANTHA: a person PCO lists has no slide	looks again
TIMECODE	every song's presentation has a timeline in ProPresenter, and PCO's LTC note for every song is the timecode the tracks actually play at that song's start; the other campuses' plans (ready.ltc_campuses: Apollo Beach, Southcoast, Woonsocket) get the same timecode for the same songs in this campus's order, and a song that is missing or out of order there is left alone. That timecode is worked out from the set: where its locator ("2 - REJOICE AGAIN - Bb") sits, through the tempo map, on the LTC track's file (LTC_00000000_..._30fps). NO LTC / LTC WRONG name the song.	writes the tracks' timecode into each song's LTC note in Planning Center where it is missing or different (nothing is written when they already match), then looks again
BKGDS	every song has at least one slide with a background	looks again
LED CROSS	every Resolume column Sunday's playlist calls up has content: each slide's macro (read from the ProPresenter Mac's Macros and .pro files) sends RossTalk CC page/row/col to Companion, that button (and the buttons it presses, e.g. page 17 → page 6 "Arena") sends /composition/columns/N/connect; COL 11 EMPTY names the column and the items that use it. Also: Resolume on the Service deck.	selects the Service deck, then looks again (an empty column is filled in Resolume)
LINEUP TV	ProdDash's Band Lineup shows this plan, nobody who is not on it or who declined, and everyone on the Band and Communication teams	re-syncs the board from Planning Center
TRACKS OUT	every location in "locations" (AB, SC) has this weekend's project folder	copies NA's project to the locations that have none (as 261018 AB Project, without Backup/). It is copied under a temporary name and renamed at the end, so a half-finished copy never looks sent. It never overwrites a folder that is already there and never runs by itself.
PROP OUT	a .probundle was saved to Location Assets/ProPresenter since Monday	looks again

Where the answers come from:

• Planning Center: ProdDash's own login on this Mac (~/Library/Application Support/ProdDash/modules.json, module pco-plan). The plan is today's, else the next one, which is the same plan the lineup TV picks.
• ProPresenter: its network API.
• The Ableton set: read over SSH on the Tracks Mac at low priority, and only re-read when the file changed. It reads the saved file, so save in Live, then hold the key to look again.
• Resolume: its web server on port 8080.

Nothing on the READY page writes to Companion, ProPresenter or the Ableton set.

Another campus: copy "ready_page", "ready", the four ready_* fixes and the READY tile from this config, then change these in "ready":

• pco_service_type
• prop_url
• tracks_host and site_folder
• lineup_url
• resolume_url

For a campus that receives tracks rather than sending them, leave locations empty and take out the prop_bundle_* lines; TRACKS OUT and PROP OUT then show NOT SET UP. A line whose address is blank shows NOT SET UP instead of an alarm.

python3 make_page.py writes ready-page.companionconfig; python3 push_page.py --which ready imports it into page 24, which was empty (pages 21 to 23 are in use).

Page keys on every page

The page number key on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is "nav_keys" in config.json (["pagenum"]; add "pageup" / "pagedown" to colour those too). The coloured key is Companion's own key converted to a regular button (nav_templates.json).

• Tap the page number: go to page 1 ("nav_tap_page" in config.json). It acts when you let go, not when you press.

• Hold the page number for 1 second: jump to the status page (page 19). The hold time is "nav_hold_ms" in config.json.

• python3 nav_keys.py applies them to every page. Run it again after adding pages.

• python3 nav_keys.py --revert puts Companion's built-in nav keys back.

What each button checks

Button	Checks
HD1–HD6	Companion connection; NO DISK; record time LOW (<30 min, amber) / FULL (<5 min); NOT REC when another deck is recording and this one isn't
	HD2 and HD4 are "optional": true in config.json: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove "optional": true on any deck to change that.
TASCAM	connection; the recorder's own Error / Caution. Use the 1.3.1-quiet Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection.
X32, ARENA, OBS	Companion connection
DIGICO (inside AUDIO)	Companion's DiGiCo_FOH connection; the relay helper org.waterschurch.digico-relay on NA Ingest (~/digico-bridge/relay.py: Companion and the DiGiCo tablets share the console's one External Control slot through it), DIGICO RELAY STOPPED / NOT LOADED, hold AUDIO restarts it; the console itself answers ping at 10.1.60.100, DIGICO NO PING. Companion's OSC connection reads "good" even with the console off, so the ping is what catches that.
STB AUDIO (inside AUDIO)	the Studio B audio source for the stream: Studio B answers ping (10.1.13.175, every second, STUDIO B NO PING after 2 misses); Pro Tools running on the Studio B Mac Studio (over SSH every 2 s, PRO TOOLS NOT RUNNING); the Dante device STUDIO-B-RedNet-TNX (STB TNX GONE). Hold 2 s presses FOH Backup; on Sundays that press happens by itself (see Audio failover).
ATEM	Companion connection; the switcher's audio (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and camera settings (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file atem-reference.xml (a copy of 4k plus 2026-10-07 14-59-55.xml from Google Drive Backup/NA/ATEM): ATEM AUDIO WRONG / ATEM CAMERA WRONG; ATEM Software Control's Enable MIDI control must be off: ATEM SW MIDI ON. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). atem_watch.js keeps one connection to the switcher. New restore file: copy it over atem-reference.xml and re-run install.sh.
LED	VX1000 and UHD Jr: Companion connection and each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion.
DANTE	All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE".
MIDI	All six network MIDI links on the FOH / Studio B checklist, each named when it drops: LIGHT PC (Ingest → Lighting), SUPERRACK (Ingest → Livebox), FX LIGHTS (Tracks → Lighting, the FX lighting cues), TRK LIVEBOX (Tracks → Livebox), TRK STUDIO B (Tracks → Studio B), PROP LIGHTS (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped.
APPS	ProPresenter API (offline or error only: its harmless yellow warning, every 15–20 min, is ignored with "ignore_warning": true); Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections.; Pro Tools on the Studio B Mac Studio set to Solo Mode AFL + Latch ("PT SOLO NOT AFL"; hold, or Sunday auto-fix, ticks them back). A helper on that Mac (remote-access/solo-mode-agent.swift, installed with install-solo-mode.sh) reads the menu; it needs Accessibility allowed once, then a restart of the helper. "PT SOLO CHECK OFF" = that helper is not running or not allowed.
VISTA	Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise SILENT (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens Prop Takeover from its show folder (E:\media\Documents\Vista 3\Shows) by itself, through its own autoload preference (<autoloadpath> in E:\media\Documents\Vista 3\.preferences, set 2026-10-08). Resolume is the ARENA button.
RESI	decoder (RossTalk connection and ping, 10.1.10.203); encoder (ping and its web page, 10.1.10.202, LAOEncoder); resi-studio-cues; Resi cloud reachable. Only shows they are on the network, not that the encoder is streaming.
NET	NA Ingest's link; gateway 10.1.0.1 pings; 4+ connections from 3+ groups dropping within 10 s (what happened Sunday 10/5 at 10:21). A blip flashes amber for 5 minutes, then goes by itself; press NET to stop the flashing, hold NET 2 s to clear it now.
INGEST	the bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, ProdCom (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk
SPOTIFY	Spotify on NA Ingest must be playing the Christian EDM 2026 playlist (spotify:playlist:2uTpJVYQLFnKF1WnSOuco8): SPOTIFY CLOSED, NOT PLAYING, or WRONG PLAYLIST (it reads all of the playlist's songs from Spotify every hour, and again whenever a song plays that isn't on the list, since the playlist keeps getting new songs). Hold 2 s (and Sunday auto-fix) plays the playlist, opening Spotify first if needed.
RF (group)	Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; "host" in config.json is only the fallback.

A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately.

The settings page

http://na-ingest.local:8020/ (the Ingest Mac, LaunchAgent org.waterschurch.status-settings, log ~/Library/Logs/status-settings.log). Anyone on the church network can look: the live deck, every key and what holding it does, every computer and what runs on it (with a light each after "Check computers now"), the gear, the logs, the history, the help pages (What each key does, the cheat sheet, Where things are). Nothing changes without the editing code, which is site.editing_code in config.json (top right: Editing, the code and your first name).

With Editing on every screen grows inputs. Changes go into one shared draft on the Ingest Mac (two people editing see each other's version; a stale tab is told); the badge at the top counts them and opens Review & Apply, which reads every change back in plain words, says what will happen (the deck goes purple for about 5 s only when something the watchdog reads changed; pages re-import only when the layout changed) and runs the pipeline: check the config, keep the old version in history/, write, restart the watchdog and wait for its start stamp, re-import the pages (refused if any unmarked key sits on them), recolour the page-number keys. If the watchdog does not come back the previous version is put back by itself. Apply is refused while a HyperDeck is recording (the Emergency button needs a reason and the word LIVE, and is logged). History keeps every version with who/when/note; Go back and Undo make the chosen version the draft.

Keys draws both pages as the deck; with Editing on a key is a chip you drag to move (or drop on the tray to hide), + New key / + New restart key add one, Delete this key removes one, the page numbers and "this campus has a Restart page" sit under the tabs. Overview carries the jobs people actually do: Change the playlist (paste the link; the key, the play fix and the key guide follow), Add or remove a HyperDeck (tick the decks in use; the HD keys and the REC group follow, "may have no disk" per deck), Add a MIDI link (the Mac, its session and peer from the live list or typed, which key), and under More jobs: Add a computer, Make a PC program startable. Site setup › Companion offers "Add a key for it" on every connection nothing watches. Computers › Setup is the one-time checklist per computer with a light and a [how] per line and the set-up scripts generated from that computer's record (settings/scripts.py fills the templates in remote-access/: the login key, this Mac's addresses, one scheduled task per program listed); Remove this computer drops the fixes and checks that used it and says so. Site setup › Site shows the running and staged code with Install the staged folder (install.sh runs as a one-shot launchd job with the config left alone), Export site bundle (a zip to drop in as sites/<id>/) and New campus from a bundle (this campus's name, addresses and code stay; every copied computer is flagged until touched).

Fixes and the remote checks are built from fields (settings/builders.py): a builder (Start a Windows program, Restart a Mac program, Restart a helper, Reopen Ableton with its last set, Restart DVS, Press a Companion button (page, row, column)...) and its fields make the command; the stored command must equal what the fields build or the config is refused. Every Mac value is shell-quoted, every Windows value is regex-checked. A Custom command (Advanced) is the one place a command is typed by hand; Review shows commands in red. A fix built with Press a Companion button can be ticked failover (under more) and given an undo; the Failover calendar and its after_seconds sit under Settings › Auto-fix, under the auto-fix window.

The page reads what the watchdog writes next to its config (snapshot.json every 5 s, started.json, live-now.json, fixes-running.json) instead of asking the gear again. Its own look at the computers (settings/probes.py) runs fixed read-only commands over the same login key, fetches whole lists and matches them here, never while a fix runs.

Other campuses: auto-setup

A campus sets itself up. All a person does is let the panel in: one line on the campus's Mac that runs Companion, one line on each other computer.

1. The Companion Mac. Terminal, paste the campus's line from Settings › Auto-setup on any running campus (North Attleboro's page shows the Apollo Beach, Southcoast and Woonsocket lines), for example curl -fsSL "http://10.1.1.1:8020/campus/install?id=ab" | zsh. It downloads the program from that page (code only, never that campus's settings or keys), installs it, makes the panel's login key, and runs python3 -m settings.campus setup: Companion gets the panel's custom variables and the blink-clock trigger (through Companion's own internal API, as its web UI does), the first two pages from 19 that hold nothing made by hand are used, the gear is found and the config is applied like any Apply. It prints the editing PIN. North Attleboro's key is added so it can help remotely (turn on Remote Login for that). Running the line again updates the program.
2. Each other computer. Mac: Terminal, curl -fsSL http://<that Mac>:8020/join | sh (adds the key, turns on Remote Login when it can, says it is here). PC: PowerShell as administrator, irm http://<that Mac>:8020/join.ps1 | iex (OpenSSH, the key, a StatusPanel start task for each known program running on it, Wake-on-LAN, then says it is here). The lines are on that campus's Settings › Auto-setup.

What gets a key (settings/autoconf.py): each Companion connection of a known kind (HyperDecks, Tascam, consoles, Dante with every device it sees, Shure receivers and PSMs with their channel count, ATEM, Videohub, Resolume, NovaStar, OBS, Resi, ProPresenter, Ableton/AbleSet), each known program running on a joined computer (ProPresenter, Ableton, Arena, OBS, Church Translate, Pro Tools on a Mac; Vista, Arena, OBS, ProPresenter on a PC), and every network MIDI peer at its wired address (with the wired-network rule). A program's key also carries its Companion connection, its start fix (auto-fixable) and a Restart page key. rtpMIDI rides inside the MIDI key. When everything fits there are no group keys; when it does not, REC AUDIO RF SHOW VIDEO STREAM group keys take the top row and wireless receivers fold into RF first. Generic connections and unknown programs are listed under Settings › Auto-setup, not guessed at.

After that (discover.auto, "add new gear by itself", on for a new campus, off at North Attleboro): the settings page looks every ten minutes and soon after a computer joins, and adds what is new (a key in the first free cell, a Dante device or MIDI link into its key). Never during the auto-fix hours or a recording; nothing is ever removed; a thing that was there before and was taken off by someone is not added back (discover-seen.json). "Look now" adds what is new at once. Everything it makes is ordinary settings, changed on the page like North Attleboro's.

The campuses at a glance

Campus	Settings page (bookmark these)	Companion Mac	Status / Restart pages
North Attleboro (hub)	http://10.1.1.1:8020/	NA Ingest	19 / 20, READY 24, CAMPUSES 98, CAMPUS 99
Woonsocket	http://10.2.2.210:8020/	WS ProPresenter Mac (wsprop)	19 / 20

Every campus: alarms armed all day; auto-fix on NA's hours (Tue 14:00–22:00, Wed 09:00–21:00, Sun 05:00–14:00); a hold on any key fixes it at any time, on that campus's own deck or from NA's CAMPUS page; ALARMS stops auto-fix. python3 -m settings.campus rebuild (on a campus's Companion Mac, in the panel folder) makes its setup again from what is found now, keeping its name, PIN and pages. python3 -m settings.campus upgrade-companion brings an older Companion to 5.0.7 from NA's copy (settings come along; the old version stays installed).

Backups

Nightly at 04:30 NA's LaunchAgent org.waterschurch.status-backup (backup/backup-campuses.sh) copies every campus's settings, Apply history and page-key record into Google Drive: Shared drives > Waters Creative > Apps > Status Panel Backups > > -.tgz, newest 30 kept, the same pattern and alert email as the checklist's backup-to-drive.sh. Other campuses are reached over SSH as the account they report. The repo keeps each campus's settings too: config.json (NA) and sites/<id>/config.json; a reinstall with --site <id> starts from it. Smart-plug keys (outlets.json) are never copied.

Every campus on North Attleboro's deck (NA only)

Each campus set up from NA's install line tells NA's settings page where it is (/api/campus/hello) and learns nothing back: no campus ever shows another, and only NA (the hub: no "home" in campuses.json) does (settings/campuses.py). NA's settings page reads each campus's key states every two seconds (/api/state) and keeps two pages in NA's Companion, added at the end when no page is empty (today 98 and 99):

• CAMPUSES (98): row 0, one key per campus, its ALL key with its name ("WS / ALL OK"); purple "WS ?" when it stops answering. Row 1, its RESTART key. Tap NA's: NA's own pages. Tap another's: the CAMPUS page showing it.
• CAMPUS (99): the tapped campus's status (or Restart) page, same places, colours and words, live. Tap = acknowledge, hold = fix, at that campus, exactly as on its own deck (a Restart key's hold restarts it there). Bottom-left ("WS / ALL SITES") goes back.

SSH from the admin laptop to every machine

python3 tools/fleet.py ard prints a command for Apple Remote Desktop (Send UNIX Command, as root): on each Mac it turns Remote Login on, lets the logged-in user take the laptop's key (~/.ssh/id_ed25519_waters) and reports to NA's settings page (/api/fleet/hello). python3 tools/fleet.py config then writes ~/.ssh/config.d/waters: ssh na-translation, ssh ws-tracks... A Mac where nobody is logged in is skipped (run it again later).

One-time Companion setup

1. Variables → Custom Variables: add three variables. Leave the start values empty and "persist" off:
   • sp_state
   • sp_hb
   • sp_cmd
2. Import / Export → Import: choose status-page.companionconfig (it's on NA Ingest's Desktop) and import it into page 19. Page 19 is empty today.
3. Make sure the flash_tick_500 trigger stays enabled. It's the blink clock.

Changing things

This folder is the source of truth. Edit config.json here, then copy the folder to NA Ingest:

scp -r status_panel.py readiness.py coremidi.py netgear.py schema.py make_page.py push_page.py nav_keys.py nav_templates.json config.json install.sh atem_watch.js atem_state.js atem-reference.xml settings checklist-ingest:status-panel-install/

Then run the installer there. It checks the config with the new code first (a bad config never gets as far as the watchdog), keeps the code that was running in prev/, backs up the old config, writes the LaunchAgent with the exit timeout this config needs (the longest fix plus its verify time, so a restart never cuts a fix in half; launchd honours at most 60 s of it) and restarts the watchdog:

ssh checklist-ingest 'zsh ~/status-panel-install/install.sh'

install.sh --keep-config updates the code and leaves the live config alone; install.sh --rollback puts the previous code back; install.sh --site <id> (a new campus) installs sites/<id>/config.json when the folder exists, otherwise sites/_template/config.json, and never replaces a config that is already installed.

If you add or move tiles, rebuild the page:

python3 make_page.py

Then import it into page 19, either by hand (Import/Export → Import) or with:

python3 push_page.py

push_page.py does the same import over Companion's own API. It refuses to overwrite page 19 unless that page is empty or already the status page.

Other commands:

python3 status_panel.py --once          # poll once and print every tile; writes nothing
python3 -m unittest discover -s tests   # offline tests, no Companion or gear needed
zsh install.sh --remove                 # stop the watchdog

The log is ~/Library/Logs/status-panel.log on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.

What the watchdog keeps in its data folder (~/Library/Application Support/status-panel/): state.json (the ALARMS override, acknowledged faults, auto-fix tries and when each fix last ran, so a restart forgets nothing), started.json (when it started, with which config), live-now.json (whether a HyperDeck is recording: changes to the panel wait while it is), fixes-running.json (the fixes running right now). A restart (launchctl kickstart -k, or the installer) lets a running fix finish first and starts nothing new meanwhile: keys say RESTARTING SOON. Windows in config.json may run past midnight ("end" earlier than "start"), and a remote check on a Windows PC ("os": "windows" on the host) is batched for cmd.exe.

Optional: see what the tracks send to the lights

The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to listen to the "NA Tracks" session. Use a separate session so nothing Companion sends can reach Ableton:

1. On NA Ingest, open Audio MIDI Setup → Window → Show MIDI Studio → Network (the globe icon).
2. Under My Sessions, press +, rename the new session to NA Ingest Listen, and tick it to enable it.
3. With NA Ingest Listen selected, choose NA Tracks in the Directory and press Connect.

The watchdog picks it up within about 10 seconds. Until then that line just stays hidden.

The LIGHT PC tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set "max_silence_seconds": 10 on the LIGHT PC midi_rx check. The tile then goes amber if the lighting PC stops answering during service.

Quiet Tascam module

The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the recorder rejects one of those polls, and the module logs a warning every time: about 430,000 lines a day in Companion's log. tascam-quiet/make_quiet_tascam.py builds 1.3.1-quiet from the stock module: the same module, but that warning is logged at most once a minute and names the commands that were just sent.

1. In Companion: Modules → Import module package, choose tascam-da-6400-1.3.1-quiet.tgz (on NA Ingest's Desktop).
2. Connections → Tascam → edit, set the module version to 1.3.1-quiet, save.

To go back, pick 1.3.0 on the connection again.

Windows PCs

remote-access/windows-setup.ps1 sets each PC up once (SSH for NA Ingest, start tasks, Wake-on-LAN). An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) gets a start task that runs with highest privileges; otherwise Windows refuses to start it (error 0x800702E4). Windows restarts the SSH service and rtpMIDI by itself if they stop.

Files

File	Purpose
status_panel.py	the watchdog
netgear.py	reads the Netgear Dante switches (MAC tables, PoE) and power-cycles one PoE port
coremidi.py	reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs)
config.json	tiles, positions, checks, fixes, hosts, windows: every setting (the site block names the campus; spotify holds the playlist; timing and page_hold_ms the numbers behind HOLD 2s and WAIT 1 MIN)
schema.py	the registry of every setting: type, default, help, which screen of the settings page owns it, and the rules a config must pass (tests/test_schema.py keeps it in step with the code)
make_page.py	builds status-page.companionconfig from config.json
install.sh	installs or updates the LaunchAgent on NA Ingest
push_page.py	imports the status page into page 19 over Companion's API (--which restart / --which ready for pages 20 and 24)
readiness.py	the READY page: looks at Planning Center, ProPresenter, the Ableton set, ProdDash's lineup, Resolume and the shared drive, and runs the READY fixes
nav_keys.py	colours (or restores) the page nav keys on every page
nav_templates.json	Companion's own nav key designs, coloured by make_page.py
remote-access/	one-time setup scripts for SSH to the Macs and Windows PCs
tascam-quiet/	builds the quiet Tascam module
tests/	offline tests, including a fake Companion and a MIDI loopback; test_schema.py checks that every key the code reads is declared in schema.py with the same default