One Companion page (page 19, **"19 Status"**) that is solid green when everything is fine and **blinks red with the actual problem** ("HD3 NO DISK", "LIGHT PC MIDI LOST") when something needs attention. The page number key on every other page follows the same colour, so you see a problem from any page.
- **`status_panel.py`** — a watchdog that runs on NA Ingest as the LaunchAgent `org.waterschurch.status-panel`. Every second it reads Companion's connection statuses and module variables. It also checks what Companion can't see: network MIDI peers, Bonjour MIDI sessions, pings, launchd services, disk space, the Bible TV app, and whether the Resi cloud is reachable. It writes the result into one Companion custom variable.
| Solid dark red | Acknowledged. It stays red until the problem clears, then goes green. A new failure blinks again. |
| Amber | Warning: low disk, battery at 2 bars, RF interference, a device that keeps dropping out ("WING GONE x3"), a network blip in the last 3 minutes ("BLIP 10:21"). |
- **Hold any button for 2 seconds:** fix what *that button is failing on right now*. A green button does nothing, the same fix won't re-run within 60 s, and every command comes from `"fixes"` in `config.json`. Hold a group button to fix all its members.
| Failing | Hold does |
|---|---|
| APPS: PROPRES OFFLINE / NOT RUNNING | restarts ProPresenter on its Mac (the newest version installed) |
| APPS: ABLETON / TRANSLATE NOT RUNNING | starts Ableton on the Tracks Mac / Church Translate on NA Translation |
| ARENA | starts Resolume Arena on the Lighting PC if it is not running, then reconnects Companion |
| OBS | starts OBS on the LiveStream PC if it is not running, then reconnects Companion. If the PC is off, it is woken (Wake-on-LAN) and the button shows WAKING PC; hold again once someone has logged in. |
| MIDI: LIGHT PC / FX LIGHTS / SUPERRACK MIDI LOST | reconnects that peer to the session it dropped out of, from the Mac's side (NA Ingest's own session, or NA Tracks' over SSH), like Connect in Audio MIDI Setup. Other peers are left alone and nothing is restarted. The address is where the peer was last seen, else `"address"` on the check. Not for a missing Bonjour advert ("... MIDI GONE"). **Keep the Lighting PC's network on Public:** its Bonjour firewall rules only allow Public, and on Private it vanishes from the MIDI directory and cannot rejoin a session (2026-10-08). Never restart rtpMIDI as a fix: it drops every session and does not rejoin them. |
| MIDI: LIGHT PC MIDI GONE | starts rtpMIDI on the Lighting PC if it is not running (its service, then the tray app). Its service also restarts itself if it crashes (Windows service recovery, set by `windows-setup.ps1`). Audio MIDI Setup does not need to be open on the Macs: their sessions run in macOS's MIDI server (tested 2026-10-08: quitting it left NA Ingest's session connected). |
| DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
| INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH | restarts that one on NA Ingest |
| TASCAM (always, even when it looks fine) | reconnects Companion to the recorder and looks again after 20 s. **HUNG - POWER CYCLE IT**: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). **NO NETWORK / CHECK POWER**: it does not answer at all. |
| HD1–6 (always, even when it looks fine) | switches Companion's connection to that deck off and on, which is what gets a restarted deck green, and checks again after 15 s. Never reboots the deck, and leaves a recording deck alone. |
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.
During that window, with ALARMS on, the panel fixes a problem by itself once it has lasted 20 s, but only with **gentle fixes**: Companion reconnects, HyperDeck off/on, the Tascam reconnect, MIDI reconnects, starting rtpMIDI, and starting Vista, Resolume, OBS, Ableton or Church Translate if they are closed, playing the Spotify playlist, plus DVS on one Mac that dropped out of Dante. The key shows **AUTO-FIXING**.
- At most 2 tries per problem, 90 s apart; then it is yours to hold.
- It waits while the network is having a blip, and does nothing to Dante when several devices vanish at once (that is NA Ingest's view or the Dante network, not the Macs).
- Never automatic: restarting a running ProPresenter, Bible app, ProdCom (or DVS on NA Ingest, which restarts ProdCom), captions or the bridge, power-cycling a switch port, or turning on a connection someone switched off. `load_config` refuses an auto fix that closes a program.
- **Daily log:** everything auto-fix does is written as it happens to `~/Desktop/Auto-Fix Logs/<date> <weekday> Auto-Fix.log` on NA Ingest (e.g. `2026-10-11 Sunday Auto-Fix.log`): the problem, what it did, whether it worked, when the fault cleared, when it gave up or held back. Double-click it to open it in Console, which shows new lines live. One file per day; each is deleted a calendar month after its date (the 11 Oct file on 11 Nov).
Tap **RESTART PAGE** on the status page (row 2). Each key restarts one thing **even when the status page says it is fine**; hold it 2 s. A key **flashes red (HOLD TO FIX)** while the thing it restarts is down, so you can see which one to hold. It shows RESTARTING, then DONE (or FAILED / what to do). The same thing will not run twice within a minute. **STATUS PAGE** (bottom left) goes back.
| DVS PROP / TRACKS / TRANSL / INGEST | Dante Virtual Soundcard's engine on that Mac (audio to and from it drops 5–10 s). DVS INGEST also restarts ProdCom afterwards, which otherwise keeps a dead input. |
| ABLETON | Ableton on the Tracks Mac, reopening the set it had open (from Live's own log; templates skipped). Unsaved changes are lost, and Live may ask to recover - click it on the Tracks Mac. APPS's hold, which starts a closed Ableton, opens that set too. |
Add or move keys in `"restart_page"` in `config.json`, then `python3 make_page.py` and `python3 push_page.py --which restart`. `push_page.py` refuses any page that holds real buttons unless it already is that page.
The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`).
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection. |
| ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. |
| APPS | ProPresenter API (offline or error only: its harmless yellow warning, every 15–20 min, is ignored with `"ignore_warning": true`); Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections.; **Pro Tools on the Studio B Mac Studio** set to Solo Mode **AFL + Latch** ("PT SOLO NOT AFL"; hold, or Sunday auto-fix, ticks them back). A helper on that Mac (`remote-access/solo-mode-agent.swift`, installed with `install-solo-mode.sh`) reads the menu; it needs Accessibility allowed once, then a restart of the helper. "PT SOLO CHECK OFF" = that helper is not running or not allowed. |
| VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`<autoloadpath>` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. |
| RESI | decoder (RossTalk connection and ping, 10.1.10.203); **encoder** (ping and its web page, 10.1.10.202, LAOEncoder); resi-studio-cues; Resi cloud reachable. Only shows they are on the network, not that the encoder is streaming. |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21). A blip **flashes amber** for 5 minutes, then goes by itself; press NET to stop the flashing, hold NET 2 s to clear it now. |
| INGEST | the bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, **ProdCom** (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk |
| SPOTIFY | Spotify on NA Ingest must be playing the **Christian EDM 2026** playlist (`spotify:playlist:2uTpJVYQLFnKF1WnSOuco8`): **SPOTIFY CLOSED**, **NOT PLAYING**, or **WRONG PLAYLIST** (it reads all of the playlist's songs from Spotify every hour, and again whenever a song plays that isn't on the list, since the playlist keeps getting new songs). Hold 2 s (and Sunday auto-fix) plays the playlist, opening Spotify first if needed. |
| RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; `"host"` in `config.json` is only the fallback. |
`http://na-ingest.local:8020/` (the Ingest Mac, LaunchAgent `org.waterschurch.status-settings`, log `~/Library/Logs/status-settings.log`). Anyone on the church network can look: the live deck, every key and what holding it does, every computer and what runs on it (with a light each after "Check computers now"), the gear, the logs, the history, the help pages (What each key does, the cheat sheet, Where things are). Nothing changes without the editing code, which is `site.editing_code` in `config.json` (top right: Editing, the code and your first name).
With Editing on every screen grows inputs. Changes go into one shared draft on the Ingest Mac (two people editing see each other's version; a stale tab is told); the badge at the top counts them and opens **Review & Apply**, which reads every change back in plain words, says what will happen (the deck goes purple for about 5 s only when something the watchdog reads changed; pages re-import only when the layout changed) and runs the pipeline: check the config, keep the old version in `history/`, write, restart the watchdog and wait for its start stamp, re-import the pages (refused if any unmarked key sits on them), recolour the page-number keys. If the watchdog does not come back the previous version is put back by itself. Apply is refused while a HyperDeck is recording (the Emergency button needs a reason and the word LIVE, and is logged). History keeps every version with who/when/note; **Go back** and **Undo** make the chosen version the draft.
**Keys** draws both pages as the deck; with Editing on a key is a chip you drag to move (or drop on the tray to hide), + New key / + New restart key add one, Delete this key removes one, the page numbers and "this campus has a Restart page" sit under the tabs. **Overview** carries the jobs people actually do: Change the playlist (paste the link; the key, the play fix and the key guide follow), Add or remove a HyperDeck (tick the decks in use; the HD keys and the REC group follow, "may have no disk" per deck), Add a MIDI link (the Mac, its session and peer from the live list or typed, which key), and under More jobs: Add a computer, Make a PC program startable. Site setup › Companion offers "Add a key for it" on every connection nothing watches. **Computers › Setup** is the one-time checklist per computer with a light and a [how] per line and the set-up scripts generated from that computer's record (`settings/scripts.py` fills the templates in `remote-access/`: the login key, this Mac's addresses, one scheduled task per program listed); Remove this computer drops the fixes and checks that used it and says so. **Site setup › Site** shows the running and staged code with **Install the staged folder** (install.sh runs as a one-shot launchd job with the config left alone), **Export site bundle** (a zip to drop in as `sites/<id>/`) and **New campus from a bundle** (this campus's name, addresses and code stay; every copied computer is flagged until touched).
Fixes and the remote checks are built from fields (`settings/builders.py`): a builder (Start a Windows program, Restart a Mac program, Restart a helper, Reopen Ableton with its last set, Restart DVS...) and its fields make the command; the stored command must equal what the fields build or the config is refused. Every Mac value is shell-quoted, every Windows value is regex-checked. A Custom command (Advanced) is the one place a command is typed by hand; Review shows commands in red.
The page reads what the watchdog writes next to its config (`snapshot.json` every 5 s, `started.json`, `live-now.json`, `fixes-running.json`) instead of asking the gear again. Its own look at the computers (`settings/probes.py`) runs fixed read-only commands over the same login key, fetches whole lists and matches them here, never while a fix runs.
1.**Variables → Custom Variables:** add three variables. Leave the start values empty and "persist" off:
-`sp_state`
-`sp_hb`
-`sp_cmd`
2.**Import / Export → Import:** choose `status-page.companionconfig` (it's on NA Ingest's Desktop) and import it into **page 19**. Page 19 is empty today.
3. Make sure the **flash_tick_500** trigger stays enabled. It's the blink clock.
## Changing things
This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:
Then run the installer there. It checks the config with the new code first (a bad config never gets as far as the watchdog), keeps the code that was running in `prev/`, backs up the old config, writes the LaunchAgent with the exit timeout this config needs (the longest fix plus its verify time, so a restart never cuts a fix in half; launchd honours at most 60 s of it) and restarts the watchdog:
`install.sh --keep-config` updates the code and leaves the live config alone; `install.sh --rollback` puts the previous code back; `install.sh --site <id>` (a new campus) installs `sites/<id>/config.json` when the folder exists, otherwise `sites/_template/config.json`, and never replaces a config that is already installed.
python3 status_panel.py --once # poll once and print every tile; writes nothing
python3 -m unittest discover -s tests # offline tests, no Companion or gear needed
zsh install.sh --remove # stop the watchdog
```
The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.
What the watchdog keeps in its data folder (`~/Library/Application Support/status-panel/`): `state.json` (the ALARMS override, acknowledged faults, auto-fix tries and when each fix last ran, so a restart forgets nothing), `started.json` (when it started, with which config), `live-now.json` (whether a HyperDeck is recording: changes to the panel wait while it is), `fixes-running.json` (the fixes running right now). A restart (`launchctl kickstart -k`, or the installer) lets a running fix finish first and starts nothing new meanwhile: keys say RESTARTING SOON. Windows in `config.json` may run past midnight (`"end"` earlier than `"start"`), and a remote check on a Windows PC (`"os": "windows"` on the host) is batched for cmd.exe.
## Optional: see what the tracks send to the lights
The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:
1. On NA Ingest, open **Audio MIDI Setup → Window → Show MIDI Studio → Network** (the globe icon).
2. Under **My Sessions**, press **+**, rename the new session to `NA Ingest Listen`, and tick it to enable it.
3. With `NA Ingest Listen` selected, choose **NA Tracks** in the **Directory** and press **Connect**.
The watchdog picks it up within about 10 seconds. Until then that line just stays hidden.
The **LIGHT PC** tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set `"max_silence_seconds": 10` on the LIGHT PC `midi_rx` check. The tile then goes amber if the lighting PC stops answering during service.
The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the recorder rejects one of those polls, and the module logs a warning every time: about 430,000 lines a day in Companion's log. `tascam-quiet/make_quiet_tascam.py` builds **1.3.1-quiet** from the stock module: the same module, but that warning is logged at most once a minute and names the commands that were just sent.
1. In Companion: **Modules → Import module package**, choose `tascam-da-6400-1.3.1-quiet.tgz` (on NA Ingest's Desktop).
2.**Connections → Tascam → edit**, set the module version to **1.3.1-quiet**, save.
`remote-access/windows-setup.ps1` sets each PC up once (SSH for NA Ingest, start tasks, Wake-on-LAN). An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) gets a start task that runs with highest privileges; otherwise Windows refuses to start it (error 0x800702E4). Windows restarts the SSH service and rtpMIDI by itself if they stop.
| `config.json` | tiles, positions, checks, fixes, hosts, windows: every setting (the `site` block names the campus; `spotify` holds the playlist; `timing` and `page_hold_ms` the numbers behind HOLD 2s and WAIT 1 MIN) |
| `schema.py` | the registry of every setting: type, default, help, which screen of the settings page owns it, and the rules a config must pass (`tests/test_schema.py` keeps it in step with the code) |
| `tests/` | offline tests, including a fake Companion and a MIDI loopback; `test_schema.py` checks that every key the code reads is declared in `schema.py` with the same default |
