One Companion page (page 19, **"19 Status"**) that is solid green when everything is fine and **blinks red with the actual problem** ("HD3 NO DISK", "LIGHT PC MIDI LOST") when something needs attention. The page number key on every other page follows the same colour, so you see a problem from any page.
- **`status_panel.py`** — a watchdog that runs on NA Ingest as the LaunchAgent `org.waterschurch.status-panel`. Every second it reads Companion's connection statuses and module variables. It also checks what Companion can't see: network MIDI peers, Bonjour MIDI sessions, pings, launchd services, disk space, the Bible TV app, and whether the Resi cloud is reachable. It writes the result into one Companion custom variable.
| Solid dark red | Acknowledged. It stays red until the problem clears, then goes green. A new failure blinks again. |
| Amber | Warning: low disk, battery at 2 bars, RF interference, a device that keeps dropping out ("WING GONE x3"), a network blip in the last 3 minutes ("BLIP 10:21"). |
- **Hold any button for 2 seconds:** fix what *that button is failing on right now*. A green button does nothing, the same fix won't re-run within 60 s, and every command comes from `"fixes"` in `config.json`. Hold a group button to fix all its members.
| Failing | Hold does |
|---|---|
| APPS: PROPRES OFFLINE / NOT RUNNING | restarts ProPresenter on its Mac (the newest version installed) |
| APPS: ABLETON / TRANSLATE NOT RUNNING | starts Ableton on the Tracks Mac / Church Translate on NA Translation |
| ARENA | starts Resolume Arena on the Lighting PC if it is not running, then reconnects Companion |
| OBS | starts OBS on the LiveStream PC if it is not running, then reconnects Companion. If the PC is off, it is woken (Wake-on-LAN) and the button shows WAKING PC; hold again once someone has logged in. |
| MIDI: LIGHT PC / FX LIGHTS / SUPERRACK MIDI LOST | reconnects that peer to the session it dropped out of, from the Mac's side (NA Ingest's own session, or NA Tracks' over SSH), like Connect in Audio MIDI Setup. Other peers are left alone and nothing is restarted. The address is where the peer was last seen, else `"address"` on the check. Not for a missing Bonjour advert ("... MIDI GONE"). **Keep the Lighting PC's network on Public:** its Bonjour firewall rules only allow Public, and on Private it vanishes from the MIDI directory and cannot rejoin a session (2026-10-08). Never restart rtpMIDI as a fix: it drops every session and does not rejoin them. |
| MIDI: LIGHT PC MIDI GONE | starts rtpMIDI on the Lighting PC if it is not running (its service, then the tray app). Its service also restarts itself if it crashes (Windows service recovery, set by `windows-setup.ps1`). Audio MIDI Setup does not need to be open on the Macs: their sessions run in macOS's MIDI server (tested 2026-10-08: quitting it left NA Ingest's session connected). |
| DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
| INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest |
| any button whose Companion connection is **disabled** | turns the connection back on (nothing else) |
| HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. |
| anything else with a Companion connection | restarts that Companion connection |
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.
The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`).
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. |
| APPS | ProPresenter API; Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. |
| VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`<autoloadpath>` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) |
| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk |
| RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted |
A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately.
## One-time Companion setup
1.**Variables → Custom Variables:** add three variables. Leave the start values empty and "persist" off:
-`sp_state`
-`sp_hb`
-`sp_cmd`
2.**Import / Export → Import:** choose `status-page.companionconfig` (it's on NA Ingest's Desktop) and import it into **page 19**. Page 19 is empty today.
3. Make sure the **flash_tick_500** trigger stays enabled. It's the blink clock.
## Changing things
This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:
python3 status_panel.py --once # poll once and print every tile; writes nothing
python3 -m unittest discover -s tests # offline tests, no Companion or gear needed
zsh install.sh --remove # stop the watchdog
```
The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.
## Optional: see what the tracks send to the lights
The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:
1. On NA Ingest, open **Audio MIDI Setup → Window → Show MIDI Studio → Network** (the globe icon).
2. Under **My Sessions**, press **+**, rename the new session to `NA Ingest Listen`, and tick it to enable it.
3. With `NA Ingest Listen` selected, choose **NA Tracks** in the **Directory** and press **Connect**.
The watchdog picks it up within about 10 seconds. Until then that line just stays hidden.
The **LIGHT PC** tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set `"max_silence_seconds": 10` on the LIGHT PC `midi_rx` check. The tile then goes amber if the lighting PC stops answering during service.
The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the recorder rejects one of those polls, and the module logs a warning every time: about 430,000 lines a day in Companion's log. `tascam-quiet/make_quiet_tascam.py` builds **1.3.1-quiet** from the stock module: the same module, but that warning is logged at most once a minute and names the commands that were just sent.
1. In Companion: **Modules → Import module package**, choose `tascam-da-6400-1.3.1-quiet.tgz` (on NA Ingest's Desktop).
2.**Connections → Tascam → edit**, set the module version to **1.3.1-quiet**, save.
