status-indicators/README.md

# Status Indicators — Companion STATUS page for NA Ingest

One Companion page (page 19, **"19 Status"**) that is solid green when everything is fine and **blinks red with the actual problem** ("HD3 NO DISK", "LIGHT PC MIDI LOST") when something needs attention. The page number key on every other page follows the same colour, so you see a problem from any page.

There are two parts:

- **`status_panel.py`** — a watchdog that runs on NA Ingest as the LaunchAgent `org.waterschurch.status-panel`. Every second it reads Companion's connection statuses and module variables. It also checks what Companion can't see: network MIDI peers, Bonjour MIDI sessions, pings, launchd services, disk space, the Bible TV app, and whether the Resi cloud is reachable. It writes the result into one Companion custom variable.
- **`status-page.companionconfig`** — the page. Every button just draws what the watchdog wrote and blinks on your existing `flash_tick` trigger.

## Layout (Stream Deck XL; MK.2 and Mini show the top-left corner)

|        | col 0 | col 1 | col 2 | col 3 | col 4 | col 5 | col 6 | col 7 |
|--------|-------|-------|-------|-------|-------|-------|-------|-------|
| row 0  | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM |
| row 1  | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM |
| row 2  | DOWN | X32 | DANTE | MIDI | APPS | | | |
| row 3  | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA |

Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more:

- REC: HD1–6 and the Tascam.
- AUDIO: X32 and Dante.
- RF: all six Shure units (AD4Q ×2, ULXD4, PSM1000 ×3).
- SHOW: MIDI and APPS.
- VIDEO: ATEM, LED and Arena.
- STREAM: OBS and Resi.

## Colours

| Colour | Meaning |
|---|---|
| Green | OK. The second line shows something useful, like `28h` of record time or `19/19` Dante devices. |
| **Blinking red** | Something is wrong. |
| Solid dark red | Acknowledged. It stays red until the problem clears, then goes green. A new failure blinks again. |
| Amber | Warning: low disk, battery at 2 bars, RF interference, a device that keeps dropping out ("WING GONE x3"), a network blip in the last 3 minutes ("BLIP 10:21"). |
| Dim red, not blinking | A fault while alarms are muted. |
| Grey | Not in use: an optional HyperDeck with no disk, or a disabled connection. |
| **Purple** | The watchdog has stopped. The master shows `WATCHDOG DOWN` and Companion's own error count. |

## Buttons

- **Press any button:** acknowledge its alarms (stops the blinking). Press **ALL** to acknowledge everything.
- **Hold any button for 2 seconds:** fix what *that button is failing on right now*. A green button does nothing, the same fix won't re-run within 60 s, and every command comes from `"fixes"` in `config.json`. Hold a group button to fix all its members.

  | Failing | Hold does |
  |---|---|
  | APPS: PROPRES OFFLINE / NOT RUNNING | restarts ProPresenter on its Mac (the newest version installed) |
  | APPS: ABLETON / TRANSLATE NOT RUNNING | starts Ableton on the Tracks Mac / Church Translate on NA Translation |
  | DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) |
  | DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
  | DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) |

  **The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
  | INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest |
  | HD1–6 lost by Companion | reboots that deck. Never one that is recording, and never for NO DISK or NOT REC |
  | anything else with a Companion connection | restarts that Companion connection |

  It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.

## Page keys on every page

The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`).

- **Tap the page number:** go to page 1 (`"nav_tap_page"` in `config.json`). It acts when you let go, not when you press.
- **Hold the page number for 1 second:** jump to the status page (page 19). The hold time is `"nav_hold_ms"` in `config.json`.

- `python3 nav_keys.py` applies them to every page. Run it again after adding pages.
- `python3 nav_keys.py --revert` puts Companion's built-in nav keys back.

## What each button checks

| Button | Checks |
|---|---|
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| TASCAM | connection; the recorder's own Error / Caution |
| X32, ATEM, ARENA, OBS | Companion connection |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". |
| MIDI | Lighting PC and SuperRack still joined to NA Ingest's network MIDI session; the Lighting PC and NA Tracks MIDI sessions still on the network. A clean disconnect shows within seconds; a crash or pulled cable takes about a minute (macOS's own timeout, measured). |
| APPS | ProPresenter API; Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. |
| RESI | decoder (RossTalk connection and ping); resi-studio-cues; Resi cloud reachable |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) |
| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk |
| RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted |

A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately.

## One-time Companion setup

1. **Variables → Custom Variables:** add three variables. Leave the start values empty and "persist" off:
   - `sp_state`
   - `sp_hb`
   - `sp_cmd`
2. **Import / Export → Import:** choose `status-page.companionconfig` (it's on NA Ingest's Desktop) and import it into **page 19**. Page 19 is empty today.
3. Make sure the **flash_tick_500** trigger stays enabled. It's the blink clock.

## Changing things

This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:

```bash
scp status_panel.py coremidi.py config.json install.sh checklist-ingest:status-panel-install/
```

Then run the installer there. It backs up the old config and restarts the watchdog:

```bash
ssh checklist-ingest 'zsh ~/status-panel-install/install.sh'
```

If you add or move tiles, rebuild the page:

```bash
python3 make_page.py
```

Then import it into page 19, either by hand (Import/Export → Import) or with:

```bash
python3 push_page.py
```

`push_page.py` does the same import over Companion's own API. It refuses to overwrite page 19 unless that page is empty or already the status page.

Other commands:

```bash
python3 status_panel.py --once          # poll once and print every tile; writes nothing
python3 -m unittest discover -s tests   # offline tests, no Companion or gear needed
zsh install.sh --remove                 # stop the watchdog
```

The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.

## Optional: see what the tracks send to the lights

The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:

1. On NA Ingest, open **Audio MIDI Setup → Window → Show MIDI Studio → Network** (the globe icon).
2. Under **My Sessions**, press **+**, rename the new session to `NA Ingest Listen`, and tick it to enable it.
3. With `NA Ingest Listen` selected, choose **NA Tracks** in the **Directory** and press **Connect**.

The watchdog picks it up within about 10 seconds. Until then that line just stays hidden.

The **LIGHT PC** tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set `"max_silence_seconds": 10` on the LIGHT PC `midi_rx` check. The tile then goes amber if the lighting PC stops answering during service.

## Files

| File | Purpose |
|---|---|
| `status_panel.py` | the watchdog |
| `coremidi.py` | reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs) |
| `config.json` | tiles, positions, checks, service windows |
| `make_page.py` | builds `status-page.companionconfig` from `config.json` |
| `install.sh` | installs or updates the LaunchAgent on NA Ingest |
| `push_page.py` | imports the status page into page 19 over Companion's API |
| `nav_keys.py` | colours (or restores) the page nav keys on every page |
| `nav_templates.json` | Companion's own nav key designs, coloured by `make_page.py` |
| `tests/` | offline tests, including a fake Companion and a MIDI loopback |