status-indicators/README.md

# Status Indicators — Companion STATUS page for NA Ingest

One Companion page (page 19, **"19 STATUS"**) that is solid green when everything is fine and **blinks red with the actual problem** ("HD2 NO DISK", "LIGHT PC MIDI LOST") when something needs attention.

There are two parts:

- **`status_panel.py`** — a watchdog that runs on NA Ingest as the LaunchAgent `org.waterschurch.status-panel`. Every second it reads Companion's connection statuses and module variables. It also checks what Companion can't see: network MIDI peers, Bonjour MIDI sessions, pings, launchd services, disk space, and whether the Resi cloud is reachable. It writes the result into one Companion custom variable.
- **`status-page.companionconfig`** — the page. Every button just draws what the watchdog wrote and blinks on your existing `flash_tick` trigger.

## Layout (Stream Deck XL; MK.2 and Mini show the top-left corner)

|        | col 1 | col 2 | col 3 | col 4 | col 5 | col 6 | col 7 |
|--------|-------|-------|-------|-------|-------|-------|-------|
| row 0  | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM |
| row 1  | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM |
| row 2  | X32 | DANTE | TRACKS | LIGHT PC | RACK (SuperRack) | BIBLE TV | PROP (ProPresenter) |
| row 3  | ATEM | OBS | RESI | NET | INGEST | LED | ARENA |

Column 0 holds page up / page number / page down, plus **SERVICE** at row 3.

Row 0 holds the group tiles. Each one shows the first problem in its group, plus "+N" if there are more. RF covers all six Shure units: AD4Q ×2, ULXD4, and PSM1000 ×3.

## Colours

| Tile | Meaning |
|---|---|
| Green | OK. The second line shows something useful, like `28h` of record time or `10/10` Dante devices. |
| **Blinking red** | Show-stopping, during service mode. |
| Solid dark red | Acknowledged. It stays red until the problem clears, then goes green. A new failure blinks again. |
| Amber | Warning: low disk, battery at 2 bars, RF interference, a device that keeps dropping out ("WING GONE x3"), a network blip in the last 3 minutes ("BLIP 10:21"). |
| Dim red, not blinking | A fault outside service mode. |
| Grey | Expected off outside service: OBS, a HyperDeck with no disk, Bible TV. |
| **Purple** | The watchdog has stopped. The master shows `WATCHDOG DOWN` and Companion's own error count. |

## Buttons

- **Press any tile:** acknowledge its alarms (stops the blinking). Press **ALL** to acknowledge everything.
- **Hold a device tile for 2 seconds:** restart that tile's *Companion connection*. This is the fix for a connection stuck after a network blip. It only restarts Companion's side and never reboots the device.
- **SERVICE:** service mode turns on automatically **Sunday 07:00–12:30** (`service_windows` in `config.json`). Press SERVICE to switch it on or off by hand, for example for a Wednesday event. The manual setting lasts until the next scheduled start or end.

## What each tile checks

| Tile | Checks |
|---|---|
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| TASCAM | connection; the recorder's own Error / Caution |
| X32, ATEM, ARENA, LED, OBS | Companion connection |
| DANTE | 10 key devices present (DiGiCo DMIs, WING, DQ rack, AD4Qs, Studio A X32 / Resi AJA / Ingest, Translation) |
| TRACKS | NA-Tracks Dante device; tracks Mac pings; "NA Tracks" MIDI session still on the network; Ableton OSC connection |
| LIGHT PC | Lighting PC still joined to NA Ingest's network MIDI session; its MIDI session still on the network |
| RACK | SuperRack-LiveBox still joined to the MIDI session; its Dante device |
| BIBLE TV | TV pings; OSC connection |
| PROP | ProPresenter API; NA-ProPresenter-Mac Dante device |
| RESI | decoder (RossTalk); resi-studio-cues; Resi cloud reachable |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) |
| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk |
| RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted |

A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately.

## One-time Companion setup

1. **Variables → Custom Variables:** add three variables. Leave the start values empty and "persist" off:
   - `sp_state`
   - `sp_hb`
   - `sp_cmd`
2. **Import / Export → Import:** choose `status-page.companionconfig` (it's on NA Ingest's Desktop) and import it into **page 19**. Page 19 is empty today.
3. Make sure the **flash_tick_500** trigger stays enabled. It's the blink clock.

## Changing things

This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:

```bash
scp status_panel.py coremidi.py config.json install.sh checklist-ingest:status-panel-install/
```

Then run the installer there. It backs up the old config and restarts the watchdog:

```bash
ssh checklist-ingest 'zsh ~/status-panel-install/install.sh'
```

If you add or move tiles, also rebuild the page and import it into page 19 again:

```bash
python3 make_page.py
```

Other commands:

```bash
python3 status_panel.py --once          # poll once and print every tile; writes nothing
python3 -m unittest discover -s tests   # offline tests, no Companion or gear needed
zsh install.sh --remove                 # stop the watchdog
```

The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.

## Optional: see what the tracks send to the lights

The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:

1. On NA Ingest, open **Audio MIDI Setup → Window → Show MIDI Studio → Network** (the globe icon).
2. Under **My Sessions**, press **+**, rename the new session to `NA Ingest Listen`, and tick it to enable it.
3. With `NA Ingest Listen` selected, choose **NA Tracks** in the **Directory** and press **Connect**.

The watchdog picks it up within about 10 seconds. Until then that line just stays hidden.

The **LIGHT PC** tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set `"max_silence_seconds": 10` on the LIGHT PC `midi_rx` check. The tile then goes amber if the lighting PC stops answering during service.

## Files

| File | Purpose |
|---|---|
| `status_panel.py` | the watchdog |
| `coremidi.py` | reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs) |
| `config.json` | tiles, positions, checks, service windows |
| `make_page.py` | builds `status-page.companionconfig` from `config.json` |
| `install.sh` | installs or updates the LaunchAgent on NA Ingest |
| `tests/` | offline tests, including a fake Companion and a MIDI loopback |