christian/status-indicators
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restart page (23), force restarts, Tascam reconnect, ProdCom, Resi encoder

Browse Source 
- Page 23 "Restart": hold a key 2 s to restart DVS (4 Macs), ProPresenter,
  Ableton, Church Translate, ProdCom, ProdDash, captions, the bridge or the
  Tascam connection even when the status page says it is fine ("force",
  shared 60 s guard). RESTART PAGE key on page 19, STATUS PAGE back.
- Holds carry Companion's clock; ones queued while the watchdog was down
  are dropped instead of firing together later.
- TASCAM hold always reconnects and checks again after 20 s: HUNG - POWER
  CYCLE IT (pings, control port hung), NO NETWORK, or turns a disabled
  connection on. Tascam pinged at its Companion address.
- INGEST: ProdCom (:24480), ProdDash's page (:24500), the caption screen;
  DVS on Ingest restarts ProdCom afterwards. RESI: the encoder (10.1.10.202).
- Windows starts check the app really started (FIX FAILED otherwise); OBS's
  start task runs elevated (OBS is "Run as administrator"); setup script
  does that for any such app and gives sshd restart-on-failure.
- push_page.py checks what is on the target page, not its name (pages 22
  and 97 are named "PAGE" but hold buttons); --which restart.
- Companion custom variables are written in the request body (no URL limit).
Reviewed by a 3-lens review with per-finding verification; 7 confirmed
issues fixed. Tested live: OBS hold opens OBS; Restart page TASCAM key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-08 18:08:29 -04:00
co-authored by Claude Opus 5.5
parent 17347a5d9a
commit a0c72183bd
7 changed files with 544 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26 -5
View File
@@ -13,7 +13,7 @@ There are two parts:
|--------|-------|-------|-------|-------|-------|-------|-------|-------| |--------|-------|-------|-------|-------|-------|-------|-------|-------|
| row 0 | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM | | row 0 | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM |
| row 1 | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM | | row 1 | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM |
| row 2 | DOWN | X32 | DANTE | MIDI | APPS | VISTA | | | | row 2 | DOWN | X32 | DANTE | MIDI | APPS | VISTA | RESTART PAGE | |
| row 3 | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA | | row 3 | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA |
Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more: Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more:
@@ -55,7 +55,8 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
| DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) | | DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) |
| DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches | | DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
| DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) | | DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) |
| INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest | | INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH | restarts that one on NA Ingest |
| TASCAM (always, even when it looks fine) | reconnects Companion to the recorder and looks again after 20 s. **HUNG - POWER CYCLE IT**: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). **NO NETWORK / CHECK POWER**: it does not answer at all. |
| any button whose Companion connection is **disabled** | turns the connection back on (nothing else) | | any button whose Companion connection is **disabled** | turns the connection back on (nothing else) |
| HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. | | HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. |
| anything else with a Companion connection | restarts that Companion connection | | anything else with a Companion connection | restarts that Companion connection |
@@ -65,6 +66,22 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`. It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight. - **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.
## Restart page (page 23)
Tap **RESTART PAGE** on the status page (row 2). Each key restarts one thing **even when the status page says it is fine**; hold it 2 s. It shows RESTARTING, then DONE (or FAILED / what to do). The same thing will not run twice within a minute. **STATUS PAGE** (bottom left) goes back.
| Key | Restarts |
|---|---|
| DVS PROP / TRACKS / TRANSL / INGEST | Dante Virtual Soundcard's engine on that Mac (audio to and from it drops 5–10 s). DVS INGEST also restarts ProdCom afterwards, which otherwise keeps a dead input. |
| PROPRES | ProPresenter (quits and reopens the newest version) |
| ABLETON | Ableton on the Tracks Mac (quits it: unsaved set changes are lost, and Live may ask to recover - click it on the Tracks Mac) |
| TRANSLATE | Church Translate on NA Translation |
| PRODCOM | ProdCom on NA Ingest (lobby captions blank for 5–15 s) |
| PRODDASH / CAPTIONS / BRIDGE | that service on NA Ingest |
| TASCAM | reconnects Companion to the recorder (see TASCAM above) |
Add or move keys in `"restart_page"` in `config.json`, then `python3 make_page.py` and `python3 push_page.py --which restart`. `push_page.py` refuses any page that holds real buttons unless it already is that page.
## Page keys on every page ## Page keys on every page
The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`). The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`).
@@ -81,16 +98,16 @@ The **page number key** on every page follows the ALL button. It's green when ev
|---|---| |---|---|
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't | | HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. | | | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded. | | TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection. |
| X32, ATEM, ARENA, OBS | Companion connection | | X32, ATEM, ARENA, OBS | Companion connection |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. | | LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". | | DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". |
| MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. | | MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. |
| APPS | ProPresenter API (offline or error only: its harmless yellow warning, every 15–20 min, is ignored with `"ignore_warning": true`); Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. | | APPS | ProPresenter API (offline or error only: its harmless yellow warning, every 15–20 min, is ignored with `"ignore_warning": true`); Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. |
| VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`<autoloadpath>` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. | | VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`<autoloadpath>` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. |
| RESI | decoder (RossTalk connection and ping); resi-studio-cues; Resi cloud reachable | | RESI | decoder (RossTalk connection and ping, 10.1.10.203); **encoder** (ping and its web page, 10.1.10.202, LAOEncoder); resi-studio-cues; Resi cloud reachable. Only shows they are on the network, not that the encoder is streaming. |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) | | NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) |
| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk | | INGEST | the bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, **ProdCom** (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk |
| RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; `"host"` in `config.json` is only the fallback. | | RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; `"host"` in `config.json` is only the fallback. |
A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately. A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately.
@@ -163,6 +180,10 @@ The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the
To go back, pick 1.3.0 on the connection again. To go back, pick 1.3.0 on the connection again.
## Windows PCs
`remote-access/windows-setup.ps1` sets each PC up once (SSH for NA Ingest, start tasks, Wake-on-LAN). An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) gets a start task that runs with highest privileges; otherwise Windows refuses to start it (error 0x800702E4). Windows restarts the SSH service and rtpMIDI by itself if they stop.
## Files ## Files
| File | Purpose | | File | Purpose |
config.json
+48 -12
View File
@@ -44,29 +44,57 @@
"run": "pgrep -xq Live || open -a 'Ableton Live 11 Suite'" }, "run": "pgrep -xq Live || open -a 'Ableton Live 11 Suite'" },
"translate": { "host": "translation", "short": "TRANSLATE", "label": "start Church Translate if it is not running", "translate": { "host": "translation", "short": "TRANSLATE", "label": "start Church Translate if it is not running",
"run": "pgrep -xq 'Church Translate' || open -a 'Church Translate'" }, "run": "pgrep -xq 'Church Translate' || open -a 'Church Translate'" },
"dvs_prop": { "host": "prop", "short": "DVS", "label": "start Dante Virtual Soundcard on the ProPresenter Mac", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, "dvs_prop": { "host": "prop", "short": "DVS", "label": "restart Dante Virtual Soundcard on the ProPresenter Mac (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" },
"dvs_tracks": { "host": "tracks", "short": "DVS", "label": "start Dante Virtual Soundcard on the Tracks Mac", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, "dvs_tracks": { "host": "tracks", "short": "DVS", "label": "restart Dante Virtual Soundcard on the Tracks Mac (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" },
"dvs_translation": { "host": "translation", "short": "DVS", "label": "start Dante Virtual Soundcard on NA Translation", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, "dvs_translation": { "host": "translation", "short": "DVS", "label": "restart Dante Virtual Soundcard on NA Translation (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" },
"dvs_ingest": { "kind": "local", "short": "DVS", "label": "start Dante Virtual Soundcard on NA Ingest", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, "dvs_ingest": { "kind": "local", "short": "DVS", "label": "restart Dante Virtual Soundcard on NA Ingest, then ProdCom (it keeps a dead DVS input otherwise)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard' || exit 1; sleep 5; pkill -TERM -x ProdCom; i=0; while pgrep -xq ProdCom && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq ProdCom && pkill -KILL -x ProdCom; sleep 1; open -g /Applications/ProdCom.app" },
"bridge": { "kind": "local", "short": "BRIDGE", "label": "restart the Resi bridge", "bridge": { "kind": "local", "short": "BRIDGE", "label": "restart the Resi bridge",
"run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.resi-propresenter || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.resi-propresenter.plist" }, "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.resi-propresenter || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.resi-propresenter.plist" },
"captions": { "kind": "local", "short": "CAPTIONS", "label": "restart lobby captions", "captions": { "kind": "local", "short": "CAPTIONS", "label": "restart lobby captions (server and display)",
"run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions.plist" }, "run": "s=0; { launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions.plist; } || s=1; { launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions-display || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions-display.plist; } || s=1; exit $s" },
"bibleapp": { "host": "bibletv", "short": "BIBLE APP", "label": "restart the Bible app on the Sanctuary TV PC (Windows: cmd.exe)", "bibleapp": { "host": "bibletv", "short": "BIBLE APP", "label": "restart the Bible app on the Sanctuary TV PC (Windows: cmd.exe)",
"run": "taskkill /F /T /IM \"Bible App.exe\" >nul 2>&1 & ping -n 3 127.0.0.1 >nul & schtasks /Run /TN \"\\StatusPanel\\Start BibleApp\"" }, "run": "taskkill /F /T /IM \"Bible App.exe\" >nul 2>&1 & ping -n 3 127.0.0.1 >nul & schtasks /Run /TN \"\\StatusPanel\\Start BibleApp\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Bible App.exe\" | find /I \"Bible App.exe\" >nul" },
"obs": { "host": "livestream", "short": "OBS", "label": "start OBS on the LiveStream PC if it is not running (wakes the PC if it is off), then reconnect Companion", "reconnect": true, "obs": { "host": "livestream", "short": "OBS", "label": "start OBS on the LiveStream PC if it is not running (wakes the PC if it is off), then reconnect Companion", "reconnect": true,
"run": "tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start OBS\"" }, "run": "tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start OBS\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul)" },
"arena": { "host": "lighting", "short": "ARENA", "label": "start Resolume Arena on the Lighting PC if it is not running, then reconnect Companion", "reconnect": true, "arena": { "host": "lighting", "short": "ARENA", "label": "start Resolume Arena on the Lighting PC if it is not running, then reconnect Companion", "reconnect": true,
"run": "tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start Arena\"" }, "run": "tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start Arena\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul)" },
"vista": { "host": "lighting", "short": "VISTA", "label": "start Vista on the Lighting PC if it is not running (a running Vista is never closed)", "vista": { "host": "lighting", "short": "VISTA", "label": "start Vista on the Lighting PC if it is not running (a running Vista is never closed)",
"run": "tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start Vista\"" }, "run": "tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start Vista\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul)" },
"rtpmidi_start": { "host": "lighting", "short": "RTPMIDI", "label": "start rtpMIDI on the Lighting PC if it is not running (service, then tray app). A running rtpMIDI is never restarted: that drops every session.", "rtpmidi_start": { "host": "lighting", "short": "RTPMIDI", "label": "start rtpMIDI on the Lighting PC if it is not running (service, then tray app). A running rtpMIDI is never restarted: that drops every session.",
"run": "(sc query rtpMIDIService | find \"RUNNING\" >nul || net start rtpMIDIService) && (tasklist /NH /FI \"IMAGENAME eq rtpMIDI.exe\" | find /I \"rtpMIDI.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start rtpMIDI\")" }, "run": "(sc query rtpMIDIService | find \"RUNNING\" >nul || net start rtpMIDIService) && (tasklist /NH /FI \"IMAGENAME eq rtpMIDI.exe\" | find /I \"rtpMIDI.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start rtpMIDI\")" },
"midi_reconnect": { "kind": "midi_connect", "short": "RECONNECT", "label": "reconnect the peer to the network MIDI session it dropped out of (like Connect in Audio MIDI Setup); the address is where it was last seen, else the check's \"address\"" }, "midi_reconnect": { "kind": "midi_connect", "short": "RECONNECT", "label": "reconnect the peer to the network MIDI session it dropped out of (like Connect in Audio MIDI Setup); the address is where it was last seen, else the check's \"address\"" },
"proddash": { "kind": "local", "short": "PRODDASH", "label": "start ProdDash", "run": "open /Applications/ProdDash.app" }, "caption_screen": { "kind": "local", "short": "CAPTION SCREEN", "label": "restart the lobby caption screen (display app only)", "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions-display || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions-display.plist" },
"proddash": { "kind": "local", "short": "PRODDASH", "label": "restart ProdDash (its launcher stops its server cleanly)",
"run": "pkill -TERM -x ProdDash; i=0; while pgrep -xq ProdDash && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; open -g /Applications/ProdDash.app" },
"prodcom": { "kind": "local", "short": "PRODCOM", "label": "restart ProdCom (transcription; lobby captions blank for 5-15 s)",
"run": "pkill -TERM -x ProdCom; i=0; while pgrep -xq ProdCom && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq ProdCom && pkill -KILL -x ProdCom; sleep 1; open -g /Applications/ProdCom.app" },
"ableton_restart": { "host": "tracks", "short": "ABLETON", "label": "restart Ableton on the Tracks Mac (quits it; unsaved set changes are lost, and Live may ask to recover on the Tracks Mac)", "conn": "ableton", "reconnect": true, "timeout": 60,
"run": "pkill -TERM -x 'Live'; i=0; while pgrep -xq 'Live' && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'Live' && pkill -KILL -x 'Live'; sleep 1; open -a 'Ableton Live 11 Suite'" },
"translate_restart": { "host": "translation", "short": "TRANSLATE", "label": "restart Church Translate on NA Translation (and its translation server)", "timeout": 60,
"run": "pg=$(ps -o pgid= -p \"$(pgrep -x 'Church Translate' | head -1)\" 2>/dev/null | tr -d ' '); pkill -TERM -x 'Church Translate'; i=0; while pgrep -xq 'Church Translate' && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'Church Translate' && pkill -KILL -x 'Church Translate'; [ -n \"$pg\" ] && [ \"$pg\" != \"$(ps -o pgid= -p $$ | tr -d ' ')\" ] && kill -KILL -- -\"$pg\" 2>/dev/null; sleep 1; open -a 'Church Translate'" },
"tascam_reconnect": { "kind": "reconnect", "conn": "tascam", "short": "TASCAM", "verify_seconds": 20, "hung_note": "HUNG - POWER\nCYCLE IT",
"label": "reconnect Companion to the Tascam; if it still does not answer but pings, its control port has hung: power-cycle it at the rack" },
"hyperdeck": { "kind": "hyperdeck", "short": "HYPERDECK", "label": "reconnect Companion to the HyperDeck, or reboot the deck only if it is hung" } "hyperdeck": { "kind": "hyperdeck", "short": "HYPERDECK", "label": "reconnect Companion to the HyperDeck, or reboot the deck only if it is hung" }
}, },
"restart_page": {
"page": 23, "page_name": "23\\nRestart", "jump_pos": [2, 6], "back_pos": [3, 0],
"keys": [
{ "id": "rs_dvs_prop", "label": "DVS PROP", "pos": [0, 1], "fix": "dvs_prop" },
{ "id": "rs_dvs_tracks", "label": "DVS TRACKS", "pos": [0, 2], "fix": "dvs_tracks" },
{ "id": "rs_dvs_transl", "label": "DVS TRANSL", "pos": [0, 3], "fix": "dvs_translation" },
{ "id": "rs_dvs_ingest", "label": "DVS INGEST", "pos": [0, 4], "fix": "dvs_ingest" },
{ "id": "rs_propres", "label": "PROPRES", "pos": [1, 1], "fix": "propresenter" },
{ "id": "rs_ableton", "label": "ABLETON", "pos": [1, 2], "fix": "ableton_restart" },
{ "id": "rs_translate", "label": "TRANSLATE", "pos": [1, 3], "fix": "translate_restart" },
{ "id": "rs_prodcom", "label": "PRODCOM", "pos": [2, 1], "fix": "prodcom" },
{ "id": "rs_proddash", "label": "PRODDASH", "pos": [2, 2], "fix": "proddash" },
{ "id": "rs_captions", "label": "CAPTIONS", "pos": [2, 3], "fix": "captions" },
{ "id": "rs_bridge", "label": "BRIDGE", "pos": [2, 4], "fix": "bridge" },
{ "id": "rs_tascam", "label": "TASCAM", "pos": [3, 1], "fix": "tascam_reconnect" }
]
},
"tiles": [ "tiles": [
{ "id": "all", "label": "ALL", "pos": [0, 1], "kind": "master" }, { "id": "all", "label": "ALL", "pos": [0, 1], "kind": "master" },
{ "id": "rec", "label": "REC", "pos": [0, 2], "members": ["hd1", "hd2", "hd3", "hd4", "hd5", "hd6", "tascam"] }, { "id": "rec", "label": "REC", "pos": [0, 2], "members": ["hd1", "hd2", "hd3", "hd4", "hd5", "hd6", "tascam"] },
@@ -82,7 +110,10 @@
{ "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] }, { "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] }, { "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] }, { "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] },
{ "id": "tascam", "label": "TASCAM", "pos": [1, 7], "checks": [{ "type": "tascam", "conn": "tascam" }] }, { "id": "tascam", "label": "TASCAM", "pos": [1, 7], "hold_always": "tascam_reconnect", "checks": [
{ "type": "tascam", "conn": "tascam", "fix": "tascam_reconnect" },
{ "type": "ping", "conn": "tascam", "host": "10.1.11.74", "name": "TASCAM" }
] },
{ "id": "x32", "label": "X32", "pos": [2, 1], "checks": [{ "type": "conn", "conn": "x32" }] }, { "id": "x32", "label": "X32", "pos": [2, 1], "checks": [{ "type": "conn", "conn": "x32" }] },
{ "id": "dante", "label": "DANTE", "pos": [2, 2], "checks": [ { "id": "dante", "label": "DANTE", "pos": [2, 2], "checks": [
@@ -131,6 +162,8 @@
{ "id": "resi", "label": "RESI", "pos": [3, 3], "checks": [ { "id": "resi", "label": "RESI", "pos": [3, 3], "checks": [
{ "type": "conn", "conn": "RESI_Decoder", "tag": "DECODER" }, { "type": "conn", "conn": "RESI_Decoder", "tag": "DECODER" },
{ "type": "ping", "host": "10.1.10.203", "name": "DECODER" }, { "type": "ping", "host": "10.1.10.203", "name": "DECODER" },
{ "type": "ping", "host": "10.1.10.202", "name": "ENCODER" },
{ "type": "http", "url": "http://10.1.10.202/", "msg": "ENCODER WEB DOWN", "interval_seconds": 10 },
{ "type": "conn", "conn": "resi-studio-cues" }, { "type": "conn", "conn": "resi-studio-cues" },
{ "type": "http", "url": "https://central.resi.io/", "name": "CLOUD" } { "type": "http", "url": "https://central.resi.io/", "name": "CLOUD" }
] }, ] },
@@ -138,7 +171,10 @@
{ "id": "ingest", "label": "INGEST", "pos": [3, 5], "checks": [ { "id": "ingest", "label": "INGEST", "pos": [3, 5], "checks": [
{ "type": "launchd", "label": "org.waterschurch.resi-propresenter", "name": "BRIDGE", "fix": "bridge" }, { "type": "launchd", "label": "org.waterschurch.resi-propresenter", "name": "BRIDGE", "fix": "bridge" },
{ "type": "launchd", "label": "org.waterschurch.lobby-captions", "name": "CAPTIONS", "fix": "captions" }, { "type": "launchd", "label": "org.waterschurch.lobby-captions", "name": "CAPTIONS", "fix": "captions" },
{ "type": "launchd", "label": "org.waterschurch.lobby-captions-display", "name": "CAPTION SCREEN", "fix": "caption_screen" },
{ "type": "http", "url": "http://127.0.0.1:24480/api/v1/ping", "msg": "PRODCOM DOWN", "interval_seconds": 5, "fix": "prodcom" },
{ "type": "process", "match": "/Applications/ProdDash.app/Contents/MacOS/ProdDash", "name": "PRODDASH", "fix": "proddash" }, { "type": "process", "match": "/Applications/ProdDash.app/Contents/MacOS/ProdDash", "name": "PRODDASH", "fix": "proddash" },
{ "type": "http", "url": "http://127.0.0.1:24500/", "msg": "PRODDASH DOWN", "interval_seconds": 10, "fix": "proddash" },
{ "type": "disk", "path": "/", "warn_gb": 25, "err_gb": 8 } { "type": "disk", "path": "/", "warn_gb": 25, "err_gb": 8 }
] }, ] },
{ "id": "led", "label": "LED", "pos": [3, 6], "checks": [ { "id": "led", "label": "LED", "pos": [3, 6], "checks": [
make_page.py
+108 -20
View File
@@ -13,9 +13,10 @@ watchdog can never leave a frozen green page behind.
Buttons write commands into sp_cmd, which status_panel.py reads: Buttons write commands into sp_cmd, which status_panel.py reads:
press ack <tile> (the master acks everything) press ack <tile> (the master acks everything)
hold 2 seconds restart <tile> (restarts that tile's Companion connections) hold 2 seconds restart <tile> (fixes what that tile is failing on)
Restart page force <key> (hold 2 s: restart that thing even when it looks fine)
python3 make_page.py -> status-page.companionconfig python3 make_page.py -> status-page.companionconfig and restart-page.companionconfig
""" """
import gzip import gzip
import hashlib import hashlib
@@ -96,6 +97,57 @@ def command_action(tid, role, cmd_var, command):
} }
def set_page_action(tid, role, page):
return {"type": "action", "id": stable_id(tid, role), "definitionId": "set_page", "connectionId": "internal",
"options": {"surfaceId": prop("self"), "page": prop(page)}, "children": {}}
def static_button(tid, text, bg, fg, page, note):
"""A plain key (no watchdog state): tap goes to `page`."""
geometry = {"enabled": prop(True), "opacity": prop(100), "x": prop(0), "y": prop(0),
"width": prop(100), "height": prop(100), "rotation": prop(0)}
layers = [
{"id": "canvas", "name": "Canvas", "usage": "auto", "type": "canvas",
"decoration": prop("border"), "showStatusIcons": prop("default")},
dict({"id": "box0", "name": "Background", "usage": "auto", "type": "box"}, **geometry,
color=prop(bg), borderWidth=prop(0), borderColor=prop(0), borderPosition=prop("inside")),
dict({"id": "text0", "name": "Text", "usage": "auto", "type": "text"}, **geometry,
text=prop(text), color=prop(fg), halign=prop("center"), valign=prop("center"),
fontsize=prop("auto"), fontsizeAllowShrink=prop(True), font=prop("companion-sans"),
outlineColor=prop(4278190080)),
]
return {
"type": "button-layered",
"style": {"layers": layers},
"options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
"canModifyStyleInApis": False, "notes": note},
"feedbacks": [],
"steps": {"0": {"action_sets": {"down": [], "up": [set_page_action(tid, "go", page)]},
"options": {"runWhileHeld": []}}},
"localVariables": [],
}
def restart_key(key, cfg):
"""A Restart page key: drawn by the watchdog like a tile; hold 2 s = "force <id>"."""
names = cfg["variables"]
stale_text = "'%s\\n?'" % key["label"]
return {
"type": "button-layered",
"style": {"layers": layers(key["id"], key, names, stale_text)},
"options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
"canModifyStyleInApis": True,
"notes": "RESTART key '%s' - hold 2 s: restart it even when the status page says it is fine "
"(status_panel.py on NA Ingest runs fix '%s')." % (key["id"], key["fix"])},
"feedbacks": [],
"steps": {"0": {"action_sets": {"down": [], "up": [],
str(HOLD_MS): [command_action(key["id"], "hold", names["command"],
"force %s $(internal:time_unix)" % key["id"])]},
"options": {"runWhileHeld": [HOLD_MS]}}},
"localVariables": [],
}
def button(tid, tile, cfg): def button(tid, tile, cfg):
names = cfg["variables"] names = cfg["variables"]
kind = tile.get("kind") kind = tile.get("kind")
@@ -114,13 +166,17 @@ def button(tid, tile, cfg):
press = "ack " + tid press = "ack " + tid
fixable = "members" in tile or any((c["type"] in RESTARTABLE and c.get("conn")) or c.get("fix") fixable = "members" in tile or any((c["type"] in RESTARTABLE and c.get("conn")) or c.get("fix")
for c in tile.get("checks", [])) for c in tile.get("checks", []))
hold = ("restart " + tid) if fixable else None hold = ("restart " + tid) if fixable or tile.get("hold_always") else None
note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)." if tile.get("hold_always"):
if hold else "") note = "Press: acknowledge. Hold 2 s: %s." % cfg["fixes"][tile["hold_always"]]["label"]
else:
note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)."
if hold else "")
action_sets = {"down": [], "up": [command_action(tid, "press", names["command"], press)]} action_sets = {"down": [], "up": [command_action(tid, "press", names["command"], press)]}
run_while_held = [] run_while_held = []
if hold: if hold:
action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold)] # Stamped with Companion's clock, so a hold made while the watchdog was down is dropped later.
action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")]
run_while_held = [HOLD_MS] run_while_held = [HOLD_MS]
return { return {
"type": "button-layered", "type": "button-layered",
@@ -177,41 +233,73 @@ def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1):
return key return key
def build(cfg): def nav_controls(cfg):
controls = {} controls = {}
coloured = cfg.get("nav_keys", list(NAV_LOCATIONS)) # which nav keys follow the ALL button coloured = cfg.get("nav_keys", list(NAV_LOCATIONS)) # which nav keys follow the ALL button
for kind, (r, c) in NAV_LOCATIONS.items(): for kind, (r, c) in NAV_LOCATIONS.items():
controls.setdefault(r, {})[c] = (nav_key(kind, cfg["variables"], cfg["page"], cfg.get("nav_hold_ms", 1000), controls.setdefault(r, {})[c] = (nav_key(kind, cfg["variables"], cfg["page"], cfg.get("nav_hold_ms", 1000),
cfg.get("nav_tap_page", 1)) cfg.get("nav_tap_page", 1))
if kind in coloured else {"type": kind}) if kind in coloured else {"type": kind})
for tile in cfg["tiles"]: return controls
if "pos" not in tile:
continue
r, c = (str(n) for n in tile["pos"]) def place(controls, pos, control):
if controls.get(r, {}).get(c): r, c = (str(n) for n in pos)
raise ValueError("two buttons at row %s column %s" % (r, c)) if controls.get(r, {}).get(c):
controls.setdefault(r, {})[c] = button(tile["id"], tile, cfg) raise ValueError("two buttons at row %s column %s" % (r, c))
controls.setdefault(r, {})[c] = control
def page_export(name, number, controls):
return { return {
"version": EXPORT_VERSION, "version": EXPORT_VERSION,
"type": "page", "type": "page",
"companionBuild": COMPANION_BUILD, "companionBuild": COMPANION_BUILD,
"page": {"id": stable_id("page", cfg["page_name"]), "name": cfg["page_name"], "page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": GRID},
"controls": controls, "gridSize": GRID},
"instances": {}, "instances": {},
"connectionCollections": [], "connectionCollections": [],
"oldPageNumber": cfg["page"], "oldPageNumber": number,
"imageLibrary": [], "imageLibrary": [],
"imageLibraryCollections": [], "imageLibraryCollections": [],
} }
def build(cfg):
controls = nav_controls(cfg)
for tile in cfg["tiles"]:
if "pos" in tile:
place(controls, tile["pos"], button(tile["id"], tile, cfg))
rp = cfg.get("restart_page")
if rp:
place(controls, rp["jump_pos"], static_button("restart_jump", "RESTART\\nPAGE", 0x37474F, 0xFFFFFF,
rp["page"], "Tap: open the Restart page (page %d)." % rp["page"]))
return page_export(cfg["page_name"], cfg["page"], controls)
def build_restart(cfg):
rp = cfg["restart_page"]
controls = nav_controls(cfg)
place(controls, rp["back_pos"], static_button("restart_back", "STATUS\\nPAGE", 0x37474F, 0xFFFFFF, cfg["page"],
"Tap: back to the status page (page %d)." % cfg["page"]))
for key in rp["keys"]:
place(controls, key["pos"], restart_key(key, cfg))
return page_export(rp["page_name"], rp["page"], controls)
def write(path, export):
with gzip.open(path, "wb") as f: # Companion's own exports are gzipped JSON
f.write(json.dumps(export, indent=1).encode())
def main(): def main():
cfg = load_config(os.path.join(HERE, "config.json")) cfg = load_config(os.path.join(HERE, "config.json"))
out = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "status-page.companionconfig") out = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "status-page.companionconfig")
data = json.dumps(build(cfg), indent=1).encode() write(out, build(cfg))
with gzip.open(out, "wb") as f: # Companion's own exports are gzipped JSON
f.write(data)
print("wrote %s (%d buttons)" % (out, sum(1 for t in cfg["tiles"] if "pos" in t))) print("wrote %s (%d buttons)" % (out, sum(1 for t in cfg["tiles"] if "pos" in t)))
if cfg.get("restart_page") and len(sys.argv) <= 1:
out = os.path.join(HERE, "restart-page.companionconfig")
write(out, build_restart(cfg))
print("wrote %s (%d restart keys)" % (out, len(cfg["restart_page"]["keys"])))
if __name__ == "__main__": if __name__ == "__main__":
push_page.py
+67 -15
View File
@@ -1,20 +1,23 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Import status-page.companionconfig into Companion's page 19 without clicking through the UI. """Import the status page (19) or the Restart page (23) into Companion without clicking through the UI.
Does exactly what Import/Export -> Import -> "import into page 19" does, over the Does exactly what Import/Export -> Import -> "import into page N" does, over the
same internal API the Companion web UI uses (tRPC over a WebSocket at /trpc): same internal API the Companion web UI uses (tRPC over a WebSocket at /trpc):
upload the file in chunks, check its SHA-1, then import it as a single page. upload the file in chunks, check its SHA-1, then import it as a single page.
Safety: it refuses unless the target page is empty or already the status page Safety: it looks at what is on the target page, not just its name. It refuses unless the
(its name contains "status"), so it cannot overwrite one of your other pages. page holds nothing but page up / page number / page down keys, or is already the page this
file builds (same name). A page named "PAGE" can still hold real buttons (pages 22 and 97 do).
python3 push_page.py # na-ingest.local:8000, page and file from config.json python3 push_page.py # the status page, from config.json
python3 push_page.py --which restart # the Restart page
python3 push_page.py --host 10.6.3.222 --page 19 --file status-page.companionconfig python3 push_page.py --host 10.6.3.222 --page 19 --file status-page.companionconfig
Stdlib only (a minimal WebSocket client is included). Written against Companion 5.0.7. Stdlib only (a minimal WebSocket client is included). Written against Companion 5.0.7.
""" """
import argparse import argparse
import base64 import base64
import gzip
import hashlib import hashlib
import json import json
import os import os
@@ -116,6 +119,46 @@ class Trpc:
return msg.get("result", {}).get("data") return msg.get("result", {}).get("data")
NAV_TYPES = ("pageup", "pagedown", "pagenum")
NAV_MARKER = "STATUS nav key" # our coloured page number key (make_page.nav_key)
def fetch_page(host, port, page):
"""The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}})."""
with urllib.request.urlopen("http://%s:%d/int/export/page/%d" % (host, port, page), timeout=10) as r:
raw = r.read()
try:
raw = gzip.decompress(raw)
except OSError:
pass
return json.loads(raw)
def real_buttons(export):
"""How many buttons a page holds besides its page up / number / down keys."""
page = export.get("page") or {}
n = 0
for row in (page.get("controls") or {}).values():
for ctl in (row or {}).values():
if not ctl or ctl.get("type") in NAV_TYPES:
continue
if NAV_MARKER in ((ctl.get("options") or {}).get("notes") or ""):
continue
n += 1
return n
def safe_target(export, expected_name):
"""(ok, reason). OK only for a page with nothing but nav keys, or the page this file builds."""
name = ((export.get("page") or {}).get("name") or "")
if name == expected_name:
return True, "it is already %r" % name
n = real_buttons(export)
if n == 0:
return True, "it holds only page keys"
return False, "it is called %r and holds %d buttons" % (name, n)
def page_name(host, port, page): def page_name(host, port, page):
url = "http://%s:%d/api/variable/internal/page_number_%d_name/value" % (host, port, page) url = "http://%s:%d/api/variable/internal/page_number_%d_name/value" % (host, port, page)
with urllib.request.urlopen(url, timeout=5) as r: with urllib.request.urlopen(url, timeout=5) as r:
@@ -125,21 +168,30 @@ def page_name(host, port, page):
def main(): def main():
cfg = json.load(open(os.path.join(HERE, "config.json"))) cfg = json.load(open(os.path.join(HERE, "config.json")))
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--which", choices=("status", "restart"), default="status")
ap.add_argument("--host", default="na-ingest.local") ap.add_argument("--host", default="na-ingest.local")
ap.add_argument("--port", type=int, default=8000) ap.add_argument("--port", type=int, default=8000)
ap.add_argument("--page", type=int, default=cfg["page"]) ap.add_argument("--page", type=int)
ap.add_argument("--file", default=os.path.join(HERE, "status-page.companionconfig")) ap.add_argument("--file")
ap.add_argument("--force", action="store_true", help="overwrite the page whatever its name") ap.add_argument("--force", action="store_true", help="overwrite the page whatever is on it")
args = ap.parse_args() args = ap.parse_args()
if args.which == "restart":
name = page_name(args.host, args.port, args.page) page, expected = cfg["restart_page"]["page"], cfg["restart_page"]["page_name"]
plain = name.replace("\\n", " ").strip() default_file = "restart-page.companionconfig"
empty = plain.upper() in ("", "PAGE", "%d PAGE" % args.page) else:
if not (empty or "status" in plain.lower() or args.force): page, expected, default_file = cfg["page"], cfg["page_name"], "status-page.companionconfig"
sys.exit("page %d is called %r - not empty and not the status page; refusing (use --force)" args.page = args.page or page
% (args.page, plain)) args.file = args.file or os.path.join(HERE, default_file)
data = open(args.file, "rb").read() data = open(args.file, "rb").read()
built = json.loads(gzip.decompress(data))
if (built.get("page") or {}).get("name") != expected:
sys.exit("%s builds page %r, not %r; refusing" % (os.path.basename(args.file),
(built.get("page") or {}).get("name"), expected))
ok, why = safe_target(fetch_page(args.host, args.port, args.page), expected)
if not ok and not args.force:
sys.exit("page %d: %s - refusing (use --force only if you mean to replace it)" % (args.page, why))
plain = page_name(args.host, args.port, args.page).replace("\\n", " ").strip()
ws = WebSocket(args.host, args.port, "/trpc") ws = WebSocket(args.host, args.port, "/trpc")
try: try:
api = Trpc(ws) api = Trpc(ws)
remote-access/windows-setup.ps1
+12 -1
View File
@@ -36,6 +36,7 @@ if (-not (Get-Service sshd -ErrorAction SilentlyContinue)) {
return return
} }
Set-Service -Name sshd -StartupType Automatic Set-Service -Name sshd -StartupType Automatic
sc.exe failure sshd reset= 86400 actions= restart/5000/restart/5000/restart/30000 | Out-Null # comes back if it stops
Start-Service sshd Start-Service sshd
if (-not (Get-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -ErrorAction SilentlyContinue)) { if (-not (Get-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -ErrorAction SilentlyContinue)) {
New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True ` New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True `
@@ -80,6 +81,15 @@ $apps = @(
@{ Name = 'BibleApp'; Exe = $bible; Args = ''; AtLogon = $true } # also opens it 15 s after login @{ Name = 'BibleApp'; Exe = $bible; Args = ''; AtLogon = $true } # also opens it 15 s after login
) )
$principal = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Limited $principal = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Limited
# An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) cannot start from a
# Limited task: Windows refuses with 0x800702E4. Its task runs with highest privileges instead.
$elevated = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Highest
$layers = @{}
foreach ($k in 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers',
'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers') {
$p = Get-ItemProperty $k -ErrorAction SilentlyContinue
if ($p) { $p.PSObject.Properties | ForEach-Object { $layers[$_.Name] = "$($_.Value)" } }
}
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries ` $settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
-ExecutionTimeLimit ([TimeSpan]::Zero) -MultipleInstances IgnoreNew -ExecutionTimeLimit ([TimeSpan]::Zero) -MultipleInstances IgnoreNew
$tasks = @() $tasks = @()
@@ -88,8 +98,9 @@ foreach ($app in $apps) {
$actionArgs = @{ Execute = $app.Exe; WorkingDirectory = (Split-Path $app.Exe) } $actionArgs = @{ Execute = $app.Exe; WorkingDirectory = (Split-Path $app.Exe) }
if ($app.Args) { $actionArgs.Argument = $app.Args } if ($app.Args) { $actionArgs.Argument = $app.Args }
$action = New-ScheduledTaskAction @actionArgs $action = New-ScheduledTaskAction @actionArgs
$runAsAdmin = $layers[$app.Exe] -match 'RUNASADMIN'
$taskArgs = @{ TaskPath = '\StatusPanel\'; TaskName = ('Start ' + $app.Name); Action = $action $taskArgs = @{ TaskPath = '\StatusPanel\'; TaskName = ('Start ' + $app.Name); Action = $action
Principal = $principal; Settings = $settings; Force = $true } Principal = $(if ($runAsAdmin) { $elevated } else { $principal }); Settings = $settings; Force = $true }
if ($app.AtLogon) { if ($app.AtLogon) {
$taskArgs.Trigger = New-ScheduledTaskTrigger -AtLogOn -User "$env:USERDOMAIN\$env:USERNAME" $taskArgs.Trigger = New-ScheduledTaskTrigger -AtLogOn -User "$env:USERDOMAIN\$env:USERNAME"
$taskArgs.Trigger.Delay = 'PT15S' $taskArgs.Trigger.Delay = 'PT15S'
status_panel.py
+131 -13
View File
@@ -111,9 +111,14 @@ class Companion:
return text if status == 200 else None return text if status == 200 else None
def set_custom(self, name, value): def set_custom(self, name, value):
path = "/api/custom-variable/%s/value?value=%s" % ( path = "/api/custom-variable/%s/value" % urllib.parse.quote(name, safe="")
urllib.parse.quote(name, safe=""), urllib.parse.quote(value, safe="")) if value == "":
status, text = self._request("POST", path) # Companion answers 400 to an empty body; an empty ?value= clears it.
status, text = self._request("POST", path + "?value=")
else:
# In the body, so sp_state (several KB) never runs into a URL length limit.
status, text = self._request("POST", path, body=value.encode("utf-8"),
headers={"Content-Type": "text/plain; charset=utf-8"})
return status == 200, status return status == 200, status
def restart_connection(self, conn_id): def restart_connection(self, conn_id):
@@ -641,6 +646,8 @@ class Engine:
self.executor = execute_fix self.executor = execute_fix
self.poe_executor = execute_poe self.poe_executor = execute_poe
self.notes = {} # tile id -> hold feedback {"text", "level", "until"} self.notes = {} # tile id -> hold feedback {"text", "level", "until"}
# The Restart page: one key per thing, hold 2 s = restart it even when it looks fine.
self.restart_keys = {k["id"]: k for k in cfg.get("restart_page", {}).get("keys", [])}
self.deck_ips = {} # HyperDeck connection -> last address Companion reported self.deck_ips = {} # HyperDeck connection -> last address Companion reported
self.now = 0.0 self.now = 0.0
@@ -867,6 +874,10 @@ class Engine:
def run_fixes(self, tile_id, snap, comp, now, cooldown=60): def run_fixes(self, tile_id, snap, comp, now, cooldown=60):
plan = self.fix_plan(tile_id, snap) plan = self.fix_plan(tile_id, snap)
always = self.tiles.get(tile_id, {}).get("hold_always")
if not plan and always:
# e.g. TASCAM: its hold reconnects even when it looks fine (its control port hangs).
return self.run_force(tile_id, snap, comp, now, cooldown, fix_id=always)
if not plan: if not plan:
if self.failing(tile_id): if self.failing(tile_id):
log.info("HOLD %s: failing, but there is no automatic fix for it", tile_id) log.info("HOLD %s: failing, but there is no automatic fix for it", tile_id)
@@ -897,12 +908,12 @@ class Engine:
elif kind == "poe": elif kind == "poe":
log.info("FIX %s: power-cycle %s's PoE port (found by MAC)", tile_id, what) log.info("FIX %s: power-cycle %s's PoE port (found by MAC)", tile_id, what)
names.append("POWER CYCLE") names.append("POWER CYCLE")
started.append((self.poe_executor, (self.cfg, what, dict(snap.dante_map.get(what, {}))))) started.append((self.poe_executor, (self.cfg, what, dict(snap.dante_map.get(what, {}))), what))
else: else:
fix = self.cfg["fixes"][what] fix = self.cfg["fixes"][what]
log.info("FIX %s: %s", tile_id, fix.get("label", what)) log.info("FIX %s: %s", tile_id, fix.get("label", what))
names.append(fix.get("short", what.upper())) names.append(fix.get("short", what.upper()))
started.append((self.executor, (self.cfg, what, chk))) started.append((self.executor, (self.cfg, what, chk), fix.get("short", what.upper())))
if not names: if not names:
self.note(tile_id, "JUST TRIED\nWAIT 1 MIN" if waited else "NOTHING\nTO FIX", "idle", now, 4) self.note(tile_id, "JUST TRIED\nWAIT 1 MIN" if waited else "NOTHING\nTO FIX", "idle", now, 4)
return plan return plan
@@ -913,21 +924,63 @@ class Engine:
self.note(tile_id, "FIXING\n" + summary, "fixing", now, 180) self.note(tile_id, "FIXING\n" + summary, "fixing", now, 180)
results = [] results = []
def work(fn, args): group = "members" in self.tiles.get(tile_id, {}) or len(started) > 1
def work(fn, args, short):
ok = fn(*args) ok = fn(*args)
results.append(ok) results.append(ok)
if isinstance(ok, str) and ok != WAKING:
said.append(((short + " ") if group else "") + ok) # on a group button, say which
if len(results) == len(started): # the last one to finish reports for all if len(results) == len(started): # the last one to finish reports for all
done = time.time() done = time.time()
if WAKING in results: if WAKING in results:
self.note(tile_id, "WAKING PC\n" + summary, "fixing", done, 120) self.note(tile_id, "WAKING PC\n" + summary, "fixing", done, 120)
elif said: # the fix says what to do next
self.note(tile_id, said[0], "failed", done, 60)
elif all(r is not False for r in results): elif all(r is not False for r in results):
self.note(tile_id, "CHECKING\n" + summary, "fixing", done, 20) self.note(tile_id, "CHECKING\n" + summary, "fixing", done, 20)
else: else:
self.note(tile_id, "FIX FAILED\n" + summary, "failed", done, 20) self.note(tile_id, "FIX FAILED\n" + summary, "failed", done, 20)
for fn, args in started: said = []
threading.Thread(target=work, args=(fn, args), daemon=True).start() for fn, args, short in started:
threading.Thread(target=work, args=(fn, args, short), daemon=True).start()
return plan return plan
def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None):
"""Run one fix whatever the status says: a Restart page key, or a button's "hold_always".
Shares the 60 s guard with holds, so a hold and a restart key never double up."""
if fix_id is None:
key = self.restart_keys.get(key_id)
if not key:
log.warning("FORCE %s: no such restart key", key_id)
return None
fix_id = key["fix"]
ident = ("fix", fix_id, "")
if now - self.fix_last.get(ident, 0) < cooldown:
log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, fix_id, cooldown)
cur = self.notes.get(key_id)
if not (cur and cur["level"] == "fixing" and now < cur["until"]): # still running: keep saying so
self.note(key_id, "JUST DONE\nWAIT 1 MIN", "idle", now, 4)
return None
self.fix_last[ident] = now
fix = self.cfg["fixes"][fix_id]
log.info("FORCE %s: %s", key_id, fix.get("label", fix_id))
self.note(key_id, "RESTARTING", "fixing", now, 180)
def work():
r = self.executor(self.cfg, fix_id, {})
done = time.time()
if r == WAKING:
self.note(key_id, "WAKING PC", "fixing", done, 120)
elif isinstance(r, str):
self.note(key_id, r, "failed", done, 60)
elif r is False:
self.note(key_id, "FAILED", "failed", done, 30)
else:
self.note(key_id, "DONE", "ok", done, 20)
threading.Thread(target=work, daemon=True).start()
return fix_id
def display_level(res, acked, service_on): def display_level(res, acked, service_on):
if res.level == OFF: if res.level == OFF:
@@ -996,6 +1049,12 @@ def colored(text, level):
def render(engine, service_on): def render(engine, service_on):
out = {} out = {}
for kid, key in engine.restart_keys.items():
note = engine.notes.get(kid)
if note and engine.now < note["until"]:
out[kid] = colored(key["label"] + "\n" + note["text"], note["level"])
else:
out[kid] = colored(key["label"] + "\nHOLD 2s", "idle")
for tile in engine.cfg["tiles"]: for tile in engine.cfg["tiles"]:
tid, kind = tile["id"], tile.get("kind") tid, kind = tile["id"], tile.get("kind")
note = engine.notes.get(tid) note = engine.notes.get(tid)
@@ -1449,13 +1508,16 @@ def execute_fix(cfg, fix_id, chk):
return WAKING return WAKING
log.info("FIX %s -> %s%s", label, "ok" if r.returncode == 0 else "exit %d" % r.returncode, log.info("FIX %s -> %s%s", label, "ok" if r.returncode == 0 else "exit %d" % r.returncode,
(": " + detail) if detail else "") (": " + detail) if detail else "")
if r.returncode == 0 and fix.get("reconnect") and chk.get("conn"): conn = chk.get("conn") or fix.get("conn")
if r.returncode == 0 and fix.get("reconnect") and conn:
# The app is running now: make Companion reconnect to it instead of waiting for its retry. # The app is running now: make Companion reconnect to it instead of waiting for its retry.
comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000)) comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000))
ids = {c["label"]: c["id"] for c in comp.connections()} ids = {c["label"]: c["id"] for c in comp.connections()}
ok = chk["conn"] in ids and comp.restart_connection(ids[chk["conn"]]) ok = conn in ids and comp.restart_connection(ids[conn])
log.info("FIX %s -> Companion connection %s restarted %s", label, chk["conn"], "ok" if ok else "FAILED") log.info("FIX %s -> Companion connection %s restarted %s", label, conn, "ok" if ok else "FAILED")
return r.returncode == 0 return r.returncode == 0
if kind == "reconnect":
return reconnect_and_verify(cfg, fix, label)
if kind == "midi_connect": if kind == "midi_connect":
# Like Connect in Audio MIDI Setup: add the peer back to the session it dropped out of. # Like Connect in Audio MIDI Setup: add the peer back to the session it dropped out of.
args = (chk["session"], chk["peer"], chk["address"]) args = (chk["session"], chk["peer"], chk["address"])
@@ -1505,6 +1567,41 @@ def execute_fix(cfg, fix_id, chk):
return False return False
def reconnect_and_verify(cfg, fix, label, sleep=time.sleep):
"""Restart one Companion connection, then look again after "verify_seconds". Returns True if it
came back; otherwise what to tell the operator: "hung_note" when the device still answers ping
(its control side has hung - only a power-cycle helps), else that it is not on the network."""
comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000))
conn = {c["label"]: c for c in comp.connections()}.get(fix["conn"])
if conn is None:
log.info("FIX %s -> no Companion connection %s", label, fix["conn"])
return False
if not conn.get("enabled"):
# Switched off in Companion: turn it on (never call that a hung device).
ok = comp.enable_connection(conn["id"])
log.info("FIX %s -> %s was disabled in Companion: turned on %s", label, fix["conn"], "ok" if ok else "FAILED")
if not ok:
return "DISABLED IN\nCOMPANION"
elif not comp.restart_connection(conn["id"]):
log.info("FIX %s -> Companion refused to restart %s", label, fix["conn"])
return False
sleep(fix.get("verify_seconds", 20))
after = {c["label"]: c for c in comp.connections()}.get(fix["conn"]) or {}
status = (after.get("status") or {}).get("category")
if status == "good":
log.info("FIX %s -> %s reconnected", label, fix["conn"])
return True
if status not in ("warning", "error"): # still starting up, or unknown: no verdict yet
log.info("FIX %s -> %s is %s after %ss", label, fix["conn"], status, fix.get("verify_seconds", 20))
return "STILL\nCONNECTING"
host = companion_conn_hosts(companion_db_path(cfg)).get(fix["conn"])
if host and ping_once(host):
log.info("FIX %s -> %s still %s but %s answers ping: control port hung", label, fix["conn"], status, host)
return fix.get("hung_note", "STILL DOWN")
log.info("FIX %s -> %s still %s and %s does not answer ping", label, fix["conn"], status, host or "its address")
return "NO NETWORK\nCHECK POWER"
def start_artnet_listener(snap, stop, port=6454): def start_artnet_listener(snap, stop, port=6454):
"""Note when each source last sent Art-Net. Shares the port (SO_REUSEPORT) so it never """Note when each source last sent Art-Net. Shares the port (SO_REUSEPORT) so it never
keeps another program from listening too. Passive: nothing is ever sent.""" keeps another program from listening too. Passive: nothing is ever sent."""
@@ -1580,6 +1677,13 @@ def load_config(path):
raise ValueError("fix %s: unknown host" % f) raise ValueError("fix %s: unknown host" % f)
if c["type"] == "remote" and c["host"] not in cfg.get("hosts", {}): if c["type"] == "remote" and c["host"] not in cfg.get("hosts", {}):
raise ValueError("tile %s: unknown host %s" % (t["id"], c["host"])) raise ValueError("tile %s: unknown host %s" % (t["id"], c["host"]))
if t.get("hold_always") and t["hold_always"] not in cfg.get("fixes", {}):
raise ValueError("tile %s: unknown hold_always fix %s" % (t["id"], t["hold_always"]))
for k in cfg.get("restart_page", {}).get("keys", []):
if k["id"] in ids:
raise ValueError("restart key %s has a tile's id" % k["id"])
if k["fix"] not in cfg.get("fixes", {}):
raise ValueError("restart key %s: unknown fix %s" % (k["id"], k["fix"]))
return cfg return cfg
@@ -1641,6 +1745,9 @@ def post_webhook(url, text):
log.warning("webhook failed: %s", e) log.warning("webhook failed: %s", e)
HOLD_MAX_AGE = 15 # seconds: a hold older than this (queued while the watchdog was down) is dropped
def handle_commands(text, engine, comp, snap, now): def handle_commands(text, engine, comp, snap, now):
for part in text.split(";"): for part in text.split(";"):
words = part.strip().split() words = part.strip().split()
@@ -1655,8 +1762,19 @@ def handle_commands(text, engine, comp, snap, now):
target = {"on": True, "off": False}.get(arg, not engine.service.on(now)) target = {"on": True, "off": False}.get(arg, not engine.service.on(now))
engine.service.set(target, now) engine.service.set(target, now)
log.info("SERVICE %s (button)", "ON" if target else "OFF") log.info("SERVICE %s (button)", "ON" if target else "OFF")
elif verb == "restart": # sent by a 2 second hold elif verb in ("restart", "force"):
engine.run_fixes(arg, snap, comp, now) # Holds carry Companion's clock ("force rs_x 1791496225"). One pressed while the watchdog
# was down must not fire minutes later in a burst with the others.
stamp = words[2] if len(words) > 2 else None
if stamp is not None and (not stamp.isdigit() or now - int(stamp) > HOLD_MAX_AGE):
log.info("%s %s: pressed %s, too long ago - ignored", verb.upper(), arg,
"at %s" % stamp if stamp.isdigit() else "at an unknown time")
elif stamp is None and verb == "force":
log.info("FORCE %s: no time on it - ignored", arg)
elif verb == "restart": # a 2 second hold on a status button
engine.run_fixes(arg, snap, comp, now)
else: # a Restart page key, held 2 seconds
engine.run_force(arg, snap, comp, now)
else: else:
log.warning("unknown command %r", part) log.warning("unknown command %r", part)
tests/test_status_panel.py
+152 -6
View File
@@ -66,7 +66,8 @@ def healthy_snapshot(now):
snap.mdns, snap.mdns_time = {"NA Tracks", "Lighting PC", "NA Ingest"}, now snap.mdns, snap.mdns_time = {"NA Tracks", "Lighting PC", "NA Ingest"}, now
snap.ping = {"10.1.3.227": 0, "10.6.7.13": 0, "10.1.0.1": 0} snap.ping = {"10.1.3.227": 0, "10.6.7.13": 0, "10.1.0.1": 0}
snap.link = True snap.link = True
snap.launchd = {"org.waterschurch.resi-propresenter": 884, "org.waterschurch.lobby-captions": 900} snap.launchd = {"org.waterschurch.resi-propresenter": 884, "org.waterschurch.lobby-captions": 900,
"org.waterschurch.lobby-captions-display": 901}
snap.launchd_time = now snap.launchd_time = now
snap.procs = {"/Applications/ProdDash.app/Contents/MacOS/ProdDash": True} snap.procs = {"/Applications/ProdDash.app/Contents/MacOS/ProdDash": True}
snap.http = {"https://central.resi.io/": True, "http://10.6.7.13:8080/": True, "http://10.1.14.225:1025/": True} snap.http = {"https://central.resi.io/": True, "http://10.6.7.13:8080/": True, "http://10.1.14.225:1025/": True}
@@ -98,8 +99,10 @@ class TestChecks(unittest.TestCase):
def test_all_green_when_healthy(self): def test_all_green_when_healthy(self):
tiles = sp.Engine(CONFIG).step(self.snap, self.now) tiles = sp.Engine(CONFIG).step(self.snap, self.now)
bad = {k: v["x"] for k, v in tiles.items() if level(v) not in ("ok", "service", "off")} bad = {k: v["x"] for k, v in tiles.items() if level(v) not in ("ok", "service", "off")
and not k.startswith("rs_")} # Restart page keys sit grey
self.assertEqual(bad, {}) self.assertEqual(bad, {})
self.assertEqual((tiles["rs_dvs_tracks"]["x"], level(tiles["rs_dvs_tracks"])), ("DVS TRACKS\nHOLD 2s", "idle"))
self.assertEqual(tiles["all"]["x"], "ALL OK") self.assertEqual(tiles["all"]["x"], "ALL OK")
self.assertEqual(tiles["hd1"]["x"], "HD1\n28h") self.assertEqual(tiles["hd1"]["x"], "HD1\n28h")
self.assertEqual(tiles["dante"]["x"], "DANTE\n19/19") self.assertEqual(tiles["dante"]["x"], "DANTE\n19/19")
@@ -608,6 +611,116 @@ class TestFixes(unittest.TestCase):
self.assertEqual(self.plan("rf"), []) # nothing left to "fix" self.assertEqual(self.plan("rf"), []) # nothing left to "fix"
self.assertFalse(self.engine.failing("rf")) self.assertFalse(self.engine.failing("rf"))
def test_restart_page_keys_run_even_when_all_is_fine(self):
ran = []
self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True
run_steps(self.engine, self.snap, SUNDAY, 4) # everything green
self.assertEqual(self.engine.run_force("rs_dvs_tracks", self.snap, None, SUNDAY + 5), "dvs_tracks")
time.sleep(0.2)
self.assertEqual(ran, ["dvs_tracks"])
self.assertEqual(self.engine.notes["rs_dvs_tracks"]["text"], "DONE")
self.assertIsNone(self.engine.run_force("rs_dvs_tracks", self.snap, None, SUNDAY + 20)) # 60 s guard
self.assertEqual(self.engine.notes["rs_dvs_tracks"]["text"], "JUST DONE\nWAIT 1 MIN")
self.assertIsNone(self.engine.run_force("rs_nope", self.snap, None, SUNDAY + 90))
# the hold on the status page and the Restart key share the guard
self.engine.run_force("rs_prodcom", self.snap, None, SUNDAY + 100)
self.engine.notes.clear()
self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = False
run_steps(self.engine, self.snap, SUNDAY + 100, 4)
self.engine.run_fixes("ingest", self.snap, None, SUNDAY + 106)
self.assertEqual(self.engine.notes["ingest"]["text"], "JUST TRIED\nWAIT 1 MIN")
# what each key restarts
keys = {k["id"]: k["fix"] for k in CONFIG["restart_page"]["keys"]}
self.assertEqual(keys["rs_ableton"], "ableton_restart")
self.assertIn("pkill -TERM -x 'Live'", CONFIG["fixes"]["ableton_restart"]["run"])
self.assertIn("kickstart -k system/com.audinate.dante.DanteVirtualSoundcard",
open(os.path.join(os.path.dirname(os.path.abspath(sp.__file__)), "remote-access", "mac-dvs-helper.sh")).read())
self.assertIn("open -g /Applications/ProdCom.app", CONFIG["fixes"]["dvs_ingest"]["run"]) # ProdCom after DVS
def test_tascam_hold_always_reconnects(self):
ran = []
self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or "HUNG - POWER\nCYCLE IT"
run_steps(self.engine, self.snap, SUNDAY, 4) # TASCAM looks fine
self.engine.run_fixes("tascam", self.snap, None, SUNDAY + 5)
time.sleep(0.2)
self.assertEqual(ran, ["tascam_reconnect"])
tiles = self.engine.step(self.snap, time.time())
self.assertEqual((tiles["tascam"]["x"], level(tiles["tascam"])), ("TASCAM\nHUNG - POWER\nCYCLE IT", "failed"))
def test_reconnect_and_verify(self):
from unittest import mock
fix = CONFIG["fixes"]["tascam_reconnect"]
conn = lambda cat, enabled=True: [{"label": "tascam", "id": "t1", "enabled": enabled, "status": {"category": cat}}]
def run(before, after, pings=True, restart_ok=True, enable_ok=True):
comp = mock.Mock()
comp.connections.side_effect = [before, after]
comp.restart_connection.return_value = restart_ok
comp.enable_connection.return_value = enable_ok
with mock.patch.object(sp, "Companion", return_value=comp), \
mock.patch.object(sp, "companion_conn_hosts", return_value={"tascam": "10.1.11.74"}), \
mock.patch.object(sp, "ping_once", return_value=pings):
return sp.reconnect_and_verify(CONFIG, fix, "x", sleep=lambda s: None), comp
r, comp = run(conn("error"), conn("good"))
self.assertTrue(r)
comp.restart_connection.assert_called_once_with("t1")
self.assertEqual(run(conn("error"), conn("error"))[0], "HUNG - POWER\nCYCLE IT") # pings: hung
self.assertEqual(run(conn("error"), conn("error"), pings=False)[0], "NO NETWORK\nCHECK POWER")
self.assertEqual(run(conn("good"), conn(None))[0], "STILL\nCONNECTING") # no verdict yet
self.assertFalse(run(conn("error"), conn("error"), restart_ok=False)[0])
r, comp = run(conn(None, enabled=False), conn("good")) # switched off
self.assertTrue(r)
comp.enable_connection.assert_called_once_with("t1")
comp.restart_connection.assert_not_called()
self.assertEqual(run(conn(None, enabled=False), conn(None), enable_ok=False)[0], "DISABLED IN\nCOMPANION")
self.assertFalse(run([], [])[0])
def test_stale_or_unstamped_holds_are_dropped(self):
ran = []
self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True
now = SUNDAY + 100
sp.handle_commands(";force rs_bridge %d;force rs_prodcom %d;force rs_proddash" % (now - 300, now - 2),
self.engine, None, self.snap, now)
time.sleep(0.2)
self.assertEqual(ran, ["prodcom"]) # only the fresh one: not the old or unstamped
def test_a_second_press_keeps_saying_restarting(self):
gate = threading.Event()
self.engine.executor = lambda cfg, fix_id, chk: gate.wait(2) or True
self.engine.run_force("rs_bridge", self.snap, None, SUNDAY)
self.engine.run_force("rs_bridge", self.snap, None, SUNDAY + 3)
self.assertEqual(self.engine.notes["rs_bridge"]["text"], "RESTARTING")
gate.set()
def test_group_button_names_the_device_that_needs_a_power_cycle(self):
self.engine.executor = lambda cfg, fix_id, chk: "HUNG - POWER\nCYCLE IT"
self.snap.conns["tascam"]["category"] = "error"
_, t = run_steps(self.engine, self.snap, SUNDAY, 4)
self.engine.run_fixes("rec", self.snap, None, t)
time.sleep(0.2)
self.assertEqual(self.engine.notes["rec"]["text"], "TASCAM HUNG - POWER\nCYCLE IT")
def test_windows_starts_check_the_app_really_started(self):
for fix_id, exe in (("obs", "obs64.exe"), ("arena", "Arena.exe"), ("vista", "Vista.exe"), ("bibleapp", "Bible App.exe")):
run = CONFIG["fixes"][fix_id]["run"]
self.assertTrue(run.rstrip(")").endswith('find /I "%s" >nul' % exe), fix_id)
self.assertIn("ping -n 9 127.0.0.1", run)
def test_ingest_resi_and_tascam_checks(self):
self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = False
self.assertEqual(self.plan("ingest"), [("fix", "prodcom")])
self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = True
self.snap.launchd.pop("org.waterschurch.lobby-captions-display")
self.assertEqual(self.plan("ingest"), [("fix", "caption_screen")]) # the screen only
self.snap.launchd["org.waterschurch.lobby-captions-display"] = 1
self.snap.ping["10.1.10.202"] = 3
tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, SUNDAY, 4)
self.assertEqual(tiles["resi"]["x"], "RESI\nENCODER NO PING")
self.snap.conn_hosts = {"tascam": "10.1.11.74"}
self.snap.ping["10.1.11.74"] = 3
tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, SUNDAY, 4)
self.assertEqual(tiles["tascam"]["x"], "TASCAM\nTASCAM NO PING")
def test_midi_reconnect_where_the_peer_was_last_seen(self): def test_midi_reconnect_where_the_peer_was_last_seen(self):
others = [{"name": "SuperRack-LiveBox", "address": "10.1.13.140:5004", "state": "running"}, others = [{"name": "SuperRack-LiveBox", "address": "10.1.13.140:5004", "state": "running"},
{"name": "Studio B Mac Studio", "address": "\tStudio B Mac Studio\tlocal.", "state": "running"}] {"name": "Studio B Mac Studio", "address": "\tStudio B Mac Studio\tlocal.", "state": "running"}]
@@ -853,7 +966,7 @@ class TestFixes(unittest.TestCase):
def test_every_fix_is_valid(self): def test_every_fix_is_valid(self):
for f in CONFIG["fixes"].values(): for f in CONFIG["fixes"].values():
self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect")) self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect", "reconnect"))
class FakeDeck: class FakeDeck:
@@ -1000,7 +1113,11 @@ class FakeCompanion(BaseHTTPRequestHandler):
if m: if m:
if m.group(1) not in s["custom"]: if m.group(1) not in s["custom"]:
return self._send(404, "Not found") return self._send(404, "Not found")
s["custom"][m.group(1)] = urllib.parse.parse_qs(u.query, keep_blank_values=True)["value"][0] body = self.rfile.read(int(self.headers.get("Content-Length") or 0)).decode()
query = urllib.parse.parse_qs(u.query, keep_blank_values=True)
if not body and "value" not in query:
return self._send(400, "No value") # like Companion 5.0.7 with an empty body
s["custom"][m.group(1)] = body if body else query["value"][0]
return self._send(200, "ok") return self._send(200, "ok")
m = re.match(r"^/api/connections/([^/]+)/restart$", u.path) m = re.match(r"^/api/connections/([^/]+)/restart$", u.path)
if m: if m:
@@ -1134,10 +1251,38 @@ class TestPage(unittest.TestCase):
self.assertIn("'$.all.b'", box["color"]["value"]) # follows the ALL button self.assertIn("'$.all.b'", box["color"]["value"]) # follows the ALL button
positioned = [t for t in CONFIG["tiles"] if "pos" in t] positioned = [t for t in CONFIG["tiles"] if "pos" in t]
n = sum(1 for r in self.controls.values() for c in r.values() if c["type"] == "button-layered") n = sum(1 for r in self.controls.values() for c in r.values() if c["type"] == "button-layered")
self.assertEqual(n, len(positioned) + 1) # + the page number key self.assertEqual(n, len(positioned) + 2) # + the page number and RESTART keys
jump = self.controls["2"]["6"]["steps"]["0"]["action_sets"]["up"][0]
self.assertEqual((jump["definitionId"], jump["options"]["page"]["value"]), ("set_page", 23))
# The master is where the smallest Stream Deck can see it. # The master is where the smallest Stream Deck can see it.
self.assertEqual(CONFIG["tiles"][0]["pos"], [0, 1]) self.assertEqual(CONFIG["tiles"][0]["pos"], [0, 1])
def test_restart_page(self):
export = make_page.build_restart(CONFIG)
self.assertEqual((export["oldPageNumber"], export["page"]["name"]), (23, "23\\nRestart"))
c = export["page"]["controls"]
holds = {}
for r in c.values():
for ctl in r.values():
sets = ctl.get("steps", {}).get("0", {}).get("action_sets", {})
if "2000" in sets:
holds[sets["2000"][0]["options"]["value"]["value"]] = sets["up"]
self.assertEqual(len(holds), len(CONFIG["restart_page"]["keys"]))
self.assertIn("$(custom:sp_cmd);force rs_tascam $(internal:time_unix)", holds)
self.assertTrue(all(up == [] for up in holds.values())) # a tap does nothing
back = c["3"]["0"]["steps"]["0"]["action_sets"]["up"][0]
self.assertEqual((back["definitionId"], back["options"]["page"]["value"]), ("set_page", 19))
def test_push_page_looks_at_what_is_on_the_page(self):
import push_page
nav = {"0": {"0": {"type": "pageup"}}, "1": {"0": {"type": "button-layered",
"options": {"notes": "STATUS nav key (pagenum): x"}}}}
self.assertTrue(push_page.safe_target({"page": {"name": "PAGE", "controls": nav}}, "23\\nRestart")[0])
busy = {"page": {"name": "PAGE", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}}
self.assertFalse(push_page.safe_target(busy, "23\\nRestart")[0]) # pages 22 and 97
busy["page"]["name"] = "23\\nRestart"
self.assertTrue(push_page.safe_target(busy, "23\\nRestart")[0]) # our own page again
def test_each_button_reads_its_own_tile(self): def test_each_button_reads_its_own_tile(self):
for t in CONFIG["tiles"]: for t in CONFIG["tiles"]:
if "pos" not in t: if "pos" not in t:
@@ -1155,7 +1300,8 @@ class TestPage(unittest.TestCase):
hd1 = self.controls["1"]["1"]["steps"]["0"] hd1 = self.controls["1"]["1"]["steps"]["0"]
up = hd1["action_sets"]["up"][0]["options"]["value"] up = hd1["action_sets"]["up"][0]["options"]["value"]
self.assertEqual(up, {"value": "$(custom:sp_cmd);ack hd1", "isExpression": False}) self.assertEqual(up, {"value": "$(custom:sp_cmd);ack hd1", "isExpression": False})
self.assertEqual(hd1["action_sets"]["2000"][0]["options"]["value"]["value"], "$(custom:sp_cmd);restart hd1") self.assertEqual(hd1["action_sets"]["2000"][0]["options"]["value"]["value"],
"$(custom:sp_cmd);restart hd1 $(internal:time_unix)")
self.assertEqual(hd1["options"]["runWhileHeld"], [2000]) self.assertEqual(hd1["options"]["runWhileHeld"], [2000])
master = self.controls["0"]["1"]["steps"]["0"] master = self.controls["0"]["1"]["steps"]["0"]
self.assertIn("ackall", master["action_sets"]["up"][0]["options"]["value"]["value"]) self.assertIn("ackall", master["action_sets"]["up"][0]["options"]["value"]["value"])