diff --git a/README.md b/README.md index da1a74e..5e7aef0 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ There are two parts: |--------|-------|-------|-------|-------|-------|-------|-------|-------| | row 0 | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM | | row 1 | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM | -| row 2 | DOWN | X32 | DANTE | MIDI | APPS | VISTA | | | +| row 2 | DOWN | X32 | DANTE | MIDI | APPS | VISTA | RESTART PAGE | | | row 3 | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA | Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more: @@ -55,7 +55,8 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl | DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) | | DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches | | DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) | - | INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest | + | INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH | restarts that one on NA Ingest | + | TASCAM (always, even when it looks fine) | reconnects Companion to the recorder and looks again after 20 s. **HUNG - POWER CYCLE IT**: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). **NO NETWORK / CHECK POWER**: it does not answer at all. | | any button whose Companion connection is **disabled** | turns the connection back on (nothing else) | | HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. | | anything else with a Companion connection | restarts that Companion connection | @@ -65,6 +66,22 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`. - **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight. +## Restart page (page 23) + +Tap **RESTART PAGE** on the status page (row 2). Each key restarts one thing **even when the status page says it is fine**; hold it 2 s. It shows RESTARTING, then DONE (or FAILED / what to do). The same thing will not run twice within a minute. **STATUS PAGE** (bottom left) goes back. + +| Key | Restarts | +|---|---| +| DVS PROP / TRACKS / TRANSL / INGEST | Dante Virtual Soundcard's engine on that Mac (audio to and from it drops 5–10 s). DVS INGEST also restarts ProdCom afterwards, which otherwise keeps a dead input. | +| PROPRES | ProPresenter (quits and reopens the newest version) | +| ABLETON | Ableton on the Tracks Mac (quits it: unsaved set changes are lost, and Live may ask to recover - click it on the Tracks Mac) | +| TRANSLATE | Church Translate on NA Translation | +| PRODCOM | ProdCom on NA Ingest (lobby captions blank for 5–15 s) | +| PRODDASH / CAPTIONS / BRIDGE | that service on NA Ingest | +| TASCAM | reconnects Companion to the recorder (see TASCAM above) | + +Add or move keys in `"restart_page"` in `config.json`, then `python3 make_page.py` and `python3 push_page.py --which restart`. `push_page.py` refuses any page that holds real buttons unless it already is that page. + ## Page keys on every page The **page number key** on every page follows the ALL button. It's green when everything is fine, flashes red when something is wrong, and turns purple if the watchdog stops. UP and DOWN stay normal. Which keys follow ALL is `"nav_keys"` in `config.json` (`["pagenum"]`; add `"pageup"` / `"pagedown"` to colour those too). The coloured key is Companion's own key converted to a regular button (`nav_templates.json`). @@ -81,16 +98,16 @@ The **page number key** on every page follows the ALL button. It's green when ev |---|---| | HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't | | | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. | -| TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded. | +| TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection. | | X32, ATEM, ARENA, OBS | Companion connection | | LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. | | DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". | | MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. | | APPS | ProPresenter API (offline or error only: its harmless yellow warning, every 15–20 min, is ignored with `"ignore_warning": true`); Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. | | VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. | -| RESI | decoder (RossTalk connection and ping); resi-studio-cues; Resi cloud reachable | +| RESI | decoder (RossTalk connection and ping, 10.1.10.203); **encoder** (ping and its web page, 10.1.10.202, LAOEncoder); resi-studio-cues; Resi cloud reachable. Only shows they are on the network, not that the encoder is streaming. | | NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) | -| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk | +| INGEST | the bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, **ProdCom** (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk | | RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; `"host"` in `config.json` is only the fallback. | A problem has to last 3 seconds before it shows, so one bad reading doesn't trigger an alarm. Recovery shows immediately. @@ -163,6 +180,10 @@ The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the To go back, pick 1.3.0 on the connection again. +## Windows PCs + +`remote-access/windows-setup.ps1` sets each PC up once (SSH for NA Ingest, start tasks, Wake-on-LAN). An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) gets a start task that runs with highest privileges; otherwise Windows refuses to start it (error 0x800702E4). Windows restarts the SSH service and rtpMIDI by itself if they stop. + ## Files | File | Purpose | diff --git a/config.json b/config.json index f72ac1f..51fec2e 100644 --- a/config.json +++ b/config.json @@ -44,29 +44,57 @@ "run": "pgrep -xq Live || open -a 'Ableton Live 11 Suite'" }, "translate": { "host": "translation", "short": "TRANSLATE", "label": "start Church Translate if it is not running", "run": "pgrep -xq 'Church Translate' || open -a 'Church Translate'" }, - "dvs_prop": { "host": "prop", "short": "DVS", "label": "start Dante Virtual Soundcard on the ProPresenter Mac", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, - "dvs_tracks": { "host": "tracks", "short": "DVS", "label": "start Dante Virtual Soundcard on the Tracks Mac", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, - "dvs_translation": { "host": "translation", "short": "DVS", "label": "start Dante Virtual Soundcard on NA Translation", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, - "dvs_ingest": { "kind": "local", "short": "DVS", "label": "start Dante Virtual Soundcard on NA Ingest", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -a 'Dante Virtual Soundcard'" }, + "dvs_prop": { "host": "prop", "short": "DVS", "label": "restart Dante Virtual Soundcard on the ProPresenter Mac (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" }, + "dvs_tracks": { "host": "tracks", "short": "DVS", "label": "restart Dante Virtual Soundcard on the Tracks Mac (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" }, + "dvs_translation": { "host": "translation", "short": "DVS", "label": "restart Dante Virtual Soundcard on NA Translation (its engine restarts; audio to and from that Mac drops for 5-10 s)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard'" }, + "dvs_ingest": { "kind": "local", "short": "DVS", "label": "restart Dante Virtual Soundcard on NA Ingest, then ProdCom (it keeps a dead DVS input otherwise)", "run": "sudo -n /usr/local/sbin/status-panel-dvs-start && open -g -a 'Dante Virtual Soundcard' || exit 1; sleep 5; pkill -TERM -x ProdCom; i=0; while pgrep -xq ProdCom && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq ProdCom && pkill -KILL -x ProdCom; sleep 1; open -g /Applications/ProdCom.app" }, "bridge": { "kind": "local", "short": "BRIDGE", "label": "restart the Resi bridge", "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.resi-propresenter || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.resi-propresenter.plist" }, - "captions": { "kind": "local", "short": "CAPTIONS", "label": "restart lobby captions", - "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions.plist" }, + "captions": { "kind": "local", "short": "CAPTIONS", "label": "restart lobby captions (server and display)", + "run": "s=0; { launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions.plist; } || s=1; { launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions-display || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions-display.plist; } || s=1; exit $s" }, "bibleapp": { "host": "bibletv", "short": "BIBLE APP", "label": "restart the Bible app on the Sanctuary TV PC (Windows: cmd.exe)", - "run": "taskkill /F /T /IM \"Bible App.exe\" >nul 2>&1 & ping -n 3 127.0.0.1 >nul & schtasks /Run /TN \"\\StatusPanel\\Start BibleApp\"" }, + "run": "taskkill /F /T /IM \"Bible App.exe\" >nul 2>&1 & ping -n 3 127.0.0.1 >nul & schtasks /Run /TN \"\\StatusPanel\\Start BibleApp\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Bible App.exe\" | find /I \"Bible App.exe\" >nul" }, "obs": { "host": "livestream", "short": "OBS", "label": "start OBS on the LiveStream PC if it is not running (wakes the PC if it is off), then reconnect Companion", "reconnect": true, - "run": "tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start OBS\"" }, + "run": "tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start OBS\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq obs64.exe\" | find /I \"obs64.exe\" >nul)" }, "arena": { "host": "lighting", "short": "ARENA", "label": "start Resolume Arena on the Lighting PC if it is not running, then reconnect Companion", "reconnect": true, - "run": "tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start Arena\"" }, + "run": "tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start Arena\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Arena.exe\" | find /I \"Arena.exe\" >nul)" }, "vista": { "host": "lighting", "short": "VISTA", "label": "start Vista on the Lighting PC if it is not running (a running Vista is never closed)", - "run": "tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start Vista\"" }, + "run": "tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start Vista\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul)" }, "rtpmidi_start": { "host": "lighting", "short": "RTPMIDI", "label": "start rtpMIDI on the Lighting PC if it is not running (service, then tray app). A running rtpMIDI is never restarted: that drops every session.", "run": "(sc query rtpMIDIService | find \"RUNNING\" >nul || net start rtpMIDIService) && (tasklist /NH /FI \"IMAGENAME eq rtpMIDI.exe\" | find /I \"rtpMIDI.exe\" >nul || schtasks /Run /TN \"\\StatusPanel\\Start rtpMIDI\")" }, "midi_reconnect": { "kind": "midi_connect", "short": "RECONNECT", "label": "reconnect the peer to the network MIDI session it dropped out of (like Connect in Audio MIDI Setup); the address is where it was last seen, else the check's \"address\"" }, - "proddash": { "kind": "local", "short": "PRODDASH", "label": "start ProdDash", "run": "open /Applications/ProdDash.app" }, + "caption_screen": { "kind": "local", "short": "CAPTION SCREEN", "label": "restart the lobby caption screen (display app only)", "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.lobby-captions-display || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.lobby-captions-display.plist" }, + "proddash": { "kind": "local", "short": "PRODDASH", "label": "restart ProdDash (its launcher stops its server cleanly)", + "run": "pkill -TERM -x ProdDash; i=0; while pgrep -xq ProdDash && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; open -g /Applications/ProdDash.app" }, + "prodcom": { "kind": "local", "short": "PRODCOM", "label": "restart ProdCom (transcription; lobby captions blank for 5-15 s)", + "run": "pkill -TERM -x ProdCom; i=0; while pgrep -xq ProdCom && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq ProdCom && pkill -KILL -x ProdCom; sleep 1; open -g /Applications/ProdCom.app" }, + "ableton_restart": { "host": "tracks", "short": "ABLETON", "label": "restart Ableton on the Tracks Mac (quits it; unsaved set changes are lost, and Live may ask to recover on the Tracks Mac)", "conn": "ableton", "reconnect": true, "timeout": 60, + "run": "pkill -TERM -x 'Live'; i=0; while pgrep -xq 'Live' && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'Live' && pkill -KILL -x 'Live'; sleep 1; open -a 'Ableton Live 11 Suite'" }, + "translate_restart": { "host": "translation", "short": "TRANSLATE", "label": "restart Church Translate on NA Translation (and its translation server)", "timeout": 60, + "run": "pg=$(ps -o pgid= -p \"$(pgrep -x 'Church Translate' | head -1)\" 2>/dev/null | tr -d ' '); pkill -TERM -x 'Church Translate'; i=0; while pgrep -xq 'Church Translate' && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'Church Translate' && pkill -KILL -x 'Church Translate'; [ -n \"$pg\" ] && [ \"$pg\" != \"$(ps -o pgid= -p $$ | tr -d ' ')\" ] && kill -KILL -- -\"$pg\" 2>/dev/null; sleep 1; open -a 'Church Translate'" }, + "tascam_reconnect": { "kind": "reconnect", "conn": "tascam", "short": "TASCAM", "verify_seconds": 20, "hung_note": "HUNG - POWER\nCYCLE IT", + "label": "reconnect Companion to the Tascam; if it still does not answer but pings, its control port has hung: power-cycle it at the rack" }, "hyperdeck": { "kind": "hyperdeck", "short": "HYPERDECK", "label": "reconnect Companion to the HyperDeck, or reboot the deck only if it is hung" } }, + "restart_page": { + "page": 23, "page_name": "23\\nRestart", "jump_pos": [2, 6], "back_pos": [3, 0], + "keys": [ + { "id": "rs_dvs_prop", "label": "DVS PROP", "pos": [0, 1], "fix": "dvs_prop" }, + { "id": "rs_dvs_tracks", "label": "DVS TRACKS", "pos": [0, 2], "fix": "dvs_tracks" }, + { "id": "rs_dvs_transl", "label": "DVS TRANSL", "pos": [0, 3], "fix": "dvs_translation" }, + { "id": "rs_dvs_ingest", "label": "DVS INGEST", "pos": [0, 4], "fix": "dvs_ingest" }, + { "id": "rs_propres", "label": "PROPRES", "pos": [1, 1], "fix": "propresenter" }, + { "id": "rs_ableton", "label": "ABLETON", "pos": [1, 2], "fix": "ableton_restart" }, + { "id": "rs_translate", "label": "TRANSLATE", "pos": [1, 3], "fix": "translate_restart" }, + { "id": "rs_prodcom", "label": "PRODCOM", "pos": [2, 1], "fix": "prodcom" }, + { "id": "rs_proddash", "label": "PRODDASH", "pos": [2, 2], "fix": "proddash" }, + { "id": "rs_captions", "label": "CAPTIONS", "pos": [2, 3], "fix": "captions" }, + { "id": "rs_bridge", "label": "BRIDGE", "pos": [2, 4], "fix": "bridge" }, + { "id": "rs_tascam", "label": "TASCAM", "pos": [3, 1], "fix": "tascam_reconnect" } + ] + }, + "tiles": [ { "id": "all", "label": "ALL", "pos": [0, 1], "kind": "master" }, { "id": "rec", "label": "REC", "pos": [0, 2], "members": ["hd1", "hd2", "hd3", "hd4", "hd5", "hd6", "tascam"] }, @@ -82,7 +110,10 @@ { "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] }, { "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] }, { "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] }, - { "id": "tascam", "label": "TASCAM", "pos": [1, 7], "checks": [{ "type": "tascam", "conn": "tascam" }] }, + { "id": "tascam", "label": "TASCAM", "pos": [1, 7], "hold_always": "tascam_reconnect", "checks": [ + { "type": "tascam", "conn": "tascam", "fix": "tascam_reconnect" }, + { "type": "ping", "conn": "tascam", "host": "10.1.11.74", "name": "TASCAM" } + ] }, { "id": "x32", "label": "X32", "pos": [2, 1], "checks": [{ "type": "conn", "conn": "x32" }] }, { "id": "dante", "label": "DANTE", "pos": [2, 2], "checks": [ @@ -131,6 +162,8 @@ { "id": "resi", "label": "RESI", "pos": [3, 3], "checks": [ { "type": "conn", "conn": "RESI_Decoder", "tag": "DECODER" }, { "type": "ping", "host": "10.1.10.203", "name": "DECODER" }, + { "type": "ping", "host": "10.1.10.202", "name": "ENCODER" }, + { "type": "http", "url": "http://10.1.10.202/", "msg": "ENCODER WEB DOWN", "interval_seconds": 10 }, { "type": "conn", "conn": "resi-studio-cues" }, { "type": "http", "url": "https://central.resi.io/", "name": "CLOUD" } ] }, @@ -138,7 +171,10 @@ { "id": "ingest", "label": "INGEST", "pos": [3, 5], "checks": [ { "type": "launchd", "label": "org.waterschurch.resi-propresenter", "name": "BRIDGE", "fix": "bridge" }, { "type": "launchd", "label": "org.waterschurch.lobby-captions", "name": "CAPTIONS", "fix": "captions" }, + { "type": "launchd", "label": "org.waterschurch.lobby-captions-display", "name": "CAPTION SCREEN", "fix": "caption_screen" }, + { "type": "http", "url": "http://127.0.0.1:24480/api/v1/ping", "msg": "PRODCOM DOWN", "interval_seconds": 5, "fix": "prodcom" }, { "type": "process", "match": "/Applications/ProdDash.app/Contents/MacOS/ProdDash", "name": "PRODDASH", "fix": "proddash" }, + { "type": "http", "url": "http://127.0.0.1:24500/", "msg": "PRODDASH DOWN", "interval_seconds": 10, "fix": "proddash" }, { "type": "disk", "path": "/", "warn_gb": 25, "err_gb": 8 } ] }, { "id": "led", "label": "LED", "pos": [3, 6], "checks": [ diff --git a/make_page.py b/make_page.py index d0215f0..02e0772 100644 --- a/make_page.py +++ b/make_page.py @@ -13,9 +13,10 @@ watchdog can never leave a frozen green page behind. Buttons write commands into sp_cmd, which status_panel.py reads: press ack (the master acks everything) - hold 2 seconds restart (restarts that tile's Companion connections) + hold 2 seconds restart (fixes what that tile is failing on) + Restart page force (hold 2 s: restart that thing even when it looks fine) - python3 make_page.py -> status-page.companionconfig + python3 make_page.py -> status-page.companionconfig and restart-page.companionconfig """ import gzip import hashlib @@ -96,6 +97,57 @@ def command_action(tid, role, cmd_var, command): } +def set_page_action(tid, role, page): + return {"type": "action", "id": stable_id(tid, role), "definitionId": "set_page", "connectionId": "internal", + "options": {"surfaceId": prop("self"), "page": prop(page)}, "children": {}} + + +def static_button(tid, text, bg, fg, page, note): + """A plain key (no watchdog state): tap goes to `page`.""" + geometry = {"enabled": prop(True), "opacity": prop(100), "x": prop(0), "y": prop(0), + "width": prop(100), "height": prop(100), "rotation": prop(0)} + layers = [ + {"id": "canvas", "name": "Canvas", "usage": "auto", "type": "canvas", + "decoration": prop("border"), "showStatusIcons": prop("default")}, + dict({"id": "box0", "name": "Background", "usage": "auto", "type": "box"}, **geometry, + color=prop(bg), borderWidth=prop(0), borderColor=prop(0), borderPosition=prop("inside")), + dict({"id": "text0", "name": "Text", "usage": "auto", "type": "text"}, **geometry, + text=prop(text), color=prop(fg), halign=prop("center"), valign=prop("center"), + fontsize=prop("auto"), fontsizeAllowShrink=prop(True), font=prop("companion-sans"), + outlineColor=prop(4278190080)), + ] + return { + "type": "button-layered", + "style": {"layers": layers}, + "options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False, + "canModifyStyleInApis": False, "notes": note}, + "feedbacks": [], + "steps": {"0": {"action_sets": {"down": [], "up": [set_page_action(tid, "go", page)]}, + "options": {"runWhileHeld": []}}}, + "localVariables": [], + } + + +def restart_key(key, cfg): + """A Restart page key: drawn by the watchdog like a tile; hold 2 s = "force ".""" + names = cfg["variables"] + stale_text = "'%s\\n?'" % key["label"] + return { + "type": "button-layered", + "style": {"layers": layers(key["id"], key, names, stale_text)}, + "options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False, + "canModifyStyleInApis": True, + "notes": "RESTART key '%s' - hold 2 s: restart it even when the status page says it is fine " + "(status_panel.py on NA Ingest runs fix '%s')." % (key["id"], key["fix"])}, + "feedbacks": [], + "steps": {"0": {"action_sets": {"down": [], "up": [], + str(HOLD_MS): [command_action(key["id"], "hold", names["command"], + "force %s $(internal:time_unix)" % key["id"])]}, + "options": {"runWhileHeld": [HOLD_MS]}}}, + "localVariables": [], + } + + def button(tid, tile, cfg): names = cfg["variables"] kind = tile.get("kind") @@ -114,13 +166,17 @@ def button(tid, tile, cfg): press = "ack " + tid fixable = "members" in tile or any((c["type"] in RESTARTABLE and c.get("conn")) or c.get("fix") for c in tile.get("checks", [])) - hold = ("restart " + tid) if fixable else None - note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)." - if hold else "") + hold = ("restart " + tid) if fixable or tile.get("hold_always") else None + if tile.get("hold_always"): + note = "Press: acknowledge. Hold 2 s: %s." % cfg["fixes"][tile["hold_always"]]["label"] + else: + note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)." + if hold else "") action_sets = {"down": [], "up": [command_action(tid, "press", names["command"], press)]} run_while_held = [] if hold: - action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold)] + # Stamped with Companion's clock, so a hold made while the watchdog was down is dropped later. + action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")] run_while_held = [HOLD_MS] return { "type": "button-layered", @@ -177,41 +233,73 @@ def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1): return key -def build(cfg): +def nav_controls(cfg): controls = {} coloured = cfg.get("nav_keys", list(NAV_LOCATIONS)) # which nav keys follow the ALL button for kind, (r, c) in NAV_LOCATIONS.items(): controls.setdefault(r, {})[c] = (nav_key(kind, cfg["variables"], cfg["page"], cfg.get("nav_hold_ms", 1000), cfg.get("nav_tap_page", 1)) if kind in coloured else {"type": kind}) - for tile in cfg["tiles"]: - if "pos" not in tile: - continue - r, c = (str(n) for n in tile["pos"]) - if controls.get(r, {}).get(c): - raise ValueError("two buttons at row %s column %s" % (r, c)) - controls.setdefault(r, {})[c] = button(tile["id"], tile, cfg) + return controls + + +def place(controls, pos, control): + r, c = (str(n) for n in pos) + if controls.get(r, {}).get(c): + raise ValueError("two buttons at row %s column %s" % (r, c)) + controls.setdefault(r, {})[c] = control + + +def page_export(name, number, controls): return { "version": EXPORT_VERSION, "type": "page", "companionBuild": COMPANION_BUILD, - "page": {"id": stable_id("page", cfg["page_name"]), "name": cfg["page_name"], - "controls": controls, "gridSize": GRID}, + "page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": GRID}, "instances": {}, "connectionCollections": [], - "oldPageNumber": cfg["page"], + "oldPageNumber": number, "imageLibrary": [], "imageLibraryCollections": [], } +def build(cfg): + controls = nav_controls(cfg) + for tile in cfg["tiles"]: + if "pos" in tile: + place(controls, tile["pos"], button(tile["id"], tile, cfg)) + rp = cfg.get("restart_page") + if rp: + place(controls, rp["jump_pos"], static_button("restart_jump", "RESTART\\nPAGE", 0x37474F, 0xFFFFFF, + rp["page"], "Tap: open the Restart page (page %d)." % rp["page"])) + return page_export(cfg["page_name"], cfg["page"], controls) + + +def build_restart(cfg): + rp = cfg["restart_page"] + controls = nav_controls(cfg) + place(controls, rp["back_pos"], static_button("restart_back", "STATUS\\nPAGE", 0x37474F, 0xFFFFFF, cfg["page"], + "Tap: back to the status page (page %d)." % cfg["page"])) + for key in rp["keys"]: + place(controls, key["pos"], restart_key(key, cfg)) + return page_export(rp["page_name"], rp["page"], controls) + + +def write(path, export): + with gzip.open(path, "wb") as f: # Companion's own exports are gzipped JSON + f.write(json.dumps(export, indent=1).encode()) + + def main(): cfg = load_config(os.path.join(HERE, "config.json")) out = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "status-page.companionconfig") - data = json.dumps(build(cfg), indent=1).encode() - with gzip.open(out, "wb") as f: # Companion's own exports are gzipped JSON - f.write(data) + write(out, build(cfg)) print("wrote %s (%d buttons)" % (out, sum(1 for t in cfg["tiles"] if "pos" in t))) + if cfg.get("restart_page") and len(sys.argv) <= 1: + out = os.path.join(HERE, "restart-page.companionconfig") + write(out, build_restart(cfg)) + print("wrote %s (%d restart keys)" % (out, len(cfg["restart_page"]["keys"]))) if __name__ == "__main__": diff --git a/push_page.py b/push_page.py index 4540a70..6df100a 100644 --- a/push_page.py +++ b/push_page.py @@ -1,20 +1,23 @@ #!/usr/bin/env python3 -"""Import status-page.companionconfig into Companion's page 19 without clicking through the UI. +"""Import the status page (19) or the Restart page (23) into Companion without clicking through the UI. -Does exactly what Import/Export -> Import -> "import into page 19" does, over the +Does exactly what Import/Export -> Import -> "import into page N" does, over the same internal API the Companion web UI uses (tRPC over a WebSocket at /trpc): upload the file in chunks, check its SHA-1, then import it as a single page. -Safety: it refuses unless the target page is empty or already the status page -(its name contains "status"), so it cannot overwrite one of your other pages. +Safety: it looks at what is on the target page, not just its name. It refuses unless the +page holds nothing but page up / page number / page down keys, or is already the page this +file builds (same name). A page named "PAGE" can still hold real buttons (pages 22 and 97 do). - python3 push_page.py # na-ingest.local:8000, page and file from config.json + python3 push_page.py # the status page, from config.json + python3 push_page.py --which restart # the Restart page python3 push_page.py --host 10.6.3.222 --page 19 --file status-page.companionconfig Stdlib only (a minimal WebSocket client is included). Written against Companion 5.0.7. """ import argparse import base64 +import gzip import hashlib import json import os @@ -116,6 +119,46 @@ class Trpc: return msg.get("result", {}).get("data") +NAV_TYPES = ("pageup", "pagedown", "pagenum") +NAV_MARKER = "STATUS nav key" # our coloured page number key (make_page.nav_key) + + +def fetch_page(host, port, page): + """The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}}).""" + with urllib.request.urlopen("http://%s:%d/int/export/page/%d" % (host, port, page), timeout=10) as r: + raw = r.read() + try: + raw = gzip.decompress(raw) + except OSError: + pass + return json.loads(raw) + + +def real_buttons(export): + """How many buttons a page holds besides its page up / number / down keys.""" + page = export.get("page") or {} + n = 0 + for row in (page.get("controls") or {}).values(): + for ctl in (row or {}).values(): + if not ctl or ctl.get("type") in NAV_TYPES: + continue + if NAV_MARKER in ((ctl.get("options") or {}).get("notes") or ""): + continue + n += 1 + return n + + +def safe_target(export, expected_name): + """(ok, reason). OK only for a page with nothing but nav keys, or the page this file builds.""" + name = ((export.get("page") or {}).get("name") or "") + if name == expected_name: + return True, "it is already %r" % name + n = real_buttons(export) + if n == 0: + return True, "it holds only page keys" + return False, "it is called %r and holds %d buttons" % (name, n) + + def page_name(host, port, page): url = "http://%s:%d/api/variable/internal/page_number_%d_name/value" % (host, port, page) with urllib.request.urlopen(url, timeout=5) as r: @@ -125,21 +168,30 @@ def page_name(host, port, page): def main(): cfg = json.load(open(os.path.join(HERE, "config.json"))) ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--which", choices=("status", "restart"), default="status") ap.add_argument("--host", default="na-ingest.local") ap.add_argument("--port", type=int, default=8000) - ap.add_argument("--page", type=int, default=cfg["page"]) - ap.add_argument("--file", default=os.path.join(HERE, "status-page.companionconfig")) - ap.add_argument("--force", action="store_true", help="overwrite the page whatever its name") + ap.add_argument("--page", type=int) + ap.add_argument("--file") + ap.add_argument("--force", action="store_true", help="overwrite the page whatever is on it") args = ap.parse_args() - - name = page_name(args.host, args.port, args.page) - plain = name.replace("\\n", " ").strip() - empty = plain.upper() in ("", "PAGE", "%d PAGE" % args.page) - if not (empty or "status" in plain.lower() or args.force): - sys.exit("page %d is called %r - not empty and not the status page; refusing (use --force)" - % (args.page, plain)) + if args.which == "restart": + page, expected = cfg["restart_page"]["page"], cfg["restart_page"]["page_name"] + default_file = "restart-page.companionconfig" + else: + page, expected, default_file = cfg["page"], cfg["page_name"], "status-page.companionconfig" + args.page = args.page or page + args.file = args.file or os.path.join(HERE, default_file) data = open(args.file, "rb").read() + built = json.loads(gzip.decompress(data)) + if (built.get("page") or {}).get("name") != expected: + sys.exit("%s builds page %r, not %r; refusing" % (os.path.basename(args.file), + (built.get("page") or {}).get("name"), expected)) + ok, why = safe_target(fetch_page(args.host, args.port, args.page), expected) + if not ok and not args.force: + sys.exit("page %d: %s - refusing (use --force only if you mean to replace it)" % (args.page, why)) + plain = page_name(args.host, args.port, args.page).replace("\\n", " ").strip() ws = WebSocket(args.host, args.port, "/trpc") try: api = Trpc(ws) diff --git a/remote-access/windows-setup.ps1 b/remote-access/windows-setup.ps1 index b241e90..dd890af 100644 --- a/remote-access/windows-setup.ps1 +++ b/remote-access/windows-setup.ps1 @@ -36,6 +36,7 @@ if (-not (Get-Service sshd -ErrorAction SilentlyContinue)) { return } Set-Service -Name sshd -StartupType Automatic +sc.exe failure sshd reset= 86400 actions= restart/5000/restart/5000/restart/30000 | Out-Null # comes back if it stops Start-Service sshd if (-not (Get-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -ErrorAction SilentlyContinue)) { New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True ` @@ -80,6 +81,15 @@ $apps = @( @{ Name = 'BibleApp'; Exe = $bible; Args = ''; AtLogon = $true } # also opens it 15 s after login ) $principal = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Limited +# An app ticked "Run this program as an administrator" (OBS on the LiveStream PC) cannot start from a +# Limited task: Windows refuses with 0x800702E4. Its task runs with highest privileges instead. +$elevated = New-ScheduledTaskPrincipal -UserId "$env:USERDOMAIN\$env:USERNAME" -LogonType Interactive -RunLevel Highest +$layers = @{} +foreach ($k in 'HKCU:\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers', + 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers') { + $p = Get-ItemProperty $k -ErrorAction SilentlyContinue + if ($p) { $p.PSObject.Properties | ForEach-Object { $layers[$_.Name] = "$($_.Value)" } } +} $settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries ` -ExecutionTimeLimit ([TimeSpan]::Zero) -MultipleInstances IgnoreNew $tasks = @() @@ -88,8 +98,9 @@ foreach ($app in $apps) { $actionArgs = @{ Execute = $app.Exe; WorkingDirectory = (Split-Path $app.Exe) } if ($app.Args) { $actionArgs.Argument = $app.Args } $action = New-ScheduledTaskAction @actionArgs + $runAsAdmin = $layers[$app.Exe] -match 'RUNASADMIN' $taskArgs = @{ TaskPath = '\StatusPanel\'; TaskName = ('Start ' + $app.Name); Action = $action - Principal = $principal; Settings = $settings; Force = $true } + Principal = $(if ($runAsAdmin) { $elevated } else { $principal }); Settings = $settings; Force = $true } if ($app.AtLogon) { $taskArgs.Trigger = New-ScheduledTaskTrigger -AtLogOn -User "$env:USERDOMAIN\$env:USERNAME" $taskArgs.Trigger.Delay = 'PT15S' diff --git a/status_panel.py b/status_panel.py index 5d2254a..adf95a8 100644 --- a/status_panel.py +++ b/status_panel.py @@ -111,9 +111,14 @@ class Companion: return text if status == 200 else None def set_custom(self, name, value): - path = "/api/custom-variable/%s/value?value=%s" % ( - urllib.parse.quote(name, safe=""), urllib.parse.quote(value, safe="")) - status, text = self._request("POST", path) + path = "/api/custom-variable/%s/value" % urllib.parse.quote(name, safe="") + if value == "": + # Companion answers 400 to an empty body; an empty ?value= clears it. + status, text = self._request("POST", path + "?value=") + else: + # In the body, so sp_state (several KB) never runs into a URL length limit. + status, text = self._request("POST", path, body=value.encode("utf-8"), + headers={"Content-Type": "text/plain; charset=utf-8"}) return status == 200, status def restart_connection(self, conn_id): @@ -641,6 +646,8 @@ class Engine: self.executor = execute_fix self.poe_executor = execute_poe self.notes = {} # tile id -> hold feedback {"text", "level", "until"} + # The Restart page: one key per thing, hold 2 s = restart it even when it looks fine. + self.restart_keys = {k["id"]: k for k in cfg.get("restart_page", {}).get("keys", [])} self.deck_ips = {} # HyperDeck connection -> last address Companion reported self.now = 0.0 @@ -867,6 +874,10 @@ class Engine: def run_fixes(self, tile_id, snap, comp, now, cooldown=60): plan = self.fix_plan(tile_id, snap) + always = self.tiles.get(tile_id, {}).get("hold_always") + if not plan and always: + # e.g. TASCAM: its hold reconnects even when it looks fine (its control port hangs). + return self.run_force(tile_id, snap, comp, now, cooldown, fix_id=always) if not plan: if self.failing(tile_id): log.info("HOLD %s: failing, but there is no automatic fix for it", tile_id) @@ -897,12 +908,12 @@ class Engine: elif kind == "poe": log.info("FIX %s: power-cycle %s's PoE port (found by MAC)", tile_id, what) names.append("POWER CYCLE") - started.append((self.poe_executor, (self.cfg, what, dict(snap.dante_map.get(what, {}))))) + started.append((self.poe_executor, (self.cfg, what, dict(snap.dante_map.get(what, {}))), what)) else: fix = self.cfg["fixes"][what] log.info("FIX %s: %s", tile_id, fix.get("label", what)) names.append(fix.get("short", what.upper())) - started.append((self.executor, (self.cfg, what, chk))) + started.append((self.executor, (self.cfg, what, chk), fix.get("short", what.upper()))) if not names: self.note(tile_id, "JUST TRIED\nWAIT 1 MIN" if waited else "NOTHING\nTO FIX", "idle", now, 4) return plan @@ -913,21 +924,63 @@ class Engine: self.note(tile_id, "FIXING\n" + summary, "fixing", now, 180) results = [] - def work(fn, args): + group = "members" in self.tiles.get(tile_id, {}) or len(started) > 1 + + def work(fn, args, short): ok = fn(*args) results.append(ok) + if isinstance(ok, str) and ok != WAKING: + said.append(((short + " ") if group else "") + ok) # on a group button, say which if len(results) == len(started): # the last one to finish reports for all done = time.time() if WAKING in results: self.note(tile_id, "WAKING PC\n" + summary, "fixing", done, 120) + elif said: # the fix says what to do next + self.note(tile_id, said[0], "failed", done, 60) elif all(r is not False for r in results): self.note(tile_id, "CHECKING\n" + summary, "fixing", done, 20) else: self.note(tile_id, "FIX FAILED\n" + summary, "failed", done, 20) - for fn, args in started: - threading.Thread(target=work, args=(fn, args), daemon=True).start() + said = [] + for fn, args, short in started: + threading.Thread(target=work, args=(fn, args, short), daemon=True).start() return plan + def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None): + """Run one fix whatever the status says: a Restart page key, or a button's "hold_always". + Shares the 60 s guard with holds, so a hold and a restart key never double up.""" + if fix_id is None: + key = self.restart_keys.get(key_id) + if not key: + log.warning("FORCE %s: no such restart key", key_id) + return None + fix_id = key["fix"] + ident = ("fix", fix_id, "") + if now - self.fix_last.get(ident, 0) < cooldown: + log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, fix_id, cooldown) + cur = self.notes.get(key_id) + if not (cur and cur["level"] == "fixing" and now < cur["until"]): # still running: keep saying so + self.note(key_id, "JUST DONE\nWAIT 1 MIN", "idle", now, 4) + return None + self.fix_last[ident] = now + fix = self.cfg["fixes"][fix_id] + log.info("FORCE %s: %s", key_id, fix.get("label", fix_id)) + self.note(key_id, "RESTARTING", "fixing", now, 180) + + def work(): + r = self.executor(self.cfg, fix_id, {}) + done = time.time() + if r == WAKING: + self.note(key_id, "WAKING PC", "fixing", done, 120) + elif isinstance(r, str): + self.note(key_id, r, "failed", done, 60) + elif r is False: + self.note(key_id, "FAILED", "failed", done, 30) + else: + self.note(key_id, "DONE", "ok", done, 20) + threading.Thread(target=work, daemon=True).start() + return fix_id + def display_level(res, acked, service_on): if res.level == OFF: @@ -996,6 +1049,12 @@ def colored(text, level): def render(engine, service_on): out = {} + for kid, key in engine.restart_keys.items(): + note = engine.notes.get(kid) + if note and engine.now < note["until"]: + out[kid] = colored(key["label"] + "\n" + note["text"], note["level"]) + else: + out[kid] = colored(key["label"] + "\nHOLD 2s", "idle") for tile in engine.cfg["tiles"]: tid, kind = tile["id"], tile.get("kind") note = engine.notes.get(tid) @@ -1449,13 +1508,16 @@ def execute_fix(cfg, fix_id, chk): return WAKING log.info("FIX %s -> %s%s", label, "ok" if r.returncode == 0 else "exit %d" % r.returncode, (": " + detail) if detail else "") - if r.returncode == 0 and fix.get("reconnect") and chk.get("conn"): + conn = chk.get("conn") or fix.get("conn") + if r.returncode == 0 and fix.get("reconnect") and conn: # The app is running now: make Companion reconnect to it instead of waiting for its retry. comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000)) ids = {c["label"]: c["id"] for c in comp.connections()} - ok = chk["conn"] in ids and comp.restart_connection(ids[chk["conn"]]) - log.info("FIX %s -> Companion connection %s restarted %s", label, chk["conn"], "ok" if ok else "FAILED") + ok = conn in ids and comp.restart_connection(ids[conn]) + log.info("FIX %s -> Companion connection %s restarted %s", label, conn, "ok" if ok else "FAILED") return r.returncode == 0 + if kind == "reconnect": + return reconnect_and_verify(cfg, fix, label) if kind == "midi_connect": # Like Connect in Audio MIDI Setup: add the peer back to the session it dropped out of. args = (chk["session"], chk["peer"], chk["address"]) @@ -1505,6 +1567,41 @@ def execute_fix(cfg, fix_id, chk): return False +def reconnect_and_verify(cfg, fix, label, sleep=time.sleep): + """Restart one Companion connection, then look again after "verify_seconds". Returns True if it + came back; otherwise what to tell the operator: "hung_note" when the device still answers ping + (its control side has hung - only a power-cycle helps), else that it is not on the network.""" + comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000)) + conn = {c["label"]: c for c in comp.connections()}.get(fix["conn"]) + if conn is None: + log.info("FIX %s -> no Companion connection %s", label, fix["conn"]) + return False + if not conn.get("enabled"): + # Switched off in Companion: turn it on (never call that a hung device). + ok = comp.enable_connection(conn["id"]) + log.info("FIX %s -> %s was disabled in Companion: turned on %s", label, fix["conn"], "ok" if ok else "FAILED") + if not ok: + return "DISABLED IN\nCOMPANION" + elif not comp.restart_connection(conn["id"]): + log.info("FIX %s -> Companion refused to restart %s", label, fix["conn"]) + return False + sleep(fix.get("verify_seconds", 20)) + after = {c["label"]: c for c in comp.connections()}.get(fix["conn"]) or {} + status = (after.get("status") or {}).get("category") + if status == "good": + log.info("FIX %s -> %s reconnected", label, fix["conn"]) + return True + if status not in ("warning", "error"): # still starting up, or unknown: no verdict yet + log.info("FIX %s -> %s is %s after %ss", label, fix["conn"], status, fix.get("verify_seconds", 20)) + return "STILL\nCONNECTING" + host = companion_conn_hosts(companion_db_path(cfg)).get(fix["conn"]) + if host and ping_once(host): + log.info("FIX %s -> %s still %s but %s answers ping: control port hung", label, fix["conn"], status, host) + return fix.get("hung_note", "STILL DOWN") + log.info("FIX %s -> %s still %s and %s does not answer ping", label, fix["conn"], status, host or "its address") + return "NO NETWORK\nCHECK POWER" + + def start_artnet_listener(snap, stop, port=6454): """Note when each source last sent Art-Net. Shares the port (SO_REUSEPORT) so it never keeps another program from listening too. Passive: nothing is ever sent.""" @@ -1580,6 +1677,13 @@ def load_config(path): raise ValueError("fix %s: unknown host" % f) if c["type"] == "remote" and c["host"] not in cfg.get("hosts", {}): raise ValueError("tile %s: unknown host %s" % (t["id"], c["host"])) + if t.get("hold_always") and t["hold_always"] not in cfg.get("fixes", {}): + raise ValueError("tile %s: unknown hold_always fix %s" % (t["id"], t["hold_always"])) + for k in cfg.get("restart_page", {}).get("keys", []): + if k["id"] in ids: + raise ValueError("restart key %s has a tile's id" % k["id"]) + if k["fix"] not in cfg.get("fixes", {}): + raise ValueError("restart key %s: unknown fix %s" % (k["id"], k["fix"])) return cfg @@ -1641,6 +1745,9 @@ def post_webhook(url, text): log.warning("webhook failed: %s", e) +HOLD_MAX_AGE = 15 # seconds: a hold older than this (queued while the watchdog was down) is dropped + + def handle_commands(text, engine, comp, snap, now): for part in text.split(";"): words = part.strip().split() @@ -1655,8 +1762,19 @@ def handle_commands(text, engine, comp, snap, now): target = {"on": True, "off": False}.get(arg, not engine.service.on(now)) engine.service.set(target, now) log.info("SERVICE %s (button)", "ON" if target else "OFF") - elif verb == "restart": # sent by a 2 second hold - engine.run_fixes(arg, snap, comp, now) + elif verb in ("restart", "force"): + # Holds carry Companion's clock ("force rs_x 1791496225"). One pressed while the watchdog + # was down must not fire minutes later in a burst with the others. + stamp = words[2] if len(words) > 2 else None + if stamp is not None and (not stamp.isdigit() or now - int(stamp) > HOLD_MAX_AGE): + log.info("%s %s: pressed %s, too long ago - ignored", verb.upper(), arg, + "at %s" % stamp if stamp.isdigit() else "at an unknown time") + elif stamp is None and verb == "force": + log.info("FORCE %s: no time on it - ignored", arg) + elif verb == "restart": # a 2 second hold on a status button + engine.run_fixes(arg, snap, comp, now) + else: # a Restart page key, held 2 seconds + engine.run_force(arg, snap, comp, now) else: log.warning("unknown command %r", part) diff --git a/tests/test_status_panel.py b/tests/test_status_panel.py index 209aee7..c5978f8 100644 --- a/tests/test_status_panel.py +++ b/tests/test_status_panel.py @@ -66,7 +66,8 @@ def healthy_snapshot(now): snap.mdns, snap.mdns_time = {"NA Tracks", "Lighting PC", "NA Ingest"}, now snap.ping = {"10.1.3.227": 0, "10.6.7.13": 0, "10.1.0.1": 0} snap.link = True - snap.launchd = {"org.waterschurch.resi-propresenter": 884, "org.waterschurch.lobby-captions": 900} + snap.launchd = {"org.waterschurch.resi-propresenter": 884, "org.waterschurch.lobby-captions": 900, + "org.waterschurch.lobby-captions-display": 901} snap.launchd_time = now snap.procs = {"/Applications/ProdDash.app/Contents/MacOS/ProdDash": True} snap.http = {"https://central.resi.io/": True, "http://10.6.7.13:8080/": True, "http://10.1.14.225:1025/": True} @@ -98,8 +99,10 @@ class TestChecks(unittest.TestCase): def test_all_green_when_healthy(self): tiles = sp.Engine(CONFIG).step(self.snap, self.now) - bad = {k: v["x"] for k, v in tiles.items() if level(v) not in ("ok", "service", "off")} + bad = {k: v["x"] for k, v in tiles.items() if level(v) not in ("ok", "service", "off") + and not k.startswith("rs_")} # Restart page keys sit grey self.assertEqual(bad, {}) + self.assertEqual((tiles["rs_dvs_tracks"]["x"], level(tiles["rs_dvs_tracks"])), ("DVS TRACKS\nHOLD 2s", "idle")) self.assertEqual(tiles["all"]["x"], "ALL OK") self.assertEqual(tiles["hd1"]["x"], "HD1\n28h") self.assertEqual(tiles["dante"]["x"], "DANTE\n19/19") @@ -608,6 +611,116 @@ class TestFixes(unittest.TestCase): self.assertEqual(self.plan("rf"), []) # nothing left to "fix" self.assertFalse(self.engine.failing("rf")) + def test_restart_page_keys_run_even_when_all_is_fine(self): + ran = [] + self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True + run_steps(self.engine, self.snap, SUNDAY, 4) # everything green + self.assertEqual(self.engine.run_force("rs_dvs_tracks", self.snap, None, SUNDAY + 5), "dvs_tracks") + time.sleep(0.2) + self.assertEqual(ran, ["dvs_tracks"]) + self.assertEqual(self.engine.notes["rs_dvs_tracks"]["text"], "DONE") + self.assertIsNone(self.engine.run_force("rs_dvs_tracks", self.snap, None, SUNDAY + 20)) # 60 s guard + self.assertEqual(self.engine.notes["rs_dvs_tracks"]["text"], "JUST DONE\nWAIT 1 MIN") + self.assertIsNone(self.engine.run_force("rs_nope", self.snap, None, SUNDAY + 90)) + # the hold on the status page and the Restart key share the guard + self.engine.run_force("rs_prodcom", self.snap, None, SUNDAY + 100) + self.engine.notes.clear() + self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = False + run_steps(self.engine, self.snap, SUNDAY + 100, 4) + self.engine.run_fixes("ingest", self.snap, None, SUNDAY + 106) + self.assertEqual(self.engine.notes["ingest"]["text"], "JUST TRIED\nWAIT 1 MIN") + # what each key restarts + keys = {k["id"]: k["fix"] for k in CONFIG["restart_page"]["keys"]} + self.assertEqual(keys["rs_ableton"], "ableton_restart") + self.assertIn("pkill -TERM -x 'Live'", CONFIG["fixes"]["ableton_restart"]["run"]) + self.assertIn("kickstart -k system/com.audinate.dante.DanteVirtualSoundcard", + open(os.path.join(os.path.dirname(os.path.abspath(sp.__file__)), "remote-access", "mac-dvs-helper.sh")).read()) + self.assertIn("open -g /Applications/ProdCom.app", CONFIG["fixes"]["dvs_ingest"]["run"]) # ProdCom after DVS + + def test_tascam_hold_always_reconnects(self): + ran = [] + self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or "HUNG - POWER\nCYCLE IT" + run_steps(self.engine, self.snap, SUNDAY, 4) # TASCAM looks fine + self.engine.run_fixes("tascam", self.snap, None, SUNDAY + 5) + time.sleep(0.2) + self.assertEqual(ran, ["tascam_reconnect"]) + tiles = self.engine.step(self.snap, time.time()) + self.assertEqual((tiles["tascam"]["x"], level(tiles["tascam"])), ("TASCAM\nHUNG - POWER\nCYCLE IT", "failed")) + + def test_reconnect_and_verify(self): + from unittest import mock + fix = CONFIG["fixes"]["tascam_reconnect"] + conn = lambda cat, enabled=True: [{"label": "tascam", "id": "t1", "enabled": enabled, "status": {"category": cat}}] + + def run(before, after, pings=True, restart_ok=True, enable_ok=True): + comp = mock.Mock() + comp.connections.side_effect = [before, after] + comp.restart_connection.return_value = restart_ok + comp.enable_connection.return_value = enable_ok + with mock.patch.object(sp, "Companion", return_value=comp), \ + mock.patch.object(sp, "companion_conn_hosts", return_value={"tascam": "10.1.11.74"}), \ + mock.patch.object(sp, "ping_once", return_value=pings): + return sp.reconnect_and_verify(CONFIG, fix, "x", sleep=lambda s: None), comp + r, comp = run(conn("error"), conn("good")) + self.assertTrue(r) + comp.restart_connection.assert_called_once_with("t1") + self.assertEqual(run(conn("error"), conn("error"))[0], "HUNG - POWER\nCYCLE IT") # pings: hung + self.assertEqual(run(conn("error"), conn("error"), pings=False)[0], "NO NETWORK\nCHECK POWER") + self.assertEqual(run(conn("good"), conn(None))[0], "STILL\nCONNECTING") # no verdict yet + self.assertFalse(run(conn("error"), conn("error"), restart_ok=False)[0]) + r, comp = run(conn(None, enabled=False), conn("good")) # switched off + self.assertTrue(r) + comp.enable_connection.assert_called_once_with("t1") + comp.restart_connection.assert_not_called() + self.assertEqual(run(conn(None, enabled=False), conn(None), enable_ok=False)[0], "DISABLED IN\nCOMPANION") + self.assertFalse(run([], [])[0]) + + def test_stale_or_unstamped_holds_are_dropped(self): + ran = [] + self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True + now = SUNDAY + 100 + sp.handle_commands(";force rs_bridge %d;force rs_prodcom %d;force rs_proddash" % (now - 300, now - 2), + self.engine, None, self.snap, now) + time.sleep(0.2) + self.assertEqual(ran, ["prodcom"]) # only the fresh one: not the old or unstamped + + def test_a_second_press_keeps_saying_restarting(self): + gate = threading.Event() + self.engine.executor = lambda cfg, fix_id, chk: gate.wait(2) or True + self.engine.run_force("rs_bridge", self.snap, None, SUNDAY) + self.engine.run_force("rs_bridge", self.snap, None, SUNDAY + 3) + self.assertEqual(self.engine.notes["rs_bridge"]["text"], "RESTARTING") + gate.set() + + def test_group_button_names_the_device_that_needs_a_power_cycle(self): + self.engine.executor = lambda cfg, fix_id, chk: "HUNG - POWER\nCYCLE IT" + self.snap.conns["tascam"]["category"] = "error" + _, t = run_steps(self.engine, self.snap, SUNDAY, 4) + self.engine.run_fixes("rec", self.snap, None, t) + time.sleep(0.2) + self.assertEqual(self.engine.notes["rec"]["text"], "TASCAM HUNG - POWER\nCYCLE IT") + + def test_windows_starts_check_the_app_really_started(self): + for fix_id, exe in (("obs", "obs64.exe"), ("arena", "Arena.exe"), ("vista", "Vista.exe"), ("bibleapp", "Bible App.exe")): + run = CONFIG["fixes"][fix_id]["run"] + self.assertTrue(run.rstrip(")").endswith('find /I "%s" >nul' % exe), fix_id) + self.assertIn("ping -n 9 127.0.0.1", run) + + def test_ingest_resi_and_tascam_checks(self): + self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = False + self.assertEqual(self.plan("ingest"), [("fix", "prodcom")]) + self.snap.http["http://127.0.0.1:24480/api/v1/ping"] = True + self.snap.launchd.pop("org.waterschurch.lobby-captions-display") + self.assertEqual(self.plan("ingest"), [("fix", "caption_screen")]) # the screen only + self.snap.launchd["org.waterschurch.lobby-captions-display"] = 1 + self.snap.ping["10.1.10.202"] = 3 + tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, SUNDAY, 4) + self.assertEqual(tiles["resi"]["x"], "RESI\nENCODER NO PING") + self.snap.conn_hosts = {"tascam": "10.1.11.74"} + self.snap.ping["10.1.11.74"] = 3 + tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, SUNDAY, 4) + self.assertEqual(tiles["tascam"]["x"], "TASCAM\nTASCAM NO PING") + def test_midi_reconnect_where_the_peer_was_last_seen(self): others = [{"name": "SuperRack-LiveBox", "address": "10.1.13.140:5004", "state": "running"}, {"name": "Studio B Mac Studio", "address": "\tStudio B Mac Studio\tlocal.", "state": "running"}] @@ -853,7 +966,7 @@ class TestFixes(unittest.TestCase): def test_every_fix_is_valid(self): for f in CONFIG["fixes"].values(): - self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect")) + self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect", "reconnect")) class FakeDeck: @@ -1000,7 +1113,11 @@ class FakeCompanion(BaseHTTPRequestHandler): if m: if m.group(1) not in s["custom"]: return self._send(404, "Not found") - s["custom"][m.group(1)] = urllib.parse.parse_qs(u.query, keep_blank_values=True)["value"][0] + body = self.rfile.read(int(self.headers.get("Content-Length") or 0)).decode() + query = urllib.parse.parse_qs(u.query, keep_blank_values=True) + if not body and "value" not in query: + return self._send(400, "No value") # like Companion 5.0.7 with an empty body + s["custom"][m.group(1)] = body if body else query["value"][0] return self._send(200, "ok") m = re.match(r"^/api/connections/([^/]+)/restart$", u.path) if m: @@ -1134,10 +1251,38 @@ class TestPage(unittest.TestCase): self.assertIn("'$.all.b'", box["color"]["value"]) # follows the ALL button positioned = [t for t in CONFIG["tiles"] if "pos" in t] n = sum(1 for r in self.controls.values() for c in r.values() if c["type"] == "button-layered") - self.assertEqual(n, len(positioned) + 1) # + the page number key + self.assertEqual(n, len(positioned) + 2) # + the page number and RESTART keys + jump = self.controls["2"]["6"]["steps"]["0"]["action_sets"]["up"][0] + self.assertEqual((jump["definitionId"], jump["options"]["page"]["value"]), ("set_page", 23)) # The master is where the smallest Stream Deck can see it. self.assertEqual(CONFIG["tiles"][0]["pos"], [0, 1]) + def test_restart_page(self): + export = make_page.build_restart(CONFIG) + self.assertEqual((export["oldPageNumber"], export["page"]["name"]), (23, "23\\nRestart")) + c = export["page"]["controls"] + holds = {} + for r in c.values(): + for ctl in r.values(): + sets = ctl.get("steps", {}).get("0", {}).get("action_sets", {}) + if "2000" in sets: + holds[sets["2000"][0]["options"]["value"]["value"]] = sets["up"] + self.assertEqual(len(holds), len(CONFIG["restart_page"]["keys"])) + self.assertIn("$(custom:sp_cmd);force rs_tascam $(internal:time_unix)", holds) + self.assertTrue(all(up == [] for up in holds.values())) # a tap does nothing + back = c["3"]["0"]["steps"]["0"]["action_sets"]["up"][0] + self.assertEqual((back["definitionId"], back["options"]["page"]["value"]), ("set_page", 19)) + + def test_push_page_looks_at_what_is_on_the_page(self): + import push_page + nav = {"0": {"0": {"type": "pageup"}}, "1": {"0": {"type": "button-layered", + "options": {"notes": "STATUS nav key (pagenum): x"}}}} + self.assertTrue(push_page.safe_target({"page": {"name": "PAGE", "controls": nav}}, "23\\nRestart")[0]) + busy = {"page": {"name": "PAGE", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}} + self.assertFalse(push_page.safe_target(busy, "23\\nRestart")[0]) # pages 22 and 97 + busy["page"]["name"] = "23\\nRestart" + self.assertTrue(push_page.safe_target(busy, "23\\nRestart")[0]) # our own page again + def test_each_button_reads_its_own_tile(self): for t in CONFIG["tiles"]: if "pos" not in t: @@ -1155,7 +1300,8 @@ class TestPage(unittest.TestCase): hd1 = self.controls["1"]["1"]["steps"]["0"] up = hd1["action_sets"]["up"][0]["options"]["value"] self.assertEqual(up, {"value": "$(custom:sp_cmd);ack hd1", "isExpression": False}) - self.assertEqual(hd1["action_sets"]["2000"][0]["options"]["value"]["value"], "$(custom:sp_cmd);restart hd1") + self.assertEqual(hd1["action_sets"]["2000"][0]["options"]["value"]["value"], + "$(custom:sp_cmd);restart hd1 $(internal:time_unix)") self.assertEqual(hd1["options"]["runWhileHeld"], [2000]) master = self.controls["0"]["1"]["steps"]["0"] self.assertIn("ackall", master["action_sets"]["up"][0]["options"]["value"]["value"])