Step 0 of the settings page: registry, safe restart, remembered state, test verb, page markers
- schema.py: every setting with type, default, help, owner screen and the rules a config must pass; tests/test_schema.py keeps it in step with the code (every key the code reads is declared, every default matches, check types and fix kinds agree, the glossary covers every word a key can show, today's config still builds today's pages apart from the key notes) - status_panel.py: VERSION + started.json; SIGTERM lets a running fix finish and starts nothing new (RESTARTING SOON); fixes-running.json and live-now.json for the settings page; state.json keeps the ALARMS override, acknowledgements, auto-fix tries and fix times across restarts, keyed by what a check is about rather than its position; 'test' and 'resurvey' commands; queued commands dropped at start; overnight windows; a HyperDeck switched off is always switched back on; the spotify fix kind (playlist set once, in config 'spotify'); Windows remote checks batched for cmd.exe with conditional echoes; every fix's kind and host validated; never_auto; wording and numbers from config (site block, timing, page_hold_ms, auto-fix keep_months); hyperdeck_probe removed; --check and --exit-timeout - make_page.py: numbers from config, markers on every key, nav templates per Companion version, --out; push_page.py: a page is overwritten only when every key carries a marker (foreign keys named by position), secret-free fetch, host from the site; nav_keys.py: host from the site; netgear.py: switches and the shared-port limit from config - install.sh: validates with the new code first, keeps prev/ for --rollback, --keep-config, derived ExitTimeOut, wider deploy list, waits for the start stamp - config.json: site, timing, spotify sections; Windows hosts marked; deployed to NA Ingest and both pages re-imported once (--force) so every key carries its marker Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
@@ -139,15 +139,17 @@ A problem has to last 3 seconds before it shows, so one bad reading doesn't trig
|
||||
This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:
|
||||
|
||||
```bash
|
||||
scp status_panel.py coremidi.py netgear.py config.json install.sh checklist-ingest:status-panel-install/
|
||||
scp status_panel.py coremidi.py netgear.py make_page.py push_page.py nav_keys.py nav_templates.json config.json install.sh atem_watch.js atem_state.js atem-reference.xml checklist-ingest:status-panel-install/
|
||||
```
|
||||
|
||||
Then run the installer there. It backs up the old config and restarts the watchdog:
|
||||
Then run the installer there. It checks the config with the new code first (a bad config never gets as far as the watchdog), keeps the code that was running in `prev/`, backs up the old config, writes the LaunchAgent with the exit timeout this config needs (the longest fix plus its verify time, so a restart never cuts a fix in half) and restarts the watchdog:
|
||||
|
||||
```bash
|
||||
ssh checklist-ingest 'zsh ~/status-panel-install/install.sh'
|
||||
```
|
||||
|
||||
`install.sh --keep-config` updates the code and leaves the live config alone; `install.sh --rollback` puts the previous code back.
|
||||
|
||||
If you add or move tiles, rebuild the page:
|
||||
|
||||
```bash
|
||||
@@ -172,6 +174,8 @@ zsh install.sh --remove # stop the watchdog
|
||||
|
||||
The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.
|
||||
|
||||
What the watchdog keeps in its data folder (`~/Library/Application Support/status-panel/`): `state.json` (the ALARMS override, acknowledged faults, auto-fix tries and when each fix last ran, so a restart forgets nothing), `started.json` (when it started, with which config), `live-now.json` (whether a HyperDeck is recording: changes to the panel wait while it is), `fixes-running.json` (the fixes running right now). A restart (`launchctl kickstart -k`, or the installer) lets a running fix finish first and starts nothing new meanwhile: keys say RESTARTING SOON. Windows in `config.json` may run past midnight (`"end"` earlier than `"start"`), and a remote check on a Windows PC (`"os": "windows"` on the host) is batched for cmd.exe.
|
||||
|
||||
## Optional: see what the tracks send to the lights
|
||||
|
||||
The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:
|
||||
@@ -204,7 +208,8 @@ To go back, pick 1.3.0 on the connection again.
|
||||
| `status_panel.py` | the watchdog |
|
||||
| `netgear.py` | reads the Netgear Dante switches (MAC tables, PoE) and power-cycles one PoE port |
|
||||
| `coremidi.py` | reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs) |
|
||||
| `config.json` | tiles, positions, checks, service windows |
|
||||
| `config.json` | tiles, positions, checks, fixes, hosts, windows: every setting (the `site` block names the campus; `spotify` holds the playlist; `timing` and `page_hold_ms` the numbers behind HOLD 2s and WAIT 1 MIN) |
|
||||
| `schema.py` | the registry of every setting: type, default, help, which screen of the settings page owns it, and the rules a config must pass (`tests/test_schema.py` keeps it in step with the code) |
|
||||
| `make_page.py` | builds `status-page.companionconfig` from `config.json` |
|
||||
| `install.sh` | installs or updates the LaunchAgent on NA Ingest |
|
||||
| `push_page.py` | imports the status page into page 19 over Companion's API |
|
||||
@@ -212,4 +217,4 @@ To go back, pick 1.3.0 on the connection again.
|
||||
| `nav_templates.json` | Companion's own nav key designs, coloured by `make_page.py` |
|
||||
| `remote-access/` | one-time setup scripts for SSH to the Macs and Windows PCs |
|
||||
| `tascam-quiet/` | builds the quiet Tascam module |
|
||||
| `tests/` | offline tests, including a fake Companion and a MIDI loopback |
|
||||
| `tests/` | offline tests, including a fake Companion and a MIDI loopback; `test_schema.py` checks that every key the code reads is declared in `schema.py` with the same default |
|
||||
|
||||
+14
-6
@@ -1,4 +1,9 @@
|
||||
{
|
||||
"site": {
|
||||
"id": "na", "name": "North Attleboro", "ingest_name": "NA Ingest", "ingest_hostname": "na-ingest.local",
|
||||
"ingest_ips": ["10.1.1.1", "10.6.3.222"],
|
||||
"live_services": []
|
||||
},
|
||||
"companion": { "host": "127.0.0.1", "port": 8000 },
|
||||
"page": 19,
|
||||
"page_name": "19\\nStatus",
|
||||
@@ -24,6 +29,10 @@
|
||||
},
|
||||
|
||||
"alert_webhook": "",
|
||||
"timing": { "fix_cooldown_seconds": 60, "alert_after_seconds": 30 },
|
||||
"page_hold_ms": 2000,
|
||||
|
||||
"spotify": { "playlist": "spotify:playlist:2uTpJVYQLFnKF1WnSOuco8", "name": "Christian EDM 2026", "open_wait": 8, "play_wait": 3 },
|
||||
|
||||
"ssh_key": "~/.ssh/id_status_panel",
|
||||
|
||||
@@ -32,10 +41,10 @@
|
||||
"prop": { "ssh": "napropresenter@10.1.14.225" },
|
||||
"tracks": { "ssh": "waterschurchmedia@10.1.3.227" },
|
||||
"translation": { "ssh": "studio@10.1.7.39" },
|
||||
"bibletv": { "ssh": "media@10.6.7.13" },
|
||||
"lighting": { "ssh": "media@10.1.10.206" },
|
||||
"bibletv": { "ssh": "media@10.6.7.13", "os": "windows" },
|
||||
"lighting": { "ssh": "media@10.1.10.206", "os": "windows" },
|
||||
"studiob": { "ssh": "studiob@10.1.13.175" },
|
||||
"livestream": { "ssh": "media@10.1.9.126", "mac": "6c:24:08:d7:fb:57", "wake_broadcast": "10.1.255.255" }
|
||||
"livestream": { "ssh": "media@10.1.9.126", "os": "windows", "mac": "6c:24:08:d7:fb:57", "wake_broadcast": "10.1.255.255" }
|
||||
},
|
||||
|
||||
"fixes": {
|
||||
@@ -72,8 +81,7 @@
|
||||
"atem_midi": { "kind": "local", "short": "ATEM MIDI", "restart_ok": true,
|
||||
"label": "turn off MIDI control in ATEM Software Control (quits and reopens the app; the switcher is not affected)",
|
||||
"run": "was=$(pgrep -f \"ATEM Software Control.app/Contents/MacOS\"); pkill -TERM -f \"ATEM Software Control.app/Contents/MacOS\"; i=0; while pgrep -fq \"ATEM Software Control.app/Contents/MacOS\" && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; defaults write \"com.blackmagic-design.ATEM Software Control\" midiEnabled -bool false; [ -z \"$was\" ] || open -g -a \"ATEM Software Control\"; [ \"$(defaults read \"com.blackmagic-design.ATEM Software Control\" midiEnabled)\" = 0 ]" },
|
||||
"spotify_play": { "kind": "local", "short": "SPOTIFY", "label": "play the Christian EDM 2026 playlist in Spotify on NA Ingest (opens Spotify if it is closed)",
|
||||
"run": "pgrep -xq Spotify || { open -g -a Spotify; sleep 8; }; osascript -e 'tell application \"Spotify\" to play track \"spotify:playlist:2uTpJVYQLFnKF1WnSOuco8\"'; sleep 3; [ \"$(osascript -e 'tell application \"Spotify\" to player state as text')\" = playing ]" },
|
||||
"spotify_play": { "kind": "spotify", "short": "SPOTIFY", "label": "play the Christian EDM 2026 playlist in Spotify on NA Ingest (opens Spotify if it is closed)" },
|
||||
"proddash": { "kind": "local", "short": "PRODDASH", "label": "restart ProdDash (its launcher stops its server cleanly)",
|
||||
"run": "pkill -TERM -x ProdDash; i=0; while pgrep -xq ProdDash && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; open -g /Applications/ProdDash.app" },
|
||||
"prodcom": { "kind": "local", "short": "PRODCOM", "label": "restart ProdCom (transcription; lobby captions blank for 5-15 s)",
|
||||
@@ -170,7 +178,7 @@
|
||||
{ "type": "midi_rx", "source": "Network NA Ingest Listen", "name": "TRACKS IN" }
|
||||
] },
|
||||
{ "id": "spotify", "label": "SPOTIFY", "pos": [2, 7], "hold_always": "spotify_play", "checks": [
|
||||
{ "type": "spotify", "playlist": "spotify:playlist:2uTpJVYQLFnKF1WnSOuco8", "name": "Christian EDM 2026", "fix": "spotify_play" }
|
||||
{ "type": "spotify", "fix": "spotify_play" }
|
||||
] },
|
||||
{ "id": "vista", "label": "VISTA", "pos": [2, 5], "checks": [
|
||||
{ "type": "artnet", "source": "10.1.10.206", "max_silence_seconds": 5, "fix": "vista" }
|
||||
|
||||
+101
-27
@@ -1,9 +1,14 @@
|
||||
#!/bin/zsh
|
||||
# Install or update the status panel watchdog on this Mac (NA Ingest) as a LaunchAgent.
|
||||
# Re-run after changing status_panel.py, coremidi.py or config.json in this folder.
|
||||
# Install or update the status panel watchdog on this Mac (the Ingest Mac) as a LaunchAgent.
|
||||
# Re-run after changing any file in this folder.
|
||||
#
|
||||
# zsh install.sh install / update and (re)start
|
||||
# zsh install.sh --remove stop it and remove the LaunchAgent (files stay)
|
||||
# zsh install.sh install / update and (re)start
|
||||
# zsh install.sh --keep-config update the code but leave the live config.json as it is
|
||||
# zsh install.sh --rollback put the previous code back (kept in prev/) and restart
|
||||
# zsh install.sh --remove stop it and remove the LaunchAgent (files stay)
|
||||
#
|
||||
# The config is validated with the NEW code before anything is touched, so a stricter check can
|
||||
# never leave the watchdog crash-looping. The previous code is kept in prev/ for --rollback.
|
||||
set -e
|
||||
SRC=${0:A:h}
|
||||
LABEL=org.waterschurch.status-panel
|
||||
@@ -11,23 +16,29 @@ LABEL=org.waterschurch.status-panel
|
||||
DEST="$HOME/Library/Application Support/status-panel"
|
||||
PLIST="$HOME/Library/LaunchAgents/$LABEL.plist"
|
||||
DOMAIN="gui/$(id -u)"
|
||||
CODE=(status_panel.py coremidi.py netgear.py make_page.py push_page.py nav_keys.py atem_watch.js atem_state.js nav_templates.json)
|
||||
KEEP_CONFIG=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--keep-config) KEEP_CONFIG=1 ;;
|
||||
--remove|--rollback) ;;
|
||||
*) echo "unknown option $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ "$1" == "--remove" ]]; then
|
||||
restart_agent() {
|
||||
launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
|
||||
rm -f "$PLIST"
|
||||
echo "removed $LABEL (files left in $DEST)"
|
||||
exit 0
|
||||
fi
|
||||
# bootout returns before the old job is gone; bootstrapping too early fails with "5: Input/output error".
|
||||
for i in {1..50}; do launchctl print "$DOMAIN/$LABEL" >/dev/null 2>&1 || break; sleep 0.2; done
|
||||
for attempt in 1 2 3; do
|
||||
launchctl bootstrap "$DOMAIN" "$PLIST" && break
|
||||
[[ $attempt == 3 ]] && { echo "launchctl bootstrap failed" >&2; exit 1; }
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
/usr/bin/python3 -c "import json,sys; json.load(open(sys.argv[1]))" "$SRC/config.json"
|
||||
mkdir -p "$DEST"
|
||||
if [[ -f "$DEST/config.json" ]] && ! cmp -s "$SRC/config.json" "$DEST/config.json"; then
|
||||
cp "$DEST/config.json" "$DEST/config.json.bak-$(date +%Y%m%d-%H%M%S)"
|
||||
fi
|
||||
cp "$SRC/status_panel.py" "$SRC/coremidi.py" "$SRC/netgear.py" "$SRC/config.json" "$SRC/atem_watch.js" "$DEST/"
|
||||
[[ -f "$SRC/atem-reference.xml" ]] && cp "$SRC/atem-reference.xml" "$DEST/"
|
||||
|
||||
cat > "$PLIST" <<EOF
|
||||
write_plist() { # $1 = exit timeout in seconds
|
||||
cat > "$PLIST" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
@@ -43,18 +54,81 @@ cat > "$PLIST" <<EOF
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>ThrottleInterval</key><integer>10</integer>
|
||||
<key>ExitTimeOut</key><integer>$1</integer>
|
||||
<key>StandardErrorPath</key><string>$HOME/Library/Logs/status-panel.err.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
}
|
||||
|
||||
launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
|
||||
# bootout returns before the old job is gone; bootstrapping too early fails with "5: Input/output error".
|
||||
for i in {1..50}; do launchctl print "$DOMAIN/$LABEL" >/dev/null 2>&1 || break; sleep 0.2; done
|
||||
for attempt in 1 2 3; do
|
||||
launchctl bootstrap "$DOMAIN" "$PLIST" && break
|
||||
[[ $attempt == 3 ]] && { echo "launchctl bootstrap failed" >&2; exit 1; }
|
||||
sleep 1
|
||||
done
|
||||
echo "installed $LABEL -> $DEST"
|
||||
wait_for_start() { # $1 = install time (unix): wait for a start stamp newer than that
|
||||
for i in {1..60}; do
|
||||
if [[ -f "$DEST/started.json" ]] && /usr/bin/python3 - "$DEST/started.json" "$1" <<'PY' 2>/dev/null
|
||||
import json, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
sys.exit(0 if d.get("started_at", 0) >= float(sys.argv[2]) else 1)
|
||||
PY
|
||||
then
|
||||
echo "running (code $(/usr/bin/python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("version"))' "$DEST/started.json"))"
|
||||
return 0
|
||||
fi
|
||||
sleep 0.5
|
||||
done
|
||||
echo "WARNING: no start stamp after 30 s - check ~/Library/Logs/status-panel.log and status-panel.err.log" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ "$1" == "--remove" ]]; then
|
||||
launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
|
||||
rm -f "$PLIST"
|
||||
echo "removed $LABEL (files left in $DEST)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$1" == "--rollback" ]]; then
|
||||
[[ -d "$DEST/prev" ]] || { echo "nothing to roll back to ($DEST/prev is missing)" >&2; exit 1; }
|
||||
for f in "$DEST"/prev/*(N); do cp "$f" "$DEST/"; done
|
||||
/usr/bin/python3 "$DEST/status_panel.py" --check --config "$DEST/config.json"
|
||||
T=$(date +%s)
|
||||
write_plist "$(/usr/bin/python3 "$DEST/status_panel.py" --exit-timeout --config "$DEST/config.json")"
|
||||
restart_agent
|
||||
echo "rolled back to the previous code (version $(/usr/bin/python3 -c 'import sys; sys.path.insert(0, sys.argv[1]); import status_panel; print(status_panel.VERSION)' "$DEST"))"
|
||||
wait_for_start "$T" || true
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# 1. The config that will run, checked with the code that will run it.
|
||||
mkdir -p "$DEST"
|
||||
if [[ $KEEP_CONFIG == 1 && -f "$DEST/config.json" ]]; then
|
||||
CFG="$DEST/config.json"
|
||||
else
|
||||
CFG="$SRC/config.json"
|
||||
fi
|
||||
/usr/bin/python3 "$SRC/status_panel.py" --check --config "$CFG"
|
||||
EXIT_TIMEOUT=$(/usr/bin/python3 "$SRC/status_panel.py" --exit-timeout --config "$CFG")
|
||||
|
||||
# 2. Keep the code that runs now, for --rollback.
|
||||
mkdir -p "$DEST/prev"
|
||||
for f in $CODE; do [[ -f "$DEST/$f" ]] && cp "$DEST/$f" "$DEST/prev/"; done
|
||||
|
||||
# 3. Files. The live config is backed up when it changes; --keep-config leaves it alone.
|
||||
if [[ $KEEP_CONFIG == 0 ]]; then
|
||||
if [[ -f "$DEST/config.json" ]] && ! cmp -s "$SRC/config.json" "$DEST/config.json"; then
|
||||
cp "$DEST/config.json" "$DEST/config.json.bak-$(date +%Y%m%d-%H%M%S)"
|
||||
echo "config.json replaced (the old one is kept as config.json.bak-...; use --keep-config to leave it)"
|
||||
fi
|
||||
cp "$SRC/config.json" "$DEST/"
|
||||
fi
|
||||
for f in $CODE; do [[ -f "$SRC/$f" ]] && cp "$SRC/$f" "$DEST/"; done
|
||||
[[ -f "$SRC/atem-reference.xml" ]] && cp "$SRC/atem-reference.xml" "$DEST/"
|
||||
if [[ -d "$SRC/companion-objects" ]]; then
|
||||
rm -rf "$DEST/companion-objects" && cp -R "$SRC/companion-objects" "$DEST/"
|
||||
fi
|
||||
|
||||
# 4. The LaunchAgent, with the exit timeout this config needs (a running fix finishes before a restart).
|
||||
T=$(date +%s)
|
||||
write_plist "$EXIT_TIMEOUT"
|
||||
restart_agent
|
||||
echo "installed $LABEL -> $DEST (exit timeout ${EXIT_TIMEOUT}s)"
|
||||
echo "log: ~/Library/Logs/status-panel.log"
|
||||
wait_for_start "$T" || true
|
||||
|
||||
+85
-44
@@ -17,6 +17,7 @@ Buttons write commands into sp_cmd, which status_panel.py reads:
|
||||
Restart page force <key> (hold 2 s: restart that thing even when it looks fine)
|
||||
|
||||
python3 make_page.py -> status-page.companionconfig and restart-page.companionconfig
|
||||
python3 make_page.py --out DIR -> the same files in DIR (the data dir on the Ingest Mac)
|
||||
"""
|
||||
import gzip
|
||||
import hashlib
|
||||
@@ -28,12 +29,25 @@ HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
from status_panel import RESTARTABLE, load_config # noqa: E402
|
||||
|
||||
# Defaults; "companion", "page_grid", "page_stale_seconds" and "page_hold_ms" in config.json override them.
|
||||
EXPORT_VERSION = 12 # Companion 5.0.7's export format
|
||||
COMPANION_BUILD = "5.0.7+9763-stable-cec2f88e6f"
|
||||
GRID = {"minColumn": 0, "maxColumn": 8, "minRow": 0, "maxRow": 5}
|
||||
STALE_SECONDS = 15
|
||||
PURPLE = 0x6A1B9A
|
||||
HOLD_MS = 2000
|
||||
# Every key this tool makes carries one of these in its notes; push_page.py overwrites a page only
|
||||
# when every key on it (besides Companion's own page keys) does.
|
||||
MARKERS = ("STATUS tile", "RESTART key", "STATUS jump key", "RESTART back key", "STATUS nav key")
|
||||
|
||||
|
||||
def settings(cfg):
|
||||
"""The page-building numbers, from config.json where set."""
|
||||
comp = cfg.get("companion", {})
|
||||
return {"export_version": comp.get("export_version", EXPORT_VERSION), "build": comp.get("build", COMPANION_BUILD),
|
||||
"grid": cfg.get("page_grid", GRID), "stale": cfg.get("page_stale_seconds", STALE_SECONDS),
|
||||
"hold_ms": cfg.get("page_hold_ms", HOLD_MS), "flash": cfg.get("variables", {}).get("flash", "flash_tick"),
|
||||
"ingest": cfg.get("site", {}).get("ingest_name", "NA Ingest")}
|
||||
|
||||
ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-"
|
||||
|
||||
@@ -48,9 +62,10 @@ def prop(value, expression=False):
|
||||
return {"value": value, "isExpression": expression}
|
||||
|
||||
|
||||
def layers(tid, tile, names, stale_text):
|
||||
state, hb = names["state"], names["heartbeat"]
|
||||
stale = "$(internal:time_unix) - $(custom:%s) > %d" % (hb, STALE_SECONDS)
|
||||
def layers(tid, tile, names, stale_text, opts=None):
|
||||
opts = opts or settings({"variables": names})
|
||||
state, hb, flash = names["state"], names["heartbeat"], opts["flash"]
|
||||
stale = "$(internal:time_unix) - $(custom:%s) > %d" % (hb, opts["stale"])
|
||||
|
||||
def pick(on_key, off_key):
|
||||
return "jsonpath($(custom:%s), '$.%s.%s')" % (state, tid, on_key), \
|
||||
@@ -58,8 +73,8 @@ def layers(tid, tile, names, stale_text):
|
||||
|
||||
b_on, b_off = pick("b", "b2")
|
||||
f_on, f_off = pick("f", "f2")
|
||||
bg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, PURPLE, b_on, b_off)
|
||||
fg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, 0xFFFFFF, f_on, f_off)
|
||||
bg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, PURPLE, flash, b_on, b_off)
|
||||
fg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, 0xFFFFFF, flash, f_on, f_off)
|
||||
text = "%s ? %s : jsonpath($(custom:%s), '$.%s.x')" % (stale, stale_text, state, tid)
|
||||
# The watchdog picks a size per key so words never break mid-word.
|
||||
size = "%s ? 20 : jsonpath($(custom:%s), '$.%s.s')" % (stale, state, tid)
|
||||
@@ -102,8 +117,8 @@ def set_page_action(tid, role, page):
|
||||
"options": {"surfaceId": prop("self"), "page": prop(page)}, "children": {}}
|
||||
|
||||
|
||||
def static_button(tid, text, bg, fg, page, note):
|
||||
"""A plain key (no watchdog state): tap goes to `page`."""
|
||||
def static_button(tid, text, bg, fg, page, note, marker):
|
||||
"""A plain key (no watchdog state): tap goes to `page`. `marker` names it as ours in its notes."""
|
||||
geometry = {"enabled": prop(True), "opacity": prop(100), "x": prop(0), "y": prop(0),
|
||||
"width": prop(100), "height": prop(100), "rotation": prop(0)}
|
||||
layers = [
|
||||
@@ -120,7 +135,7 @@ def static_button(tid, text, bg, fg, page, note):
|
||||
"type": "button-layered",
|
||||
"style": {"layers": layers},
|
||||
"options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
|
||||
"canModifyStyleInApis": False, "notes": note},
|
||||
"canModifyStyleInApis": False, "notes": "%s - %s" % (marker, note)},
|
||||
"feedbacks": [],
|
||||
"steps": {"0": {"action_sets": {"down": [], "up": [set_page_action(tid, "go", page)]},
|
||||
"options": {"runWhileHeld": []}}},
|
||||
@@ -129,27 +144,29 @@ def static_button(tid, text, bg, fg, page, note):
|
||||
|
||||
|
||||
def restart_key(key, cfg):
|
||||
"""A Restart page key: drawn by the watchdog like a tile; hold 2 s = "force <id>"."""
|
||||
names = cfg["variables"]
|
||||
"""A Restart page key: drawn by the watchdog like a tile; hold = "force <id>"."""
|
||||
names, opts = cfg["variables"], settings(cfg)
|
||||
hold_ms, secs = opts["hold_ms"], "%g" % (opts["hold_ms"] / 1000.0)
|
||||
stale_text = "'%s\\n?'" % key["label"]
|
||||
return {
|
||||
"type": "button-layered",
|
||||
"style": {"layers": layers(key["id"], key, names, stale_text)},
|
||||
"style": {"layers": layers(key["id"], key, names, stale_text, opts)},
|
||||
"options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
|
||||
"canModifyStyleInApis": True,
|
||||
"notes": "RESTART key '%s' - hold 2 s: restart it even when the status page says it is fine "
|
||||
"(status_panel.py on NA Ingest runs fix '%s')." % (key["id"], key["fix"])},
|
||||
"notes": "RESTART key '%s' - hold %s s: restart it even when the status page says it is fine "
|
||||
"(status_panel.py on %s runs fix '%s')." % (key["id"], secs, opts["ingest"], key["fix"])},
|
||||
"feedbacks": [],
|
||||
"steps": {"0": {"action_sets": {"down": [], "up": [],
|
||||
str(HOLD_MS): [command_action(key["id"], "hold", names["command"],
|
||||
str(hold_ms): [command_action(key["id"], "hold", names["command"],
|
||||
"force %s $(internal:time_unix)" % key["id"])]},
|
||||
"options": {"runWhileHeld": [HOLD_MS]}}},
|
||||
"options": {"runWhileHeld": [hold_ms]}}},
|
||||
"localVariables": [],
|
||||
}
|
||||
|
||||
|
||||
def button(tid, tile, cfg):
|
||||
names = cfg["variables"]
|
||||
names, opts = cfg["variables"], settings(cfg)
|
||||
hold_ms, secs = opts["hold_ms"], "%g" % (opts["hold_ms"] / 1000.0)
|
||||
kind = tile.get("kind")
|
||||
label = tile["label"]
|
||||
if kind == "master":
|
||||
@@ -168,24 +185,26 @@ def button(tid, tile, cfg):
|
||||
for c in tile.get("checks", []))
|
||||
hold = ("restart " + tid) if fixable or tile.get("hold_always") or kind == "network" else None
|
||||
if kind == "network":
|
||||
note = "Press: stop the flashing. Hold 2 s: clear a network blip (it also clears itself after 5 minutes)."
|
||||
minutes = cfg.get("network", {}).get("remember_seconds", 300) // 60
|
||||
note = ("Press: stop the flashing. Hold %s s: clear a network blip (it also clears itself after %d minutes)."
|
||||
% (secs, minutes))
|
||||
elif tile.get("hold_always"):
|
||||
note = "Press: acknowledge. Hold 2 s: %s." % cfg["fixes"][tile["hold_always"]]["label"]
|
||||
note = "Press: acknowledge. Hold %s s: %s." % (secs, cfg["fixes"][tile["hold_always"]]["label"])
|
||||
else:
|
||||
note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)."
|
||||
if hold else "")
|
||||
note = "Press: acknowledge." + (" Hold %s s: fix what this button is failing on (nothing if it is green)."
|
||||
% secs if hold else "")
|
||||
action_sets = {"down": [], "up": [command_action(tid, "press", names["command"], press)]}
|
||||
run_while_held = []
|
||||
if hold:
|
||||
# Stamped with Companion's clock, so a hold made while the watchdog was down is dropped later.
|
||||
action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")]
|
||||
run_while_held = [HOLD_MS]
|
||||
action_sets[str(hold_ms)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")]
|
||||
run_while_held = [hold_ms]
|
||||
return {
|
||||
"type": "button-layered",
|
||||
"style": {"layers": layers(tid, tile, names, stale_text)},
|
||||
"style": {"layers": layers(tid, tile, names, stale_text, opts)},
|
||||
"options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
|
||||
"canModifyStyleInApis": True,
|
||||
"notes": "STATUS tile '%s' - drawn by status_panel.py on NA Ingest. %s" % (tid, note)},
|
||||
"notes": "STATUS tile '%s' - drawn by status_panel.py on %s. %s" % (tid, opts["ingest"], note)},
|
||||
"feedbacks": [],
|
||||
"steps": {"0": {"action_sets": action_sets, "options": {"runWhileHeld": run_while_held}}},
|
||||
"localVariables": [],
|
||||
@@ -196,7 +215,18 @@ NAV_LOCATIONS = {"pageup": ("0", "0"), "pagenum": ("1", "0"), "pagedown": ("2",
|
||||
NAV_MARKER = "STATUS nav key"
|
||||
|
||||
|
||||
def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1):
|
||||
def nav_templates(cfg):
|
||||
"""Companion's own nav key designs for the version in use: companion-objects/nav_templates/<major.minor>.json
|
||||
when that exists, else this folder's nav_templates.json (captured from 5.0.7)."""
|
||||
version = ".".join(str(settings(cfg)["build"]).split("+")[0].split(".")[:2])
|
||||
for path in (os.path.join(HERE, "companion-objects", "nav_templates", version + ".json"),
|
||||
os.path.join(HERE, "nav_templates.json")):
|
||||
if os.path.exists(path):
|
||||
return json.load(open(path))
|
||||
raise FileNotFoundError("no nav key templates for Companion %s" % version)
|
||||
|
||||
|
||||
def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1, cfg=None):
|
||||
"""Companion's own UP / page number / DOWN key, coloured like the ALL button.
|
||||
|
||||
For the page number key, status_page adds: tap = go to tap_page (on release,
|
||||
@@ -204,12 +234,13 @@ def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1):
|
||||
status page. The tap names the page outright: the converted key's "startup"
|
||||
means the current page on surfaces set to "use last page".
|
||||
"""
|
||||
templates = json.load(open(os.path.join(HERE, "nav_templates.json")))
|
||||
key = json.loads(json.dumps(templates[kind]))
|
||||
stale = "$(internal:time_unix) - $(custom:%s) > %d" % (names["heartbeat"], STALE_SECONDS)
|
||||
cfg = cfg or {"variables": names}
|
||||
opts = settings(cfg)
|
||||
key = json.loads(json.dumps(nav_templates(cfg)[kind]))
|
||||
stale = "$(internal:time_unix) - $(custom:%s) > %d" % (names["heartbeat"], opts["stale"])
|
||||
pick = lambda field: "jsonpath($(custom:%s), '$.all.%s')" % (names["state"], field)
|
||||
bg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, PURPLE, pick("b"), pick("b2"))
|
||||
fg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, 0xFFFFFF, pick("f"), pick("f2"))
|
||||
bg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, PURPLE, opts["flash"], pick("b"), pick("b2"))
|
||||
fg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, 0xFFFFFF, opts["flash"], pick("f"), pick("f2"))
|
||||
|
||||
def paint(layers):
|
||||
for layer in layers:
|
||||
@@ -240,7 +271,7 @@ def nav_controls(cfg):
|
||||
coloured = cfg.get("nav_keys", list(NAV_LOCATIONS)) # which nav keys follow the ALL button
|
||||
for kind, (r, c) in NAV_LOCATIONS.items():
|
||||
controls.setdefault(r, {})[c] = (nav_key(kind, cfg["variables"], cfg["page"], cfg.get("nav_hold_ms", 1000),
|
||||
cfg.get("nav_tap_page", 1))
|
||||
cfg.get("nav_tap_page", 1), cfg)
|
||||
if kind in coloured else {"type": kind})
|
||||
return controls
|
||||
|
||||
@@ -252,12 +283,13 @@ def place(controls, pos, control):
|
||||
controls.setdefault(r, {})[c] = control
|
||||
|
||||
|
||||
def page_export(name, number, controls):
|
||||
def page_export(name, number, controls, cfg=None):
|
||||
opts = settings(cfg or {})
|
||||
return {
|
||||
"version": EXPORT_VERSION,
|
||||
"version": opts["export_version"],
|
||||
"type": "page",
|
||||
"companionBuild": COMPANION_BUILD,
|
||||
"page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": GRID},
|
||||
"companionBuild": opts["build"],
|
||||
"page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": opts["grid"]},
|
||||
"instances": {},
|
||||
"connectionCollections": [],
|
||||
"oldPageNumber": number,
|
||||
@@ -274,18 +306,20 @@ def build(cfg):
|
||||
rp = cfg.get("restart_page")
|
||||
if rp:
|
||||
place(controls, rp["jump_pos"], static_button("restart_jump", "RESTART\\nPAGE", 0x37474F, 0xFFFFFF,
|
||||
rp["page"], "Tap: open the Restart page (page %d)." % rp["page"]))
|
||||
return page_export(cfg["page_name"], cfg["page"], controls)
|
||||
rp["page"], "Tap: open the Restart page (page %d)." % rp["page"],
|
||||
"STATUS jump key"))
|
||||
return page_export(cfg["page_name"], cfg["page"], controls, cfg)
|
||||
|
||||
|
||||
def build_restart(cfg):
|
||||
rp = cfg["restart_page"]
|
||||
controls = nav_controls(cfg)
|
||||
place(controls, rp["back_pos"], static_button("restart_back", "STATUS\\nPAGE", 0x37474F, 0xFFFFFF, cfg["page"],
|
||||
"Tap: back to the status page (page %d)." % cfg["page"]))
|
||||
"Tap: back to the status page (page %d)." % cfg["page"],
|
||||
"RESTART back key"))
|
||||
for key in rp["keys"]:
|
||||
place(controls, key["pos"], restart_key(key, cfg))
|
||||
return page_export(rp["page_name"], rp["page"], controls)
|
||||
return page_export(rp["page_name"], rp["page"], controls, cfg)
|
||||
|
||||
|
||||
def write(path, export):
|
||||
@@ -293,13 +327,20 @@ def write(path, export):
|
||||
f.write(json.dumps(export, indent=1).encode())
|
||||
|
||||
|
||||
def main():
|
||||
cfg = load_config(os.path.join(HERE, "config.json"))
|
||||
out = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "status-page.companionconfig")
|
||||
def main(argv=None):
|
||||
import argparse
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--config", default=os.path.join(HERE, "config.json"))
|
||||
ap.add_argument("--out", help="folder for the page files (default: this folder)")
|
||||
ap.add_argument("status_file", nargs="?", help="write only the status page, to this file")
|
||||
args = ap.parse_args(argv)
|
||||
cfg = load_config(args.config)
|
||||
folder = args.out or HERE
|
||||
out = args.status_file or os.path.join(folder, "status-page.companionconfig")
|
||||
write(out, build(cfg))
|
||||
print("wrote %s (%d buttons)" % (out, sum(1 for t in cfg["tiles"] if "pos" in t)))
|
||||
if cfg.get("restart_page") and len(sys.argv) <= 1:
|
||||
out = os.path.join(HERE, "restart-page.companionconfig")
|
||||
if cfg.get("restart_page") and not args.status_file:
|
||||
out = os.path.join(folder, "restart-page.companionconfig")
|
||||
write(out, build_restart(cfg))
|
||||
print("wrote %s (%d restart keys)" % (out, len(cfg["restart_page"]["keys"])))
|
||||
|
||||
|
||||
+2
-2
@@ -78,8 +78,8 @@ def find_nav_keys(host, port, status_page):
|
||||
def main():
|
||||
cfg = json.load(open(os.path.join(HERE, "config.json")))
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--host", default="na-ingest.local")
|
||||
ap.add_argument("--port", type=int, default=8000)
|
||||
ap.add_argument("--host", default=cfg.get("site", {}).get("ingest_hostname") or cfg["companion"].get("host", "127.0.0.1"))
|
||||
ap.add_argument("--port", type=int, default=cfg["companion"].get("port", 8000))
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--revert", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
+15
-2
@@ -18,10 +18,23 @@ import subprocess
|
||||
import sys
|
||||
import telnetlib # stdlib in the macOS python3 (3.9)
|
||||
|
||||
SWITCHES = ["10.66.1.100", "10.66.1.102"]
|
||||
SWITCHES = ["10.66.1.100", "10.66.1.102"] # the CLI's default; the watchdog passes "dante_switches" from config
|
||||
KEYCHAIN_SERVICE = "netgear-status-panel"
|
||||
|
||||
|
||||
def config_switches():
|
||||
""""dante_switches" from this folder's config.json, for the command line (the watchdog passes its own)."""
|
||||
import json, os
|
||||
try:
|
||||
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "config.json")) as f:
|
||||
cfg = json.load(f)
|
||||
global MAX_SHARED
|
||||
MAX_SHARED = cfg.get("dante_max_shared", MAX_SHARED)
|
||||
return cfg.get("dante_switches") or SWITCHES
|
||||
except (OSError, ValueError):
|
||||
return SWITCHES
|
||||
|
||||
|
||||
def credentials(service=KEYCHAIN_SERVICE):
|
||||
attrs = subprocess.run(["/usr/bin/security", "find-generic-password", "-s", service],
|
||||
capture_output=True, text=True).stdout
|
||||
@@ -71,7 +84,7 @@ class Switch:
|
||||
def lookup(macs):
|
||||
user, password = credentials()
|
||||
wanted = {m.lower(): m for m in macs}
|
||||
for host in SWITCHES:
|
||||
for host in config_switches():
|
||||
sw = Switch(host, user, password)
|
||||
try:
|
||||
model = [l.strip() for l in sw.run("show version").splitlines()
|
||||
|
||||
+36
-23
@@ -5,9 +5,10 @@ Does exactly what Import/Export -> Import -> "import into page N" does, over the
|
||||
same internal API the Companion web UI uses (tRPC over a WebSocket at /trpc):
|
||||
upload the file in chunks, check its SHA-1, then import it as a single page.
|
||||
|
||||
Safety: it looks at what is on the target page, not just its name. It refuses unless the
|
||||
page holds nothing but page up / page number / page down keys, or is already the page this
|
||||
file builds (same name). A page named "PAGE" can still hold real buttons (pages 22 and 97 do).
|
||||
Safety: it looks at what is on the target page, not just its name. It refuses unless every
|
||||
key on the page (besides Companion's own page up / number / down keys) carries one of the
|
||||
markers make_page.py writes into its notes, i.e. the page holds nothing a person made by hand.
|
||||
A key someone added is named by its position, so it can be moved first.
|
||||
|
||||
python3 push_page.py # the status page, from config.json
|
||||
python3 push_page.py --which restart # the Restart page
|
||||
@@ -29,6 +30,9 @@ import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
from make_page import MARKERS # noqa: E402
|
||||
|
||||
CHUNK = 256 * 1024
|
||||
|
||||
|
||||
@@ -124,8 +128,10 @@ NAV_MARKER = "STATUS nav key" # our coloured page number key (make_page
|
||||
|
||||
|
||||
def fetch_page(host, port, page):
|
||||
"""The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}})."""
|
||||
with urllib.request.urlopen("http://%s:%d/int/export/page/%d" % (host, port, page), timeout=10) as r:
|
||||
"""The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}}),
|
||||
without any connection secrets."""
|
||||
url = "http://%s:%d/int/export/page/%d?includeSecrets=false" % (host, port, page)
|
||||
with urllib.request.urlopen(url, timeout=10) as r:
|
||||
raw = r.read()
|
||||
try:
|
||||
raw = gzip.decompress(raw)
|
||||
@@ -134,29 +140,36 @@ def fetch_page(host, port, page):
|
||||
return json.loads(raw)
|
||||
|
||||
|
||||
def real_buttons(export):
|
||||
"""How many buttons a page holds besides its page up / number / down keys."""
|
||||
def foreign_buttons(export):
|
||||
"""[(row, col)] of every key on the page that neither Companion nor this tool made."""
|
||||
page = export.get("page") or {}
|
||||
n = 0
|
||||
for row in (page.get("controls") or {}).values():
|
||||
for ctl in (row or {}).values():
|
||||
found = []
|
||||
for r, row in sorted((page.get("controls") or {}).items(), key=lambda kv: int(kv[0])):
|
||||
for c, ctl in sorted((row or {}).items(), key=lambda kv: int(kv[0])):
|
||||
if not ctl or ctl.get("type") in NAV_TYPES:
|
||||
continue
|
||||
if NAV_MARKER in ((ctl.get("options") or {}).get("notes") or ""):
|
||||
notes = (ctl.get("options") or {}).get("notes") or ""
|
||||
if any(m in notes for m in MARKERS):
|
||||
continue
|
||||
n += 1
|
||||
return n
|
||||
found.append((int(r), int(c)))
|
||||
return found
|
||||
|
||||
|
||||
def safe_target(export, expected_name):
|
||||
"""(ok, reason). OK only for a page with nothing but nav keys, or the page this file builds."""
|
||||
def real_buttons(export):
|
||||
"""How many keys a page holds that this tool did not make."""
|
||||
return len(foreign_buttons(export))
|
||||
|
||||
|
||||
def safe_target(export, expected_name=None):
|
||||
"""(ok, reason). OK only when every key on the page is Companion's own page key or one of ours,
|
||||
whatever the page is called: nothing a person made by hand is ever overwritten."""
|
||||
name = ((export.get("page") or {}).get("name") or "")
|
||||
if name == expected_name:
|
||||
return True, "it is already %r" % name
|
||||
n = real_buttons(export)
|
||||
if n == 0:
|
||||
return True, "it holds only page keys"
|
||||
return False, "it is called %r and holds %d buttons" % (name, n)
|
||||
foreign = foreign_buttons(export)
|
||||
if not foreign:
|
||||
return True, "it holds only page keys and keys made by this tool"
|
||||
where = ", ".join("row %d col %d" % rc for rc in foreign[:6]) + (" ..." if len(foreign) > 6 else "")
|
||||
return False, "it is called %r and holds %d key%s not made by this tool (%s) - move or delete them first" % (
|
||||
name, len(foreign), "" if len(foreign) == 1 else "s", where)
|
||||
|
||||
|
||||
def page_name(host, port, page):
|
||||
@@ -169,8 +182,8 @@ def main():
|
||||
cfg = json.load(open(os.path.join(HERE, "config.json")))
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--which", choices=("status", "restart"), default="status")
|
||||
ap.add_argument("--host", default="na-ingest.local")
|
||||
ap.add_argument("--port", type=int, default=8000)
|
||||
ap.add_argument("--host", default=cfg.get("site", {}).get("ingest_hostname") or cfg["companion"].get("host", "127.0.0.1"))
|
||||
ap.add_argument("--port", type=int, default=cfg["companion"].get("port", 8000))
|
||||
ap.add_argument("--page", type=int)
|
||||
ap.add_argument("--file")
|
||||
ap.add_argument("--force", action="store_true", help="overwrite the page whatever is on it")
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,658 @@
|
||||
"""The registry of every setting the status panel has.
|
||||
|
||||
One record per config key (path, type, default, plain-words help, which screen of the settings
|
||||
page owns it, whether it sits under Advanced), the check types with their fields, the fix kinds,
|
||||
the words a key can show, and the rules a config must pass. The settings page draws its forms
|
||||
from this file; tests/test_schema.py keeps it in step with the code (every key the watchdog reads
|
||||
must be declared here, every default in the code must equal the one here).
|
||||
|
||||
Paths: "companion.host", "tiles[].checks[].conn" (a list element), "fixes.*.run" (a map entry).
|
||||
"""
|
||||
import copy
|
||||
import re
|
||||
|
||||
import status_panel as sp
|
||||
|
||||
# --- Field records -----------------------------------------------------------
|
||||
# type: str | int | float | bool | list | dict | enum | path | ip | mac | url | hhmm | days | id
|
||||
# apply: what changes when the value does: "restart" (the watchdog restarts), "pages" (the Companion
|
||||
# pages are rebuilt and re-imported when the built page differs), "none" (page-only data), "settings".
|
||||
|
||||
F = {}
|
||||
|
||||
|
||||
def field(path, type, default=None, help="", screen="", advanced=False, secret=False, apply="restart",
|
||||
required=False, options=None, lo=None, hi=None, regex=None, example=None, shows=None):
|
||||
F[path] = {"path": path, "type": type, "default": default, "help": help, "screen": screen,
|
||||
"advanced": advanced, "secret": secret, "apply": apply, "required": required,
|
||||
"options": options, "lo": lo, "hi": hi, "regex": regex, "example": example, "shows": shows}
|
||||
|
||||
|
||||
# -- site --
|
||||
field("site", "dict", {}, "This campus: its name, this Mac, and when changes must wait.", "Site")
|
||||
field("site.id", "id", "na", "Short name used for folders and files.", "Site", advanced=True, regex=r"[a-z0-9_-]+")
|
||||
field("site.name", "str", "", "The campus, as people say it.", "Site", example="North Attleboro", apply="none")
|
||||
field("site.ingest_name", "str", "NA Ingest", "What this Mac is called in logs, notes and alerts.", "Site", apply="pages")
|
||||
field("site.ingest_hostname", "str", "", "This Mac's network name, used in the page address and by the page tools.",
|
||||
"Site", example="na-ingest.local", apply="none")
|
||||
field("site.ingest_ips", "list", [], "This Mac's addresses, for the PCs' firewall allow-lists.", "Site", apply="none")
|
||||
field("site.contacts", "str", "", "Printed on the cheat sheet: who to call.", "Site", apply="none")
|
||||
field("site.alert_prefix", "str", "", "The first words of an alert message; blank = this Mac's name.", "Site", advanced=True)
|
||||
field("site.live_services", "list", [], "Clock windows during which changes to the panel wait (a recording "
|
||||
"HyperDeck always counts as live, whatever the clock says).", "Schedules")
|
||||
field("site.live_services[].days", "days", None, "Days of the week.", "Schedules", required=True)
|
||||
field("site.live_services[].start", "hhmm", None, "Start time.", "Schedules", required=True)
|
||||
field("site.live_services[].end", "hhmm", None, "End time; earlier than the start means it runs past midnight.",
|
||||
"Schedules", required=True)
|
||||
field("site.editing_code", "str", "", "The code that switches Editing on. Kept here, in the repo, so everyone "
|
||||
"with the repo has it.", "Site", advanced=True, secret=True, apply="none")
|
||||
field("site.settings_port", "int", 8020, "Where the settings page listens.", "Site", advanced=True, lo=1024, hi=65535,
|
||||
apply="settings")
|
||||
field("site.dvs_helper", "path", "/usr/local/sbin/status-panel-dvs-start", "The DVS helper on each Mac that runs "
|
||||
"Dante Virtual Soundcard.", "Site", advanced=True)
|
||||
field("site.solo_dir", "path", "~/Library/Application Support/status-panel", "The Pro Tools helper's folder on "
|
||||
"the Studio B Mac.", "Site", advanced=True)
|
||||
|
||||
# -- companion --
|
||||
field("companion", "dict", {}, "Where Companion answers.", "Companion", required=True)
|
||||
field("companion.host", "str", "127.0.0.1", "Companion's address from this Mac.", "Companion", advanced=True)
|
||||
field("companion.port", "int", 8000, "Companion's HTTP port.", "Companion", advanced=True, lo=1, hi=65535)
|
||||
field("companion.db", "path", None, "Companion's settings database; blank = the newest one found.", "Companion",
|
||||
advanced=True)
|
||||
field("companion.export_version", "int", sp.__dict__.get("EXPORT_VERSION", 12), "Companion's page export format "
|
||||
"number (changes with a Companion upgrade).", "Companion", advanced=True, apply="pages")
|
||||
field("companion.build", "str", "5.0.7+9763-stable-cec2f88e6f", "The Companion build the pages are written for.",
|
||||
"Companion", advanced=True, apply="pages")
|
||||
field("page", "int", 19, "The Status page's number in Companion.", "Keys", lo=1, apply="pages")
|
||||
field("page_name", "str", "19\\nStatus", "The Status page's name in Companion.", "Keys", advanced=True, apply="pages")
|
||||
field("nav_keys", "list", ["pagenum"], "Which of Companion's page keys (page up, page number, page down) are "
|
||||
"coloured like the ALL key on every page.", "Companion", advanced=True, options=["pageup", "pagenum", "pagedown"],
|
||||
apply="pages")
|
||||
field("nav_hold_ms", "int", 1000, "Holding the page-number key this long jumps to the Status page.", "Companion",
|
||||
advanced=True, lo=200, hi=5000, apply="pages")
|
||||
field("nav_tap_page", "int", 1, "Tapping the page-number key goes to this page.", "Companion", advanced=True, lo=1,
|
||||
apply="pages")
|
||||
field("variables", "dict", {}, "The custom variables the panel uses inside Companion.", "Companion", required=True)
|
||||
field("variables.state", "str", "sp_state", "The variable the keys are drawn from.", "Companion", advanced=True,
|
||||
regex=r"[A-Za-z0-9_]+", apply="pages")
|
||||
field("variables.heartbeat", "str", "sp_hb", "The variable that says the watchdog is alive.", "Companion",
|
||||
advanced=True, regex=r"[A-Za-z0-9_]+", apply="pages")
|
||||
field("variables.command", "str", "sp_cmd", "The variable key presses are written to.", "Companion", advanced=True,
|
||||
regex=r"[A-Za-z0-9_]+", apply="pages")
|
||||
field("variables.flash", "str", "flash_tick", "The blink clock. Renaming it means renaming it inside the trigger "
|
||||
"in Companion too.", "Companion", advanced=True, regex=r"[A-Za-z0-9_]+", apply="pages")
|
||||
field("page_grid", "dict", {"minColumn": 0, "maxColumn": 8, "minRow": 0, "maxRow": 5}, "The export grid "
|
||||
"(the deck is the 8x4 inside it).", "Companion", advanced=True, apply="pages")
|
||||
field("page_stale_seconds", "int", 15, "Keys turn purple when the watchdog has been silent this long.",
|
||||
"Companion", advanced=True, lo=5, hi=120, apply="pages")
|
||||
field("page_hold_ms", "int", 2000, "How long a key is held to run its fix.", "Schedules", advanced=True, lo=500,
|
||||
hi=5000, apply="pages", shows="HOLD 2s")
|
||||
|
||||
# -- schedules and timing --
|
||||
field("service_windows", "list", [{"days": list(sp.DAYS), "start": "00:00", "end": "23:59"}],
|
||||
"When alarms are armed: always (the ALARMS key silences them until midnight).", "Schedules", advanced=True)
|
||||
field("service_windows[].days", "days", None, "", "Schedules", required=True)
|
||||
field("service_windows[].start", "hhmm", None, "", "Schedules", required=True)
|
||||
field("service_windows[].end", "hhmm", None, "", "Schedules", required=True)
|
||||
field("debounce_seconds", "int", 3, "A fault must last this long before it shows (one bad reading is not an "
|
||||
"alarm).", "Schedules", advanced=True, lo=0, hi=60)
|
||||
field("flap", "dict", {}, "Repeated faults show amber 'xN'.", "Schedules", advanced=True)
|
||||
field("flap.count", "int", 3, "This many faults...", "Schedules", advanced=True, lo=1, hi=20)
|
||||
field("flap.window_seconds", "int", 600, "...within this many seconds.", "Schedules", advanced=True, lo=60, hi=86400)
|
||||
field("timing", "dict", {}, "Timing numbers.", "Schedules", advanced=True)
|
||||
field("timing.fix_cooldown_seconds", "int", 60, "The same fix never re-runs within this.", "Schedules", advanced=True,
|
||||
lo=10, hi=600, shows="WAIT 1 MIN")
|
||||
field("timing.alert_after_seconds", "int", 30, "Send the alert message when Companion has not answered for this "
|
||||
"long.", "Schedules", advanced=True, lo=5, hi=600)
|
||||
field("alert_webhook", "url", "", "Where alert messages go (a Slack-style web hook); blank = off.", "Site",
|
||||
secret=True)
|
||||
field("auto_fix", "dict", {}, "When the panel may fix things by itself.", "Schedules")
|
||||
field("auto_fix.windows", "list", [], "When auto-fix may act.", "Schedules")
|
||||
field("auto_fix.windows[].days", "days", None, "", "Schedules", required=True)
|
||||
field("auto_fix.windows[].start", "hhmm", None, "", "Schedules", required=True)
|
||||
field("auto_fix.windows[].end", "hhmm", None, "", "Schedules", required=True)
|
||||
field("auto_fix.after_seconds", "int", 20, "A fault must last this long before auto-fix acts.", "Schedules",
|
||||
advanced=True, lo=5, hi=600)
|
||||
field("auto_fix.max_tries", "int", 2, "Tries per fault, then it is yours to hold.", "Schedules", advanced=True,
|
||||
lo=1, hi=10)
|
||||
field("auto_fix.retry_seconds", "int", 90, "Between tries.", "Schedules", advanced=True, lo=30, hi=3600)
|
||||
field("auto_fix.pause_after_network_seconds", "int", 60, "Wait this long after a network blip.", "Schedules",
|
||||
advanced=True, lo=0, hi=3600)
|
||||
field("auto_fix.forget_seconds", "int", 600, "Forget a fault's tries once it has been gone this long.",
|
||||
"Schedules", advanced=True, lo=60, hi=86400)
|
||||
field("auto_fix.restart_connections", "bool", False, "Also reconnect Companion connections by itself.", "Schedules")
|
||||
field("auto_fix.fixes", "list", [], "The fixes auto-fix may run (gentle ones only).", "Schedules")
|
||||
field("auto_fix.log_dir", "path", "~/Desktop/Auto-Fix Logs", "One file per day of everything auto-fix does.",
|
||||
"Schedules", advanced=True)
|
||||
field("auto_fix.keep_months", "int", 1, "Daily log files are deleted after this many months.", "Schedules",
|
||||
advanced=True, lo=1, hi=12)
|
||||
|
||||
# -- network and Dante --
|
||||
field("network", "dict", {}, "The NET key.", "Network & Dante")
|
||||
field("network.interface", "str", "en0", "The network port to watch for link.", "Network & Dante")
|
||||
field("network.gateway", "ip", "", "The router to ping; blank = off.", "Network & Dante")
|
||||
field("network.burst_connections", "int", 4, "A blip is this many connections...", "Network & Dante", advanced=True,
|
||||
lo=1, hi=50)
|
||||
field("network.burst_groups", "int", 3, "...from this many groups...", "Network & Dante", advanced=True, lo=1, hi=20)
|
||||
field("network.burst_window_seconds", "int", 10, "...dropping within this many seconds.", "Network & Dante",
|
||||
advanced=True, lo=1, hi=300)
|
||||
field("network.remember_seconds", "int", 300, "NET shows BLIP this long after.", "Network & Dante", advanced=True,
|
||||
lo=10, hi=3600, apply="pages")
|
||||
field("ssh_key", "path", "~/.ssh/id_status_panel", "The login key this Mac uses for the other computers.", "Site",
|
||||
advanced=True)
|
||||
field("dante_switches", "list", [], "The Netgear switches on the Dante network.", "Network & Dante")
|
||||
field("dante_map_seconds", "int", 600, "Survey the switches this often.", "Network & Dante", advanced=True, lo=60,
|
||||
hi=86400)
|
||||
field("dante_max_shared", "int", 2, "Never power-cycle a port carrying more than this many devices (an uplink).",
|
||||
"Network & Dante", advanced=True, lo=1, hi=10)
|
||||
|
||||
# -- Spotify and ATEM --
|
||||
field("spotify", "dict", {}, "The service playlist.", "Spotify")
|
||||
field("spotify.playlist", "str", "", "The playlist Spotify must be playing.", "Spotify", regex=r"spotify:playlist:[A-Za-z0-9]+",
|
||||
required=True)
|
||||
field("spotify.name", "str", "", "The playlist's name, as the keys and the key guide say it.", "Spotify")
|
||||
field("spotify.open_wait", "int", 8, "Seconds to wait after opening Spotify.", "Spotify", advanced=True, lo=1, hi=60)
|
||||
field("spotify.play_wait", "int", 3, "Seconds to wait before checking it plays.", "Spotify", advanced=True, lo=1, hi=60)
|
||||
field("atem", "dict", {}, "The switcher and its saved file.", "ATEM")
|
||||
field("atem.host", "ip", "", "The switcher's address.", "ATEM", required=True)
|
||||
field("atem.node", "path", "/Applications/Companion [0-9]*.app/Contents/Resources/node-runtimes/node22/bin/node",
|
||||
"Companion's node (pattern; the newest match is used).", "ATEM", advanced=True)
|
||||
field("atem.library", "path", "~/Library/Application Support/companion/modules/bmd-atem-*/node_modules/atem-connection",
|
||||
"Companion's ATEM library (pattern).", "ATEM", advanced=True)
|
||||
field("atem.reference", "path", "atem-reference.xml", "The saved switcher file to keep in line with.", "ATEM")
|
||||
field("atem.compare", "dict", None, "What to keep in line: audio fields and camera settings; blank = everything.",
|
||||
"ATEM", advanced=True)
|
||||
field("atem.source_folder", "path", "", "Where ATEM Software Control saves its files (for 'pick the newest').",
|
||||
"ATEM", apply="none")
|
||||
|
||||
# -- hosts --
|
||||
field("hosts", "dict", {}, "The computers this Mac logs in to.", "Computers")
|
||||
field("hosts.*.ssh", "str", None, "How this Mac logs in: user@address.", "Computers", regex=r"[^@\s]+@[^@\s]+")
|
||||
field("hosts.*.mac", "mac", None, "The PC's hardware address, to wake it when it is off.", "Computers")
|
||||
field("hosts.*.wake_broadcast", "ip", None, "The broadcast address the wake-up goes to.", "Computers", advanced=True)
|
||||
field("hosts.*.name", "str", None, "The computer, as people say it.", "Computers", apply="none")
|
||||
field("hosts.*.os", "enum", "mac", "Mac or Windows (a PC's checks are written for cmd.exe).", "Computers",
|
||||
options=["mac", "windows", "other"])
|
||||
field("hosts.*.role", "str", None, "What it is for.", "Computers", apply="none")
|
||||
field("hosts.*.notes", "str", None, "Notes for operators.", "Computers", apply="none")
|
||||
field("hosts.*.pingable", "bool", True, "Off for a PC whose firewall blocks ping.", "Computers", apply="none")
|
||||
field("hosts.*.self", "bool", False, "This Mac itself.", "Computers", apply="none")
|
||||
field("hosts.*.runs", "list", [], "What runs on it (programs, tasks, services, helpers).", "Computers", apply="none")
|
||||
field("hosts.*.networks", "list", [], "Its network ports and what each is for (LAN, Dante, switch management).",
|
||||
"Computers", advanced=True, apply="none")
|
||||
field("hosts.*.manual_done", "dict", {}, "Set-up steps ticked by hand (who, when).", "Computers", apply="none")
|
||||
field("gear", "dict", {}, "Devices that are not computers: where each one is.", "Computers", apply="none")
|
||||
field("gear.*.location", "str", None, "Where it is in the building.", "Computers", apply="none")
|
||||
field("gear.*.notes", "str", None, "Notes for operators.", "Computers", apply="none")
|
||||
|
||||
# -- fixes --
|
||||
field("fixes", "dict", {}, "Everything a hold can run.", "Fixes")
|
||||
field("fixes.*.kind", "enum", "ssh", "How it is built.", "Fixes", options=list(sp.FIX_KINDS))
|
||||
field("fixes.*.host", "str", None, "The computer it runs on (ssh fixes).", "Fixes")
|
||||
field("fixes.*.run", "str", None, "The command (zsh on Macs, cmd.exe on PCs).", "Fixes", advanced=True)
|
||||
field("fixes.*.timeout", "int", 60, "Give up after this many seconds.", "Fixes", advanced=True, lo=5, hi=600)
|
||||
field("fixes.*.short", "str", None, "The word the key shows while it runs (FIXING ...).", "Fixes", apply="pages")
|
||||
field("fixes.*.label", "str", None, "Plain words: what it does.", "Fixes", required=True, apply="pages")
|
||||
field("fixes.*.reconnect", "bool", False, "Reconnect the Companion connection afterwards.", "Fixes")
|
||||
field("fixes.*.conn", "str", None, "The Companion connection it reconnects.", "Fixes")
|
||||
field("fixes.*.restart_ok", "bool", False, "May close a running program (and still run by itself).", "Fixes")
|
||||
field("fixes.*.never_auto", "bool", False, "Never runs by itself.", "Fixes")
|
||||
field("fixes.*.part", "enum", None, "Which part of the switcher.", "Fixes", options=["audio", "camera"])
|
||||
field("fixes.*.verify_seconds", "int", 20, "Wait this long before judging a reconnect.", "Fixes", advanced=True,
|
||||
lo=5, hi=120)
|
||||
field("fixes.*.off_seconds", "int", 2, "How long a HyperDeck's connection stays off.", "Fixes", advanced=True,
|
||||
lo=1, hi=10)
|
||||
field("fixes.*.hung_note", "str", None, "What the key says when the device answers ping but not Companion.",
|
||||
"Fixes", advanced=True)
|
||||
field("fixes.*.switch", "ip", None, "The PoE switch.", "Fixes")
|
||||
field("fixes.*.port", "str", None, "The PoE port (e.g. 0/14).", "Fixes", regex=r"\d+/\d+")
|
||||
field("fixes.*.mac", "mac", None, "The device expected on that port.", "Fixes")
|
||||
field("fixes.*.playlist", "str", None, "A playlist other than the site's (rare).", "Fixes", advanced=True)
|
||||
field("fixes.*.template", "str", None, "The builder this fix was made with.", "Fixes", advanced=True, apply="none")
|
||||
field("fixes.*.params", "dict", None, "The builder's fields.", "Fixes", advanced=True, apply="none")
|
||||
|
||||
# -- tiles (keys) --
|
||||
field("tiles", "list", [], "The keys on the Status page.", "Keys", required=True)
|
||||
field("tiles[].id", "id", None, "Used inside Companion.", "Keys", advanced=True, required=True, regex=r"[A-Za-z0-9_]+",
|
||||
apply="pages")
|
||||
field("tiles[].label", "str", None, "The first line on the key.", "Keys", required=True, apply="pages")
|
||||
field("tiles[].pos", "list", None, "[row, column]; absent = no key of its own (shown inside a group).", "Keys",
|
||||
apply="pages")
|
||||
field("tiles[].kind", "enum", None, "A special key: master (ALL), service (ALARMS), network (NET).", "Keys",
|
||||
advanced=True, options=["master", "service", "network"], apply="pages")
|
||||
field("tiles[].members", "list", None, "A group: the keys it sums up.", "Keys")
|
||||
field("tiles[].checks", "list", [], "What the key watches.", "Keys")
|
||||
field("tiles[].hold_always", "str", None, "A fix a hold runs even when the key is green.", "Keys", apply="pages")
|
||||
field("tiles[].checks[].type", "enum", None, "What kind of check.", "Keys", required=True)
|
||||
field("tiles[].checks[].conn", "str", None, "The Companion connection.", "Keys")
|
||||
field("tiles[].checks[].fix", "str", None, "What holding the key runs for this check (a fix, or a table per "
|
||||
"sub-check).", "Keys")
|
||||
field("tiles[].checks[].tag", "str", None, "Names the device when one key covers several.", "Keys")
|
||||
field("tiles[].checks[].service_only", "bool", False, "Legacy: only during a service (has no effect while alarms "
|
||||
"are armed all day).", "Keys", advanced=True)
|
||||
field("tiles[].checks[].debounce_seconds", "int", None, "Wait before this check's fault shows.", "Keys", advanced=True,
|
||||
lo=0, hi=60)
|
||||
field("tiles[].checks[].ignore_warning", "bool", False, "Ignore Companion's yellow warning state.", "Keys")
|
||||
field("tiles[].checks[].optional", "bool", False, "A HyperDeck that may have no disk (grey, no alarm).", "Keys")
|
||||
field("tiles[].checks[].disk_service_only", "bool", True, "NO DISK only during a service.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].full_minutes", "int", 5, "FULL below this many minutes left.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].low_minutes", "int", 30, "LOW below this many minutes left.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].rec_sync", "bool", True, "Must record whenever another deck records.", "Keys")
|
||||
field("tiles[].checks[].ip", "ip", None, "The deck's address, instead of what Companion reports.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].devices", "dict", None, "Dante devices: name -> key text.", "Keys")
|
||||
field("tiles[].checks[].count_all", "bool", False, "Count every watched Dante device on this key.", "Keys")
|
||||
field("tiles[].checks[].channels", "int", 1, "Channels on the receiver.", "Keys", lo=1, hi=8)
|
||||
field("tiles[].checks[].battery_err_bars", "int", 1, "Red at this many bars or fewer.", "Keys", advanced=True, lo=0, hi=5)
|
||||
field("tiles[].checks[].battery_warn_bars", "int", 2, "Amber at this many bars or fewer.", "Keys", advanced=True, lo=0, hi=5)
|
||||
field("tiles[].checks[].muted_is_error", "bool", False, "RF mute is red, not amber.", "Keys")
|
||||
field("tiles[].checks[].host", "str", None, "The computer (an address for ping, a computer name for the rest).", "Keys")
|
||||
field("tiles[].checks[].name", "str", None, "The name the key shows.", "Keys")
|
||||
field("tiles[].checks[].msg", "str", None, "The words the key shows when it fails.", "Keys")
|
||||
field("tiles[].checks[].failures", "int", 2, "Pings missed before NO PING.", "Keys", advanced=True, lo=1, hi=10)
|
||||
field("tiles[].checks[].label", "str", None, "The launchd helper.", "Keys")
|
||||
field("tiles[].checks[].match", "str", None, "The program (pgrep -f pattern).", "Keys")
|
||||
field("tiles[].checks[].url", "url", None, "The web address to fetch.", "Keys")
|
||||
field("tiles[].checks[].interval_seconds", "int", 30, "Fetch it this often.", "Keys", advanced=True, lo=5, hi=3600)
|
||||
field("tiles[].checks[].path", "path", None, "The disk to watch.", "Keys")
|
||||
field("tiles[].checks[].warn_gb", "int", 20, "Amber below this many GB free.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].err_gb", "int", 5, "Red below this many GB free.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].session", "str", None, "The network MIDI session.", "Keys")
|
||||
field("tiles[].checks[].peer", "str", None, "The peer that must be in it.", "Keys")
|
||||
field("tiles[].checks[].address", "str", None, "Where to reconnect the peer if it was never seen.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].domain", "str", None, "The macOS preference domain.", "Keys")
|
||||
field("tiles[].checks[].key", "str", None, "The preference key.", "Keys")
|
||||
field("tiles[].checks[].want", "str", None, "The value it must have.", "Keys")
|
||||
field("tiles[].checks[].playlist", "str", None, "A playlist other than the site's (rare).", "Keys", advanced=True)
|
||||
field("tiles[].checks[].instance", "str", None, "The Bonjour MIDI name.", "Keys")
|
||||
field("tiles[].checks[].source", "str", None, "The MIDI source, or the Art-Net sender's address.", "Keys")
|
||||
field("tiles[].checks[].max_silence_seconds", "int", None, "Amber/red after this much silence.", "Keys", lo=1, hi=600)
|
||||
field("tiles[].checks[].cmd", "str", None, "The command run on that computer.", "Keys", advanced=True)
|
||||
field("tiles[].checks[].shell", "enum", None, "sh (Macs) or cmd (PCs); blank = from the computer's kind.", "Keys",
|
||||
advanced=True, options=["sh", "cmd"])
|
||||
field("tiles[].checks[].part", "enum", None, "Which part of the switcher.", "Keys", options=["audio", "camera"])
|
||||
field("tiles[].checks[].template", "str", None, "The builder this check was made with.", "Keys", advanced=True, apply="none")
|
||||
field("tiles[].checks[].params", "dict", None, "The builder's fields.", "Keys", advanced=True, apply="none")
|
||||
|
||||
# -- restart page --
|
||||
field("restart_page", "dict", None, "The Restart page; absent = this campus has none.", "Keys", apply="pages")
|
||||
field("restart_page.page", "int", 20, "Its number in Companion.", "Keys", lo=1, apply="pages")
|
||||
field("restart_page.page_name", "str", "20\\nRestart", "Its name in Companion.", "Keys", advanced=True, apply="pages")
|
||||
field("restart_page.jump_pos", "list", None, "Where the 'go to Restart page' key sits on the Status page.", "Keys",
|
||||
apply="pages")
|
||||
field("restart_page.back_pos", "list", None, "Where the 'back to Status' key sits on the Restart page.", "Keys",
|
||||
apply="pages")
|
||||
field("restart_page.keys", "list", [], "The restart keys.", "Keys", apply="pages")
|
||||
field("restart_page.keys[].id", "id", None, "", "Keys", advanced=True, required=True, regex=r"[A-Za-z0-9_]+", apply="pages")
|
||||
field("restart_page.keys[].label", "str", None, "", "Keys", required=True, apply="pages")
|
||||
field("restart_page.keys[].pos", "list", None, "[row, column].", "Keys", required=True, apply="pages")
|
||||
field("restart_page.keys[].fix", "str", None, "What holding it runs.", "Keys", required=True)
|
||||
field("restart_page.keys[].watch", "list", [], "Keep flashing HOLD TO FIX while these fixes' checks or "
|
||||
"'conn:<label>' connections are down.", "Keys")
|
||||
|
||||
# --- Check types ---------------------------------------------------------------
|
||||
# fields: the keys that type reads; subs: the sub-check names it returns (fix tables use them);
|
||||
# identity: what it is about; module: the Companion module its connection is expected to use.
|
||||
CHECKS = {
|
||||
"conn": {"fields": ["conn", "service_only", "ignore_warning"], "subs": ["conn"], "identity": ["conn"], "module": None,
|
||||
"question": "Is a device answering?", "words": "A Companion connection is good."},
|
||||
"hyperdeck": {"fields": ["conn", "optional", "disk_service_only", "full_minutes", "low_minutes", "rec_sync", "ip"],
|
||||
"subs": ["conn", "disk", "rec"], "identity": ["conn"], "module": "bmd-hyperdeck",
|
||||
"question": "Is a recorder OK?", "words": "A HyperDeck is connected, has a disk, and records with the others."},
|
||||
"dante": {"fields": ["conn", "devices", "count_all", "service_only"], "subs": ["conn", "<device name>"], "identity": ["conn"],
|
||||
"module": "audinate-dantecontroller", "question": "Is Dante complete?", "words": "Every listed Dante device is present."},
|
||||
"shure": {"fields": ["conn", "channels", "battery_err_bars", "battery_warn_bars"], "subs": ["conn", "ch<N>_batt", "ch<N>_rf"],
|
||||
"identity": ["conn"], "module": "shure-wireless", "question": "Is a device answering?", "words": "A Shure receiver: batteries and interference."},
|
||||
"psm": {"fields": ["conn", "channels", "muted_is_error"], "subs": ["conn", "ch<N>_mute"], "identity": ["conn"],
|
||||
"module": "shure-psm1000", "question": "Is a device answering?", "words": "A PSM transmitter: RF mute."},
|
||||
"tascam": {"fields": ["conn"], "subs": ["conn", "error", "caution"], "identity": ["conn"], "module": "tascam-da-6400",
|
||||
"question": "Is a recorder OK?", "words": "The Tascam recorder reports no error or caution."},
|
||||
"ping": {"fields": ["host", "conn", "name", "msg", "failures", "service_only"], "subs": ["ping"], "identity": ["conn", "host"],
|
||||
"module": None, "question": "Is a device answering?", "words": "An address answers ping."},
|
||||
"launchd": {"fields": ["label", "name"], "subs": ["svc"], "identity": ["label"], "module": None,
|
||||
"question": "Is a program running?", "words": "A helper (LaunchAgent) on this Mac is loaded and running."},
|
||||
"process": {"fields": ["match", "name", "template", "params"], "subs": ["proc"], "identity": ["match"], "module": None,
|
||||
"question": "Is a program running?", "words": "A program on this Mac is running."},
|
||||
"http": {"fields": ["url", "msg", "name", "interval_seconds"], "subs": ["http"], "identity": ["url"], "module": None,
|
||||
"question": "Is a device answering?", "words": "A web address answers."},
|
||||
"disk": {"fields": ["path", "warn_gb", "err_gb"], "subs": ["disk"], "identity": ["path"], "module": None,
|
||||
"question": "Is a setting right?", "words": "Free space on a disk."},
|
||||
"midi_peer": {"fields": ["session", "peer", "host", "address"], "subs": ["peer"], "identity": ["host", "session", "peer"],
|
||||
"module": None, "question": "Is a MIDI link up?", "words": "A peer is in a network MIDI session (here or on another Mac)."},
|
||||
"pref": {"fields": ["domain", "key", "want", "msg", "template", "params"], "subs": ["pref"], "identity": ["domain", "key"],
|
||||
"module": None, "question": "Is a setting right?", "words": "A macOS preference has the value it should."},
|
||||
"spotify": {"fields": ["playlist", "fix"], "subs": ["spotify"], "identity": [], "module": None,
|
||||
"question": "Is Spotify playing?", "words": "Spotify is playing the service playlist."},
|
||||
"mdns": {"fields": ["instance", "name"], "subs": ["mdns"], "identity": ["instance"], "module": None,
|
||||
"question": "Is a MIDI link up?", "words": "A network MIDI session is advertised on Bonjour."},
|
||||
"midi_rx": {"fields": ["source", "name", "max_silence_seconds"], "subs": ["rx"], "identity": ["source"], "module": None,
|
||||
"question": "Is a MIDI link up?", "words": "MIDI is arriving from a source."},
|
||||
"remote": {"fields": ["host", "cmd", "msg", "shell", "template", "params"], "subs": ["remote"], "identity": ["host", "cmd"],
|
||||
"module": None, "question": "Is a program running?", "words": "A command on another computer succeeds."},
|
||||
"artnet": {"fields": ["source", "max_silence_seconds", "msg"], "subs": ["artnet"], "identity": ["source"], "module": None,
|
||||
"question": "Is the lighting console sending?", "words": "Art-Net is arriving from the lighting program."},
|
||||
"atem": {"fields": ["part", "msg"], "subs": ["audio", "camera"], "identity": ["part"], "module": "bmd-atem",
|
||||
"question": "Is a setting right?", "words": "The switcher matches the saved file."},
|
||||
}
|
||||
COMMON_CHECK_FIELDS = ["type", "fix", "tag", "service_only", "debounce_seconds"]
|
||||
FIX_KINDS = sp.FIX_KINDS
|
||||
FIX_KIND_FIELDS = {
|
||||
"ssh": ["host", "run", "timeout", "reconnect", "conn"], "local": ["run", "timeout", "reconnect", "conn"],
|
||||
"atem": ["part"], "reconnect": ["conn", "verify_seconds", "hung_note"], "midi_connect": [],
|
||||
"netgear_poe": ["switch", "port", "mac"], "hyperdeck": ["verify_seconds", "off_seconds", "hung_note"],
|
||||
"spotify": ["playlist"],
|
||||
}
|
||||
EXPECTED_MODULE = {t: spec["module"] for t, spec in CHECKS.items() if spec["module"]}
|
||||
|
||||
# --- Every word a key can show, grouped by what to do about it -------------------
|
||||
GLOSSARY = {
|
||||
"hold the key: the fix runs": [
|
||||
"NOT PLAYING", "WRONG PLAYLIST", "SPOTIFY CLOSED", "STOPPED", "NOT LOADED", "MIDI LOST", "MIDI", "GONE",
|
||||
"DISABLED", "OFFLINE", "NO SESSION", "NOT RUNNING", "DOWN", "NOT REC", "SILENT", "WARNING", "STARTING",
|
||||
"ATEM AUDIO WRONG", "ATEM CAMERA WRONG", "ATEM SW MIDI ON", "PT SOLO NOT AFL", "PROPRES NOT RUNNING",
|
||||
"ABLETON NOT RUNNING", "TRANSLATE NOT RUNNING", "PROP TV DOWN", "BIBLE APP DOWN", "PRODCOM DOWN", "PRODDASH DOWN",
|
||||
"ENCODER WEB DOWN", "PT SOLO CHECK OFF", "UNREACH", "DROPPED",
|
||||
],
|
||||
"check the device or the cable": [
|
||||
"NO PING", "NO ANSWER", "UNREACHABLE", "LINK DOWN", "NO GATEWAY", "NO DISK", "FULL", "LOW", "BATT", "INTERF",
|
||||
"RF MUTE", "HUNG - POWER CYCLE IT", "NO NETWORK CHECK POWER", "STILL DOWN", "REFUSED", "LOGIN", "BLIP",
|
||||
"NOT REACHABLE", "DISK",
|
||||
],
|
||||
"ask Chris": ["NOT IN COMPANION", "CHECK BUG", "NO DATA", "WATCHDOG DOWN", "ERR", "PROCESS", "WEB"],
|
||||
"what a key says while you hold it": [
|
||||
"NOTHING TO FIX", "NO AUTO FIX", "JUST TRIED", "WAIT 1 MIN", "WAIT", "MIN", "S", "CHECKING", "FIXING",
|
||||
"AUTO-FIXING", "WAKING PC", "FIX FAILED", "RESTARTING", "RESTARTING SOON", "SOON", "DONE", "FAILED",
|
||||
"JUST DONE", "STARTING", "RECORDING LEFT ALONE", "NOTHING TO CLEAR", "CLEARED", "STILL CONNECTING",
|
||||
"STILL OFF IN COMPANION", "DISABLED IN COMPANION", "TURN ON", "RECONNECT", "POWER CYCLE", "FIXES", "HOLD TO FIX",
|
||||
"HOLD", "REC", "PLAYING", "OK", "OFF", "ON", "ALL OK", "WARN", "TRS", "GB", "NOT", "REACHABLE", "RECORDING",
|
||||
"LEFT ALONE", "TO FIX", "TO CLEAR", "NO AUTO", "FIX", "AUTO", "IN", "COMPANION", "POWER", "CYCLE",
|
||||
"STILL", "OFF IN", "DISABLED IN", "CONNECTING", "CHECK", "NETWORK", "NO",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# --- Rules ---------------------------------------------------------------------
|
||||
|
||||
def defaults():
|
||||
"""{path: default} for every field with one."""
|
||||
return {p: copy.deepcopy(f["default"]) for p, f in F.items() if f["default"] is not None}
|
||||
|
||||
|
||||
def key_record():
|
||||
"""{receiver kind: {keys}}: which keys the code may read from each kind of config object."""
|
||||
rec = {}
|
||||
for p in F:
|
||||
parts = p.split(".")
|
||||
kind = "top"
|
||||
if p.startswith("tiles[].checks[]."):
|
||||
kind, parts = "check", parts[2:]
|
||||
elif p.startswith("tiles[]."):
|
||||
kind, parts = "tile", parts[1:]
|
||||
elif p.startswith("fixes.*."):
|
||||
kind, parts = "fix", parts[2:]
|
||||
elif p.startswith("hosts.*."):
|
||||
kind, parts = "host", parts[2:]
|
||||
elif p.startswith("restart_page.keys[]."):
|
||||
kind, parts = "restart_key", parts[2:]
|
||||
elif p.startswith("restart_page."):
|
||||
kind, parts = "restart_page", parts[1:]
|
||||
elif p.startswith("auto_fix."):
|
||||
kind, parts = "auto", parts[1:]
|
||||
elif p.startswith("network."):
|
||||
kind, parts = "network", parts[1:]
|
||||
elif p.startswith("variables."):
|
||||
kind, parts = "variables", parts[1:]
|
||||
elif p.startswith("site."):
|
||||
kind, parts = "site", parts[1:]
|
||||
elif p.startswith("atem."):
|
||||
kind, parts = "atem", parts[1:]
|
||||
elif p.startswith("spotify."):
|
||||
kind, parts = "spotify", parts[1:]
|
||||
elif p.startswith("companion."):
|
||||
kind, parts = "companion", parts[1:]
|
||||
elif p.startswith("timing."):
|
||||
kind, parts = "timing", parts[1:]
|
||||
elif p.startswith("flap."):
|
||||
kind, parts = "flap", parts[1:]
|
||||
elif p.startswith("gear.*."):
|
||||
kind, parts = "gear", parts[2:]
|
||||
if "[]" in parts[0] and len(parts) > 1:
|
||||
kind, parts = "window", parts[1:]
|
||||
rec.setdefault(kind, set()).add(parts[0].replace("[]", ""))
|
||||
return rec
|
||||
|
||||
|
||||
def _windows(ws, where, errors):
|
||||
try:
|
||||
sp.check_windows(ws, where)
|
||||
except ValueError as e:
|
||||
errors.append(str(e))
|
||||
|
||||
|
||||
def validate(cfg, live=None):
|
||||
"""(errors, warnings). Errors stop an Apply; warnings are shown. `live` (optional) is what the
|
||||
settings page knows right now: {"connections": {label: module id}, "hosts_up": {alias: bool}}."""
|
||||
errors, warnings = [], []
|
||||
live = live or {}
|
||||
|
||||
def pos_ok(pos, where):
|
||||
if not (isinstance(pos, list) and len(pos) == 2 and all(isinstance(n, int) for n in pos)):
|
||||
errors.append("%s: position must be [row, column]" % where)
|
||||
return None
|
||||
grid = cfg.get("page_grid", F["page_grid"]["default"])
|
||||
if not (grid["minRow"] <= pos[0] <= grid["maxRow"] and grid["minColumn"] <= pos[1] <= grid["maxColumn"]):
|
||||
errors.append("%s: row %d column %d is off the grid" % (where, pos[0], pos[1]))
|
||||
if pos[1] == 0 and pos[0] in (0, 1, 2):
|
||||
errors.append("%s: row %d column 0 is a page key" % (where, pos[0]))
|
||||
return tuple(pos)
|
||||
|
||||
for req in ("companion", "variables", "tiles"):
|
||||
if req not in cfg:
|
||||
errors.append("missing %s" % req)
|
||||
if errors:
|
||||
return errors, warnings
|
||||
ids, seen_pos = set(), {}
|
||||
kinds = {"master": 0, "service": 0, "network": 0}
|
||||
fixes, hosts = cfg.get("fixes", {}), cfg.get("hosts", {})
|
||||
for t in cfg["tiles"]:
|
||||
tid = t.get("id", "")
|
||||
where = "key %s" % (tid or "?")
|
||||
if not re.fullmatch(r"[A-Za-z0-9_]+", str(tid)):
|
||||
errors.append("%s: the id may only hold letters, digits and _" % where)
|
||||
if tid in ids:
|
||||
errors.append("%s: two keys with the same id" % where)
|
||||
ids.add(tid)
|
||||
if not t.get("label"):
|
||||
errors.append("%s: no label" % where)
|
||||
if t.get("kind") in kinds:
|
||||
kinds[t["kind"]] += 1
|
||||
if "pos" in t:
|
||||
pos = pos_ok(t["pos"], where)
|
||||
if pos in seen_pos:
|
||||
errors.append("%s and %s share row %d column %d" % (seen_pos[pos], where, pos[0], pos[1]))
|
||||
seen_pos[pos] = where
|
||||
if t.get("hold_always") and t["hold_always"] not in fixes:
|
||||
errors.append("%s: hold runs unknown fix %s" % (where, t["hold_always"]))
|
||||
for i, c in enumerate(t.get("checks", [])):
|
||||
cw = "%s check %d" % (where, i + 1)
|
||||
if c.get("type") not in CHECKS:
|
||||
errors.append("%s: unknown check type %r" % (cw, c.get("type")))
|
||||
continue
|
||||
spec = CHECKS[c["type"]]
|
||||
for k in c:
|
||||
if k not in spec["fields"] and k not in COMMON_CHECK_FIELDS:
|
||||
warnings.append("%s: %s is not a field of a %s check" % (cw, k, c["type"]))
|
||||
if c["type"] in ("conn", "hyperdeck", "dante", "shure", "psm", "tascam") and not c.get("conn"):
|
||||
errors.append("%s: a %s check needs a Companion connection" % (cw, c["type"]))
|
||||
if c["type"] == "ping" and not (c.get("host") or c.get("conn")):
|
||||
errors.append("%s: a ping needs an address or a connection" % cw)
|
||||
if c["type"] == "midi_peer" and not (c.get("session") and c.get("peer")):
|
||||
errors.append("%s: a MIDI link needs a session and a peer" % cw)
|
||||
if c["type"] == "remote" and not (c.get("host") and c.get("cmd") and c.get("msg")):
|
||||
errors.append("%s: a remote check needs a computer, a command and the words to show" % cw)
|
||||
fx = c.get("fix")
|
||||
for f in (fx.values() if isinstance(fx, dict) else [fx]):
|
||||
if f and f not in fixes and f not in (sp.RESTART_CONNECTION, sp.DANTE_AUTO):
|
||||
errors.append("%s: unknown fix %s" % (cw, f))
|
||||
if c.get("service_only"):
|
||||
warnings.append("%s: 'only during a service' has no effect while alarms are armed all day" % cw)
|
||||
conn, mod = c.get("conn"), spec["module"]
|
||||
if conn and mod and live.get("connections") and conn in live["connections"] \
|
||||
and live["connections"][conn] and not live["connections"][conn].startswith(mod):
|
||||
warnings.append("%s: %s uses module %s; this check expects %s" % (cw, conn, live["connections"][conn], mod))
|
||||
if conn and live.get("connections") and conn not in live["connections"]:
|
||||
warnings.append("%s: Companion has no connection called %s" % (cw, conn))
|
||||
for t in cfg["tiles"]:
|
||||
for m in t.get("members", []):
|
||||
if m not in ids:
|
||||
errors.append("key %s: unknown member %s" % (t.get("id"), m))
|
||||
if m == t.get("id"):
|
||||
errors.append("key %s: is its own member" % t.get("id"))
|
||||
in_group = {m for t in cfg["tiles"] for m in t.get("members", [])}
|
||||
for t in cfg["tiles"]:
|
||||
if "pos" not in t and "members" not in t and t.get("id") not in in_group and t.get("kind") != "master":
|
||||
warnings.append("key %s has no place on the deck and is in no group: nobody will see it" % t.get("id"))
|
||||
if kinds["master"] != 1:
|
||||
errors.append("there must be exactly one ALL key (kind master); found %d" % kinds["master"])
|
||||
if kinds["service"] != 1:
|
||||
errors.append("there must be exactly one ALARMS key (kind service); found %d" % kinds["service"])
|
||||
if kinds["network"] > 1:
|
||||
errors.append("at most one NET key (kind network); found %d" % kinds["network"])
|
||||
for fid, fix in fixes.items():
|
||||
where = "fix %s" % fid
|
||||
kind = fix.get("kind", "ssh")
|
||||
if kind not in FIX_KINDS:
|
||||
errors.append("%s: unknown kind %s" % (where, kind))
|
||||
if not fix.get("label"):
|
||||
errors.append("%s: needs plain words (label)" % where)
|
||||
if kind == "ssh":
|
||||
h = hosts.get(fix.get("host"))
|
||||
if h is None:
|
||||
errors.append("%s: unknown computer %s" % (where, fix.get("host")))
|
||||
elif h.get("self"):
|
||||
errors.append("%s: %s is this Mac - use the local kind" % (where, fix.get("host")))
|
||||
elif not h.get("ssh"):
|
||||
errors.append("%s: computer %s has no log-in address" % (where, fix.get("host")))
|
||||
if kind in ("ssh", "local") and not fix.get("run"):
|
||||
errors.append("%s: nothing to run" % where)
|
||||
if kind == "netgear_poe" and not (fix.get("switch") and fix.get("port")):
|
||||
errors.append("%s: needs a switch and a port" % where)
|
||||
if kind == "reconnect" and not fix.get("conn"):
|
||||
errors.append("%s: needs the connection to reconnect" % where)
|
||||
if kind == "atem" and fix.get("part") not in ("audio", "camera"):
|
||||
errors.append("%s: part must be audio or camera" % where)
|
||||
for k in fix:
|
||||
if k not in FIX_KIND_FIELDS.get(kind, []) and k not in ("kind", "short", "label", "restart_ok", "never_auto",
|
||||
"template", "params", "timeout"):
|
||||
warnings.append("%s: %s is not a field of a %s fix" % (where, k, kind))
|
||||
for alias, h in hosts.items():
|
||||
if not re.fullmatch(r"[a-z0-9_]+", alias):
|
||||
errors.append("computer %s: the short name may only hold lowercase letters, digits and _" % alias)
|
||||
if not h.get("self") and not re.fullmatch(r"[^@\s]+@[^@\s]+", str(h.get("ssh", ""))):
|
||||
errors.append("computer %s: the log-in address must be user@address" % alias)
|
||||
if h.get("mac") and not re.fullmatch(r"(?:[0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}", h["mac"]):
|
||||
errors.append("computer %s: not a hardware (MAC) address: %s" % (alias, h["mac"]))
|
||||
for t in cfg["tiles"]:
|
||||
for i, c in enumerate(t.get("checks", [])):
|
||||
if c.get("type") == "remote" or (c.get("type") == "midi_peer" and c.get("host")):
|
||||
h = hosts.get(c["host"])
|
||||
if h is None:
|
||||
errors.append("key %s check %d: unknown computer %s" % (t.get("id"), i + 1, c["host"]))
|
||||
elif h.get("self"):
|
||||
errors.append("key %s check %d: %s is this Mac" % (t.get("id"), i + 1, c["host"]))
|
||||
a = cfg.get("auto_fix") or {}
|
||||
for f in a.get("fixes", []):
|
||||
if f not in fixes:
|
||||
errors.append("auto-fix: unknown fix %s" % f)
|
||||
elif fixes[f].get("never_auto"):
|
||||
errors.append("auto-fix: %s is marked never to run by itself" % f)
|
||||
elif re.search(r"pkill|taskkill|killall", fixes[f].get("run", "")) and not fixes[f].get("restart_ok"):
|
||||
errors.append("auto-fix: %s closes a running program; tick 'may close a running program' or take it out" % f)
|
||||
_windows(cfg.get("service_windows"), "alarms window", errors)
|
||||
_windows(a.get("windows"), "auto-fix window", errors)
|
||||
_windows((cfg.get("site") or {}).get("live_services"), "live services", errors)
|
||||
rp = cfg.get("restart_page")
|
||||
if rp:
|
||||
rpos = {}
|
||||
for k in rp.get("keys", []):
|
||||
kid = k.get("id", "")
|
||||
where = "restart key %s" % (kid or "?")
|
||||
if kid in ids:
|
||||
errors.append("%s: a key on the Status page has that id" % where)
|
||||
if k.get("fix") not in fixes:
|
||||
errors.append("%s: unknown fix %s" % (where, k.get("fix")))
|
||||
for w in k.get("watch", []):
|
||||
if not w.startswith("conn:") and w not in fixes:
|
||||
errors.append("%s: unknown fix %s in its watch list" % (where, w))
|
||||
pos = pos_ok(k.get("pos"), where)
|
||||
if pos in rpos:
|
||||
errors.append("%s and %s share row %d column %d" % (rpos[pos], where, pos[0], pos[1]))
|
||||
rpos[pos] = where
|
||||
for name in ("jump_pos", "back_pos"):
|
||||
if name in rp:
|
||||
pos_ok(rp[name], "the Restart page's %s" % name.replace("_pos", " key"))
|
||||
if "jump_pos" in rp and tuple(rp["jump_pos"]) in seen_pos:
|
||||
errors.append("the 'go to Restart page' key sits on %s" % seen_pos[tuple(rp["jump_pos"])])
|
||||
if "back_pos" in rp and tuple(rp["back_pos"]) in rpos:
|
||||
errors.append("the 'back to Status' key sits on %s" % rpos[tuple(rp["back_pos"])])
|
||||
if rp.get("page") == cfg.get("page"):
|
||||
errors.append("the Status page and the Restart page cannot be the same page")
|
||||
spo = cfg.get("spotify") or {}
|
||||
if any(c.get("type") == "spotify" for t in cfg["tiles"] for c in t.get("checks", [])):
|
||||
if not re.fullmatch(r"spotify:playlist:[A-Za-z0-9]+", str(spo.get("playlist", ""))):
|
||||
errors.append("Spotify: the playlist must look like spotify:playlist:...")
|
||||
for p, f in F.items():
|
||||
v = _get(cfg, p)
|
||||
if v is None:
|
||||
continue
|
||||
if f["type"] == "int" and not isinstance(v, int):
|
||||
errors.append("%s: must be a whole number" % p)
|
||||
elif f["type"] == "int":
|
||||
if f["lo"] is not None and v < f["lo"]:
|
||||
errors.append("%s: %s is below %s" % (p, v, f["lo"]))
|
||||
if f["hi"] is not None and v > f["hi"]:
|
||||
errors.append("%s: %s is above %s" % (p, v, f["hi"]))
|
||||
if f["regex"] and isinstance(v, str) and not re.fullmatch(f["regex"], v):
|
||||
errors.append("%s: %r is not in the right form" % (p, v))
|
||||
if f["type"] == "enum" and f["options"] and v not in f["options"]:
|
||||
errors.append("%s: %r is not one of %s" % (p, v, ", ".join(f["options"])))
|
||||
return errors, warnings
|
||||
|
||||
|
||||
def _get(cfg, path):
|
||||
"""The value at a plain path (no [] or *); None when absent."""
|
||||
if "[]" in path or ".*." in path:
|
||||
return None
|
||||
cur = cfg
|
||||
for part in path.split("."):
|
||||
if not isinstance(cur, dict) or part not in cur:
|
||||
return None
|
||||
cur = cur[part]
|
||||
return cur
|
||||
|
||||
|
||||
def mask_secrets(obj):
|
||||
"""A copy with every secret replaced by "set (N characters)" or "" (for API responses and history)."""
|
||||
secret_paths = {p for p, f in F.items() if f["secret"]}
|
||||
|
||||
def walk(node, path):
|
||||
if isinstance(node, dict):
|
||||
return {k: ("set (%d characters)" % len(v) if v else "") if "%s.%s" % (path, k) in secret_paths
|
||||
or k in secret_paths else walk(v, "%s.%s" % (path, k) if path else k) for k, v in node.items()}
|
||||
if isinstance(node, list):
|
||||
return [walk(v, path + "[]") for v in node]
|
||||
return node
|
||||
return walk(copy.deepcopy(obj), "")
|
||||
|
||||
|
||||
def phrases(cfg):
|
||||
"""The sentences the page and the logs build from the config."""
|
||||
a = cfg.get("auto_fix") or {}
|
||||
site = cfg.get("site") or {}
|
||||
return {
|
||||
"auto_fix": "may run by itself %s" % sp.windows_phrase(a.get("windows", [])) if a.get("windows") else "never runs by itself",
|
||||
"live": ("changes wait while a HyperDeck is recording" + (" or %s" % sp.windows_phrase(site["live_services"])
|
||||
if site.get("live_services") else "")),
|
||||
"alarms": "Alarms are on every day. The ALARMS key silences them until midnight, then they come back by themselves.",
|
||||
"hold": "HOLD %gs" % (cfg.get("page_hold_ms", 2000) / 1000.0),
|
||||
"ingest": site.get("ingest_name", "NA Ingest"),
|
||||
}
|
||||
Binary file not shown.
+458
-109
@@ -16,6 +16,7 @@ Stdlib only; runs on the macOS /usr/bin/python3 (3.9).
|
||||
"""
|
||||
import argparse
|
||||
import calendar
|
||||
import hashlib
|
||||
import http.client
|
||||
import json
|
||||
import logging
|
||||
@@ -24,6 +25,7 @@ import logging.handlers
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
@@ -38,6 +40,10 @@ from datetime import datetime
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
log = logging.getLogger("status-panel")
|
||||
|
||||
VERSION = "2026.10.13" # logged at start ("status panel running (code ...)") and written to started.json
|
||||
# Every kind of fix the watchdog can run (validated by load_config).
|
||||
FIX_KINDS = ("ssh", "local", "atem", "reconnect", "midi_connect", "netgear_poe", "hyperdeck", "spotify")
|
||||
|
||||
OK, WARN, ERR, OFF = "ok", "warn", "err", "off"
|
||||
|
||||
# Display levels, least to most urgent. "idle" is a service-only fault while
|
||||
@@ -629,13 +635,14 @@ class AutoLog:
|
||||
|
||||
NAME = re.compile(r"^(\d{4})-(\d{2})-(\d{2}) [A-Za-z]+ Auto-Fix\.log$")
|
||||
|
||||
def __init__(self, directory):
|
||||
def __init__(self, directory, who="NA Ingest", when="on Sundays 06:30-13:30", keep_months=1):
|
||||
self.dir = os.path.expanduser(directory)
|
||||
self.lock = threading.Lock()
|
||||
self.who, self.when, self.keep_months = who, when, max(1, int(keep_months))
|
||||
|
||||
def cleanup(self, now):
|
||||
"""Delete day files once they are a calendar month old (the 11 Oct file goes on 11 Nov).
|
||||
Only files named exactly like these logs are ever touched."""
|
||||
"""Delete day files once they are keep_months calendar months old (the 11 Oct file goes on
|
||||
11 Nov). Only files named exactly like these logs are ever touched."""
|
||||
today = datetime.fromtimestamp(now).date()
|
||||
removed = []
|
||||
try:
|
||||
@@ -647,7 +654,8 @@ class AutoLog:
|
||||
if not m:
|
||||
continue
|
||||
y, mo, d = (int(x) for x in m.groups())
|
||||
y2, mo2 = (y + 1, 1) if mo == 12 else (y, mo + 1)
|
||||
total = mo - 1 + self.keep_months
|
||||
y2, mo2 = y + total // 12, total % 12 + 1
|
||||
try:
|
||||
due = datetime(y2, mo2, min(d, calendar.monthrange(y2, mo2)[1])).date()
|
||||
except ValueError:
|
||||
@@ -678,41 +686,111 @@ class AutoLog:
|
||||
with open(path, "a", encoding="utf-8") as f:
|
||||
if new:
|
||||
f.write("Status panel auto-fixes - %s\n"
|
||||
"(NA Ingest fixes these by itself on Sundays 06:30-13:30; anything that closes a\n"
|
||||
"(%s fixes these by itself %s; anything that closes a\n"
|
||||
" running program stays a hold. Newest lines at the bottom.)\n\n"
|
||||
% dt.strftime("%A %-d %B %Y"))
|
||||
% (dt.strftime("%A %-d %B %Y"), self.who, self.when))
|
||||
f.write(line)
|
||||
except OSError as e:
|
||||
log.warning("auto-fix log %s: %s", self.dir, e)
|
||||
|
||||
|
||||
def window_holds(w, day, hm, prev_day):
|
||||
"""One window at one moment. "end" <= "start" means the window runs past midnight: it starts on
|
||||
one of its days and ends early the next morning (a Saturday night service: sat 18:00-00:30)."""
|
||||
if w["start"] < w["end"]:
|
||||
return day in w["days"] and w["start"] <= hm < w["end"]
|
||||
return (day in w["days"] and hm >= w["start"]) or (prev_day in w["days"] and hm < w["end"])
|
||||
|
||||
|
||||
def in_windows(windows, now):
|
||||
"""Is `now` inside one of [{"days": [...], "start": "HH:MM", "end": "HH:MM"}]?"""
|
||||
dt = datetime.fromtimestamp(now)
|
||||
hm, day = dt.strftime("%H:%M"), DAYS[dt.weekday()]
|
||||
return any(day in w["days"] and w["start"] <= hm < w["end"] for w in windows)
|
||||
return any(window_holds(w, day, hm, DAYS[dt.weekday() - 1]) for w in windows)
|
||||
|
||||
|
||||
def windows_phrase(windows):
|
||||
"""Windows in words, for log headers and notes: "on Sundays 06:30-13:30"."""
|
||||
parts = []
|
||||
for w in windows:
|
||||
days = [d for d in DAYS if d in w["days"]]
|
||||
names = [datetime(2024, 1, 1 + DAYS.index(d)).strftime("%As") for d in days] # 2024-01-01 is a Monday
|
||||
when = "every day" if len(days) == 7 else " and ".join(names)
|
||||
parts.append("%s %s-%s" % (when, w["start"], w["end"]))
|
||||
return ("on " + " and ".join(parts)) if parts else "never"
|
||||
|
||||
|
||||
def check_identity(chk):
|
||||
"""What a check is about, independent of its position in the list: the key that remembered
|
||||
state (acknowledgements, auto-fix tries) is filed under, so editing the list keeps it."""
|
||||
t = chk["type"]
|
||||
if t == "midi_peer":
|
||||
return "%s/%s/%s" % (chk.get("host", ""), chk["session"], chk["peer"])
|
||||
if t == "remote":
|
||||
return "%s/%s" % (chk["host"], hashlib.sha1(chk["cmd"].encode()).hexdigest()[:8])
|
||||
if t == "ping":
|
||||
return chk.get("conn") or chk.get("host", "")
|
||||
if t == "pref":
|
||||
return "%s/%s" % (chk["domain"], chk["key"])
|
||||
single = {"http": "url", "mdns": "instance", "launchd": "label", "process": "match", "artnet": "source",
|
||||
"atem": "part", "disk": "path", "midi_rx": "source"}
|
||||
if t in single:
|
||||
return str(chk.get(single[t], ""))
|
||||
return chk.get("conn") or t
|
||||
|
||||
|
||||
def write_json(path, obj):
|
||||
"""Write a small JSON file in one go (a reader never sees half of it)."""
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(obj, f, indent=1, sort_keys=True)
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
class State:
|
||||
"""One file (state.json) for everything the watchdog remembers across restarts: the ALARMS
|
||||
override, acknowledged faults, auto-fix tries and when each fix last ran. One writer; saved whole."""
|
||||
|
||||
def __init__(self, path):
|
||||
self.path, self.data, self.lock = path, {}, threading.Lock()
|
||||
if not path:
|
||||
return
|
||||
try:
|
||||
with open(path) as f:
|
||||
self.data = json.load(f)
|
||||
except (OSError, ValueError):
|
||||
legacy = os.path.join(os.path.dirname(path), "service-mode.json") # the file before state.json
|
||||
try:
|
||||
with open(legacy) as f:
|
||||
old = json.load(f)
|
||||
self.data = {"service": {"override": old.get("override"), "last_sched": old.get("last_sched")}}
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
def get(self, key, default=None):
|
||||
return self.data.get(key, default)
|
||||
|
||||
def set(self, key, value):
|
||||
with self.lock:
|
||||
self.data[key] = value
|
||||
if self.path:
|
||||
try:
|
||||
write_json(self.path, self.data)
|
||||
except OSError as e:
|
||||
log.warning("could not save %s: %s", self.path, e)
|
||||
|
||||
|
||||
class ServiceMode:
|
||||
"""On inside a scheduled window; a button press overrides until the next window edge."""
|
||||
|
||||
def __init__(self, windows, path):
|
||||
self.windows, self.path = windows, path
|
||||
self.override, self.last_sched = None, None
|
||||
if not path:
|
||||
return
|
||||
try:
|
||||
with open(path) as f:
|
||||
saved = json.load(f)
|
||||
self.override, self.last_sched = saved.get("override"), saved.get("last_sched")
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
def __init__(self, windows, state):
|
||||
self.windows = windows
|
||||
self.state = state if isinstance(state, State) else State(state)
|
||||
saved = self.state.get("service") or {}
|
||||
self.override, self.last_sched = saved.get("override"), saved.get("last_sched")
|
||||
|
||||
def scheduled(self, now):
|
||||
dt = datetime.fromtimestamp(now)
|
||||
hm = dt.strftime("%H:%M")
|
||||
day = DAYS[dt.weekday()]
|
||||
return any(day in w["days"] and w["start"] <= hm < w["end"] for w in self.windows)
|
||||
return in_windows(self.windows, now)
|
||||
|
||||
def on(self, now):
|
||||
sched = self.scheduled(now)
|
||||
@@ -728,28 +806,33 @@ class ServiceMode:
|
||||
self._save()
|
||||
|
||||
def _save(self):
|
||||
if not self.path:
|
||||
return
|
||||
try:
|
||||
tmp = self.path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump({"override": self.override, "last_sched": self.last_sched}, f)
|
||||
os.replace(tmp, self.path)
|
||||
except OSError as e:
|
||||
log.warning("could not save service mode: %s", e)
|
||||
self.state.set("service", {"override": self.override, "last_sched": self.last_sched})
|
||||
|
||||
|
||||
# --- Engine ------------------------------------------------------------------
|
||||
|
||||
class Engine:
|
||||
def __init__(self, cfg, state_path=None):
|
||||
def __init__(self, cfg, state_path=None, data_dir=None):
|
||||
self.cfg = cfg
|
||||
self.tiles = {t["id"]: t for t in cfg["tiles"]}
|
||||
self.debounce = cfg.get("debounce_seconds", 3)
|
||||
self.flap_count = cfg.get("flap", {}).get("count", 3)
|
||||
self.flap_window = cfg.get("flap", {}).get("window_seconds", 600)
|
||||
self.net = cfg.get("network", {})
|
||||
self.service = ServiceMode(cfg.get("service_windows", []), state_path)
|
||||
self.site = cfg.get("site", {})
|
||||
timing = cfg.get("timing", {})
|
||||
self.cooldown = timing.get("fix_cooldown_seconds", 60) # the same fix never re-runs within this
|
||||
self.hold_ms = cfg.get("page_hold_ms", 2000) # how long a key is held to run its fix
|
||||
self.state = State(state_path)
|
||||
self.service = ServiceMode(cfg.get("service_windows", []), self.state)
|
||||
self.data_dir = data_dir
|
||||
self.stopping = False # SIGTERM arrived: finish what runs, start nothing new
|
||||
self.running = {} # fix thread name -> {"key", "what", "since"} (published as fixes-running.json)
|
||||
self.fix_threads = [] # the threads behind self.running, joined at exit
|
||||
self.any_recording = False
|
||||
self.live = None # (live?, reason) last written to live-now.json
|
||||
self.live_written = 0.0
|
||||
self.restore_acks = set(self.state.get("acks") or []) # acknowledged before the last restart
|
||||
self.trackers = {}
|
||||
self.group_of = {}
|
||||
for t in cfg["tiles"]:
|
||||
@@ -763,16 +846,19 @@ class Engine:
|
||||
self.burst = None # {"start", "conns", "cleared"}
|
||||
self.events = []
|
||||
self.meta = {} # key -> (tile id, check, sub key)
|
||||
self.fix_last = {} # fix identity -> last run time
|
||||
self.fix_last = {tuple(k.split("\0")): v for k, v in (self.state.get("fix_last") or {}).items()}
|
||||
self.executor = execute_fix
|
||||
self.poe_executor = execute_poe
|
||||
self.notes = {} # tile id -> hold feedback {"text", "level", "until"}
|
||||
# The Restart page: one key per thing, hold 2 s = restart it even when it looks fine.
|
||||
self.restart_keys = {k["id"]: k for k in cfg.get("restart_page", {}).get("keys", [])}
|
||||
self.auto = cfg.get("auto_fix") or {}
|
||||
self.auto_tries = {} # fix identity -> [tries, last try time] for the fault now going on
|
||||
self.auto_seen = {} # fix identity -> last time it was in an auto plan
|
||||
self.autolog = AutoLog(self.auto.get("log_dir", "~/Desktop/Auto-Fix Logs")) if self.auto else None
|
||||
saved = self.state.get("auto") or {}
|
||||
self.auto_tries = {tuple(k.split("\0")): v for k, v in saved.get("tries", {}).items()}
|
||||
self.auto_seen = {tuple(k.split("\0")): v for k, v in saved.get("seen", {}).items()}
|
||||
self.autolog = AutoLog(self.auto.get("log_dir", "~/Desktop/Auto-Fix Logs"),
|
||||
self.site.get("ingest_name", "NA Ingest"), windows_phrase(self.auto.get("windows", [])),
|
||||
self.auto.get("keep_months", 1)) if self.auto else None
|
||||
if self.autolog:
|
||||
self.autolog.cleanup(time.time())
|
||||
self.auto_watch = {} # "BUTTON FAULT" auto-fix went after -> when (to log when it clears)
|
||||
@@ -782,6 +868,81 @@ class Engine:
|
||||
self.deck_ips = {} # HyperDeck connection -> last address Companion reported
|
||||
self.now = 0.0
|
||||
|
||||
@property
|
||||
def hold_text(self):
|
||||
return "HOLD %ss" % ("%g" % (self.hold_ms / 1000.0))
|
||||
|
||||
@property
|
||||
def wait_text(self):
|
||||
c = int(self.cooldown)
|
||||
return "WAIT %d MIN" % (c // 60) if c % 60 == 0 else "WAIT %d S" % c
|
||||
|
||||
def identity_key(self, key):
|
||||
"""A tracker key ("tile:index:sub") as the position-free name its state is remembered by."""
|
||||
tile_id, chk, sub = self.meta[key]
|
||||
return "%s|%s|%s" % (tile_id, check_identity(chk), sub)
|
||||
|
||||
def remember(self):
|
||||
"""Save what must survive a restart: acknowledgements, auto-fix tries, when fixes last ran."""
|
||||
acks = sorted({self.identity_key(k) for k, tr in self.trackers.items()
|
||||
if tr.confirmed and tr.acked and k in self.meta} | self.restore_acks)
|
||||
self.state.set("acks", acks)
|
||||
self.state.set("fix_last", {"\0".join(k): v for k, v in self.fix_last.items()})
|
||||
self.state.set("auto", {"tries": {"\0".join(k): v for k, v in self.auto_tries.items()},
|
||||
"seen": {"\0".join(k): v for k, v in self.auto_seen.items()}})
|
||||
|
||||
def live_now(self, now):
|
||||
"""(live?, why): changes to the panel wait while this is true. A recording HyperDeck is
|
||||
always live; a campus may add clock windows ("live_services" in "site")."""
|
||||
if self.any_recording:
|
||||
return True, "a HyperDeck is recording"
|
||||
if in_windows(self.site.get("live_services", []), now):
|
||||
return True, "a live service is scheduled"
|
||||
return False, ""
|
||||
|
||||
def publish(self, now):
|
||||
"""live-now.json and fixes-running.json in the data dir (the settings page reads them)."""
|
||||
if not self.data_dir:
|
||||
return
|
||||
live = self.live_now(now)
|
||||
if live != self.live or now - self.live_written > 5:
|
||||
try:
|
||||
write_json(os.path.join(self.data_dir, "live-now.json"),
|
||||
{"live": live[0], "reason": live[1], "time": now})
|
||||
except OSError as e:
|
||||
log.warning("live-now.json: %s", e)
|
||||
self.live, self.live_written = live, now
|
||||
|
||||
def fix_started(self, name, key_id, what):
|
||||
self.running[name] = {"key": key_id, "what": what, "since": time.time()}
|
||||
self.publish_running()
|
||||
|
||||
def fix_done(self, name):
|
||||
self.running.pop(name, None)
|
||||
self.publish_running()
|
||||
|
||||
def publish_running(self):
|
||||
if self.data_dir:
|
||||
try:
|
||||
write_json(os.path.join(self.data_dir, "fixes-running.json"), self.running)
|
||||
except OSError as e:
|
||||
log.warning("fixes-running.json: %s", e)
|
||||
|
||||
def start_fix_thread(self, target, key_id, what):
|
||||
"""Every fix runs in a thread the watchdog knows about: a restart waits for them."""
|
||||
name = "fix-%s-%d" % (key_id, int(time.time() * 1000) % 100000)
|
||||
|
||||
def wrapped():
|
||||
try:
|
||||
target()
|
||||
finally:
|
||||
self.fix_done(name)
|
||||
self.fix_started(name, key_id, what)
|
||||
t = threading.Thread(target=wrapped, name=name, daemon=True)
|
||||
self.fix_threads = [x for x in self.fix_threads if x.is_alive()] + [t]
|
||||
t.start()
|
||||
return t
|
||||
|
||||
# raw results for every check key
|
||||
def evaluate(self, snap, now, service_on):
|
||||
ctx = {"now": now, "service": service_on, "any_recording": False, "dante_all": {}}
|
||||
@@ -794,6 +955,7 @@ class Engine:
|
||||
if chk["type"] == "hyperdeck" and chk.get("rec_sync", True):
|
||||
if conn_result(snap, chk["conn"], now).level == OK and hyperdeck_recording(snap, chk["conn"]):
|
||||
ctx["any_recording"] = True
|
||||
self.any_recording = ctx["any_recording"]
|
||||
raw = {}
|
||||
for t in self.cfg["tiles"]:
|
||||
for i, chk in enumerate(t.get("checks", [])):
|
||||
@@ -878,13 +1040,23 @@ class Engine:
|
||||
deb = self.debounce if debounce is None else debounce
|
||||
for key, (tile_id, res, chk) in raw.items():
|
||||
tr = self.trackers.setdefault(key, Tracker())
|
||||
was_acked = tr.acked
|
||||
# A MIDI peer leaving a session is a clean event: show it after 1 s, not the general 3 s.
|
||||
event = tr.update(res, now, 0 if chk["type"] == "network" else
|
||||
chk.get("debounce_seconds", 1 if chk["type"] == "midi_peer" else deb),
|
||||
self.flap_window)
|
||||
if event == "fault":
|
||||
self.events.append(("FAULT", "%s %s" % (self.tiles[tile_id]["label"], res.msg)))
|
||||
ident = self.identity_key(key) if key in self.meta else None
|
||||
if ident in self.restore_acks: # acknowledged before the restart: still acknowledged
|
||||
self.restore_acks.discard(ident)
|
||||
tr.acked = True
|
||||
self.remember()
|
||||
elif event == "clear":
|
||||
ident = self.identity_key(key) if key in self.meta else None
|
||||
if was_acked or ident in self.restore_acks:
|
||||
self.restore_acks.discard(ident)
|
||||
self.remember() # an acknowledgement is for one fault only
|
||||
text = "%s %s" % (self.tiles[tile_id]["label"], tr.last_fault_msg)
|
||||
self.events.append(("CLEAR", text))
|
||||
fixed_at = self.auto_watch.pop(text, None)
|
||||
@@ -896,15 +1068,19 @@ class Engine:
|
||||
# fault is over - clear it, or it lingers unseen and a hold would still try to fix it.
|
||||
for key, tr in self.trackers.items():
|
||||
if key not in raw and (tr.confirmed or tr.bad_since is not None):
|
||||
was_acked = tr.acked
|
||||
if tr.update(Result(OK), now, 0, self.flap_window) == "clear" and key in self.meta:
|
||||
tile_id = self.meta[key][0]
|
||||
self.events.append(("CLEAR", "%s %s" % (self.tiles[tile_id]["label"], tr.last_fault_msg)))
|
||||
if was_acked:
|
||||
self.remember()
|
||||
self.items = {}
|
||||
for key, (tile_id, res, chk) in raw.items():
|
||||
tr = self.trackers[key]
|
||||
eff = tr.effective(self.flap_count)
|
||||
self.items.setdefault(tile_id, []).append(
|
||||
(display_level(eff, tr.acked, service_on), eff, tr.since or now))
|
||||
self.publish(now)
|
||||
return render(self, service_on)
|
||||
|
||||
def keys_for(self, tile_id):
|
||||
@@ -925,6 +1101,8 @@ class Engine:
|
||||
if tr.confirmed and not tr.acked:
|
||||
tr.acked = True
|
||||
n += 1
|
||||
if n:
|
||||
self.remember()
|
||||
return n
|
||||
|
||||
def restart_targets(self, tile_id):
|
||||
@@ -1038,7 +1216,11 @@ class Engine:
|
||||
"""Show `text` on the button for a while (hold feedback), instead of its usual text."""
|
||||
self.notes[tile_id] = {"text": text, "level": level, "until": now + seconds}
|
||||
|
||||
def run_fixes(self, tile_id, snap, comp, now, cooldown=60):
|
||||
def run_fixes(self, tile_id, snap, comp, now, cooldown=None):
|
||||
cooldown = self.cooldown if cooldown is None else cooldown
|
||||
if self.stopping:
|
||||
self.note(tile_id, "RESTARTING\nSOON", "idle", now, 4)
|
||||
return []
|
||||
if self.tiles.get(tile_id, {}).get("kind") == "network":
|
||||
# NET: a hold clears a remembered network blip (link and gateway clear themselves).
|
||||
if self.burst is None:
|
||||
@@ -1071,8 +1253,9 @@ class Engine:
|
||||
return plan
|
||||
return self.run_plan(tile_id, plan, snap, comp, now, cooldown)
|
||||
|
||||
def run_plan(self, tile_id, plan, snap, comp, now, cooldown=60, auto=False, on_done=None):
|
||||
def run_plan(self, tile_id, plan, snap, comp, now, cooldown=None, auto=False, on_done=None):
|
||||
"""Carry out fix_plan() items for one button (a hold, or auto-fix during service)."""
|
||||
cooldown = self.cooldown if cooldown is None else cooldown
|
||||
started, names, waited, midi = [], [], False, []
|
||||
for ident, kind, what, chk in plan:
|
||||
if now - self.fix_last.get(ident, 0) < cooldown:
|
||||
@@ -1080,6 +1263,7 @@ class Engine:
|
||||
waited = True
|
||||
continue
|
||||
self.fix_last[ident] = now
|
||||
self.remember()
|
||||
if kind == "enable":
|
||||
c = snap.conns.get(what)
|
||||
if c:
|
||||
@@ -1112,7 +1296,7 @@ class Engine:
|
||||
started.append((midi_worker, (midi,), self.cfg["fixes"][midi[0][0]].get("short", "MIDI")))
|
||||
if not names:
|
||||
if not auto:
|
||||
self.note(tile_id, "JUST TRIED\nWAIT 1 MIN" if waited else "NOTHING\nTO FIX", "idle", now, 4)
|
||||
self.note(tile_id, ("JUST TRIED\n" + self.wait_text) if waited else "NOTHING\nTO FIX", "idle", now, 4)
|
||||
return plan
|
||||
summary = names[0] if len(set(names)) == 1 else "%d FIXES" % len(names)
|
||||
if not started: # only Companion reconnects: quick
|
||||
@@ -1148,7 +1332,7 @@ class Engine:
|
||||
on_done(outcome)
|
||||
said = []
|
||||
for fn, args, short in started:
|
||||
threading.Thread(target=work, args=(fn, args, short), daemon=True).start()
|
||||
self.start_fix_thread(lambda fn=fn, args=args, short=short: work(fn, args, short), tile_id, short)
|
||||
return plan
|
||||
|
||||
def auto_fix(self, snap, comp, now):
|
||||
@@ -1158,7 +1342,7 @@ class Engine:
|
||||
the network is having a blip, and never while ALARMS has muted the panel (maintenance).
|
||||
Restarting a running program stays a hold."""
|
||||
a = self.auto
|
||||
if not a or now - self.auto_last < 5:
|
||||
if not a or now - self.auto_last < 5 or self.stopping:
|
||||
return []
|
||||
self.auto_last = now
|
||||
if not in_windows(a.get("windows", []), now) or not self.service.on(now):
|
||||
@@ -1184,7 +1368,7 @@ class Engine:
|
||||
if self.auto_paused and self.autolog:
|
||||
self.autolog.write(now, "NET", "network settled: auto-fix is back on")
|
||||
self.auto_paused = False
|
||||
gentle = set(a.get("fixes", []))
|
||||
gentle = {f for f in a.get("fixes", []) if not self.cfg["fixes"].get(f, {}).get("never_auto")}
|
||||
done = []
|
||||
for t in self.cfg["tiles"]:
|
||||
if "checks" not in t:
|
||||
@@ -1243,6 +1427,8 @@ class Engine:
|
||||
# fault blink out for a moment (restarting, re-confirming), which must not reset the count.
|
||||
if now - self.auto_seen.get(ident, 0) >= forget:
|
||||
del self.auto_tries[ident]
|
||||
if done or self.auto_tries:
|
||||
self.remember()
|
||||
return done
|
||||
|
||||
def describe(self, kind, what):
|
||||
@@ -1253,9 +1439,13 @@ class Engine:
|
||||
return self.cfg["fixes"][what].get("label", what)
|
||||
return "%s %s" % (kind, what)
|
||||
|
||||
def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None, chk=None):
|
||||
def run_force(self, key_id, snap, comp, now, cooldown=None, fix_id=None, chk=None, verb="FORCE"):
|
||||
"""Run one fix whatever the status says: a Restart page key, or a button's "hold_always".
|
||||
Shares the 60 s guard with holds, so a hold and a restart key never double up."""
|
||||
Shares the cooldown guard with holds, so a hold and a restart key never double up."""
|
||||
cooldown = self.cooldown if cooldown is None else cooldown
|
||||
if self.stopping:
|
||||
self.note(key_id, "RESTARTING\nSOON", "idle", now, 4)
|
||||
return None
|
||||
if fix_id is None:
|
||||
key = self.restart_keys.get(key_id)
|
||||
if not key:
|
||||
@@ -1266,19 +1456,20 @@ class Engine:
|
||||
and now - self.fix_last.get(("fix", w, ""), 0) < cooldown]
|
||||
if busy: # e.g. auto-fix is starting Ableton right now
|
||||
log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, busy[0], cooldown)
|
||||
self.note(key_id, "STARTING\nWAIT 1 MIN", "idle", now, 4)
|
||||
self.note(key_id, "STARTING\n" + self.wait_text, "idle", now, 4)
|
||||
return None
|
||||
per_deck = self.cfg["fixes"][fix_id].get("kind") == "hyperdeck" and chk and chk.get("conn")
|
||||
ident = ("fix", fix_id, chk["conn"] if per_deck else "") # the same guard a failing hold uses
|
||||
if now - self.fix_last.get(ident, 0) < cooldown:
|
||||
log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, fix_id, cooldown)
|
||||
log.info("%s %s: %s ran less than %ds ago, skipped", verb, key_id, fix_id, cooldown)
|
||||
cur = self.notes.get(key_id)
|
||||
if not (cur and cur["level"] == "fixing" and now < cur["until"]): # still running: keep saying so
|
||||
self.note(key_id, "JUST DONE\nWAIT 1 MIN", "idle", now, 4)
|
||||
self.note(key_id, "JUST DONE\n" + self.wait_text, "idle", now, 4)
|
||||
return None
|
||||
self.fix_last[ident] = now
|
||||
self.remember()
|
||||
fix = self.cfg["fixes"][fix_id]
|
||||
log.info("FORCE %s: %s", key_id, fix.get("label", fix_id))
|
||||
log.info("%s %s: %s", verb, key_id, fix.get("label", fix_id))
|
||||
self.note(key_id, "RESTARTING", "fixing", now, 180)
|
||||
|
||||
def work():
|
||||
@@ -1292,9 +1483,51 @@ class Engine:
|
||||
self.note(key_id, "FAILED", "failed", done, 30)
|
||||
else:
|
||||
self.note(key_id, "DONE", "ok", done, 20)
|
||||
threading.Thread(target=work, daemon=True).start()
|
||||
self.start_fix_thread(work, key_id, fix.get("short", fix_id.upper()))
|
||||
return fix_id
|
||||
|
||||
def run_test(self, key_id, fix_id, which, snap, comp, now):
|
||||
"""The settings page's "Run this fix now": one check's fix, run exactly as a hold on that key
|
||||
would run it (deck address, MIDI peer address, recording guard, cooldown), logged as TEST.
|
||||
`which` picks the check: its number on the key, or its identity. Refused for anything a hold
|
||||
would not do from that key, and for fixes that act on the switcher, a PoE port or Dante."""
|
||||
tile = self.tiles.get(key_id)
|
||||
if not tile:
|
||||
log.warning("TEST %s: no such key", key_id)
|
||||
return None
|
||||
fix = self.cfg["fixes"].get(fix_id)
|
||||
if not fix or fix.get("kind") in ("atem", "netgear_poe") or fix_id in (RESTART_CONNECTION, DANTE_AUTO):
|
||||
log.warning("TEST %s: %s is not a fix the page may run", key_id, fix_id)
|
||||
return None
|
||||
checks = tile.get("checks", [])
|
||||
chk = None
|
||||
for i, c in enumerate(checks):
|
||||
if which in (str(i), check_identity(c), urllib.parse.quote(check_identity(c), safe="")):
|
||||
chk = c
|
||||
if chk is None and tile.get("hold_always") == fix_id:
|
||||
chk = next((c for c in checks if c.get("conn")), {})
|
||||
if chk is None:
|
||||
log.warning("TEST %s: no check %r on it", key_id, which)
|
||||
return None
|
||||
own = chk.get("fix")
|
||||
own = set(own.values()) if isinstance(own, dict) else {own}
|
||||
if fix_id not in own and tile.get("hold_always") != fix_id:
|
||||
log.warning("TEST %s: %s is not attached to that check", key_id, fix_id)
|
||||
return None
|
||||
if chk.get("type") == "hyperdeck":
|
||||
if hyperdeck_recording(snap, chk["conn"]):
|
||||
log.info("TEST %s: %s is recording, left alone", key_id, chk["conn"])
|
||||
self.note(key_id, "RECORDING\nLEFT ALONE", "idle", now, 4)
|
||||
return None
|
||||
chk = dict(chk, ip=chk.get("ip") or snap.var(chk["conn"], "ip") or self.deck_ips.get(chk["conn"]))
|
||||
if fix.get("kind") == "midi_connect":
|
||||
address = snap.peer_addr.get((chk.get("host", ""), chk["session"], chk["peer"])) or chk.get("address")
|
||||
if not address:
|
||||
log.info("TEST %s: no address known for %s", key_id, chk["peer"])
|
||||
return None
|
||||
chk = dict(chk, address=address)
|
||||
return self.run_force(key_id, snap, comp, now, fix_id=fix_id, chk=chk, verb="TEST")
|
||||
|
||||
|
||||
def display_level(res, acked, service_on):
|
||||
if res.level == OFF:
|
||||
@@ -1373,7 +1606,7 @@ def render(engine, service_on):
|
||||
elif down: # flashes: this is the one to hold
|
||||
out[kid] = colored(key["label"] + "\nHOLD TO FIX", "err")
|
||||
else:
|
||||
out[kid] = colored(key["label"] + "\nHOLD 2s", "idle")
|
||||
out[kid] = colored(key["label"] + "\n" + engine.hold_text, "idle")
|
||||
for tile in engine.cfg["tiles"]:
|
||||
tid, kind = tile["id"], tile.get("kind")
|
||||
note = engine.notes.get(tid)
|
||||
@@ -1616,13 +1849,18 @@ def playlist_tracks(playlist_uri):
|
||||
return tracks, False
|
||||
|
||||
|
||||
def spotify_playlist(cfg, chk=None):
|
||||
"""The service playlist: "spotify" in config is the one place it is set; a check may override it."""
|
||||
return (chk or {}).get("playlist") or (cfg.get("spotify") or {}).get("playlist")
|
||||
|
||||
|
||||
def poll_spotify(snap, cfg):
|
||||
checks = [c for t in cfg["tiles"] for c in t.get("checks", []) if c["type"] == "spotify"]
|
||||
if not checks:
|
||||
return
|
||||
def refresh():
|
||||
try:
|
||||
snap.spotify_tracks, snap.spotify_complete = playlist_tracks(checks[0]["playlist"])
|
||||
snap.spotify_tracks, snap.spotify_complete = playlist_tracks(spotify_playlist(cfg, checks[0]))
|
||||
except (OSError, ValueError, KeyError, AttributeError, TypeError) as e:
|
||||
log.warning("Spotify playlist list: %s", e)
|
||||
snap.spotify_tracks_time = time.time()
|
||||
@@ -1769,11 +2007,22 @@ def wake_on_lan(mac, broadcast=None, port=9):
|
||||
return packet
|
||||
|
||||
|
||||
def remote_script(cmds):
|
||||
"""One shell line that runs every check and prints SP<n>=<exit code>."""
|
||||
def remote_script(cmds, shell="sh"):
|
||||
"""One line that runs every check and prints SP<n>=<0 or 1>. On a Mac the shell evaluates $?
|
||||
after each command; cmd.exe expands %ERRORLEVEL% once before the line runs, so Windows gets a
|
||||
conditional echo per command instead (never a bare %ERRORLEVEL%)."""
|
||||
if shell == "cmd":
|
||||
return " & ".join("((%s) >nul 2>&1 && echo SP%d=0 || echo SP%d=1)" % (cmd, i, i) for i, cmd in enumerate(cmds))
|
||||
return "; ".join("(%s) >/dev/null 2>&1; echo SP%d=$?" % (cmd, i) for i, cmd in enumerate(cmds))
|
||||
|
||||
|
||||
def host_shell(cfg, host, chk=None):
|
||||
"""Which shell a remote check's command line is written for: "cmd" on a Windows PC, else "sh"."""
|
||||
if chk and chk.get("shell"):
|
||||
return chk["shell"]
|
||||
return "cmd" if cfg.get("hosts", {}).get(host, {}).get("os") == "windows" else "sh"
|
||||
|
||||
|
||||
def parse_remote(out, n):
|
||||
codes = dict(re.findall(r"SP(\d+)=(\d+)", out or ""))
|
||||
return [None if str(i) not in codes else codes[str(i)] == "0" for i in range(n)]
|
||||
@@ -1790,7 +2039,7 @@ def poll_remote(snap, cfg):
|
||||
result = dict(snap.remote)
|
||||
for host, cmds in by_host.items():
|
||||
try:
|
||||
out = subprocess.run(ssh_cmd(cfg, host) + [remote_script(cmds)], capture_output=True,
|
||||
out = subprocess.run(ssh_cmd(cfg, host) + [remote_script(cmds, host_shell(cfg, host))], capture_output=True,
|
||||
text=True, timeout=20).stdout
|
||||
except subprocess.TimeoutExpired:
|
||||
out = ""
|
||||
@@ -1799,41 +2048,6 @@ def poll_remote(snap, cfg):
|
||||
snap.remote = result
|
||||
|
||||
|
||||
def hyperdeck_probe(ip, timeout=3, port=9993):
|
||||
"""Ask a HyperDeck directly what it is doing, on its control port 9993.
|
||||
|
||||
Returns its transport status ("record", "preview", "stopped", ...), "busy" if it
|
||||
refused another client, "answered" if it greeted but did not report a status,
|
||||
"hung" if it accepted the connection and said nothing at all, or "unreachable".
|
||||
Only "hung" justifies a reboot.
|
||||
"""
|
||||
try:
|
||||
s = socket.create_connection((ip, port), timeout=timeout)
|
||||
except OSError:
|
||||
return "unreachable"
|
||||
buf = b""
|
||||
with s:
|
||||
s.settimeout(timeout)
|
||||
try:
|
||||
s.sendall(b"transport info\r\n")
|
||||
end = time.time() + timeout
|
||||
while time.time() < end:
|
||||
chunk = s.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
buf += chunk
|
||||
m = re.search(rb"208 transport info:.*?status: (\w+)", buf, re.S)
|
||||
if m:
|
||||
return m.group(1).decode().lower()
|
||||
if b"120 connection rejected" in buf:
|
||||
return "busy"
|
||||
except socket.timeout:
|
||||
pass
|
||||
except OSError:
|
||||
return "unreachable" if not buf else "answered"
|
||||
return "answered" if buf else "hung"
|
||||
|
||||
|
||||
# --- The self-updating Dante map: name -> IP -> MAC -> switch port -> PoE --------
|
||||
|
||||
def dante_map_path(data_dir):
|
||||
@@ -1950,6 +2164,8 @@ def execute_fix(cfg, fix_id, chk):
|
||||
ok = conn in ids and comp.restart_connection(ids[conn])
|
||||
log.info("FIX %s -> Companion connection %s restarted %s", label, conn, "ok" if ok else "FAILED")
|
||||
return r.returncode == 0
|
||||
if kind == "spotify":
|
||||
return play_spotify(cfg, fix, chk, label)
|
||||
if kind == "atem":
|
||||
ok = bool(ATEM_WATCHER) and ATEM_WATCHER.fix(fix["part"])
|
||||
log.info("FIX %s -> %s", label, "switcher matches the restore file again" if ok else "FAILED")
|
||||
@@ -1990,6 +2206,27 @@ def execute_fix(cfg, fix_id, chk):
|
||||
return False
|
||||
|
||||
|
||||
def play_spotify(cfg, fix, chk, label, run=None):
|
||||
"""Play the service playlist in Spotify on this Mac, opening Spotify first if it is closed."""
|
||||
run = run or subprocess.run
|
||||
sp_ = cfg.get("spotify") or {}
|
||||
playlist = spotify_playlist(cfg, chk) or fix.get("playlist")
|
||||
if not playlist:
|
||||
log.warning("FIX %s: no playlist set", label)
|
||||
return False
|
||||
if run(["/usr/bin/pgrep", "-xq", "Spotify"], timeout=5).returncode != 0:
|
||||
run(["/usr/bin/open", "-g", "-a", "Spotify"], timeout=15)
|
||||
time.sleep(sp_.get("open_wait", 8))
|
||||
run(["/usr/bin/osascript", "-e", 'tell application "Spotify" to play track "%s"' % playlist],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
time.sleep(sp_.get("play_wait", 3))
|
||||
r = run(["/usr/bin/osascript", "-e", 'tell application "Spotify" to player state as text'],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
ok = r.returncode == 0 and r.stdout.strip() == "playing"
|
||||
log.info("FIX %s -> %s", label, "playing" if ok else "not playing (%s)" % (r.stdout.strip() or r.stderr.strip()[-80:]))
|
||||
return ok
|
||||
|
||||
|
||||
def reconnect_and_verify(cfg, fix, label, sleep=None, conn=None, host=None, toggle=False):
|
||||
"""Restart one Companion connection (toggle=True: switch it off and on again), then look again
|
||||
after "verify_seconds". Returns True if it came back; otherwise what to tell the operator:
|
||||
@@ -2006,8 +2243,11 @@ def reconnect_and_verify(cfg, fix, label, sleep=None, conn=None, host=None, togg
|
||||
if conn.get("enabled") and not comp.disable_connection(conn["id"]):
|
||||
log.info("FIX %s -> Companion refused to switch %s off", label, fix["conn"])
|
||||
return False
|
||||
sleep(fix.get("off_seconds", 2))
|
||||
if not comp.enable_connection(conn["id"]):
|
||||
try:
|
||||
sleep(fix.get("off_seconds", 2))
|
||||
finally: # whatever happens, it is switched back on
|
||||
on = comp.enable_connection(conn["id"])
|
||||
if not on:
|
||||
log.info("FIX %s -> Companion did not switch %s back on", label, fix["conn"])
|
||||
return "STILL OFF IN\nCOMPANION"
|
||||
log.info("FIX %s -> %s switched off and on", label, fix["conn"])
|
||||
@@ -2058,10 +2298,14 @@ CCU_PARAMS = {
|
||||
}
|
||||
|
||||
|
||||
def atem_reference(xml_path):
|
||||
def atem_reference(xml_path, compare=None):
|
||||
"""The restore file (.xml from ATEM Software Control) as what atem_watch.js compares against:
|
||||
every Fairlight source's mix option and fader, and every camera's settings above."""
|
||||
every Fairlight source's mix option and fader, and every camera's settings above. `compare`
|
||||
("compare" in config "atem") narrows it: {"audio": ["mixOption", "faderGain"], "camera": [...]}."""
|
||||
import xml.etree.ElementTree as ET
|
||||
compare = compare or {}
|
||||
audio_fields = compare.get("audio", ["mixOption", "faderGain"])
|
||||
camera_params = compare.get("camera") # parameter names, e.g. ["ApertureFstop", ...]
|
||||
root = ET.parse(xml_path).getroot()
|
||||
audio = {}
|
||||
mixer = root.find("FairlightAudioMixer")
|
||||
@@ -2074,13 +2318,16 @@ def atem_reference(xml_path):
|
||||
gain = float(src.attrib.get("faderGain", "nan"))
|
||||
except ValueError:
|
||||
gain = float("nan")
|
||||
if math.isfinite(gain): # the file saves dB; the switcher uses 1/100 dB
|
||||
if math.isfinite(gain) and "faderGain" in audio_fields: # the file saves dB; the switcher 1/100 dB
|
||||
want["faderGain"] = int(round(gain * 100))
|
||||
audio["%s/%s" % (inp.attrib["id"], src.attrib["id"])] = want
|
||||
if "mixOption" in audio_fields or "faderGain" in want:
|
||||
audio["%s/%s" % (inp.attrib["id"], src.attrib["id"])] = want
|
||||
camera = {}
|
||||
cc = root.find("CameraControl")
|
||||
for p in (cc if cc is not None else []):
|
||||
spec = CCU_PARAMS.get((p.attrib.get("category"), p.attrib.get("parameter")))
|
||||
if camera_params is not None and p.attrib.get("parameter") not in camera_params:
|
||||
continue
|
||||
if spec and all(f in p.attrib for f in spec[2]):
|
||||
camera["%s/%d/%d" % (p.attrib["device"], spec[0], spec[1])] = [float(p.attrib[f]) for f in spec[2]]
|
||||
return {"audio": audio, "camera": camera}
|
||||
@@ -2100,7 +2347,7 @@ class AtemWatcher:
|
||||
|
||||
def __init__(self, cfg, snap, data_dir, stop):
|
||||
a = cfg["atem"]
|
||||
self.cfg, self.snap, self.stop = a, snap, stop
|
||||
self.atem, self.snap, self.stop = a, snap, stop
|
||||
self.proc, self.lock, self.fixed = None, threading.Lock(), {}
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
self.script = os.path.join(here, "atem_watch.js")
|
||||
@@ -2112,12 +2359,12 @@ class AtemWatcher:
|
||||
if not os.path.isabs(xml):
|
||||
xml = os.path.join(here, xml)
|
||||
with open(self.ref_path, "w") as f:
|
||||
json.dump(atem_reference(xml), f)
|
||||
json.dump(atem_reference(xml, a.get("compare")), f)
|
||||
|
||||
def run(self):
|
||||
while not self.stop.is_set():
|
||||
try:
|
||||
self.proc = subprocess.Popen([self.node, self.script, self.cfg["host"], self.library, self.ref_path],
|
||||
self.proc = subprocess.Popen([self.node, self.script, self.atem["host"], self.library, self.ref_path],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, bufsize=1)
|
||||
errors = deque(maxlen=5) # read all the time, so a chatty stderr never stalls it
|
||||
@@ -2231,9 +2478,53 @@ class Periodic(threading.Thread):
|
||||
|
||||
# --- Main --------------------------------------------------------------------
|
||||
|
||||
def check_windows(windows, where):
|
||||
for w in windows or []:
|
||||
if not isinstance(w, dict) or not isinstance(w.get("days"), list) or not w["days"]:
|
||||
raise ValueError("%s: a window needs days, start and end" % where)
|
||||
for d in w["days"]:
|
||||
if d not in DAYS:
|
||||
raise ValueError("%s: unknown day %r" % (where, d))
|
||||
for k in ("start", "end"):
|
||||
if not re.fullmatch(r"([01]\d|2[0-3]):[0-5]\d", str(w.get(k, ""))):
|
||||
raise ValueError("%s: %s must be HH:MM" % (where, k))
|
||||
|
||||
|
||||
def check_host(cfg, host, where):
|
||||
"""A machine the watchdog logs in to: listed in "hosts", with an ssh address, and not this Mac."""
|
||||
h = cfg.get("hosts", {}).get(host)
|
||||
if h is None:
|
||||
raise ValueError("%s: unknown host %s" % (where, host))
|
||||
if h.get("self"):
|
||||
raise ValueError("%s: %s is this Mac - use a local fix or check" % (where, host))
|
||||
if not h.get("ssh"):
|
||||
raise ValueError("%s: host %s has no ssh address" % (where, host))
|
||||
|
||||
|
||||
def exit_timeout(cfg):
|
||||
"""How long launchd must let the watchdog finish a running fix before killing it: the longest
|
||||
fix plus its verify time plus a margin, never under a minute (install.sh writes it into the plist)."""
|
||||
longest = 0
|
||||
for fix in cfg.get("fixes", {}).values():
|
||||
longest = max(longest, fix.get("timeout", 60) + (fix.get("verify_seconds", 20)
|
||||
if fix.get("kind") in ("reconnect", "hyperdeck") else 0))
|
||||
return max(60, longest + 10)
|
||||
|
||||
|
||||
def load_config(path):
|
||||
with open(path) as f:
|
||||
cfg = json.load(f)
|
||||
for fid, fix in cfg.get("fixes", {}).items():
|
||||
kind = fix.get("kind", "ssh")
|
||||
if kind not in FIX_KINDS:
|
||||
raise ValueError("fix %s: unknown kind %s" % (fid, kind))
|
||||
if kind == "ssh":
|
||||
check_host(cfg, fix.get("host"), "fix " + fid)
|
||||
if kind in ("ssh", "local") and not fix.get("run"):
|
||||
raise ValueError("fix %s: nothing to run" % fid)
|
||||
check_windows(cfg.get("service_windows"), "service_windows")
|
||||
check_windows((cfg.get("auto_fix") or {}).get("windows"), "auto_fix")
|
||||
check_windows((cfg.get("site") or {}).get("live_services"), "site.live_services")
|
||||
ids = set()
|
||||
for t in cfg["tiles"]:
|
||||
if t["id"] in ids:
|
||||
@@ -2254,13 +2545,15 @@ def load_config(path):
|
||||
if (f and f not in (RESTART_CONNECTION, DANTE_AUTO) and cfg["fixes"][f].get("kind", "ssh") == "ssh"
|
||||
and cfg["fixes"][f]["host"] not in cfg.get("hosts", {})):
|
||||
raise ValueError("fix %s: unknown host" % f)
|
||||
if c["type"] == "remote" and c["host"] not in cfg.get("hosts", {}):
|
||||
raise ValueError("tile %s: unknown host %s" % (t["id"], c["host"]))
|
||||
if c["type"] == "remote" or (c["type"] == "midi_peer" and c.get("host")):
|
||||
check_host(cfg, c["host"], "tile " + t["id"])
|
||||
if t.get("hold_always") and t["hold_always"] not in cfg.get("fixes", {}):
|
||||
raise ValueError("tile %s: unknown hold_always fix %s" % (t["id"], t["hold_always"]))
|
||||
for f in (cfg.get("auto_fix") or {}).get("fixes", []):
|
||||
if f not in cfg.get("fixes", {}):
|
||||
raise ValueError("auto_fix: unknown fix %s" % f)
|
||||
if cfg["fixes"][f].get("never_auto"):
|
||||
raise ValueError("auto_fix: %s is marked never to run by itself" % f)
|
||||
if re.search(r"pkill|taskkill|killall", cfg["fixes"][f].get("run", "")) and not cfg["fixes"][f].get("restart_ok"):
|
||||
raise ValueError("auto_fix: %s closes a running program - that stays a hold" % f)
|
||||
for k in cfg.get("restart_page", {}).get("keys", []):
|
||||
@@ -2372,22 +2665,57 @@ def handle_commands(text, engine, comp, snap, now):
|
||||
engine.run_fixes(arg, snap, comp, now)
|
||||
else: # a Restart page key, held 2 seconds
|
||||
engine.run_force(arg, snap, comp, now)
|
||||
elif verb == "test":
|
||||
# The settings page: "test <key> <fix> <check number or identity> <unix time>".
|
||||
if len(words) < 5 or not words[4].isdigit() or now - int(words[4]) > HOLD_MAX_AGE:
|
||||
log.info("TEST %s: no time on it, or too long ago - ignored", arg)
|
||||
else:
|
||||
engine.run_test(arg, words[2], words[3], snap, comp, now)
|
||||
elif verb == "resurvey": # the settings page: look at the Dante switches now
|
||||
snap.dante_map_time = 0.0
|
||||
log.info("RESURVEY: Dante map survey requested")
|
||||
else:
|
||||
log.warning("unknown command %r", part)
|
||||
|
||||
|
||||
def run(cfg, data_dir, dry_run=False, stop=None):
|
||||
def run(cfg, data_dir, dry_run=False, stop=None, config_sha=None):
|
||||
stop = stop or threading.Event()
|
||||
names = cfg["variables"]
|
||||
comp = make_companion(cfg)
|
||||
snap = Snapshot()
|
||||
engine = Engine(cfg, os.path.join(data_dir, "service-mode.json"))
|
||||
engine = Engine(cfg, os.path.join(data_dir, "state.json"), data_dir)
|
||||
snap.dante_map = load_dante_map(dante_map_path(data_dir))
|
||||
site = cfg.get("site", {})
|
||||
prefix = site.get("alert_prefix") or site.get("ingest_name") or "Status panel"
|
||||
alert_after = cfg.get("timing", {}).get("alert_after_seconds", 30)
|
||||
|
||||
def on_term(signum, frame):
|
||||
log.info("stop requested: %d fix(es) running will finish first", len(engine.running))
|
||||
engine.stopping = True
|
||||
stop.set()
|
||||
try:
|
||||
signal.signal(signal.SIGTERM, on_term)
|
||||
except ValueError: # not the main thread (tests)
|
||||
pass
|
||||
threads = start_pollers(cfg, snap, stop, data_dir)
|
||||
if not dry_run:
|
||||
try:
|
||||
# A hold pressed while the watchdog was down must not fire on a config it was not meant for.
|
||||
queued = comp.get_custom(names["command"])
|
||||
if queued and queued.strip():
|
||||
comp.set_custom(names["command"], "")
|
||||
log.info("dropped commands queued before the start: %s", queued.strip())
|
||||
except (OSError, http.client.HTTPException) as e:
|
||||
log.warning("could not read the command variable at start: %s", e)
|
||||
stop.wait(2.5) # let the first polls land before judging anything
|
||||
last_json, last_write, last_eval = None, 0.0, 0.0
|
||||
down_since, alerted, warned_missing = None, False, set()
|
||||
log.info("status panel running (%s)", "dry run" if dry_run else "live")
|
||||
log.info("status panel running (code %s, %s)", VERSION, "dry run" if dry_run else "live")
|
||||
try:
|
||||
write_json(os.path.join(data_dir, "started.json"),
|
||||
{"pid": os.getpid(), "started_at": time.time(), "config_sha": config_sha, "version": VERSION})
|
||||
except OSError as e:
|
||||
log.warning("started.json: %s", e)
|
||||
while not stop.is_set():
|
||||
now = time.time()
|
||||
try:
|
||||
@@ -2428,18 +2756,27 @@ def run(cfg, data_dir, dry_run=False, stop=None):
|
||||
if down_since is not None:
|
||||
log.info("Companion reachable again")
|
||||
if alerted:
|
||||
post_webhook(cfg.get("alert_webhook"), "NA Ingest: Companion is answering again.")
|
||||
post_webhook(cfg.get("alert_webhook"), "%s: Companion is answering again." % prefix)
|
||||
down_since, alerted = None, False
|
||||
except (OSError, http.client.HTTPException) as e:
|
||||
if down_since is None:
|
||||
down_since = now
|
||||
log.warning("Companion unreachable: %s", e)
|
||||
elif not alerted and now - down_since > 30:
|
||||
post_webhook(cfg.get("alert_webhook"), "NA Ingest: Companion has not answered for 30 seconds.")
|
||||
elif not alerted and now - down_since > alert_after:
|
||||
post_webhook(cfg.get("alert_webhook"), "%s: Companion has not answered for %d seconds."
|
||||
% (prefix, alert_after))
|
||||
alerted = True
|
||||
stop.wait(0.25)
|
||||
# A fix that is running finishes before the process goes (launchd waits exit_timeout for us).
|
||||
budget = time.time() + max(5, exit_timeout(cfg) - 5)
|
||||
for t in engine.fix_threads:
|
||||
t.join(timeout=max(0.1, budget - time.time()))
|
||||
if any(t.is_alive() for t in engine.fix_threads):
|
||||
log.warning("stopping with a fix still running")
|
||||
else:
|
||||
log.info("stopped")
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
t.join(timeout=2)
|
||||
|
||||
|
||||
def run_once(cfg):
|
||||
@@ -2483,8 +2820,16 @@ def main(argv=None):
|
||||
ap.add_argument("--once", action="store_true", help="poll once, print the tiles, write nothing")
|
||||
ap.add_argument("--dry-run", action="store_true", help="run but never write to Companion")
|
||||
ap.add_argument("--log", default=os.path.expanduser("~/Library/Logs/status-panel.log"))
|
||||
ap.add_argument("--check", action="store_true", help="load and validate the config, then exit")
|
||||
ap.add_argument("--exit-timeout", action="store_true", help="print the launchd exit timeout this config needs")
|
||||
args = ap.parse_args(argv)
|
||||
cfg = load_config(args.config)
|
||||
if args.check:
|
||||
print("config OK: %d keys, %d fixes" % (len(cfg["tiles"]), len(cfg.get("fixes", {}))))
|
||||
return
|
||||
if args.exit_timeout:
|
||||
print(exit_timeout(cfg))
|
||||
return
|
||||
if args.once:
|
||||
run_once(cfg)
|
||||
return
|
||||
@@ -2494,7 +2839,11 @@ def main(argv=None):
|
||||
if sys.stderr.isatty():
|
||||
log.addHandler(logging.StreamHandler())
|
||||
log.setLevel(logging.INFO)
|
||||
run(cfg, args.data_dir, dry_run=args.dry_run)
|
||||
if not (3, 9) <= sys.version_info[:2] <= (3, 12):
|
||||
log.warning("python %s: the Dante switch reader (telnetlib) needs 3.9-3.12", sys.version.split()[0])
|
||||
with open(args.config, "rb") as f:
|
||||
config_sha = hashlib.sha256(f.read()).hexdigest()
|
||||
run(cfg, args.data_dir, dry_run=args.dry_run, config_sha=config_sha)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
"""schema.py must describe exactly what the code reads: every config key the watchdog and the page
|
||||
tools read is declared, every default in the code equals the schema's, every check type and fix
|
||||
kind is listed, every word a key can show is in the glossary, and today's config still builds
|
||||
today's pages."""
|
||||
import ast
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
import make_page # noqa: E402
|
||||
import schema # noqa: E402
|
||||
import status_panel as sp # noqa: E402
|
||||
|
||||
WALKED = ["status_panel.py", "make_page.py", "push_page.py", "nav_keys.py", "netgear.py"]
|
||||
|
||||
# What a variable holds, by name, when nothing in the function says otherwise (parameters mostly).
|
||||
NAMES = {"cfg": "top", "config": "top", "chk": "check", "c": "check", "fix": "fix", "f": "fix", "t": "tile",
|
||||
"tile": "tile", "k": "restart_key", "key": "restart_key", "rp": "restart_page", "w": "window",
|
||||
"names": "variables", "host": "host", "h": "host", "windows": "windows", "site": "site"}
|
||||
ATTRS = {"self.cfg": "top", "engine.cfg": "top", "self.net": "network", "self.auto": "auto", "self.site": "site",
|
||||
"self.atem": "atem",
|
||||
"self.tiles": "tiles", "engine.tiles": "tiles", "self.restart_keys": "restart_keys",
|
||||
"engine.restart_keys": "restart_keys"}
|
||||
# A key of a top-level dict that holds a record of another kind.
|
||||
TOP = {"companion": "companion", "network": "network", "variables": "variables", "auto_fix": "auto", "site": "site",
|
||||
"atem": "atem", "spotify": "spotify", "timing": "timing", "flap": "flap", "hosts": "hosts", "fixes": "fixes",
|
||||
"tiles": "tiles", "restart_page": "restart_page", "gear": "gears", "service_windows": "windows"}
|
||||
SUB = {("auto", "windows"): "windows", ("site", "live_services"): "windows", ("tile", "checks"): "checks",
|
||||
("restart_page", "keys"): "restart_keys", ("tile", "members"): "names"}
|
||||
ELEMENT = {"tiles": "tile", "checks": "check", "restart_keys": "restart_key", "windows": "window", "fixes": "fix",
|
||||
"hosts": "host", "gears": "gear"}
|
||||
# Keys the code reads that are the watchdog's own working data, not settings.
|
||||
NOT_SETTINGS = {("check", "ip")}
|
||||
|
||||
|
||||
def dotted(node):
|
||||
if isinstance(node, ast.Name):
|
||||
return node.id
|
||||
if isinstance(node, ast.Attribute):
|
||||
base = dotted(node.value)
|
||||
return base + "." + node.attr if base else None
|
||||
return None
|
||||
|
||||
|
||||
def collect(path):
|
||||
"""[(kind, key, default node or None, line)] for every config read in one file."""
|
||||
tree = ast.parse(open(path, encoding="utf-8").read())
|
||||
reads = []
|
||||
|
||||
def kind_of(node, local):
|
||||
if isinstance(node, ast.Name):
|
||||
return local.get(node.id, NAMES.get(node.id))
|
||||
if isinstance(node, ast.Attribute):
|
||||
return ATTRS.get(dotted(node))
|
||||
if isinstance(node, ast.Subscript):
|
||||
base = kind_of(node.value, local)
|
||||
key = node.slice.value if isinstance(node.slice, ast.Constant) else None
|
||||
return step(base, key)
|
||||
if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "get" and node.args:
|
||||
base = kind_of(node.func.value, local)
|
||||
key = node.args[0].value if isinstance(node.args[0], ast.Constant) else None
|
||||
return step(base, key)
|
||||
if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in ("items", "values"):
|
||||
return kind_of(node.func.value, local)
|
||||
if isinstance(node, ast.Call) and dotted(node.func) == "json.load":
|
||||
return "jsonfile" # a config file when it lands in cfg / config
|
||||
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in ("enumerate", "dict", "sorted", "list") and node.args:
|
||||
return kind_of(node.args[0], local)
|
||||
if isinstance(node, ast.BoolOp):
|
||||
return kind_of(node.values[0], local)
|
||||
if isinstance(node, ast.IfExp):
|
||||
return kind_of(node.body, local)
|
||||
return None
|
||||
|
||||
def step(base, key):
|
||||
if base is None:
|
||||
return None
|
||||
if key is None: # dict[variable]: an element
|
||||
return ELEMENT.get(base)
|
||||
if base == "top":
|
||||
return TOP.get(key)
|
||||
return SUB.get((base, key))
|
||||
|
||||
def scan(body, local):
|
||||
for node in ast.walk(ast.Module(body=body, type_ignores=[])):
|
||||
if isinstance(node, ast.FunctionDef):
|
||||
continue
|
||||
if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
|
||||
kind = kind_of(node.value, local)
|
||||
if kind == "jsonfile":
|
||||
kind = "top" if node.targets[0].id in ("cfg", "config") else None
|
||||
local[node.targets[0].id] = kind
|
||||
elif isinstance(node, (ast.For, ast.comprehension)):
|
||||
elem = ELEMENT.get(kind_of(node.iter, local))
|
||||
if isinstance(node.target, ast.Name):
|
||||
local[node.target.id] = elem
|
||||
elif isinstance(node.target, ast.Tuple) and len(node.target.elts) == 2 \
|
||||
and isinstance(node.target.elts[1], ast.Name):
|
||||
local[node.target.elts[1].id] = elem
|
||||
for node in ast.walk(ast.Module(body=body, type_ignores=[])):
|
||||
if isinstance(node, ast.FunctionDef):
|
||||
continue
|
||||
if isinstance(node, ast.Subscript) and isinstance(node.slice, ast.Constant) and isinstance(node.slice.value, str):
|
||||
kind = kind_of(node.value, local)
|
||||
if kind:
|
||||
reads.append((kind, node.slice.value, None, node.lineno))
|
||||
elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "get" \
|
||||
and node.args and isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str):
|
||||
kind = kind_of(node.func.value, local)
|
||||
if kind:
|
||||
reads.append((kind, node.args[0].value, node.args[1] if len(node.args) > 1 else None, node.lineno))
|
||||
|
||||
def functions(node):
|
||||
for child in ast.iter_child_nodes(node):
|
||||
if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
local = {a.arg: NAMES.get(a.arg) for a in child.args.args + child.args.kwonlyargs}
|
||||
# a parameter shadows the name map only when the map knows it; "page", "kind" etc. stay unknown
|
||||
scan(child.body, {k: v for k, v in local.items() if v})
|
||||
functions(child)
|
||||
elif isinstance(child, ast.ClassDef):
|
||||
functions(child)
|
||||
scan([n for n in tree.body if not isinstance(n, (ast.FunctionDef, ast.ClassDef))], {})
|
||||
functions(tree)
|
||||
return reads
|
||||
|
||||
|
||||
def all_reads():
|
||||
out = []
|
||||
for name in WALKED:
|
||||
out += [(name,) + r for r in collect(os.path.join(ROOT, name))]
|
||||
return out
|
||||
|
||||
|
||||
def path_of(kind, key):
|
||||
prefix = {"top": "", "check": "tiles[].checks[].", "tile": "tiles[].", "fix": "fixes.*.", "host": "hosts.*.",
|
||||
"restart_key": "restart_page.keys[].", "restart_page": "restart_page.", "auto": "auto_fix.",
|
||||
"network": "network.", "variables": "variables.", "site": "site.", "atem": "atem.", "spotify": "spotify.",
|
||||
"companion": "companion.", "timing": "timing.", "flap": "flap.", "gear": "gear.*.",
|
||||
"window": "auto_fix.windows[]."}.get(kind)
|
||||
return None if prefix is None else prefix + key
|
||||
|
||||
|
||||
class TestSchemaMatchesCode(unittest.TestCase):
|
||||
def test_every_config_read_is_declared(self):
|
||||
record = schema.key_record()
|
||||
missing = sorted({(kind, key) for _, kind, key, _, _ in all_reads()
|
||||
if kind in record and key not in record[kind] and (kind, key) not in NOT_SETTINGS})
|
||||
self.assertEqual(missing, [], "config keys the code reads that schema.py does not declare")
|
||||
kinds = {kind for _, kind, _, _, _ in all_reads()}
|
||||
self.assertNotIn(None, kinds)
|
||||
unknown = sorted(k for k in kinds if k not in record and k not in ("names", "windows", "checks", "tiles",
|
||||
"fixes", "hosts", "restart_keys", "gears"))
|
||||
self.assertEqual(unknown, [])
|
||||
|
||||
def test_defaults_in_code_match_the_schema(self):
|
||||
defaults = schema.defaults()
|
||||
wrong = []
|
||||
for name, kind, key, default, line in all_reads():
|
||||
if default is None or not isinstance(default, ast.Constant):
|
||||
continue
|
||||
path = path_of(kind, key)
|
||||
if path in defaults and defaults[path] != default.value:
|
||||
wrong.append("%s:%d %s is %r in the code, %r in the schema" % (name, line, path, default.value, defaults[path]))
|
||||
self.assertEqual(wrong, [])
|
||||
|
||||
def test_every_declared_key_is_read_somewhere(self):
|
||||
"""Dead settings are caught; keys only the settings page uses are listed here."""
|
||||
ui_only = {"site.id", "site.name", "site.ingest_hostname", "site.ingest_ips", "site.contacts", "site.editing_code",
|
||||
"site.settings_port", "site.dvs_helper", "site.solo_dir", "hosts.*.name", "hosts.*.role",
|
||||
"hosts.*.notes", "hosts.*.pingable", "hosts.*.runs", "hosts.*.networks", "hosts.*.manual_done",
|
||||
"gear", "gear.*.location", "gear.*.notes", "atem.source_folder", "fixes.*.template", "fixes.*.params",
|
||||
"tiles[].checks[].template", "tiles[].checks[].params", "spotify.name", "site", "timing",
|
||||
"fixes", "hosts", "tiles", "flap", "auto_fix", "network", "variables", "companion", "atem", "spotify",
|
||||
"restart_page", "service_windows", "restart_page.keys", "tiles[].checks", "auto_fix.windows",
|
||||
"site.live_services", "tiles[].checks[].type", "fixes.*.playlist", "tiles[].checks[].playlist"}
|
||||
read = {path_of(kind, key) for _, kind, key, _, _ in all_reads()}
|
||||
read |= {"service_windows[].days", "service_windows[].start", "service_windows[].end",
|
||||
"site.live_services[].days", "site.live_services[].start", "site.live_services[].end"}
|
||||
for w in ("days", "start", "end"):
|
||||
if "auto_fix.windows[]." + w in read:
|
||||
read |= {"service_windows[].%s" % w, "site.live_services[].%s" % w}
|
||||
dead = sorted(p for p in schema.F if p not in read and p not in ui_only)
|
||||
self.assertEqual(dead, [], "declared settings nothing reads")
|
||||
|
||||
def test_check_types_fix_kinds_and_subs(self):
|
||||
self.assertEqual(set(schema.CHECKS), set(sp.CHECKS))
|
||||
self.assertEqual(tuple(schema.FIX_KINDS), sp.FIX_KINDS)
|
||||
for t, spec in schema.CHECKS.items():
|
||||
for f in spec["fields"] + spec["identity"]:
|
||||
self.assertIn("tiles[].checks[]." + f, schema.F, "%s field %s" % (t, f))
|
||||
for k in sp.FIX_KINDS:
|
||||
self.assertIn(k, schema.FIX_KIND_FIELDS)
|
||||
|
||||
def test_glossary_covers_every_word_a_key_can_show(self):
|
||||
words = set()
|
||||
for group in schema.GLOSSARY.values():
|
||||
for phrase in group:
|
||||
words |= set(phrase.split())
|
||||
tree = ast.parse(open(os.path.join(ROOT, "status_panel.py"), encoding="utf-8").read())
|
||||
texts = []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Call):
|
||||
name = node.func.id if isinstance(node.func, ast.Name) else (node.func.attr if isinstance(node.func, ast.Attribute) else "")
|
||||
if name == "Result" and len(node.args) > 1 and isinstance(node.args[1], ast.Constant):
|
||||
texts.append(node.args[1].value)
|
||||
elif name == "note" and len(node.args) > 1 and isinstance(node.args[1], ast.Constant):
|
||||
texts.append(node.args[1].value)
|
||||
elif name == "note" and len(node.args) > 1 and isinstance(node.args[1], ast.BinOp):
|
||||
for part in ast.walk(node.args[1]):
|
||||
if isinstance(part, ast.Constant) and isinstance(part.value, str):
|
||||
texts.append(part.value)
|
||||
unknown = set()
|
||||
for text in texts:
|
||||
for word in re.findall(r"[A-Z][A-Z-]+", text):
|
||||
if word not in words:
|
||||
unknown.add(word)
|
||||
self.assertEqual(sorted(unknown), [], "words a key can show that the glossary does not explain")
|
||||
|
||||
def test_todays_config_builds_todays_pages(self):
|
||||
cfg = sp.load_config(os.path.join(ROOT, "config.json"))
|
||||
|
||||
def strip(export):
|
||||
for row in export["page"]["controls"].values():
|
||||
for ctl in row.values():
|
||||
if ctl and "options" in ctl:
|
||||
ctl["options"].pop("notes", None)
|
||||
return export
|
||||
for build, name in ((make_page.build, "status-page.companionconfig"), (make_page.build_restart, "restart-page.companionconfig")):
|
||||
on_disk = json.loads(gzip.open(os.path.join(ROOT, name)).read())
|
||||
self.assertEqual(strip(build(cfg)), strip(on_disk), name)
|
||||
self.assertNotIn("restart_page", {k for k in cfg if k == "nothing"})
|
||||
bare = dict(cfg)
|
||||
bare.pop("restart_page")
|
||||
export = make_page.build(bare)
|
||||
notes = [((ctl.get("options") or {}).get("notes") or "") for row in export["page"]["controls"].values()
|
||||
for ctl in row.values() if ctl]
|
||||
self.assertFalse(any("STATUS jump key" in n for n in notes)) # no Restart page: no jump key
|
||||
|
||||
def test_todays_config_passes_the_rules(self):
|
||||
cfg = sp.load_config(os.path.join(ROOT, "config.json"))
|
||||
errors, warnings = schema.validate(cfg)
|
||||
self.assertEqual(errors, [])
|
||||
self.assertTrue(all("only during a service" in w for w in warnings), warnings)
|
||||
|
||||
def test_rules_catch_the_classic_mistakes(self):
|
||||
cfg = sp.load_config(os.path.join(ROOT, "config.json"))
|
||||
bad = json.loads(json.dumps(cfg))
|
||||
bad["tiles"][7]["pos"] = bad["tiles"][8]["pos"] # two keys on one cell
|
||||
bad["tiles"].append({"id": "all", "label": "X", "pos": [3, 8], "kind": "master"})
|
||||
bad["fixes"]["vista"]["host"] = "nowhere"
|
||||
bad["restart_page"]["keys"][0]["fix"] = "no-such-fix"
|
||||
bad["hosts"]["bibletv"]["mac"] = "not-a-mac"
|
||||
bad["auto_fix"]["fixes"].append("propresenter") # closes a program
|
||||
bad["fixes"]["ableton"]["never_auto"] = True # ...and ableton is in the auto list
|
||||
errors, _ = schema.validate(bad)
|
||||
text = "\n".join(errors)
|
||||
for expect in ("share row", "two keys with the same id", "exactly one ALL key", "unknown computer nowhere",
|
||||
"unknown fix no-such-fix", "not a hardware", "closes a running program", "never to run by itself"):
|
||||
self.assertIn(expect, text)
|
||||
home = json.loads(json.dumps(cfg))
|
||||
home["hosts"]["ingest"] = {"self": True, "name": "NA Ingest"}
|
||||
home["fixes"]["vista"]["host"] = "ingest"
|
||||
self.assertIn("is this Mac", "\n".join(schema.validate(home)[0]))
|
||||
|
||||
def test_live_warnings(self):
|
||||
cfg = sp.load_config(os.path.join(ROOT, "config.json"))
|
||||
live = {"connections": {"HyperDeck_2": "generic-tcp-udp"}}
|
||||
_, warnings = schema.validate(cfg, live)
|
||||
self.assertTrue(any("HyperDeck_2 uses module generic-tcp-udp" in w for w in warnings), warnings)
|
||||
self.assertTrue(any("no connection called HyperDeck_1" in w for w in warnings))
|
||||
|
||||
def test_secrets_are_masked(self):
|
||||
masked = schema.mask_secrets({"alert_webhook": "https://hooks/abc", "site": {"editing_code": "apple bee", "name": "NA"},
|
||||
"fixes": {"x": {"run": "ls"}}})
|
||||
self.assertEqual(masked["alert_webhook"], "set (17 characters)")
|
||||
self.assertEqual(masked["site"]["editing_code"], "set (9 characters)")
|
||||
self.assertEqual(masked["site"]["name"], "NA")
|
||||
self.assertEqual(masked["fixes"]["x"]["run"], "ls")
|
||||
|
||||
def test_phrases(self):
|
||||
cfg = sp.load_config(os.path.join(ROOT, "config.json"))
|
||||
p = schema.phrases(cfg)
|
||||
self.assertEqual(p["auto_fix"], "may run by itself on Sundays 06:30-13:30")
|
||||
self.assertEqual(p["live"], "changes wait while a HyperDeck is recording")
|
||||
self.assertEqual(p["hold"], "HOLD 2s")
|
||||
self.assertEqual(sp.windows_phrase([{"days": ["sat", "sun"], "start": "18:00", "end": "00:30"}]),
|
||||
"on Saturdays and Sundays 18:00-00:30")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+148
-90
@@ -130,7 +130,8 @@ class TestChecks(unittest.TestCase):
|
||||
s.spotify_recent.append("spotify:track:q")
|
||||
self.assertEqual(msg().msg, "WRONG PLAYLIST") # full list: one unknown is enough
|
||||
fix = CONFIG["fixes"][chk["fix"]]
|
||||
self.assertIn(chk["playlist"], fix["run"])
|
||||
self.assertEqual(fix["kind"], "spotify")
|
||||
self.assertEqual(sp.spotify_playlist(CONFIG, chk), CONFIG["spotify"]["playlist"])
|
||||
self.assertIn(chk["fix"], CONFIG["auto_fix"]["fixes"])
|
||||
|
||||
def test_connection_states(self):
|
||||
@@ -1352,93 +1353,145 @@ class TestFixes(unittest.TestCase):
|
||||
self.assertEqual(sp.parse_remote("SP0=0\nSP1=1\n", 2), [True, False])
|
||||
self.assertEqual(sp.parse_remote("", 2), [None, None])
|
||||
|
||||
def test_windows_remote_checks_use_conditional_echoes(self):
|
||||
line = sp.remote_script(["tasklist | find /I \"Vista.exe\"", "sc query rtpMIDIService | find \"RUNNING\""], "cmd")
|
||||
self.assertNotIn("%ERRORLEVEL%", line)
|
||||
self.assertIn("&& echo SP0=0 || echo SP0=1", line)
|
||||
self.assertIn("&& echo SP1=0 || echo SP1=1", line)
|
||||
self.assertEqual(sp.parse_remote("SP0=0\r\nSP1=1\r\n", 2), [True, False])
|
||||
self.assertEqual(sp.host_shell(CONFIG, "lighting"), "cmd")
|
||||
self.assertEqual(sp.host_shell(CONFIG, "tracks"), "sh")
|
||||
|
||||
def test_overnight_windows(self):
|
||||
sat_2350 = datetime(2026, 10, 10, 23, 50).timestamp()
|
||||
sun_0020 = datetime(2026, 10, 11, 0, 20).timestamp()
|
||||
sun_0100 = datetime(2026, 10, 11, 1, 0).timestamp()
|
||||
w = [{"days": ["sat"], "start": "18:00", "end": "00:30"}]
|
||||
self.assertTrue(sp.in_windows(w, sat_2350))
|
||||
self.assertTrue(sp.in_windows(w, sun_0020)) # past midnight, still Saturday's service
|
||||
self.assertFalse(sp.in_windows(w, sun_0100))
|
||||
self.assertFalse(sp.in_windows([{"days": ["sun"], "start": "18:00", "end": "00:30"}], sat_2350))
|
||||
|
||||
def test_check_identity_is_about_the_thing_not_the_position(self):
|
||||
midi = [c for c in CONFIG["tiles"][9]["checks"] if c["type"] == "midi_peer"]
|
||||
self.assertEqual(len({sp.check_identity(c) for c in midi}), len(midi)) # six peers, six identities
|
||||
self.assertEqual(sp.check_identity({"type": "midi_peer", "host": "tracks", "session": "NA Tracks", "peer": "Lighting PC"}),
|
||||
"tracks/NA Tracks/Lighting PC")
|
||||
self.assertEqual(sp.check_identity({"type": "ping", "conn": "Axient_1", "host": "10.1.1.101"}), "Axient_1")
|
||||
self.assertEqual(sp.check_identity({"type": "conn", "conn": "ATEM"}), "ATEM")
|
||||
|
||||
def test_acknowledgements_survive_a_restart(self):
|
||||
state = os.path.join(tempfile.mkdtemp(prefix="sp-state-"), "state.json")
|
||||
engine = sp.Engine(CONFIG, state)
|
||||
engine.executor = engine.poe_executor = lambda *a: True
|
||||
snap = healthy_snapshot(SUNDAY)
|
||||
snap.conns["ATEM"]["category"] = "error"
|
||||
run_steps(engine, snap, SUNDAY, 5)
|
||||
engine.ack("atem")
|
||||
self.assertEqual(level(run_steps(engine, snap, SUNDAY + 6, 1)[0]["atem"]), "ack")
|
||||
saved = json.load(open(state))
|
||||
self.assertIn("atem|ATEM|conn", saved["acks"])
|
||||
again = sp.Engine(CONFIG, state) # the watchdog restarts
|
||||
again.executor = again.poe_executor = lambda *a: True
|
||||
tiles, _ = run_steps(again, snap, SUNDAY + 10, 5)
|
||||
self.assertEqual(level(tiles["atem"]), "ack") # still acknowledged, no new flashing
|
||||
snap.conns["ATEM"]["category"] = "good"
|
||||
run_steps(again, snap, SUNDAY + 20, 2)
|
||||
self.assertEqual(json.load(open(state))["acks"], []) # cleared: forgotten
|
||||
|
||||
def test_run_this_fix_now_goes_through_the_hold_path(self):
|
||||
ran = []
|
||||
self.engine.executor = lambda cfg, fix_id, chk: ran.append((fix_id, chk.get("conn"), chk.get("ip"), chk.get("peer"))) or True
|
||||
snap = self.snap
|
||||
run_steps(self.engine, snap, SUNDAY, 2)
|
||||
comp = mock_comp()
|
||||
now = SUNDAY + 3
|
||||
self.engine.run_test("hd2", "hyperdeck", "0", snap, comp, now) # the check's number...
|
||||
self.engine.run_test("midi", "midi_reconnect", "tracks/NA Tracks/Studio B Mac Studio", snap, comp, now) # ...or its identity
|
||||
self.engine.run_test("midi", "midi_reconnect", "9", snap, comp, now) # no such check
|
||||
self.engine.run_test("atem", "atem_audio", "1", snap, comp, now) # never from the page
|
||||
self.engine.run_test("hd2", "propresenter", "0", snap, comp, now) # not attached to that key
|
||||
for t in self.engine.fix_threads:
|
||||
t.join(2)
|
||||
self.assertEqual(ran, [("hyperdeck", "HyperDeck_2", "10.1.10.12", None),
|
||||
("midi_reconnect", None, None, "Studio B Mac Studio")])
|
||||
self.assertEqual(self.engine.notes["hd2"]["text"], "DONE")
|
||||
|
||||
def test_a_restart_lets_the_running_fix_finish_and_starts_nothing_new(self):
|
||||
gate = threading.Event()
|
||||
self.engine.executor = lambda cfg, fix_id, chk: gate.wait(3) or True
|
||||
run_steps(self.engine, self.snap, SUNDAY, 2)
|
||||
self.engine.run_force("rs_bridge", self.snap, mock_comp(), SUNDAY + 3)
|
||||
self.assertEqual(len(self.engine.running), 1) # published for the settings page
|
||||
self.engine.stopping = True # SIGTERM arrived
|
||||
self.engine.run_force("rs_captions", self.snap, mock_comp(), SUNDAY + 4)
|
||||
self.assertEqual(self.engine.notes["rs_captions"]["text"], "RESTARTING\nSOON")
|
||||
self.assertEqual(self.engine.run_fixes("ingest", self.snap, mock_comp(), SUNDAY + 4), [])
|
||||
gate.set()
|
||||
for t in self.engine.fix_threads:
|
||||
t.join(2)
|
||||
self.assertEqual(self.engine.running, {})
|
||||
self.assertEqual(sp.exit_timeout(CONFIG), 90) # the longest fix + its verify + 10
|
||||
|
||||
def test_a_hyperdeck_switched_off_is_always_switched_back_on(self):
|
||||
calls = []
|
||||
|
||||
class Comp:
|
||||
def connections(self):
|
||||
return [{"label": "HyperDeck_2", "id": "x", "enabled": True, "status": {"category": "good"}}]
|
||||
|
||||
def disable_connection(self, cid):
|
||||
calls.append("off")
|
||||
return True
|
||||
|
||||
def enable_connection(self, cid):
|
||||
calls.append("on")
|
||||
return True
|
||||
import unittest.mock as mock
|
||||
|
||||
def boom(seconds):
|
||||
raise RuntimeError("killed mid-way")
|
||||
with mock.patch.object(sp, "Companion", lambda *a, **k: Comp()):
|
||||
with self.assertRaises(RuntimeError):
|
||||
sp.reconnect_and_verify(CONFIG, CONFIG["fixes"]["hyperdeck"], "x", sleep=boom, conn="HyperDeck_2", toggle=True)
|
||||
self.assertEqual(calls, ["off", "on"])
|
||||
|
||||
def test_spotify_fix_opens_and_plays(self):
|
||||
calls = []
|
||||
|
||||
class R:
|
||||
def __init__(self, code, out=""):
|
||||
self.returncode, self.stdout, self.stderr = code, out, ""
|
||||
|
||||
def run(argv, **kw):
|
||||
calls.append(argv)
|
||||
if argv[1] == "-xq":
|
||||
return R(1) # Spotify is closed
|
||||
if "player state" in argv[-1]:
|
||||
return R(0, "playing\n")
|
||||
return R(0)
|
||||
import unittest.mock as mock
|
||||
with mock.patch.object(sp.time, "sleep", lambda s: None):
|
||||
ok = sp.play_spotify(CONFIG, CONFIG["fixes"]["spotify_play"], {}, "spotify", run=run)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(calls[1][:3], ["/usr/bin/open", "-g", "-a"])
|
||||
self.assertIn(CONFIG["spotify"]["playlist"], calls[2][-1])
|
||||
|
||||
def test_never_auto_fixes_stay_a_hold(self):
|
||||
cfg = json.loads(json.dumps(CONFIG))
|
||||
cfg["fixes"]["ableton"]["never_auto"] = True
|
||||
cfg["auto_fix"]["fixes"].remove("ableton")
|
||||
engine = sp.Engine(cfg, None)
|
||||
self.assertNotIn("ableton", {f for f in engine.auto.get("fixes", []) if not engine.cfg["fixes"][f].get("never_auto")})
|
||||
cfg["auto_fix"]["fixes"].append("ableton")
|
||||
path = os.path.join(tempfile.mkdtemp(prefix="sp-cfg-"), "config.json")
|
||||
json.dump(cfg, open(path, "w"))
|
||||
with self.assertRaises(ValueError):
|
||||
sp.load_config(path)
|
||||
|
||||
def test_every_fix_is_valid(self):
|
||||
for f in CONFIG["fixes"].values():
|
||||
self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect", "reconnect", "atem"))
|
||||
|
||||
|
||||
class FakeDeck:
|
||||
"""A pretend HyperDeck on a local port: answers like a real one, stays silent, or greets only."""
|
||||
|
||||
def __init__(self, mode):
|
||||
import socket as _s
|
||||
self.mode, self.received = mode, b""
|
||||
self.srv = _s.socket()
|
||||
self.srv.bind(("127.0.0.1", 0))
|
||||
self.srv.listen(5)
|
||||
self.port = self.srv.getsockname()[1]
|
||||
threading.Thread(target=self.serve, daemon=True).start()
|
||||
|
||||
def serve(self):
|
||||
while True:
|
||||
try:
|
||||
conn, _ = self.srv.accept()
|
||||
except OSError:
|
||||
return
|
||||
with conn:
|
||||
conn.settimeout(4)
|
||||
if self.mode != "silent":
|
||||
conn.sendall(b"500 connection info:\r\nprotocol version: 1.11\r\nmodel: HyperDeck Studio 4K Pro\r\n\r\n")
|
||||
try:
|
||||
data = conn.recv(1024)
|
||||
self.received += data
|
||||
if self.mode in ("record", "preview") and b"transport info" in data:
|
||||
conn.sendall(b"208 transport info:\r\nstatus: %s\r\nspeed: 100\r\n\r\n" % self.mode.encode())
|
||||
more = conn.recv(1024)
|
||||
self.received += more
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def close(self):
|
||||
self.srv.close()
|
||||
|
||||
|
||||
class TestHyperDeckProbe(unittest.TestCase):
|
||||
def test_probe_answers(self):
|
||||
for mode, expected in (("record", "record"), ("preview", "preview"), ("greet", "answered"), ("silent", "hung")):
|
||||
deck = FakeDeck(mode)
|
||||
try:
|
||||
self.assertEqual(sp.hyperdeck_probe("127.0.0.1", timeout=1, port=deck.port), expected, mode)
|
||||
finally:
|
||||
deck.close()
|
||||
deck = FakeDeck("preview")
|
||||
port = deck.port
|
||||
deck.close()
|
||||
self.assertEqual(sp.hyperdeck_probe("127.0.0.1", timeout=1, port=port), "unreachable")
|
||||
|
||||
def test_the_hold_switches_the_connection_off_and_on_and_never_reboots(self):
|
||||
from unittest import mock
|
||||
deck = FakeDeck("silent") # even a deck that would once have been rebooted
|
||||
try:
|
||||
comp = mock.Mock()
|
||||
comp.connections.side_effect = [
|
||||
[{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "error"}}],
|
||||
[{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "good"}}]]
|
||||
comp.disable_connection.return_value = comp.enable_connection.return_value = True
|
||||
chk = {"conn": "HyperDeck_1", "ip": "127.0.0.1", "port": deck.port}
|
||||
with mock.patch.object(sp, "Companion", return_value=comp), mock.patch.object(sp.time, "sleep"):
|
||||
self.assertTrue(sp.execute_fix(CONFIG, "hyperdeck", chk))
|
||||
self.assertEqual([c[0] for c in comp.method_calls if c[0].endswith("_connection")],
|
||||
["disable_connection", "enable_connection"])
|
||||
comp.restart_connection.assert_not_called()
|
||||
time.sleep(0.3)
|
||||
self.assertNotIn(b"reboot", deck.received)
|
||||
finally:
|
||||
deck.close()
|
||||
|
||||
def test_a_deck_that_stays_offline_says_so(self):
|
||||
from unittest import mock
|
||||
fix = CONFIG["fixes"]["hyperdeck"]
|
||||
for pings, expect in ((True, "STILL OFFLINE\nHOLD AGAIN"), (False, "NO NETWORK\nCHECK POWER")):
|
||||
comp = mock.Mock()
|
||||
comp.connections.side_effect = [
|
||||
[{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}],
|
||||
[{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}]]
|
||||
with mock.patch.object(sp, "Companion", return_value=comp), \
|
||||
mock.patch.object(sp, "ping_once", return_value=pings) as ping:
|
||||
self.assertEqual(sp.reconnect_and_verify(CONFIG, fix, "x", sleep=lambda s: None,
|
||||
conn="HyperDeck_2", host="10.1.10.12", toggle=True), expect)
|
||||
ping.assert_called_once_with("10.1.10.12") # the deck's own address, from Companion
|
||||
self.assertIn(f.get("kind", "ssh"), sp.FIX_KINDS)
|
||||
|
||||
|
||||
class TestNetgearParsing(unittest.TestCase):
|
||||
@@ -1679,10 +1732,15 @@ class TestPage(unittest.TestCase):
|
||||
nav = {"0": {"0": {"type": "pageup"}}, "1": {"0": {"type": "button-layered",
|
||||
"options": {"notes": "STATUS nav key (pagenum): x"}}}}
|
||||
self.assertTrue(push_page.safe_target({"page": {"name": "PAGE", "controls": nav}}, "23\\nRestart")[0])
|
||||
busy = {"page": {"name": "PAGE", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}}
|
||||
self.assertFalse(push_page.safe_target(busy, "23\\nRestart")[0]) # pages 22 and 97
|
||||
busy["page"]["name"] = "23\\nRestart"
|
||||
self.assertTrue(push_page.safe_target(busy, "23\\nRestart")[0]) # our own page again
|
||||
ours = {"page": {"name": "whatever", "controls": dict(nav, **{"3": {"7": {"type": "button-layered",
|
||||
"options": {"notes": "STATUS tile 'led' - drawn by status_panel.py on NA Ingest."}},
|
||||
"6": {"type": "button-layered", "options": {"notes": "STATUS jump key - Tap: open the Restart page."}}}})}}
|
||||
self.assertTrue(push_page.safe_target(ours, "23\\nRestart")[0]) # our keys, whatever the page is called
|
||||
busy = {"page": {"name": "23\\nRestart", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}}
|
||||
ok, why = push_page.safe_target(busy, "23\\nRestart")
|
||||
self.assertFalse(ok) # a key someone made by hand: refused
|
||||
self.assertIn("row 2 col 3", why) # ...and named by its place
|
||||
self.assertIn("move or delete them first", why)
|
||||
|
||||
def test_each_button_reads_its_own_tile(self):
|
||||
for t in CONFIG["tiles"]:
|
||||
|
||||
Reference in New Issue
Block a user