Step 0 of the settings page: registry, safe restart, remembered state, test verb, page markers

- schema.py: every setting with type, default, help, owner screen and the rules a config must pass; tests/test_schema.py keeps it in step with the code (every key the code reads is declared, every default matches, check types and fix kinds agree, the glossary covers every word a key can show, today's config still builds today's pages apart from the key notes) - status_panel.py: VERSION + started.json; SIGTERM lets a running fix finish and starts nothing new (RESTARTING SOON); fixes-running.json and live-now.json for the settings page; state.json keeps the ALARMS override, acknowledgements, auto-fix tries and fix times across restarts, keyed by what a check is about rather than its position; 'test' and 'resurvey' commands; queued commands dropped at start; overnight windows; a HyperDeck switched off is always switched back on; the spotify fix kind (playlist set once, in config 'spotify'); Windows remote checks batched for cmd.exe with conditional echoes; every fix's kind and host validated; never_auto; wording and numbers from config (site block, timing, page_hold_ms, auto-fix keep_months); hyperdeck_probe removed; --check and --exit-timeout - make_page.py: numbers from config, markers on every key, nav templates per Companion version, --out; push_page.py: a page is overwritten only when every key carries a marker (foreign keys named by position), secret-free fetch, host from the site; nav_keys.py: host from the site; netgear.py: switches and the shared-port limit from config - install.sh: validates with the new code first, keeps prev/ for --rollback, --keep-config, derived ExitTimeOut, wider deploy list, waits for the start stamp - config.json: site, timing, spotify sections; Windows hosts marked; deployed to NA Ingest and both pages re-imported once (--force) so every key carries its marker Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 16:00:41 -04:00
parent a3934b7c62
commit 5d66805687
13 changed files with 1822 additions and 307 deletions
@@ -139,15 +139,17 @@ A problem has to last 3 seconds before it shows, so one bad reading doesn't trig
 This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:
 ```bash
-scp status_panel.py coremidi.py netgear.py config.json install.sh checklist-ingest:status-panel-install/
+scp status_panel.py coremidi.py netgear.py make_page.py push_page.py nav_keys.py nav_templates.json config.json install.sh atem_watch.js atem_state.js atem-reference.xml checklist-ingest:status-panel-install/
 ```
-Then run the installer there. It backs up the old config and restarts the watchdog:
+Then run the installer there. It checks the config with the new code first (a bad config never gets as far as the watchdog), keeps the code that was running in `prev/`, backs up the old config, writes the LaunchAgent with the exit timeout this config needs (the longest fix plus its verify time, so a restart never cuts a fix in half) and restarts the watchdog:
 ```bash
 ssh checklist-ingest 'zsh ~/status-panel-install/install.sh'
 ```
 `install.sh --keep-config` updates the code and leaves the live config alone; `install.sh --rollback` puts the previous code back.
 If you add or move tiles, rebuild the page:
 ```bash
@@ -172,6 +174,8 @@ zsh install.sh --remove                 # stop the watchdog
 The log is `~/Library/Logs/status-panel.log` on NA Ingest. Every fault and clear is written there with a timestamp, which is handy for a post-service review.
 What the watchdog keeps in its data folder (`~/Library/Application Support/status-panel/`): `state.json` (the ALARMS override, acknowledged faults, auto-fix tries and when each fix last ran, so a restart forgets nothing), `started.json` (when it started, with which config), `live-now.json` (whether a HyperDeck is recording: changes to the panel wait while it is), `fixes-running.json` (the fixes running right now). A restart (`launchctl kickstart -k`, or the installer) lets a running fix finish first and starts nothing new meanwhile: keys say RESTARTING SOON. Windows in `config.json` may run past midnight (`"end"` earlier than `"start"`), and a remote check on a Windows PC (`"os": "windows"` on the host) is batched for cmd.exe.
 ## Optional: see what the tracks send to the lights
 The TRACKS tile can show when the tracks rig last sent MIDI ("TRK MIDI 3s"). For that, NA Ingest has to *listen* to the "NA Tracks" session. Use a **separate** session so nothing Companion sends can reach Ableton:
@@ -204,7 +208,8 @@ To go back, pick 1.3.0 on the connection again.
 | `status_panel.py` | the watchdog |
 | `netgear.py` | reads the Netgear Dante switches (MAC tables, PoE) and power-cycles one PoE port |
 | `coremidi.py` | reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs) |
-| `config.json` | tiles, positions, checks, service windows |
+| `config.json` | tiles, positions, checks, fixes, hosts, windows: every setting (the `site` block names the campus; `spotify` holds the playlist; `timing` and `page_hold_ms` the numbers behind HOLD 2s and WAIT 1 MIN) |
 | `schema.py` | the registry of every setting: type, default, help, which screen of the settings page owns it, and the rules a config must pass (`tests/test_schema.py` keeps it in step with the code) |
 | `make_page.py` | builds `status-page.companionconfig` from `config.json` |
 | `install.sh` | installs or updates the LaunchAgent on NA Ingest |
 | `push_page.py` | imports the status page into page 19 over Companion's API |
@@ -212,4 +217,4 @@ To go back, pick 1.3.0 on the connection again.
 | `nav_templates.json` | Companion's own nav key designs, coloured by `make_page.py` |
 | `remote-access/` | one-time setup scripts for SSH to the Macs and Windows PCs |
 | `tascam-quiet/` | builds the quiet Tascam module |
-| `tests/` | offline tests, including a fake Companion and a MIDI loopback |
+| `tests/` | offline tests, including a fake Companion and a MIDI loopback; `test_schema.py` checks that every key the code reads is declared in `schema.py` with the same default |
@@ -1,4 +1,9 @@
 {
  "site": {
    "id": "na", "name": "North Attleboro", "ingest_name": "NA Ingest", "ingest_hostname": "na-ingest.local",
    "ingest_ips": ["10.1.1.1", "10.6.3.222"],
    "live_services": []
  },
  "companion": { "host": "127.0.0.1", "port": 8000 },
  "page": 19,
  "page_name": "19\\nStatus",
@@ -24,6 +29,10 @@
  },
  "alert_webhook": "",
  "timing": { "fix_cooldown_seconds": 60, "alert_after_seconds": 30 },
  "page_hold_ms": 2000,
  "spotify": { "playlist": "spotify:playlist:2uTpJVYQLFnKF1WnSOuco8", "name": "Christian EDM 2026", "open_wait": 8, "play_wait": 3 },
  "ssh_key": "~/.ssh/id_status_panel",
@@ -32,10 +41,10 @@
    "prop":        { "ssh": "napropresenter@10.1.14.225" },
    "tracks":      { "ssh": "waterschurchmedia@10.1.3.227" },
    "translation": { "ssh": "studio@10.1.7.39" },
-    "bibletv":     { "ssh": "media@10.6.7.13" },
+    "bibletv":     { "ssh": "media@10.6.7.13", "os": "windows" },
-    "lighting":    { "ssh": "media@10.1.10.206" },
+    "lighting":    { "ssh": "media@10.1.10.206", "os": "windows" },
    "studiob":     { "ssh": "studiob@10.1.13.175" },
-    "livestream":  { "ssh": "media@10.1.9.126", "mac": "6c:24:08:d7:fb:57", "wake_broadcast": "10.1.255.255" }
+    "livestream":  { "ssh": "media@10.1.9.126", "os": "windows", "mac": "6c:24:08:d7:fb:57", "wake_broadcast": "10.1.255.255" }
  },
  "fixes": {
@@ -72,8 +81,7 @@
    "atem_midi":       { "kind": "local", "short": "ATEM MIDI", "restart_ok": true,
                         "label": "turn off MIDI control in ATEM Software Control (quits and reopens the app; the switcher is not affected)",
                         "run": "was=$(pgrep -f \"ATEM Software Control.app/Contents/MacOS\"); pkill -TERM -f \"ATEM Software Control.app/Contents/MacOS\"; i=0; while pgrep -fq \"ATEM Software Control.app/Contents/MacOS\" && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; defaults write \"com.blackmagic-design.ATEM Software Control\" midiEnabled -bool false; [ -z \"$was\" ] || open -g -a \"ATEM Software Control\"; [ \"$(defaults read \"com.blackmagic-design.ATEM Software Control\" midiEnabled)\" = 0 ]" },
-    "spotify_play":    { "kind": "local", "short": "SPOTIFY", "label": "play the Christian EDM 2026 playlist in Spotify on NA Ingest (opens Spotify if it is closed)",
+    "spotify_play":    { "kind": "spotify", "short": "SPOTIFY", "label": "play the Christian EDM 2026 playlist in Spotify on NA Ingest (opens Spotify if it is closed)" },
                         "run": "pgrep -xq Spotify || { open -g -a Spotify; sleep 8; }; osascript -e 'tell application \"Spotify\" to play track \"spotify:playlist:2uTpJVYQLFnKF1WnSOuco8\"'; sleep 3; [ \"$(osascript -e 'tell application \"Spotify\" to player state as text')\" = playing ]" },
    "proddash":        { "kind": "local", "short": "PRODDASH", "label": "restart ProdDash (its launcher stops its server cleanly)",
                         "run": "pkill -TERM -x ProdDash; i=0; while pgrep -xq ProdDash && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; open -g /Applications/ProdDash.app" },
    "prodcom":         { "kind": "local", "short": "PRODCOM", "label": "restart ProdCom (transcription; lobby captions blank for 5-15 s)",
@@ -170,7 +178,7 @@
      { "type": "midi_rx", "source": "Network NA Ingest Listen", "name": "TRACKS IN" }
    ] },
    { "id": "spotify", "label": "SPOTIFY", "pos": [2, 7], "hold_always": "spotify_play", "checks": [
-      { "type": "spotify", "playlist": "spotify:playlist:2uTpJVYQLFnKF1WnSOuco8", "name": "Christian EDM 2026", "fix": "spotify_play" }
+      { "type": "spotify", "fix": "spotify_play" }
    ] },
    { "id": "vista", "label": "VISTA", "pos": [2, 5], "checks": [
      { "type": "artnet", "source": "10.1.10.206", "max_silence_seconds": 5, "fix": "vista" }
@@ -1,9 +1,14 @@
 #!/bin/zsh
-# Install or update the status panel watchdog on this Mac (NA Ingest) as a LaunchAgent.
+# Install or update the status panel watchdog on this Mac (the Ingest Mac) as a LaunchAgent.
-# Re-run after changing status_panel.py, coremidi.py or config.json in this folder.
+# Re-run after changing any file in this folder.
 #
 #   zsh install.sh                 install / update and (re)start
 #   zsh install.sh --keep-config   update the code but leave the live config.json as it is
 #   zsh install.sh --rollback      put the previous code back (kept in prev/) and restart
 #   zsh install.sh --remove        stop it and remove the LaunchAgent (files stay)
 #
 # The config is validated with the NEW code before anything is touched, so a stricter check can
 # never leave the watchdog crash-looping. The previous code is kept in prev/ for --rollback.
 set -e
 SRC=${0:A:h}
 LABEL=org.waterschurch.status-panel
@@ -11,22 +16,28 @@ LABEL=org.waterschurch.status-panel
 DEST="$HOME/Library/Application Support/status-panel"
 PLIST="$HOME/Library/LaunchAgents/$LABEL.plist"
 DOMAIN="gui/$(id -u)"
 CODE=(status_panel.py coremidi.py netgear.py make_page.py push_page.py nav_keys.py atem_watch.js atem_state.js nav_templates.json)
 KEEP_CONFIG=0
 for arg in "$@"; do
  case "$arg" in
    --keep-config) KEEP_CONFIG=1 ;;
    --remove|--rollback) ;;
    *) echo "unknown option $arg" >&2; exit 2 ;;
  esac
 done
-if [[ "$1" == "--remove" ]]; then
+restart_agent() {
  launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
-  rm -f "$PLIST"
+  # bootout returns before the old job is gone; bootstrapping too early fails with "5: Input/output error".
-  echo "removed $LABEL (files left in $DEST)"
+  for i in {1..50}; do launchctl print "$DOMAIN/$LABEL" >/dev/null 2>&1 || break; sleep 0.2; done
-  exit 0
+  for attempt in 1 2 3; do
-fi
+    launchctl bootstrap "$DOMAIN" "$PLIST" && break
-
+    [[ $attempt == 3 ]] && { echo "launchctl bootstrap failed" >&2; exit 1; }
-/usr/bin/python3 -c "import json,sys; json.load(open(sys.argv[1]))" "$SRC/config.json"
+    sleep 1
-mkdir -p "$DEST"
+  done
-if [[ -f "$DEST/config.json" ]] && ! cmp -s "$SRC/config.json" "$DEST/config.json"; then
+}
  cp "$DEST/config.json" "$DEST/config.json.bak-$(date +%Y%m%d-%H%M%S)"
 fi
 cp "$SRC/status_panel.py" "$SRC/coremidi.py" "$SRC/netgear.py" "$SRC/config.json" "$SRC/atem_watch.js" "$DEST/"
 [[ -f "$SRC/atem-reference.xml" ]] && cp "$SRC/atem-reference.xml" "$DEST/"
 write_plist() {   # $1 = exit timeout in seconds
  cat > "$PLIST" <<EOF
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
@@ -43,18 +54,81 @@ cat > "$PLIST" <<EOF
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>ThrottleInterval</key><integer>10</integer>
  <key>ExitTimeOut</key><integer>$1</integer>
  <key>StandardErrorPath</key><string>$HOME/Library/Logs/status-panel.err.log</string>
 </dict>
 </plist>
 EOF
 }
-launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
+wait_for_start() {   # $1 = install time (unix): wait for a start stamp newer than that
-# bootout returns before the old job is gone; bootstrapping too early fails with "5: Input/output error".
+  for i in {1..60}; do
-for i in {1..50}; do launchctl print "$DOMAIN/$LABEL" >/dev/null 2>&1 || break; sleep 0.2; done
+    if [[ -f "$DEST/started.json" ]] && /usr/bin/python3 - "$DEST/started.json" "$1" <<'PY' 2>/dev/null
-for attempt in 1 2 3; do
+import json, sys
-  launchctl bootstrap "$DOMAIN" "$PLIST" && break
+d = json.load(open(sys.argv[1]))
-  [[ $attempt == 3 ]] && { echo "launchctl bootstrap failed" >&2; exit 1; }
+sys.exit(0 if d.get("started_at", 0) >= float(sys.argv[2]) else 1)
-  sleep 1
+PY
    then
      echo "running (code $(/usr/bin/python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("version"))' "$DEST/started.json"))"
      return 0
    fi
    sleep 0.5
  done
-echo "installed $LABEL -> $DEST"
+  echo "WARNING: no start stamp after 30 s - check ~/Library/Logs/status-panel.log and status-panel.err.log" >&2
  return 1
 }
 if [[ "$1" == "--remove" ]]; then
  launchctl bootout "$DOMAIN/$LABEL" 2>/dev/null || true
  rm -f "$PLIST"
  echo "removed $LABEL (files left in $DEST)"
  exit 0
 fi
 if [[ "$1" == "--rollback" ]]; then
  [[ -d "$DEST/prev" ]] || { echo "nothing to roll back to ($DEST/prev is missing)" >&2; exit 1; }
  for f in "$DEST"/prev/*(N); do cp "$f" "$DEST/"; done
  /usr/bin/python3 "$DEST/status_panel.py" --check --config "$DEST/config.json"
  T=$(date +%s)
  write_plist "$(/usr/bin/python3 "$DEST/status_panel.py" --exit-timeout --config "$DEST/config.json")"
  restart_agent
  echo "rolled back to the previous code (version $(/usr/bin/python3 -c 'import sys; sys.path.insert(0, sys.argv[1]); import status_panel; print(status_panel.VERSION)' "$DEST"))"
  wait_for_start "$T" || true
  exit 0
 fi
 # 1. The config that will run, checked with the code that will run it.
 mkdir -p "$DEST"
 if [[ $KEEP_CONFIG == 1 && -f "$DEST/config.json" ]]; then
  CFG="$DEST/config.json"
 else
  CFG="$SRC/config.json"
 fi
 /usr/bin/python3 "$SRC/status_panel.py" --check --config "$CFG"
 EXIT_TIMEOUT=$(/usr/bin/python3 "$SRC/status_panel.py" --exit-timeout --config "$CFG")
 # 2. Keep the code that runs now, for --rollback.
 mkdir -p "$DEST/prev"
 for f in $CODE; do [[ -f "$DEST/$f" ]] && cp "$DEST/$f" "$DEST/prev/"; done
 # 3. Files. The live config is backed up when it changes; --keep-config leaves it alone.
 if [[ $KEEP_CONFIG == 0 ]]; then
  if [[ -f "$DEST/config.json" ]] && ! cmp -s "$SRC/config.json" "$DEST/config.json"; then
    cp "$DEST/config.json" "$DEST/config.json.bak-$(date +%Y%m%d-%H%M%S)"
    echo "config.json replaced (the old one is kept as config.json.bak-...; use --keep-config to leave it)"
  fi
  cp "$SRC/config.json" "$DEST/"
 fi
 for f in $CODE; do [[ -f "$SRC/$f" ]] && cp "$SRC/$f" "$DEST/"; done
 [[ -f "$SRC/atem-reference.xml" ]] && cp "$SRC/atem-reference.xml" "$DEST/"
 if [[ -d "$SRC/companion-objects" ]]; then
  rm -rf "$DEST/companion-objects" && cp -R "$SRC/companion-objects" "$DEST/"
 fi
 # 4. The LaunchAgent, with the exit timeout this config needs (a running fix finishes before a restart).
 T=$(date +%s)
 write_plist "$EXIT_TIMEOUT"
 restart_agent
 echo "installed $LABEL -> $DEST (exit timeout ${EXIT_TIMEOUT}s)"
 echo "log: ~/Library/Logs/status-panel.log"
 wait_for_start "$T" || true
@@ -17,6 +17,7 @@ Buttons write commands into sp_cmd, which status_panel.py reads:
  Restart page    force <key>       (hold 2 s: restart that thing even when it looks fine)
  python3 make_page.py               -> status-page.companionconfig and restart-page.companionconfig
  python3 make_page.py --out DIR     -> the same files in DIR (the data dir on the Ingest Mac)
 """
 import gzip
 import hashlib
@@ -28,12 +29,25 @@ HERE = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, HERE)
 from status_panel import RESTARTABLE, load_config  # noqa: E402
 # Defaults; "companion", "page_grid", "page_stale_seconds" and "page_hold_ms" in config.json override them.
 EXPORT_VERSION = 12                       # Companion 5.0.7's export format
 COMPANION_BUILD = "5.0.7+9763-stable-cec2f88e6f"
 GRID = {"minColumn": 0, "maxColumn": 8, "minRow": 0, "maxRow": 5}
 STALE_SECONDS = 15
 PURPLE = 0x6A1B9A
 HOLD_MS = 2000
 # Every key this tool makes carries one of these in its notes; push_page.py overwrites a page only
 # when every key on it (besides Companion's own page keys) does.
 MARKERS = ("STATUS tile", "RESTART key", "STATUS jump key", "RESTART back key", "STATUS nav key")
 def settings(cfg):
    """The page-building numbers, from config.json where set."""
    comp = cfg.get("companion", {})
    return {"export_version": comp.get("export_version", EXPORT_VERSION), "build": comp.get("build", COMPANION_BUILD),
            "grid": cfg.get("page_grid", GRID), "stale": cfg.get("page_stale_seconds", STALE_SECONDS),
            "hold_ms": cfg.get("page_hold_ms", HOLD_MS), "flash": cfg.get("variables", {}).get("flash", "flash_tick"),
            "ingest": cfg.get("site", {}).get("ingest_name", "NA Ingest")}
 ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-"
@@ -48,9 +62,10 @@ def prop(value, expression=False):
    return {"value": value, "isExpression": expression}
-def layers(tid, tile, names, stale_text):
+def layers(tid, tile, names, stale_text, opts=None):
-    state, hb = names["state"], names["heartbeat"]
+    opts = opts or settings({"variables": names})
-    stale = "$(internal:time_unix) - $(custom:%s) > %d" % (hb, STALE_SECONDS)
+    state, hb, flash = names["state"], names["heartbeat"], opts["flash"]
    stale = "$(internal:time_unix) - $(custom:%s) > %d" % (hb, opts["stale"])
    def pick(on_key, off_key):
        return "jsonpath($(custom:%s), '$.%s.%s')" % (state, tid, on_key), \
@@ -58,8 +73,8 @@ def layers(tid, tile, names, stale_text):
    b_on, b_off = pick("b", "b2")
    f_on, f_off = pick("f", "f2")
-    bg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, PURPLE, b_on, b_off)
+    bg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, PURPLE, flash, b_on, b_off)
-    fg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, 0xFFFFFF, f_on, f_off)
+    fg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, 0xFFFFFF, flash, f_on, f_off)
    text = "%s ? %s : jsonpath($(custom:%s), '$.%s.x')" % (stale, stale_text, state, tid)
    # The watchdog picks a size per key so words never break mid-word.
    size = "%s ? 20 : jsonpath($(custom:%s), '$.%s.s')" % (stale, state, tid)
@@ -102,8 +117,8 @@ def set_page_action(tid, role, page):
            "options": {"surfaceId": prop("self"), "page": prop(page)}, "children": {}}
-def static_button(tid, text, bg, fg, page, note):
+def static_button(tid, text, bg, fg, page, note, marker):
-    """A plain key (no watchdog state): tap goes to `page`."""
+    """A plain key (no watchdog state): tap goes to `page`. `marker` names it as ours in its notes."""
    geometry = {"enabled": prop(True), "opacity": prop(100), "x": prop(0), "y": prop(0),
                "width": prop(100), "height": prop(100), "rotation": prop(0)}
    layers = [
@@ -120,7 +135,7 @@ def static_button(tid, text, bg, fg, page, note):
        "type": "button-layered",
        "style": {"layers": layers},
        "options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
-                    "canModifyStyleInApis": False, "notes": note},
+                    "canModifyStyleInApis": False, "notes": "%s - %s" % (marker, note)},
        "feedbacks": [],
        "steps": {"0": {"action_sets": {"down": [], "up": [set_page_action(tid, "go", page)]},
                        "options": {"runWhileHeld": []}}},
@@ -129,27 +144,29 @@ def static_button(tid, text, bg, fg, page, note):
 def restart_key(key, cfg):
-    """A Restart page key: drawn by the watchdog like a tile; hold 2 s = "force <id>"."""
+    """A Restart page key: drawn by the watchdog like a tile; hold = "force <id>"."""
-    names = cfg["variables"]
+    names, opts = cfg["variables"], settings(cfg)
    hold_ms, secs = opts["hold_ms"], "%g" % (opts["hold_ms"] / 1000.0)
    stale_text = "'%s\\n?'" % key["label"]
    return {
        "type": "button-layered",
-        "style": {"layers": layers(key["id"], key, names, stale_text)},
+        "style": {"layers": layers(key["id"], key, names, stale_text, opts)},
        "options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
                    "canModifyStyleInApis": True,
-                    "notes": "RESTART key '%s' - hold 2 s: restart it even when the status page says it is fine "
+                    "notes": "RESTART key '%s' - hold %s s: restart it even when the status page says it is fine "
-                             "(status_panel.py on NA Ingest runs fix '%s')." % (key["id"], key["fix"])},
+                             "(status_panel.py on %s runs fix '%s')." % (key["id"], secs, opts["ingest"], key["fix"])},
        "feedbacks": [],
        "steps": {"0": {"action_sets": {"down": [], "up": [],
-                                        str(HOLD_MS): [command_action(key["id"], "hold", names["command"],
+                                        str(hold_ms): [command_action(key["id"], "hold", names["command"],
                                                                      "force %s $(internal:time_unix)" % key["id"])]},
-                        "options": {"runWhileHeld": [HOLD_MS]}}},
+                        "options": {"runWhileHeld": [hold_ms]}}},
        "localVariables": [],
    }
 def button(tid, tile, cfg):
-    names = cfg["variables"]
+    names, opts = cfg["variables"], settings(cfg)
    hold_ms, secs = opts["hold_ms"], "%g" % (opts["hold_ms"] / 1000.0)
    kind = tile.get("kind")
    label = tile["label"]
    if kind == "master":
@@ -168,24 +185,26 @@ def button(tid, tile, cfg):
                                            for c in tile.get("checks", []))
        hold = ("restart " + tid) if fixable or tile.get("hold_always") or kind == "network" else None
        if kind == "network":
-            note = "Press: stop the flashing. Hold 2 s: clear a network blip (it also clears itself after 5 minutes)."
+            minutes = cfg.get("network", {}).get("remember_seconds", 300) // 60
            note = ("Press: stop the flashing. Hold %s s: clear a network blip (it also clears itself after %d minutes)."
                    % (secs, minutes))
        elif tile.get("hold_always"):
-            note = "Press: acknowledge. Hold 2 s: %s." % cfg["fixes"][tile["hold_always"]]["label"]
+            note = "Press: acknowledge. Hold %s s: %s." % (secs, cfg["fixes"][tile["hold_always"]]["label"])
        else:
-            note = "Press: acknowledge." + (" Hold 2 s: fix what this button is failing on (nothing if it is green)."
+            note = "Press: acknowledge." + (" Hold %s s: fix what this button is failing on (nothing if it is green)."
-                                            if hold else "")
+                                            % secs if hold else "")
    action_sets = {"down": [], "up": [command_action(tid, "press", names["command"], press)]}
    run_while_held = []
    if hold:
        # Stamped with Companion's clock, so a hold made while the watchdog was down is dropped later.
-        action_sets[str(HOLD_MS)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")]
+        action_sets[str(hold_ms)] = [command_action(tid, "hold", names["command"], hold + " $(internal:time_unix)")]
-        run_while_held = [HOLD_MS]
+        run_while_held = [hold_ms]
    return {
        "type": "button-layered",
-        "style": {"layers": layers(tid, tile, names, stale_text)},
+        "style": {"layers": layers(tid, tile, names, stale_text, opts)},
        "options": {"stepProgression": "auto", "stepExpression": "", "rotaryActions": False,
                    "canModifyStyleInApis": True,
-                    "notes": "STATUS tile '%s' - drawn by status_panel.py on NA Ingest. %s" % (tid, note)},
+                    "notes": "STATUS tile '%s' - drawn by status_panel.py on %s. %s" % (tid, opts["ingest"], note)},
        "feedbacks": [],
        "steps": {"0": {"action_sets": action_sets, "options": {"runWhileHeld": run_while_held}}},
        "localVariables": [],
@@ -196,7 +215,18 @@ NAV_LOCATIONS = {"pageup": ("0", "0"), "pagenum": ("1", "0"), "pagedown": ("2",
 NAV_MARKER = "STATUS nav key"
-def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1):
+def nav_templates(cfg):
    """Companion's own nav key designs for the version in use: companion-objects/nav_templates/<major.minor>.json
    when that exists, else this folder's nav_templates.json (captured from 5.0.7)."""
    version = ".".join(str(settings(cfg)["build"]).split("+")[0].split(".")[:2])
    for path in (os.path.join(HERE, "companion-objects", "nav_templates", version + ".json"),
                 os.path.join(HERE, "nav_templates.json")):
        if os.path.exists(path):
            return json.load(open(path))
    raise FileNotFoundError("no nav key templates for Companion %s" % version)
 def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1, cfg=None):
    """Companion's own UP / page number / DOWN key, coloured like the ALL button.
    For the page number key, status_page adds: tap = go to tap_page (on release,
@@ -204,12 +234,13 @@ def nav_key(kind, names, status_page=None, hold_ms=1000, tap_page=1):
    status page. The tap names the page outright: the converted key's "startup"
    means the current page on surfaces set to "use last page".
    """
-    templates = json.load(open(os.path.join(HERE, "nav_templates.json")))
+    cfg = cfg or {"variables": names}
-    key = json.loads(json.dumps(templates[kind]))
+    opts = settings(cfg)
-    stale = "$(internal:time_unix) - $(custom:%s) > %d" % (names["heartbeat"], STALE_SECONDS)
+    key = json.loads(json.dumps(nav_templates(cfg)[kind]))
    stale = "$(internal:time_unix) - $(custom:%s) > %d" % (names["heartbeat"], opts["stale"])
    pick = lambda field: "jsonpath($(custom:%s), '$.all.%s')" % (names["state"], field)
-    bg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, PURPLE, pick("b"), pick("b2"))
+    bg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, PURPLE, opts["flash"], pick("b"), pick("b2"))
-    fg = "%s ? %d : ($(custom:flash_tick) == 1 ? %s : %s)" % (stale, 0xFFFFFF, pick("f"), pick("f2"))
+    fg = "%s ? %d : ($(custom:%s) == 1 ? %s : %s)" % (stale, 0xFFFFFF, opts["flash"], pick("f"), pick("f2"))
    def paint(layers):
        for layer in layers:
@@ -240,7 +271,7 @@ def nav_controls(cfg):
    coloured = cfg.get("nav_keys", list(NAV_LOCATIONS))       # which nav keys follow the ALL button
    for kind, (r, c) in NAV_LOCATIONS.items():
        controls.setdefault(r, {})[c] = (nav_key(kind, cfg["variables"], cfg["page"], cfg.get("nav_hold_ms", 1000),
-                                                 cfg.get("nav_tap_page", 1))
+                                                 cfg.get("nav_tap_page", 1), cfg)
                                         if kind in coloured else {"type": kind})
    return controls
@@ -252,12 +283,13 @@ def place(controls, pos, control):
    controls.setdefault(r, {})[c] = control
-def page_export(name, number, controls):
+def page_export(name, number, controls, cfg=None):
    opts = settings(cfg or {})
    return {
-        "version": EXPORT_VERSION,
+        "version": opts["export_version"],
        "type": "page",
-        "companionBuild": COMPANION_BUILD,
+        "companionBuild": opts["build"],
-        "page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": GRID},
+        "page": {"id": stable_id("page", name), "name": name, "controls": controls, "gridSize": opts["grid"]},
        "instances": {},
        "connectionCollections": [],
        "oldPageNumber": number,
@@ -274,18 +306,20 @@ def build(cfg):
    rp = cfg.get("restart_page")
    if rp:
        place(controls, rp["jump_pos"], static_button("restart_jump", "RESTART\\nPAGE", 0x37474F, 0xFFFFFF,
-                                                      rp["page"], "Tap: open the Restart page (page %d)." % rp["page"]))
+                                                      rp["page"], "Tap: open the Restart page (page %d)." % rp["page"],
-    return page_export(cfg["page_name"], cfg["page"], controls)
+                                                      "STATUS jump key"))
    return page_export(cfg["page_name"], cfg["page"], controls, cfg)
 def build_restart(cfg):
    rp = cfg["restart_page"]
    controls = nav_controls(cfg)
    place(controls, rp["back_pos"], static_button("restart_back", "STATUS\\nPAGE", 0x37474F, 0xFFFFFF, cfg["page"],
-                                                  "Tap: back to the status page (page %d)." % cfg["page"]))
+                                                  "Tap: back to the status page (page %d)." % cfg["page"],
                                                  "RESTART back key"))
    for key in rp["keys"]:
        place(controls, key["pos"], restart_key(key, cfg))
-    return page_export(rp["page_name"], rp["page"], controls)
+    return page_export(rp["page_name"], rp["page"], controls, cfg)
 def write(path, export):
@@ -293,13 +327,20 @@ def write(path, export):
        f.write(json.dumps(export, indent=1).encode())
-def main():
+def main(argv=None):
-    cfg = load_config(os.path.join(HERE, "config.json"))
+    import argparse
-    out = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "status-page.companionconfig")
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--config", default=os.path.join(HERE, "config.json"))
    ap.add_argument("--out", help="folder for the page files (default: this folder)")
    ap.add_argument("status_file", nargs="?", help="write only the status page, to this file")
    args = ap.parse_args(argv)
    cfg = load_config(args.config)
    folder = args.out or HERE
    out = args.status_file or os.path.join(folder, "status-page.companionconfig")
    write(out, build(cfg))
    print("wrote %s (%d buttons)" % (out, sum(1 for t in cfg["tiles"] if "pos" in t)))
-    if cfg.get("restart_page") and len(sys.argv) <= 1:
+    if cfg.get("restart_page") and not args.status_file:
-        out = os.path.join(HERE, "restart-page.companionconfig")
+        out = os.path.join(folder, "restart-page.companionconfig")
        write(out, build_restart(cfg))
        print("wrote %s (%d restart keys)" % (out, len(cfg["restart_page"]["keys"])))
@@ -78,8 +78,8 @@ def find_nav_keys(host, port, status_page):
 def main():
    cfg = json.load(open(os.path.join(HERE, "config.json")))
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    ap.add_argument("--host", default="na-ingest.local")
+    ap.add_argument("--host", default=cfg.get("site", {}).get("ingest_hostname") or cfg["companion"].get("host", "127.0.0.1"))
-    ap.add_argument("--port", type=int, default=8000)
+    ap.add_argument("--port", type=int, default=cfg["companion"].get("port", 8000))
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--revert", action="store_true")
    args = ap.parse_args()
@@ -18,10 +18,23 @@ import subprocess
 import sys
 import telnetlib  # stdlib in the macOS python3 (3.9)
-SWITCHES = ["10.66.1.100", "10.66.1.102"]
+SWITCHES = ["10.66.1.100", "10.66.1.102"]        # the CLI's default; the watchdog passes "dante_switches" from config
 KEYCHAIN_SERVICE = "netgear-status-panel"
 def config_switches():
    """"dante_switches" from this folder's config.json, for the command line (the watchdog passes its own)."""
    import json, os
    try:
        with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "config.json")) as f:
            cfg = json.load(f)
        global MAX_SHARED
        MAX_SHARED = cfg.get("dante_max_shared", MAX_SHARED)
        return cfg.get("dante_switches") or SWITCHES
    except (OSError, ValueError):
        return SWITCHES
 def credentials(service=KEYCHAIN_SERVICE):
    attrs = subprocess.run(["/usr/bin/security", "find-generic-password", "-s", service],
                           capture_output=True, text=True).stdout
@@ -71,7 +84,7 @@ class Switch:
 def lookup(macs):
    user, password = credentials()
    wanted = {m.lower(): m for m in macs}
-    for host in SWITCHES:
+    for host in config_switches():
        sw = Switch(host, user, password)
        try:
            model = [l.strip() for l in sw.run("show version").splitlines()
@@ -5,9 +5,10 @@ Does exactly what Import/Export -> Import -> "import into page N" does, over the
 same internal API the Companion web UI uses (tRPC over a WebSocket at /trpc):
 upload the file in chunks, check its SHA-1, then import it as a single page.
-Safety: it looks at what is on the target page, not just its name. It refuses unless the
+Safety: it looks at what is on the target page, not just its name. It refuses unless every
-page holds nothing but page up / page number / page down keys, or is already the page this
+key on the page (besides Companion's own page up / number / down keys) carries one of the
-file builds (same name). A page named "PAGE" can still hold real buttons (pages 22 and 97 do).
+markers make_page.py writes into its notes, i.e. the page holds nothing a person made by hand.
 A key someone added is named by its position, so it can be moved first.
  python3 push_page.py                         # the status page, from config.json
  python3 push_page.py --which restart         # the Restart page
@@ -29,6 +30,9 @@ import urllib.parse
 import urllib.request
 HERE = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, HERE)
 from make_page import MARKERS  # noqa: E402
 CHUNK = 256 * 1024
@@ -124,8 +128,10 @@ NAV_MARKER = "STATUS nav key"          # our coloured page number key (make_page
 def fetch_page(host, port, page):
-    """The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}})."""
+    """The page as Companion exports it ({"page": {"name", "controls": {row: {col: control}}}}),
-    with urllib.request.urlopen("http://%s:%d/int/export/page/%d" % (host, port, page), timeout=10) as r:
+    without any connection secrets."""
    url = "http://%s:%d/int/export/page/%d?includeSecrets=false" % (host, port, page)
    with urllib.request.urlopen(url, timeout=10) as r:
        raw = r.read()
    try:
        raw = gzip.decompress(raw)
@@ -134,29 +140,36 @@ def fetch_page(host, port, page):
    return json.loads(raw)
-def real_buttons(export):
+def foreign_buttons(export):
-    """How many buttons a page holds besides its page up / number / down keys."""
+    """[(row, col)] of every key on the page that neither Companion nor this tool made."""
    page = export.get("page") or {}
-    n = 0
+    found = []
-    for row in (page.get("controls") or {}).values():
+    for r, row in sorted((page.get("controls") or {}).items(), key=lambda kv: int(kv[0])):
-        for ctl in (row or {}).values():
+        for c, ctl in sorted((row or {}).items(), key=lambda kv: int(kv[0])):
            if not ctl or ctl.get("type") in NAV_TYPES:
                continue
-            if NAV_MARKER in ((ctl.get("options") or {}).get("notes") or ""):
+            notes = (ctl.get("options") or {}).get("notes") or ""
            if any(m in notes for m in MARKERS):
                continue
-            n += 1
+            found.append((int(r), int(c)))
-    return n
+    return found
-def safe_target(export, expected_name):
+def real_buttons(export):
-    """(ok, reason). OK only for a page with nothing but nav keys, or the page this file builds."""
+    """How many keys a page holds that this tool did not make."""
    return len(foreign_buttons(export))
 def safe_target(export, expected_name=None):
    """(ok, reason). OK only when every key on the page is Companion's own page key or one of ours,
    whatever the page is called: nothing a person made by hand is ever overwritten."""
    name = ((export.get("page") or {}).get("name") or "")
-    if name == expected_name:
+    foreign = foreign_buttons(export)
-        return True, "it is already %r" % name
+    if not foreign:
-    n = real_buttons(export)
+        return True, "it holds only page keys and keys made by this tool"
-    if n == 0:
+    where = ", ".join("row %d col %d" % rc for rc in foreign[:6]) + (" ..." if len(foreign) > 6 else "")
-        return True, "it holds only page keys"
+    return False, "it is called %r and holds %d key%s not made by this tool (%s) - move or delete them first" % (
-    return False, "it is called %r and holds %d buttons" % (name, n)
+        name, len(foreign), "" if len(foreign) == 1 else "s", where)
 def page_name(host, port, page):
@@ -169,8 +182,8 @@ def main():
    cfg = json.load(open(os.path.join(HERE, "config.json")))
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--which", choices=("status", "restart"), default="status")
-    ap.add_argument("--host", default="na-ingest.local")
+    ap.add_argument("--host", default=cfg.get("site", {}).get("ingest_hostname") or cfg["companion"].get("host", "127.0.0.1"))
-    ap.add_argument("--port", type=int, default=8000)
+    ap.add_argument("--port", type=int, default=cfg["companion"].get("port", 8000))
    ap.add_argument("--page", type=int)
    ap.add_argument("--file")
    ap.add_argument("--force", action="store_true", help="overwrite the page whatever is on it")
@@ -0,0 +1,658 @@
 """The registry of every setting the status panel has.
 One record per config key (path, type, default, plain-words help, which screen of the settings
 page owns it, whether it sits under Advanced), the check types with their fields, the fix kinds,
 the words a key can show, and the rules a config must pass. The settings page draws its forms
 from this file; tests/test_schema.py keeps it in step with the code (every key the watchdog reads
 must be declared here, every default in the code must equal the one here).
 Paths: "companion.host", "tiles[].checks[].conn" (a list element), "fixes.*.run" (a map entry).
 """
 import copy
 import re
 import status_panel as sp
 # --- Field records -----------------------------------------------------------
 # type: str | int | float | bool | list | dict | enum | path | ip | mac | url | hhmm | days | id
 # apply: what changes when the value does: "restart" (the watchdog restarts), "pages" (the Companion
 # pages are rebuilt and re-imported when the built page differs), "none" (page-only data), "settings".
 F = {}
 def field(path, type, default=None, help="", screen="", advanced=False, secret=False, apply="restart",
          required=False, options=None, lo=None, hi=None, regex=None, example=None, shows=None):
    F[path] = {"path": path, "type": type, "default": default, "help": help, "screen": screen,
               "advanced": advanced, "secret": secret, "apply": apply, "required": required,
               "options": options, "lo": lo, "hi": hi, "regex": regex, "example": example, "shows": shows}
 # -- site --
 field("site", "dict", {}, "This campus: its name, this Mac, and when changes must wait.", "Site")
 field("site.id", "id", "na", "Short name used for folders and files.", "Site", advanced=True, regex=r"[a-z0-9_-]+")
 field("site.name", "str", "", "The campus, as people say it.", "Site", example="North Attleboro", apply="none")
 field("site.ingest_name", "str", "NA Ingest", "What this Mac is called in logs, notes and alerts.", "Site", apply="pages")
 field("site.ingest_hostname", "str", "", "This Mac's network name, used in the page address and by the page tools.",
      "Site", example="na-ingest.local", apply="none")
 field("site.ingest_ips", "list", [], "This Mac's addresses, for the PCs' firewall allow-lists.", "Site", apply="none")
 field("site.contacts", "str", "", "Printed on the cheat sheet: who to call.", "Site", apply="none")
 field("site.alert_prefix", "str", "", "The first words of an alert message; blank = this Mac's name.", "Site", advanced=True)
 field("site.live_services", "list", [], "Clock windows during which changes to the panel wait (a recording "
      "HyperDeck always counts as live, whatever the clock says).", "Schedules")
 field("site.live_services[].days", "days", None, "Days of the week.", "Schedules", required=True)
 field("site.live_services[].start", "hhmm", None, "Start time.", "Schedules", required=True)
 field("site.live_services[].end", "hhmm", None, "End time; earlier than the start means it runs past midnight.",
      "Schedules", required=True)
 field("site.editing_code", "str", "", "The code that switches Editing on. Kept here, in the repo, so everyone "
      "with the repo has it.", "Site", advanced=True, secret=True, apply="none")
 field("site.settings_port", "int", 8020, "Where the settings page listens.", "Site", advanced=True, lo=1024, hi=65535,
      apply="settings")
 field("site.dvs_helper", "path", "/usr/local/sbin/status-panel-dvs-start", "The DVS helper on each Mac that runs "
      "Dante Virtual Soundcard.", "Site", advanced=True)
 field("site.solo_dir", "path", "~/Library/Application Support/status-panel", "The Pro Tools helper's folder on "
      "the Studio B Mac.", "Site", advanced=True)
 # -- companion --
 field("companion", "dict", {}, "Where Companion answers.", "Companion", required=True)
 field("companion.host", "str", "127.0.0.1", "Companion's address from this Mac.", "Companion", advanced=True)
 field("companion.port", "int", 8000, "Companion's HTTP port.", "Companion", advanced=True, lo=1, hi=65535)
 field("companion.db", "path", None, "Companion's settings database; blank = the newest one found.", "Companion",
      advanced=True)
 field("companion.export_version", "int", sp.__dict__.get("EXPORT_VERSION", 12), "Companion's page export format "
      "number (changes with a Companion upgrade).", "Companion", advanced=True, apply="pages")
 field("companion.build", "str", "5.0.7+9763-stable-cec2f88e6f", "The Companion build the pages are written for.",
      "Companion", advanced=True, apply="pages")
 field("page", "int", 19, "The Status page's number in Companion.", "Keys", lo=1, apply="pages")
 field("page_name", "str", "19\\nStatus", "The Status page's name in Companion.", "Keys", advanced=True, apply="pages")
 field("nav_keys", "list", ["pagenum"], "Which of Companion's page keys (page up, page number, page down) are "
      "coloured like the ALL key on every page.", "Companion", advanced=True, options=["pageup", "pagenum", "pagedown"],
      apply="pages")
 field("nav_hold_ms", "int", 1000, "Holding the page-number key this long jumps to the Status page.", "Companion",
      advanced=True, lo=200, hi=5000, apply="pages")
 field("nav_tap_page", "int", 1, "Tapping the page-number key goes to this page.", "Companion", advanced=True, lo=1,
      apply="pages")
 field("variables", "dict", {}, "The custom variables the panel uses inside Companion.", "Companion", required=True)
 field("variables.state", "str", "sp_state", "The variable the keys are drawn from.", "Companion", advanced=True,
      regex=r"[A-Za-z0-9_]+", apply="pages")
 field("variables.heartbeat", "str", "sp_hb", "The variable that says the watchdog is alive.", "Companion",
      advanced=True, regex=r"[A-Za-z0-9_]+", apply="pages")
 field("variables.command", "str", "sp_cmd", "The variable key presses are written to.", "Companion", advanced=True,
      regex=r"[A-Za-z0-9_]+", apply="pages")
 field("variables.flash", "str", "flash_tick", "The blink clock. Renaming it means renaming it inside the trigger "
      "in Companion too.", "Companion", advanced=True, regex=r"[A-Za-z0-9_]+", apply="pages")
 field("page_grid", "dict", {"minColumn": 0, "maxColumn": 8, "minRow": 0, "maxRow": 5}, "The export grid "
      "(the deck is the 8x4 inside it).", "Companion", advanced=True, apply="pages")
 field("page_stale_seconds", "int", 15, "Keys turn purple when the watchdog has been silent this long.",
      "Companion", advanced=True, lo=5, hi=120, apply="pages")
 field("page_hold_ms", "int", 2000, "How long a key is held to run its fix.", "Schedules", advanced=True, lo=500,
      hi=5000, apply="pages", shows="HOLD 2s")
 # -- schedules and timing --
 field("service_windows", "list", [{"days": list(sp.DAYS), "start": "00:00", "end": "23:59"}],
      "When alarms are armed: always (the ALARMS key silences them until midnight).", "Schedules", advanced=True)
 field("service_windows[].days", "days", None, "", "Schedules", required=True)
 field("service_windows[].start", "hhmm", None, "", "Schedules", required=True)
 field("service_windows[].end", "hhmm", None, "", "Schedules", required=True)
 field("debounce_seconds", "int", 3, "A fault must last this long before it shows (one bad reading is not an "
      "alarm).", "Schedules", advanced=True, lo=0, hi=60)
 field("flap", "dict", {}, "Repeated faults show amber 'xN'.", "Schedules", advanced=True)
 field("flap.count", "int", 3, "This many faults...", "Schedules", advanced=True, lo=1, hi=20)
 field("flap.window_seconds", "int", 600, "...within this many seconds.", "Schedules", advanced=True, lo=60, hi=86400)
 field("timing", "dict", {}, "Timing numbers.", "Schedules", advanced=True)
 field("timing.fix_cooldown_seconds", "int", 60, "The same fix never re-runs within this.", "Schedules", advanced=True,
      lo=10, hi=600, shows="WAIT 1 MIN")
 field("timing.alert_after_seconds", "int", 30, "Send the alert message when Companion has not answered for this "
      "long.", "Schedules", advanced=True, lo=5, hi=600)
 field("alert_webhook", "url", "", "Where alert messages go (a Slack-style web hook); blank = off.", "Site",
      secret=True)
 field("auto_fix", "dict", {}, "When the panel may fix things by itself.", "Schedules")
 field("auto_fix.windows", "list", [], "When auto-fix may act.", "Schedules")
 field("auto_fix.windows[].days", "days", None, "", "Schedules", required=True)
 field("auto_fix.windows[].start", "hhmm", None, "", "Schedules", required=True)
 field("auto_fix.windows[].end", "hhmm", None, "", "Schedules", required=True)
 field("auto_fix.after_seconds", "int", 20, "A fault must last this long before auto-fix acts.", "Schedules",
      advanced=True, lo=5, hi=600)
 field("auto_fix.max_tries", "int", 2, "Tries per fault, then it is yours to hold.", "Schedules", advanced=True,
      lo=1, hi=10)
 field("auto_fix.retry_seconds", "int", 90, "Between tries.", "Schedules", advanced=True, lo=30, hi=3600)
 field("auto_fix.pause_after_network_seconds", "int", 60, "Wait this long after a network blip.", "Schedules",
      advanced=True, lo=0, hi=3600)
 field("auto_fix.forget_seconds", "int", 600, "Forget a fault's tries once it has been gone this long.",
      "Schedules", advanced=True, lo=60, hi=86400)
 field("auto_fix.restart_connections", "bool", False, "Also reconnect Companion connections by itself.", "Schedules")
 field("auto_fix.fixes", "list", [], "The fixes auto-fix may run (gentle ones only).", "Schedules")
 field("auto_fix.log_dir", "path", "~/Desktop/Auto-Fix Logs", "One file per day of everything auto-fix does.",
      "Schedules", advanced=True)
 field("auto_fix.keep_months", "int", 1, "Daily log files are deleted after this many months.", "Schedules",
      advanced=True, lo=1, hi=12)
 # -- network and Dante --
 field("network", "dict", {}, "The NET key.", "Network & Dante")
 field("network.interface", "str", "en0", "The network port to watch for link.", "Network & Dante")
 field("network.gateway", "ip", "", "The router to ping; blank = off.", "Network & Dante")
 field("network.burst_connections", "int", 4, "A blip is this many connections...", "Network & Dante", advanced=True,
      lo=1, hi=50)
 field("network.burst_groups", "int", 3, "...from this many groups...", "Network & Dante", advanced=True, lo=1, hi=20)
 field("network.burst_window_seconds", "int", 10, "...dropping within this many seconds.", "Network & Dante",
      advanced=True, lo=1, hi=300)
 field("network.remember_seconds", "int", 300, "NET shows BLIP this long after.", "Network & Dante", advanced=True,
      lo=10, hi=3600, apply="pages")
 field("ssh_key", "path", "~/.ssh/id_status_panel", "The login key this Mac uses for the other computers.", "Site",
      advanced=True)
 field("dante_switches", "list", [], "The Netgear switches on the Dante network.", "Network & Dante")
 field("dante_map_seconds", "int", 600, "Survey the switches this often.", "Network & Dante", advanced=True, lo=60,
      hi=86400)
 field("dante_max_shared", "int", 2, "Never power-cycle a port carrying more than this many devices (an uplink).",
      "Network & Dante", advanced=True, lo=1, hi=10)
 # -- Spotify and ATEM --
 field("spotify", "dict", {}, "The service playlist.", "Spotify")
 field("spotify.playlist", "str", "", "The playlist Spotify must be playing.", "Spotify", regex=r"spotify:playlist:[A-Za-z0-9]+",
      required=True)
 field("spotify.name", "str", "", "The playlist's name, as the keys and the key guide say it.", "Spotify")
 field("spotify.open_wait", "int", 8, "Seconds to wait after opening Spotify.", "Spotify", advanced=True, lo=1, hi=60)
 field("spotify.play_wait", "int", 3, "Seconds to wait before checking it plays.", "Spotify", advanced=True, lo=1, hi=60)
 field("atem", "dict", {}, "The switcher and its saved file.", "ATEM")
 field("atem.host", "ip", "", "The switcher's address.", "ATEM", required=True)
 field("atem.node", "path", "/Applications/Companion [0-9]*.app/Contents/Resources/node-runtimes/node22/bin/node",
      "Companion's node (pattern; the newest match is used).", "ATEM", advanced=True)
 field("atem.library", "path", "~/Library/Application Support/companion/modules/bmd-atem-*/node_modules/atem-connection",
      "Companion's ATEM library (pattern).", "ATEM", advanced=True)
 field("atem.reference", "path", "atem-reference.xml", "The saved switcher file to keep in line with.", "ATEM")
 field("atem.compare", "dict", None, "What to keep in line: audio fields and camera settings; blank = everything.",
      "ATEM", advanced=True)
 field("atem.source_folder", "path", "", "Where ATEM Software Control saves its files (for 'pick the newest').",
      "ATEM", apply="none")
 # -- hosts --
 field("hosts", "dict", {}, "The computers this Mac logs in to.", "Computers")
 field("hosts.*.ssh", "str", None, "How this Mac logs in: user@address.", "Computers", regex=r"[^@\s]+@[^@\s]+")
 field("hosts.*.mac", "mac", None, "The PC's hardware address, to wake it when it is off.", "Computers")
 field("hosts.*.wake_broadcast", "ip", None, "The broadcast address the wake-up goes to.", "Computers", advanced=True)
 field("hosts.*.name", "str", None, "The computer, as people say it.", "Computers", apply="none")
 field("hosts.*.os", "enum", "mac", "Mac or Windows (a PC's checks are written for cmd.exe).", "Computers",
      options=["mac", "windows", "other"])
 field("hosts.*.role", "str", None, "What it is for.", "Computers", apply="none")
 field("hosts.*.notes", "str", None, "Notes for operators.", "Computers", apply="none")
 field("hosts.*.pingable", "bool", True, "Off for a PC whose firewall blocks ping.", "Computers", apply="none")
 field("hosts.*.self", "bool", False, "This Mac itself.", "Computers", apply="none")
 field("hosts.*.runs", "list", [], "What runs on it (programs, tasks, services, helpers).", "Computers", apply="none")
 field("hosts.*.networks", "list", [], "Its network ports and what each is for (LAN, Dante, switch management).",
      "Computers", advanced=True, apply="none")
 field("hosts.*.manual_done", "dict", {}, "Set-up steps ticked by hand (who, when).", "Computers", apply="none")
 field("gear", "dict", {}, "Devices that are not computers: where each one is.", "Computers", apply="none")
 field("gear.*.location", "str", None, "Where it is in the building.", "Computers", apply="none")
 field("gear.*.notes", "str", None, "Notes for operators.", "Computers", apply="none")
 # -- fixes --
 field("fixes", "dict", {}, "Everything a hold can run.", "Fixes")
 field("fixes.*.kind", "enum", "ssh", "How it is built.", "Fixes", options=list(sp.FIX_KINDS))
 field("fixes.*.host", "str", None, "The computer it runs on (ssh fixes).", "Fixes")
 field("fixes.*.run", "str", None, "The command (zsh on Macs, cmd.exe on PCs).", "Fixes", advanced=True)
 field("fixes.*.timeout", "int", 60, "Give up after this many seconds.", "Fixes", advanced=True, lo=5, hi=600)
 field("fixes.*.short", "str", None, "The word the key shows while it runs (FIXING ...).", "Fixes", apply="pages")
 field("fixes.*.label", "str", None, "Plain words: what it does.", "Fixes", required=True, apply="pages")
 field("fixes.*.reconnect", "bool", False, "Reconnect the Companion connection afterwards.", "Fixes")
 field("fixes.*.conn", "str", None, "The Companion connection it reconnects.", "Fixes")
 field("fixes.*.restart_ok", "bool", False, "May close a running program (and still run by itself).", "Fixes")
 field("fixes.*.never_auto", "bool", False, "Never runs by itself.", "Fixes")
 field("fixes.*.part", "enum", None, "Which part of the switcher.", "Fixes", options=["audio", "camera"])
 field("fixes.*.verify_seconds", "int", 20, "Wait this long before judging a reconnect.", "Fixes", advanced=True,
      lo=5, hi=120)
 field("fixes.*.off_seconds", "int", 2, "How long a HyperDeck's connection stays off.", "Fixes", advanced=True,
      lo=1, hi=10)
 field("fixes.*.hung_note", "str", None, "What the key says when the device answers ping but not Companion.",
      "Fixes", advanced=True)
 field("fixes.*.switch", "ip", None, "The PoE switch.", "Fixes")
 field("fixes.*.port", "str", None, "The PoE port (e.g. 0/14).", "Fixes", regex=r"\d+/\d+")
 field("fixes.*.mac", "mac", None, "The device expected on that port.", "Fixes")
 field("fixes.*.playlist", "str", None, "A playlist other than the site's (rare).", "Fixes", advanced=True)
 field("fixes.*.template", "str", None, "The builder this fix was made with.", "Fixes", advanced=True, apply="none")
 field("fixes.*.params", "dict", None, "The builder's fields.", "Fixes", advanced=True, apply="none")
 # -- tiles (keys) --
 field("tiles", "list", [], "The keys on the Status page.", "Keys", required=True)
 field("tiles[].id", "id", None, "Used inside Companion.", "Keys", advanced=True, required=True, regex=r"[A-Za-z0-9_]+",
      apply="pages")
 field("tiles[].label", "str", None, "The first line on the key.", "Keys", required=True, apply="pages")
 field("tiles[].pos", "list", None, "[row, column]; absent = no key of its own (shown inside a group).", "Keys",
      apply="pages")
 field("tiles[].kind", "enum", None, "A special key: master (ALL), service (ALARMS), network (NET).", "Keys",
      advanced=True, options=["master", "service", "network"], apply="pages")
 field("tiles[].members", "list", None, "A group: the keys it sums up.", "Keys")
 field("tiles[].checks", "list", [], "What the key watches.", "Keys")
 field("tiles[].hold_always", "str", None, "A fix a hold runs even when the key is green.", "Keys", apply="pages")
 field("tiles[].checks[].type", "enum", None, "What kind of check.", "Keys", required=True)
 field("tiles[].checks[].conn", "str", None, "The Companion connection.", "Keys")
 field("tiles[].checks[].fix", "str", None, "What holding the key runs for this check (a fix, or a table per "
      "sub-check).", "Keys")
 field("tiles[].checks[].tag", "str", None, "Names the device when one key covers several.", "Keys")
 field("tiles[].checks[].service_only", "bool", False, "Legacy: only during a service (has no effect while alarms "
      "are armed all day).", "Keys", advanced=True)
 field("tiles[].checks[].debounce_seconds", "int", None, "Wait before this check's fault shows.", "Keys", advanced=True,
      lo=0, hi=60)
 field("tiles[].checks[].ignore_warning", "bool", False, "Ignore Companion's yellow warning state.", "Keys")
 field("tiles[].checks[].optional", "bool", False, "A HyperDeck that may have no disk (grey, no alarm).", "Keys")
 field("tiles[].checks[].disk_service_only", "bool", True, "NO DISK only during a service.", "Keys", advanced=True)
 field("tiles[].checks[].full_minutes", "int", 5, "FULL below this many minutes left.", "Keys", advanced=True)
 field("tiles[].checks[].low_minutes", "int", 30, "LOW below this many minutes left.", "Keys", advanced=True)
 field("tiles[].checks[].rec_sync", "bool", True, "Must record whenever another deck records.", "Keys")
 field("tiles[].checks[].ip", "ip", None, "The deck's address, instead of what Companion reports.", "Keys", advanced=True)
 field("tiles[].checks[].devices", "dict", None, "Dante devices: name -> key text.", "Keys")
 field("tiles[].checks[].count_all", "bool", False, "Count every watched Dante device on this key.", "Keys")
 field("tiles[].checks[].channels", "int", 1, "Channels on the receiver.", "Keys", lo=1, hi=8)
 field("tiles[].checks[].battery_err_bars", "int", 1, "Red at this many bars or fewer.", "Keys", advanced=True, lo=0, hi=5)
 field("tiles[].checks[].battery_warn_bars", "int", 2, "Amber at this many bars or fewer.", "Keys", advanced=True, lo=0, hi=5)
 field("tiles[].checks[].muted_is_error", "bool", False, "RF mute is red, not amber.", "Keys")
 field("tiles[].checks[].host", "str", None, "The computer (an address for ping, a computer name for the rest).", "Keys")
 field("tiles[].checks[].name", "str", None, "The name the key shows.", "Keys")
 field("tiles[].checks[].msg", "str", None, "The words the key shows when it fails.", "Keys")
 field("tiles[].checks[].failures", "int", 2, "Pings missed before NO PING.", "Keys", advanced=True, lo=1, hi=10)
 field("tiles[].checks[].label", "str", None, "The launchd helper.", "Keys")
 field("tiles[].checks[].match", "str", None, "The program (pgrep -f pattern).", "Keys")
 field("tiles[].checks[].url", "url", None, "The web address to fetch.", "Keys")
 field("tiles[].checks[].interval_seconds", "int", 30, "Fetch it this often.", "Keys", advanced=True, lo=5, hi=3600)
 field("tiles[].checks[].path", "path", None, "The disk to watch.", "Keys")
 field("tiles[].checks[].warn_gb", "int", 20, "Amber below this many GB free.", "Keys", advanced=True)
 field("tiles[].checks[].err_gb", "int", 5, "Red below this many GB free.", "Keys", advanced=True)
 field("tiles[].checks[].session", "str", None, "The network MIDI session.", "Keys")
 field("tiles[].checks[].peer", "str", None, "The peer that must be in it.", "Keys")
 field("tiles[].checks[].address", "str", None, "Where to reconnect the peer if it was never seen.", "Keys", advanced=True)
 field("tiles[].checks[].domain", "str", None, "The macOS preference domain.", "Keys")
 field("tiles[].checks[].key", "str", None, "The preference key.", "Keys")
 field("tiles[].checks[].want", "str", None, "The value it must have.", "Keys")
 field("tiles[].checks[].playlist", "str", None, "A playlist other than the site's (rare).", "Keys", advanced=True)
 field("tiles[].checks[].instance", "str", None, "The Bonjour MIDI name.", "Keys")
 field("tiles[].checks[].source", "str", None, "The MIDI source, or the Art-Net sender's address.", "Keys")
 field("tiles[].checks[].max_silence_seconds", "int", None, "Amber/red after this much silence.", "Keys", lo=1, hi=600)
 field("tiles[].checks[].cmd", "str", None, "The command run on that computer.", "Keys", advanced=True)
 field("tiles[].checks[].shell", "enum", None, "sh (Macs) or cmd (PCs); blank = from the computer's kind.", "Keys",
      advanced=True, options=["sh", "cmd"])
 field("tiles[].checks[].part", "enum", None, "Which part of the switcher.", "Keys", options=["audio", "camera"])
 field("tiles[].checks[].template", "str", None, "The builder this check was made with.", "Keys", advanced=True, apply="none")
 field("tiles[].checks[].params", "dict", None, "The builder's fields.", "Keys", advanced=True, apply="none")
 # -- restart page --
 field("restart_page", "dict", None, "The Restart page; absent = this campus has none.", "Keys", apply="pages")
 field("restart_page.page", "int", 20, "Its number in Companion.", "Keys", lo=1, apply="pages")
 field("restart_page.page_name", "str", "20\\nRestart", "Its name in Companion.", "Keys", advanced=True, apply="pages")
 field("restart_page.jump_pos", "list", None, "Where the 'go to Restart page' key sits on the Status page.", "Keys",
      apply="pages")
 field("restart_page.back_pos", "list", None, "Where the 'back to Status' key sits on the Restart page.", "Keys",
      apply="pages")
 field("restart_page.keys", "list", [], "The restart keys.", "Keys", apply="pages")
 field("restart_page.keys[].id", "id", None, "", "Keys", advanced=True, required=True, regex=r"[A-Za-z0-9_]+", apply="pages")
 field("restart_page.keys[].label", "str", None, "", "Keys", required=True, apply="pages")
 field("restart_page.keys[].pos", "list", None, "[row, column].", "Keys", required=True, apply="pages")
 field("restart_page.keys[].fix", "str", None, "What holding it runs.", "Keys", required=True)
 field("restart_page.keys[].watch", "list", [], "Keep flashing HOLD TO FIX while these fixes' checks or "
      "'conn:<label>' connections are down.", "Keys")
 # --- Check types ---------------------------------------------------------------
 # fields: the keys that type reads; subs: the sub-check names it returns (fix tables use them);
 # identity: what it is about; module: the Companion module its connection is expected to use.
 CHECKS = {
    "conn":      {"fields": ["conn", "service_only", "ignore_warning"], "subs": ["conn"], "identity": ["conn"], "module": None,
                  "question": "Is a device answering?", "words": "A Companion connection is good."},
    "hyperdeck": {"fields": ["conn", "optional", "disk_service_only", "full_minutes", "low_minutes", "rec_sync", "ip"],
                  "subs": ["conn", "disk", "rec"], "identity": ["conn"], "module": "bmd-hyperdeck",
                  "question": "Is a recorder OK?", "words": "A HyperDeck is connected, has a disk, and records with the others."},
    "dante":     {"fields": ["conn", "devices", "count_all", "service_only"], "subs": ["conn", "<device name>"], "identity": ["conn"],
                  "module": "audinate-dantecontroller", "question": "Is Dante complete?", "words": "Every listed Dante device is present."},
    "shure":     {"fields": ["conn", "channels", "battery_err_bars", "battery_warn_bars"], "subs": ["conn", "ch<N>_batt", "ch<N>_rf"],
                  "identity": ["conn"], "module": "shure-wireless", "question": "Is a device answering?", "words": "A Shure receiver: batteries and interference."},
    "psm":       {"fields": ["conn", "channels", "muted_is_error"], "subs": ["conn", "ch<N>_mute"], "identity": ["conn"],
                  "module": "shure-psm1000", "question": "Is a device answering?", "words": "A PSM transmitter: RF mute."},
    "tascam":    {"fields": ["conn"], "subs": ["conn", "error", "caution"], "identity": ["conn"], "module": "tascam-da-6400",
                  "question": "Is a recorder OK?", "words": "The Tascam recorder reports no error or caution."},
    "ping":      {"fields": ["host", "conn", "name", "msg", "failures", "service_only"], "subs": ["ping"], "identity": ["conn", "host"],
                  "module": None, "question": "Is a device answering?", "words": "An address answers ping."},
    "launchd":   {"fields": ["label", "name"], "subs": ["svc"], "identity": ["label"], "module": None,
                  "question": "Is a program running?", "words": "A helper (LaunchAgent) on this Mac is loaded and running."},
    "process":   {"fields": ["match", "name", "template", "params"], "subs": ["proc"], "identity": ["match"], "module": None,
                  "question": "Is a program running?", "words": "A program on this Mac is running."},
    "http":      {"fields": ["url", "msg", "name", "interval_seconds"], "subs": ["http"], "identity": ["url"], "module": None,
                  "question": "Is a device answering?", "words": "A web address answers."},
    "disk":      {"fields": ["path", "warn_gb", "err_gb"], "subs": ["disk"], "identity": ["path"], "module": None,
                  "question": "Is a setting right?", "words": "Free space on a disk."},
    "midi_peer": {"fields": ["session", "peer", "host", "address"], "subs": ["peer"], "identity": ["host", "session", "peer"],
                  "module": None, "question": "Is a MIDI link up?", "words": "A peer is in a network MIDI session (here or on another Mac)."},
    "pref":      {"fields": ["domain", "key", "want", "msg", "template", "params"], "subs": ["pref"], "identity": ["domain", "key"],
                  "module": None, "question": "Is a setting right?", "words": "A macOS preference has the value it should."},
    "spotify":   {"fields": ["playlist", "fix"], "subs": ["spotify"], "identity": [], "module": None,
                  "question": "Is Spotify playing?", "words": "Spotify is playing the service playlist."},
    "mdns":      {"fields": ["instance", "name"], "subs": ["mdns"], "identity": ["instance"], "module": None,
                  "question": "Is a MIDI link up?", "words": "A network MIDI session is advertised on Bonjour."},
    "midi_rx":   {"fields": ["source", "name", "max_silence_seconds"], "subs": ["rx"], "identity": ["source"], "module": None,
                  "question": "Is a MIDI link up?", "words": "MIDI is arriving from a source."},
    "remote":    {"fields": ["host", "cmd", "msg", "shell", "template", "params"], "subs": ["remote"], "identity": ["host", "cmd"],
                  "module": None, "question": "Is a program running?", "words": "A command on another computer succeeds."},
    "artnet":    {"fields": ["source", "max_silence_seconds", "msg"], "subs": ["artnet"], "identity": ["source"], "module": None,
                  "question": "Is the lighting console sending?", "words": "Art-Net is arriving from the lighting program."},
    "atem":      {"fields": ["part", "msg"], "subs": ["audio", "camera"], "identity": ["part"], "module": "bmd-atem",
                  "question": "Is a setting right?", "words": "The switcher matches the saved file."},
 }
 COMMON_CHECK_FIELDS = ["type", "fix", "tag", "service_only", "debounce_seconds"]
 FIX_KINDS = sp.FIX_KINDS
 FIX_KIND_FIELDS = {
    "ssh": ["host", "run", "timeout", "reconnect", "conn"], "local": ["run", "timeout", "reconnect", "conn"],
    "atem": ["part"], "reconnect": ["conn", "verify_seconds", "hung_note"], "midi_connect": [],
    "netgear_poe": ["switch", "port", "mac"], "hyperdeck": ["verify_seconds", "off_seconds", "hung_note"],
    "spotify": ["playlist"],
 }
 EXPECTED_MODULE = {t: spec["module"] for t, spec in CHECKS.items() if spec["module"]}
 # --- Every word a key can show, grouped by what to do about it -------------------
 GLOSSARY = {
    "hold the key: the fix runs": [
        "NOT PLAYING", "WRONG PLAYLIST", "SPOTIFY CLOSED", "STOPPED", "NOT LOADED", "MIDI LOST", "MIDI", "GONE",
        "DISABLED", "OFFLINE", "NO SESSION", "NOT RUNNING", "DOWN", "NOT REC", "SILENT", "WARNING", "STARTING",
        "ATEM AUDIO WRONG", "ATEM CAMERA WRONG", "ATEM SW MIDI ON", "PT SOLO NOT AFL", "PROPRES NOT RUNNING",
        "ABLETON NOT RUNNING", "TRANSLATE NOT RUNNING", "PROP TV DOWN", "BIBLE APP DOWN", "PRODCOM DOWN", "PRODDASH DOWN",
        "ENCODER WEB DOWN", "PT SOLO CHECK OFF", "UNREACH", "DROPPED",
    ],
    "check the device or the cable": [
        "NO PING", "NO ANSWER", "UNREACHABLE", "LINK DOWN", "NO GATEWAY", "NO DISK", "FULL", "LOW", "BATT", "INTERF",
        "RF MUTE", "HUNG - POWER CYCLE IT", "NO NETWORK CHECK POWER", "STILL DOWN", "REFUSED", "LOGIN", "BLIP",
        "NOT REACHABLE", "DISK",
    ],
    "ask Chris": ["NOT IN COMPANION", "CHECK BUG", "NO DATA", "WATCHDOG DOWN", "ERR", "PROCESS", "WEB"],
    "what a key says while you hold it": [
        "NOTHING TO FIX", "NO AUTO FIX", "JUST TRIED", "WAIT 1 MIN", "WAIT", "MIN", "S", "CHECKING", "FIXING",
        "AUTO-FIXING", "WAKING PC", "FIX FAILED", "RESTARTING", "RESTARTING SOON", "SOON", "DONE", "FAILED",
        "JUST DONE", "STARTING", "RECORDING LEFT ALONE", "NOTHING TO CLEAR", "CLEARED", "STILL CONNECTING",
        "STILL OFF IN COMPANION", "DISABLED IN COMPANION", "TURN ON", "RECONNECT", "POWER CYCLE", "FIXES", "HOLD TO FIX",
        "HOLD", "REC", "PLAYING", "OK", "OFF", "ON", "ALL OK", "WARN", "TRS", "GB", "NOT", "REACHABLE", "RECORDING",
        "LEFT ALONE", "TO FIX", "TO CLEAR", "NO AUTO", "FIX", "AUTO", "IN", "COMPANION", "POWER", "CYCLE",
        "STILL", "OFF IN", "DISABLED IN", "CONNECTING", "CHECK", "NETWORK", "NO",
    ],
 }
 # --- Rules ---------------------------------------------------------------------
 def defaults():
    """{path: default} for every field with one."""
    return {p: copy.deepcopy(f["default"]) for p, f in F.items() if f["default"] is not None}
 def key_record():
    """{receiver kind: {keys}}: which keys the code may read from each kind of config object."""
    rec = {}
    for p in F:
        parts = p.split(".")
        kind = "top"
        if p.startswith("tiles[].checks[]."):
            kind, parts = "check", parts[2:]
        elif p.startswith("tiles[]."):
            kind, parts = "tile", parts[1:]
        elif p.startswith("fixes.*."):
            kind, parts = "fix", parts[2:]
        elif p.startswith("hosts.*."):
            kind, parts = "host", parts[2:]
        elif p.startswith("restart_page.keys[]."):
            kind, parts = "restart_key", parts[2:]
        elif p.startswith("restart_page."):
            kind, parts = "restart_page", parts[1:]
        elif p.startswith("auto_fix."):
            kind, parts = "auto", parts[1:]
        elif p.startswith("network."):
            kind, parts = "network", parts[1:]
        elif p.startswith("variables."):
            kind, parts = "variables", parts[1:]
        elif p.startswith("site."):
            kind, parts = "site", parts[1:]
        elif p.startswith("atem."):
            kind, parts = "atem", parts[1:]
        elif p.startswith("spotify."):
            kind, parts = "spotify", parts[1:]
        elif p.startswith("companion."):
            kind, parts = "companion", parts[1:]
        elif p.startswith("timing."):
            kind, parts = "timing", parts[1:]
        elif p.startswith("flap."):
            kind, parts = "flap", parts[1:]
        elif p.startswith("gear.*."):
            kind, parts = "gear", parts[2:]
        if "[]" in parts[0] and len(parts) > 1:
            kind, parts = "window", parts[1:]
        rec.setdefault(kind, set()).add(parts[0].replace("[]", ""))
    return rec
 def _windows(ws, where, errors):
    try:
        sp.check_windows(ws, where)
    except ValueError as e:
        errors.append(str(e))
 def validate(cfg, live=None):
    """(errors, warnings). Errors stop an Apply; warnings are shown. `live` (optional) is what the
    settings page knows right now: {"connections": {label: module id}, "hosts_up": {alias: bool}}."""
    errors, warnings = [], []
    live = live or {}
    def pos_ok(pos, where):
        if not (isinstance(pos, list) and len(pos) == 2 and all(isinstance(n, int) for n in pos)):
            errors.append("%s: position must be [row, column]" % where)
            return None
        grid = cfg.get("page_grid", F["page_grid"]["default"])
        if not (grid["minRow"] <= pos[0] <= grid["maxRow"] and grid["minColumn"] <= pos[1] <= grid["maxColumn"]):
            errors.append("%s: row %d column %d is off the grid" % (where, pos[0], pos[1]))
        if pos[1] == 0 and pos[0] in (0, 1, 2):
            errors.append("%s: row %d column 0 is a page key" % (where, pos[0]))
        return tuple(pos)
    for req in ("companion", "variables", "tiles"):
        if req not in cfg:
            errors.append("missing %s" % req)
    if errors:
        return errors, warnings
    ids, seen_pos = set(), {}
    kinds = {"master": 0, "service": 0, "network": 0}
    fixes, hosts = cfg.get("fixes", {}), cfg.get("hosts", {})
    for t in cfg["tiles"]:
        tid = t.get("id", "")
        where = "key %s" % (tid or "?")
        if not re.fullmatch(r"[A-Za-z0-9_]+", str(tid)):
            errors.append("%s: the id may only hold letters, digits and _" % where)
        if tid in ids:
            errors.append("%s: two keys with the same id" % where)
        ids.add(tid)
        if not t.get("label"):
            errors.append("%s: no label" % where)
        if t.get("kind") in kinds:
            kinds[t["kind"]] += 1
        if "pos" in t:
            pos = pos_ok(t["pos"], where)
            if pos in seen_pos:
                errors.append("%s and %s share row %d column %d" % (seen_pos[pos], where, pos[0], pos[1]))
            seen_pos[pos] = where
        if t.get("hold_always") and t["hold_always"] not in fixes:
            errors.append("%s: hold runs unknown fix %s" % (where, t["hold_always"]))
        for i, c in enumerate(t.get("checks", [])):
            cw = "%s check %d" % (where, i + 1)
            if c.get("type") not in CHECKS:
                errors.append("%s: unknown check type %r" % (cw, c.get("type")))
                continue
            spec = CHECKS[c["type"]]
            for k in c:
                if k not in spec["fields"] and k not in COMMON_CHECK_FIELDS:
                    warnings.append("%s: %s is not a field of a %s check" % (cw, k, c["type"]))
            if c["type"] in ("conn", "hyperdeck", "dante", "shure", "psm", "tascam") and not c.get("conn"):
                errors.append("%s: a %s check needs a Companion connection" % (cw, c["type"]))
            if c["type"] == "ping" and not (c.get("host") or c.get("conn")):
                errors.append("%s: a ping needs an address or a connection" % cw)
            if c["type"] == "midi_peer" and not (c.get("session") and c.get("peer")):
                errors.append("%s: a MIDI link needs a session and a peer" % cw)
            if c["type"] == "remote" and not (c.get("host") and c.get("cmd") and c.get("msg")):
                errors.append("%s: a remote check needs a computer, a command and the words to show" % cw)
            fx = c.get("fix")
            for f in (fx.values() if isinstance(fx, dict) else [fx]):
                if f and f not in fixes and f not in (sp.RESTART_CONNECTION, sp.DANTE_AUTO):
                    errors.append("%s: unknown fix %s" % (cw, f))
            if c.get("service_only"):
                warnings.append("%s: 'only during a service' has no effect while alarms are armed all day" % cw)
            conn, mod = c.get("conn"), spec["module"]
            if conn and mod and live.get("connections") and conn in live["connections"] \
                    and live["connections"][conn] and not live["connections"][conn].startswith(mod):
                warnings.append("%s: %s uses module %s; this check expects %s" % (cw, conn, live["connections"][conn], mod))
            if conn and live.get("connections") and conn not in live["connections"]:
                warnings.append("%s: Companion has no connection called %s" % (cw, conn))
    for t in cfg["tiles"]:
        for m in t.get("members", []):
            if m not in ids:
                errors.append("key %s: unknown member %s" % (t.get("id"), m))
            if m == t.get("id"):
                errors.append("key %s: is its own member" % t.get("id"))
    in_group = {m for t in cfg["tiles"] for m in t.get("members", [])}
    for t in cfg["tiles"]:
        if "pos" not in t and "members" not in t and t.get("id") not in in_group and t.get("kind") != "master":
            warnings.append("key %s has no place on the deck and is in no group: nobody will see it" % t.get("id"))
    if kinds["master"] != 1:
        errors.append("there must be exactly one ALL key (kind master); found %d" % kinds["master"])
    if kinds["service"] != 1:
        errors.append("there must be exactly one ALARMS key (kind service); found %d" % kinds["service"])
    if kinds["network"] > 1:
        errors.append("at most one NET key (kind network); found %d" % kinds["network"])
    for fid, fix in fixes.items():
        where = "fix %s" % fid
        kind = fix.get("kind", "ssh")
        if kind not in FIX_KINDS:
            errors.append("%s: unknown kind %s" % (where, kind))
        if not fix.get("label"):
            errors.append("%s: needs plain words (label)" % where)
        if kind == "ssh":
            h = hosts.get(fix.get("host"))
            if h is None:
                errors.append("%s: unknown computer %s" % (where, fix.get("host")))
            elif h.get("self"):
                errors.append("%s: %s is this Mac - use the local kind" % (where, fix.get("host")))
            elif not h.get("ssh"):
                errors.append("%s: computer %s has no log-in address" % (where, fix.get("host")))
        if kind in ("ssh", "local") and not fix.get("run"):
            errors.append("%s: nothing to run" % where)
        if kind == "netgear_poe" and not (fix.get("switch") and fix.get("port")):
            errors.append("%s: needs a switch and a port" % where)
        if kind == "reconnect" and not fix.get("conn"):
            errors.append("%s: needs the connection to reconnect" % where)
        if kind == "atem" and fix.get("part") not in ("audio", "camera"):
            errors.append("%s: part must be audio or camera" % where)
        for k in fix:
            if k not in FIX_KIND_FIELDS.get(kind, []) and k not in ("kind", "short", "label", "restart_ok", "never_auto",
                                                                      "template", "params", "timeout"):
                warnings.append("%s: %s is not a field of a %s fix" % (where, k, kind))
    for alias, h in hosts.items():
        if not re.fullmatch(r"[a-z0-9_]+", alias):
            errors.append("computer %s: the short name may only hold lowercase letters, digits and _" % alias)
        if not h.get("self") and not re.fullmatch(r"[^@\s]+@[^@\s]+", str(h.get("ssh", ""))):
            errors.append("computer %s: the log-in address must be user@address" % alias)
        if h.get("mac") and not re.fullmatch(r"(?:[0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}", h["mac"]):
            errors.append("computer %s: not a hardware (MAC) address: %s" % (alias, h["mac"]))
    for t in cfg["tiles"]:
        for i, c in enumerate(t.get("checks", [])):
            if c.get("type") == "remote" or (c.get("type") == "midi_peer" and c.get("host")):
                h = hosts.get(c["host"])
                if h is None:
                    errors.append("key %s check %d: unknown computer %s" % (t.get("id"), i + 1, c["host"]))
                elif h.get("self"):
                    errors.append("key %s check %d: %s is this Mac" % (t.get("id"), i + 1, c["host"]))
    a = cfg.get("auto_fix") or {}
    for f in a.get("fixes", []):
        if f not in fixes:
            errors.append("auto-fix: unknown fix %s" % f)
        elif fixes[f].get("never_auto"):
            errors.append("auto-fix: %s is marked never to run by itself" % f)
        elif re.search(r"pkill|taskkill|killall", fixes[f].get("run", "")) and not fixes[f].get("restart_ok"):
            errors.append("auto-fix: %s closes a running program; tick 'may close a running program' or take it out" % f)
    _windows(cfg.get("service_windows"), "alarms window", errors)
    _windows(a.get("windows"), "auto-fix window", errors)
    _windows((cfg.get("site") or {}).get("live_services"), "live services", errors)
    rp = cfg.get("restart_page")
    if rp:
        rpos = {}
        for k in rp.get("keys", []):
            kid = k.get("id", "")
            where = "restart key %s" % (kid or "?")
            if kid in ids:
                errors.append("%s: a key on the Status page has that id" % where)
            if k.get("fix") not in fixes:
                errors.append("%s: unknown fix %s" % (where, k.get("fix")))
            for w in k.get("watch", []):
                if not w.startswith("conn:") and w not in fixes:
                    errors.append("%s: unknown fix %s in its watch list" % (where, w))
            pos = pos_ok(k.get("pos"), where)
            if pos in rpos:
                errors.append("%s and %s share row %d column %d" % (rpos[pos], where, pos[0], pos[1]))
            rpos[pos] = where
        for name in ("jump_pos", "back_pos"):
            if name in rp:
                pos_ok(rp[name], "the Restart page's %s" % name.replace("_pos", " key"))
        if "jump_pos" in rp and tuple(rp["jump_pos"]) in seen_pos:
            errors.append("the 'go to Restart page' key sits on %s" % seen_pos[tuple(rp["jump_pos"])])
        if "back_pos" in rp and tuple(rp["back_pos"]) in rpos:
            errors.append("the 'back to Status' key sits on %s" % rpos[tuple(rp["back_pos"])])
        if rp.get("page") == cfg.get("page"):
            errors.append("the Status page and the Restart page cannot be the same page")
    spo = cfg.get("spotify") or {}
    if any(c.get("type") == "spotify" for t in cfg["tiles"] for c in t.get("checks", [])):
        if not re.fullmatch(r"spotify:playlist:[A-Za-z0-9]+", str(spo.get("playlist", ""))):
            errors.append("Spotify: the playlist must look like spotify:playlist:...")
    for p, f in F.items():
        v = _get(cfg, p)
        if v is None:
            continue
        if f["type"] == "int" and not isinstance(v, int):
            errors.append("%s: must be a whole number" % p)
        elif f["type"] == "int":
            if f["lo"] is not None and v < f["lo"]:
                errors.append("%s: %s is below %s" % (p, v, f["lo"]))
            if f["hi"] is not None and v > f["hi"]:
                errors.append("%s: %s is above %s" % (p, v, f["hi"]))
        if f["regex"] and isinstance(v, str) and not re.fullmatch(f["regex"], v):
            errors.append("%s: %r is not in the right form" % (p, v))
        if f["type"] == "enum" and f["options"] and v not in f["options"]:
            errors.append("%s: %r is not one of %s" % (p, v, ", ".join(f["options"])))
    return errors, warnings
 def _get(cfg, path):
    """The value at a plain path (no [] or *); None when absent."""
    if "[]" in path or ".*." in path:
        return None
    cur = cfg
    for part in path.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return None
        cur = cur[part]
    return cur
 def mask_secrets(obj):
    """A copy with every secret replaced by "set (N characters)" or "" (for API responses and history)."""
    secret_paths = {p for p, f in F.items() if f["secret"]}
    def walk(node, path):
        if isinstance(node, dict):
            return {k: ("set (%d characters)" % len(v) if v else "") if "%s.%s" % (path, k) in secret_paths
                    or k in secret_paths else walk(v, "%s.%s" % (path, k) if path else k) for k, v in node.items()}
        if isinstance(node, list):
            return [walk(v, path + "[]") for v in node]
        return node
    return walk(copy.deepcopy(obj), "")
 def phrases(cfg):
    """The sentences the page and the logs build from the config."""
    a = cfg.get("auto_fix") or {}
    site = cfg.get("site") or {}
    return {
        "auto_fix": "may run by itself %s" % sp.windows_phrase(a.get("windows", [])) if a.get("windows") else "never runs by itself",
        "live": ("changes wait while a HyperDeck is recording" + (" or %s" % sp.windows_phrase(site["live_services"])
                 if site.get("live_services") else "")),
        "alarms": "Alarms are on every day. The ALARMS key silences them until midnight, then they come back by themselves.",
        "hold": "HOLD %gs" % (cfg.get("page_hold_ms", 2000) / 1000.0),
        "ingest": site.get("ingest_name", "NA Ingest"),
    }
@@ -16,6 +16,7 @@ Stdlib only; runs on the macOS /usr/bin/python3 (3.9).
 """
 import argparse
 import calendar
 import hashlib
 import http.client
 import json
 import logging
@@ -24,6 +25,7 @@ import logging.handlers
 import os
 import re
 import shutil
 import signal
 import socket
 import ssl
 import subprocess
@@ -38,6 +40,10 @@ from datetime import datetime
 HERE = os.path.dirname(os.path.abspath(__file__))
 log = logging.getLogger("status-panel")
 VERSION = "2026.10.13"     # logged at start ("status panel running (code ...)") and written to started.json
 # Every kind of fix the watchdog can run (validated by load_config).
 FIX_KINDS = ("ssh", "local", "atem", "reconnect", "midi_connect", "netgear_poe", "hyperdeck", "spotify")
 OK, WARN, ERR, OFF = "ok", "warn", "err", "off"
 # Display levels, least to most urgent. "idle" is a service-only fault while
@@ -629,13 +635,14 @@ class AutoLog:
    NAME = re.compile(r"^(\d{4})-(\d{2})-(\d{2}) [A-Za-z]+ Auto-Fix\.log$")
-    def __init__(self, directory):
+    def __init__(self, directory, who="NA Ingest", when="on Sundays 06:30-13:30", keep_months=1):
        self.dir = os.path.expanduser(directory)
        self.lock = threading.Lock()
        self.who, self.when, self.keep_months = who, when, max(1, int(keep_months))
    def cleanup(self, now):
-        """Delete day files once they are a calendar month old (the 11 Oct file goes on 11 Nov).
+        """Delete day files once they are keep_months calendar months old (the 11 Oct file goes on
-        Only files named exactly like these logs are ever touched."""
+        11 Nov). Only files named exactly like these logs are ever touched."""
        today = datetime.fromtimestamp(now).date()
        removed = []
        try:
@@ -647,7 +654,8 @@ class AutoLog:
            if not m:
                continue
            y, mo, d = (int(x) for x in m.groups())
-            y2, mo2 = (y + 1, 1) if mo == 12 else (y, mo + 1)
+            total = mo - 1 + self.keep_months
            y2, mo2 = y + total // 12, total % 12 + 1
            try:
                due = datetime(y2, mo2, min(d, calendar.monthrange(y2, mo2)[1])).date()
            except ValueError:
@@ -678,41 +686,111 @@ class AutoLog:
                with open(path, "a", encoding="utf-8") as f:
                    if new:
                        f.write("Status panel auto-fixes - %s\n"
-                                "(NA Ingest fixes these by itself on Sundays 06:30-13:30; anything that closes a\n"
+                                "(%s fixes these by itself %s; anything that closes a\n"
                                " running program stays a hold. Newest lines at the bottom.)\n\n"
-                                % dt.strftime("%A %-d %B %Y"))
+                                % (dt.strftime("%A %-d %B %Y"), self.who, self.when))
                    f.write(line)
        except OSError as e:
            log.warning("auto-fix log %s: %s", self.dir, e)
 def window_holds(w, day, hm, prev_day):
    """One window at one moment. "end" <= "start" means the window runs past midnight: it starts on
    one of its days and ends early the next morning (a Saturday night service: sat 18:00-00:30)."""
    if w["start"] < w["end"]:
        return day in w["days"] and w["start"] <= hm < w["end"]
    return (day in w["days"] and hm >= w["start"]) or (prev_day in w["days"] and hm < w["end"])
 def in_windows(windows, now):
    """Is `now` inside one of [{"days": [...], "start": "HH:MM", "end": "HH:MM"}]?"""
    dt = datetime.fromtimestamp(now)
    hm, day = dt.strftime("%H:%M"), DAYS[dt.weekday()]
-    return any(day in w["days"] and w["start"] <= hm < w["end"] for w in windows)
+    return any(window_holds(w, day, hm, DAYS[dt.weekday() - 1]) for w in windows)
 def windows_phrase(windows):
    """Windows in words, for log headers and notes: "on Sundays 06:30-13:30"."""
    parts = []
    for w in windows:
        days = [d for d in DAYS if d in w["days"]]
        names = [datetime(2024, 1, 1 + DAYS.index(d)).strftime("%As") for d in days]   # 2024-01-01 is a Monday
        when = "every day" if len(days) == 7 else " and ".join(names)
        parts.append("%s %s-%s" % (when, w["start"], w["end"]))
    return ("on " + " and ".join(parts)) if parts else "never"
 def check_identity(chk):
    """What a check is about, independent of its position in the list: the key that remembered
    state (acknowledgements, auto-fix tries) is filed under, so editing the list keeps it."""
    t = chk["type"]
    if t == "midi_peer":
        return "%s/%s/%s" % (chk.get("host", ""), chk["session"], chk["peer"])
    if t == "remote":
        return "%s/%s" % (chk["host"], hashlib.sha1(chk["cmd"].encode()).hexdigest()[:8])
    if t == "ping":
        return chk.get("conn") or chk.get("host", "")
    if t == "pref":
        return "%s/%s" % (chk["domain"], chk["key"])
    single = {"http": "url", "mdns": "instance", "launchd": "label", "process": "match", "artnet": "source",
              "atem": "part", "disk": "path", "midi_rx": "source"}
    if t in single:
        return str(chk.get(single[t], ""))
    return chk.get("conn") or t
 def write_json(path, obj):
    """Write a small JSON file in one go (a reader never sees half of it)."""
    tmp = path + ".tmp"
    with open(tmp, "w") as f:
        json.dump(obj, f, indent=1, sort_keys=True)
    os.replace(tmp, path)
 class State:
    """One file (state.json) for everything the watchdog remembers across restarts: the ALARMS
    override, acknowledged faults, auto-fix tries and when each fix last ran. One writer; saved whole."""
    def __init__(self, path):
        self.path, self.data, self.lock = path, {}, threading.Lock()
        if not path:
            return
        try:
            with open(path) as f:
                self.data = json.load(f)
        except (OSError, ValueError):
            legacy = os.path.join(os.path.dirname(path), "service-mode.json")   # the file before state.json
            try:
                with open(legacy) as f:
                    old = json.load(f)
                self.data = {"service": {"override": old.get("override"), "last_sched": old.get("last_sched")}}
            except (OSError, ValueError):
                pass
    def get(self, key, default=None):
        return self.data.get(key, default)
    def set(self, key, value):
        with self.lock:
            self.data[key] = value
            if self.path:
                try:
                    write_json(self.path, self.data)
                except OSError as e:
                    log.warning("could not save %s: %s", self.path, e)
 class ServiceMode:
    """On inside a scheduled window; a button press overrides until the next window edge."""
-    def __init__(self, windows, path):
+    def __init__(self, windows, state):
-        self.windows, self.path = windows, path
+        self.windows = windows
-        self.override, self.last_sched = None, None
+        self.state = state if isinstance(state, State) else State(state)
-        if not path:
+        saved = self.state.get("service") or {}
            return
        try:
            with open(path) as f:
                saved = json.load(f)
        self.override, self.last_sched = saved.get("override"), saved.get("last_sched")
        except (OSError, ValueError):
            pass
    def scheduled(self, now):
-        dt = datetime.fromtimestamp(now)
+        return in_windows(self.windows, now)
        hm = dt.strftime("%H:%M")
        day = DAYS[dt.weekday()]
        return any(day in w["days"] and w["start"] <= hm < w["end"] for w in self.windows)
    def on(self, now):
        sched = self.scheduled(now)
@@ -728,28 +806,33 @@ class ServiceMode:
        self._save()
    def _save(self):
-        if not self.path:
+        self.state.set("service", {"override": self.override, "last_sched": self.last_sched})
            return
        try:
            tmp = self.path + ".tmp"
            with open(tmp, "w") as f:
                json.dump({"override": self.override, "last_sched": self.last_sched}, f)
            os.replace(tmp, self.path)
        except OSError as e:
            log.warning("could not save service mode: %s", e)
 # --- Engine ------------------------------------------------------------------
 class Engine:
-    def __init__(self, cfg, state_path=None):
+    def __init__(self, cfg, state_path=None, data_dir=None):
        self.cfg = cfg
        self.tiles = {t["id"]: t for t in cfg["tiles"]}
        self.debounce = cfg.get("debounce_seconds", 3)
        self.flap_count = cfg.get("flap", {}).get("count", 3)
        self.flap_window = cfg.get("flap", {}).get("window_seconds", 600)
        self.net = cfg.get("network", {})
-        self.service = ServiceMode(cfg.get("service_windows", []), state_path)
+        self.site = cfg.get("site", {})
        timing = cfg.get("timing", {})
        self.cooldown = timing.get("fix_cooldown_seconds", 60)     # the same fix never re-runs within this
        self.hold_ms = cfg.get("page_hold_ms", 2000)                # how long a key is held to run its fix
        self.state = State(state_path)
        self.service = ServiceMode(cfg.get("service_windows", []), self.state)
        self.data_dir = data_dir
        self.stopping = False       # SIGTERM arrived: finish what runs, start nothing new
        self.running = {}           # fix thread name -> {"key", "what", "since"} (published as fixes-running.json)
        self.fix_threads = []       # the threads behind self.running, joined at exit
        self.any_recording = False
        self.live = None            # (live?, reason) last written to live-now.json
        self.live_written = 0.0
        self.restore_acks = set(self.state.get("acks") or [])        # acknowledged before the last restart
        self.trackers = {}
        self.group_of = {}
        for t in cfg["tiles"]:
@@ -763,16 +846,19 @@ class Engine:
        self.burst = None           # {"start", "conns", "cleared"}
        self.events = []
        self.meta = {}              # key -> (tile id, check, sub key)
-        self.fix_last = {}          # fix identity -> last run time
+        self.fix_last = {tuple(k.split("\0")): v for k, v in (self.state.get("fix_last") or {}).items()}
        self.executor = execute_fix
        self.poe_executor = execute_poe
        self.notes = {}             # tile id -> hold feedback {"text", "level", "until"}
        # The Restart page: one key per thing, hold 2 s = restart it even when it looks fine.
        self.restart_keys = {k["id"]: k for k in cfg.get("restart_page", {}).get("keys", [])}
        self.auto = cfg.get("auto_fix") or {}
-        self.auto_tries = {}        # fix identity -> [tries, last try time] for the fault now going on
+        saved = self.state.get("auto") or {}
-        self.auto_seen = {}         # fix identity -> last time it was in an auto plan
+        self.auto_tries = {tuple(k.split("\0")): v for k, v in saved.get("tries", {}).items()}
-        self.autolog = AutoLog(self.auto.get("log_dir", "~/Desktop/Auto-Fix Logs")) if self.auto else None
+        self.auto_seen = {tuple(k.split("\0")): v for k, v in saved.get("seen", {}).items()}
        self.autolog = AutoLog(self.auto.get("log_dir", "~/Desktop/Auto-Fix Logs"),
                               self.site.get("ingest_name", "NA Ingest"), windows_phrase(self.auto.get("windows", [])),
                               self.auto.get("keep_months", 1)) if self.auto else None
        if self.autolog:
            self.autolog.cleanup(time.time())
        self.auto_watch = {}        # "BUTTON FAULT" auto-fix went after -> when (to log when it clears)
@@ -782,6 +868,81 @@ class Engine:
        self.deck_ips = {}          # HyperDeck connection -> last address Companion reported
        self.now = 0.0
    @property
    def hold_text(self):
        return "HOLD %ss" % ("%g" % (self.hold_ms / 1000.0))
    @property
    def wait_text(self):
        c = int(self.cooldown)
        return "WAIT %d MIN" % (c // 60) if c % 60 == 0 else "WAIT %d S" % c
    def identity_key(self, key):
        """A tracker key ("tile:index:sub") as the position-free name its state is remembered by."""
        tile_id, chk, sub = self.meta[key]
        return "%s|%s|%s" % (tile_id, check_identity(chk), sub)
    def remember(self):
        """Save what must survive a restart: acknowledgements, auto-fix tries, when fixes last ran."""
        acks = sorted({self.identity_key(k) for k, tr in self.trackers.items()
                       if tr.confirmed and tr.acked and k in self.meta} | self.restore_acks)
        self.state.set("acks", acks)
        self.state.set("fix_last", {"\0".join(k): v for k, v in self.fix_last.items()})
        self.state.set("auto", {"tries": {"\0".join(k): v for k, v in self.auto_tries.items()},
                                "seen": {"\0".join(k): v for k, v in self.auto_seen.items()}})
    def live_now(self, now):
        """(live?, why): changes to the panel wait while this is true. A recording HyperDeck is
        always live; a campus may add clock windows ("live_services" in "site")."""
        if self.any_recording:
            return True, "a HyperDeck is recording"
        if in_windows(self.site.get("live_services", []), now):
            return True, "a live service is scheduled"
        return False, ""
    def publish(self, now):
        """live-now.json and fixes-running.json in the data dir (the settings page reads them)."""
        if not self.data_dir:
            return
        live = self.live_now(now)
        if live != self.live or now - self.live_written > 5:
            try:
                write_json(os.path.join(self.data_dir, "live-now.json"),
                           {"live": live[0], "reason": live[1], "time": now})
            except OSError as e:
                log.warning("live-now.json: %s", e)
            self.live, self.live_written = live, now
    def fix_started(self, name, key_id, what):
        self.running[name] = {"key": key_id, "what": what, "since": time.time()}
        self.publish_running()
    def fix_done(self, name):
        self.running.pop(name, None)
        self.publish_running()
    def publish_running(self):
        if self.data_dir:
            try:
                write_json(os.path.join(self.data_dir, "fixes-running.json"), self.running)
            except OSError as e:
                log.warning("fixes-running.json: %s", e)
    def start_fix_thread(self, target, key_id, what):
        """Every fix runs in a thread the watchdog knows about: a restart waits for them."""
        name = "fix-%s-%d" % (key_id, int(time.time() * 1000) % 100000)
        def wrapped():
            try:
                target()
            finally:
                self.fix_done(name)
        self.fix_started(name, key_id, what)
        t = threading.Thread(target=wrapped, name=name, daemon=True)
        self.fix_threads = [x for x in self.fix_threads if x.is_alive()] + [t]
        t.start()
        return t
    # raw results for every check key
    def evaluate(self, snap, now, service_on):
        ctx = {"now": now, "service": service_on, "any_recording": False, "dante_all": {}}
@@ -794,6 +955,7 @@ class Engine:
                if chk["type"] == "hyperdeck" and chk.get("rec_sync", True):
                    if conn_result(snap, chk["conn"], now).level == OK and hyperdeck_recording(snap, chk["conn"]):
                        ctx["any_recording"] = True
        self.any_recording = ctx["any_recording"]
        raw = {}
        for t in self.cfg["tiles"]:
            for i, chk in enumerate(t.get("checks", [])):
@@ -878,13 +1040,23 @@ class Engine:
        deb = self.debounce if debounce is None else debounce
        for key, (tile_id, res, chk) in raw.items():
            tr = self.trackers.setdefault(key, Tracker())
            was_acked = tr.acked
            # A MIDI peer leaving a session is a clean event: show it after 1 s, not the general 3 s.
            event = tr.update(res, now, 0 if chk["type"] == "network" else
                              chk.get("debounce_seconds", 1 if chk["type"] == "midi_peer" else deb),
                              self.flap_window)
            if event == "fault":
                self.events.append(("FAULT", "%s %s" % (self.tiles[tile_id]["label"], res.msg)))
                ident = self.identity_key(key) if key in self.meta else None
                if ident in self.restore_acks:       # acknowledged before the restart: still acknowledged
                    self.restore_acks.discard(ident)
                    tr.acked = True
                    self.remember()
            elif event == "clear":
                ident = self.identity_key(key) if key in self.meta else None
                if was_acked or ident in self.restore_acks:
                    self.restore_acks.discard(ident)
                    self.remember()                      # an acknowledgement is for one fault only
                text = "%s %s" % (self.tiles[tile_id]["label"], tr.last_fault_msg)
                self.events.append(("CLEAR", text))
                fixed_at = self.auto_watch.pop(text, None)
@@ -896,15 +1068,19 @@ class Engine:
        # fault is over - clear it, or it lingers unseen and a hold would still try to fix it.
        for key, tr in self.trackers.items():
            if key not in raw and (tr.confirmed or tr.bad_since is not None):
                was_acked = tr.acked
                if tr.update(Result(OK), now, 0, self.flap_window) == "clear" and key in self.meta:
                    tile_id = self.meta[key][0]
                    self.events.append(("CLEAR", "%s %s" % (self.tiles[tile_id]["label"], tr.last_fault_msg)))
                    if was_acked:
                        self.remember()
        self.items = {}
        for key, (tile_id, res, chk) in raw.items():
            tr = self.trackers[key]
            eff = tr.effective(self.flap_count)
            self.items.setdefault(tile_id, []).append(
                (display_level(eff, tr.acked, service_on), eff, tr.since or now))
        self.publish(now)
        return render(self, service_on)
    def keys_for(self, tile_id):
@@ -925,6 +1101,8 @@ class Engine:
            if tr.confirmed and not tr.acked:
                tr.acked = True
                n += 1
        if n:
            self.remember()
        return n
    def restart_targets(self, tile_id):
@@ -1038,7 +1216,11 @@ class Engine:
        """Show `text` on the button for a while (hold feedback), instead of its usual text."""
        self.notes[tile_id] = {"text": text, "level": level, "until": now + seconds}
-    def run_fixes(self, tile_id, snap, comp, now, cooldown=60):
+    def run_fixes(self, tile_id, snap, comp, now, cooldown=None):
        cooldown = self.cooldown if cooldown is None else cooldown
        if self.stopping:
            self.note(tile_id, "RESTARTING\nSOON", "idle", now, 4)
            return []
        if self.tiles.get(tile_id, {}).get("kind") == "network":
            # NET: a hold clears a remembered network blip (link and gateway clear themselves).
            if self.burst is None:
@@ -1071,8 +1253,9 @@ class Engine:
            return plan
        return self.run_plan(tile_id, plan, snap, comp, now, cooldown)
-    def run_plan(self, tile_id, plan, snap, comp, now, cooldown=60, auto=False, on_done=None):
+    def run_plan(self, tile_id, plan, snap, comp, now, cooldown=None, auto=False, on_done=None):
        """Carry out fix_plan() items for one button (a hold, or auto-fix during service)."""
        cooldown = self.cooldown if cooldown is None else cooldown
        started, names, waited, midi = [], [], False, []
        for ident, kind, what, chk in plan:
            if now - self.fix_last.get(ident, 0) < cooldown:
@@ -1080,6 +1263,7 @@ class Engine:
                waited = True
                continue
            self.fix_last[ident] = now
            self.remember()
            if kind == "enable":
                c = snap.conns.get(what)
                if c:
@@ -1112,7 +1296,7 @@ class Engine:
            started.append((midi_worker, (midi,), self.cfg["fixes"][midi[0][0]].get("short", "MIDI")))
        if not names:
            if not auto:
-                self.note(tile_id, "JUST TRIED\nWAIT 1 MIN" if waited else "NOTHING\nTO FIX", "idle", now, 4)
+                self.note(tile_id, ("JUST TRIED\n" + self.wait_text) if waited else "NOTHING\nTO FIX", "idle", now, 4)
            return plan
        summary = names[0] if len(set(names)) == 1 else "%d FIXES" % len(names)
        if not started:                               # only Companion reconnects: quick
@@ -1148,7 +1332,7 @@ class Engine:
                    on_done(outcome)
        said = []
        for fn, args, short in started:
-            threading.Thread(target=work, args=(fn, args, short), daemon=True).start()
+            self.start_fix_thread(lambda fn=fn, args=args, short=short: work(fn, args, short), tile_id, short)
        return plan
    def auto_fix(self, snap, comp, now):
@@ -1158,7 +1342,7 @@ class Engine:
        the network is having a blip, and never while ALARMS has muted the panel (maintenance).
        Restarting a running program stays a hold."""
        a = self.auto
-        if not a or now - self.auto_last < 5:
+        if not a or now - self.auto_last < 5 or self.stopping:
            return []
        self.auto_last = now
        if not in_windows(a.get("windows", []), now) or not self.service.on(now):
@@ -1184,7 +1368,7 @@ class Engine:
        if self.auto_paused and self.autolog:
            self.autolog.write(now, "NET", "network settled: auto-fix is back on")
        self.auto_paused = False
-        gentle = set(a.get("fixes", []))
+        gentle = {f for f in a.get("fixes", []) if not self.cfg["fixes"].get(f, {}).get("never_auto")}
        done = []
        for t in self.cfg["tiles"]:
            if "checks" not in t:
@@ -1243,6 +1427,8 @@ class Engine:
            # fault blink out for a moment (restarting, re-confirming), which must not reset the count.
            if now - self.auto_seen.get(ident, 0) >= forget:
                del self.auto_tries[ident]
        if done or self.auto_tries:
            self.remember()
        return done
    def describe(self, kind, what):
@@ -1253,9 +1439,13 @@ class Engine:
            return self.cfg["fixes"][what].get("label", what)
        return "%s %s" % (kind, what)
-    def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None, chk=None):
+    def run_force(self, key_id, snap, comp, now, cooldown=None, fix_id=None, chk=None, verb="FORCE"):
        """Run one fix whatever the status says: a Restart page key, or a button's "hold_always".
-        Shares the 60 s guard with holds, so a hold and a restart key never double up."""
+        Shares the cooldown guard with holds, so a hold and a restart key never double up."""
        cooldown = self.cooldown if cooldown is None else cooldown
        if self.stopping:
            self.note(key_id, "RESTARTING\nSOON", "idle", now, 4)
            return None
        if fix_id is None:
            key = self.restart_keys.get(key_id)
            if not key:
@@ -1266,19 +1456,20 @@ class Engine:
                    and now - self.fix_last.get(("fix", w, ""), 0) < cooldown]
            if busy:                                    # e.g. auto-fix is starting Ableton right now
                log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, busy[0], cooldown)
-                self.note(key_id, "STARTING\nWAIT 1 MIN", "idle", now, 4)
+                self.note(key_id, "STARTING\n" + self.wait_text, "idle", now, 4)
                return None
        per_deck = self.cfg["fixes"][fix_id].get("kind") == "hyperdeck" and chk and chk.get("conn")
        ident = ("fix", fix_id, chk["conn"] if per_deck else "")   # the same guard a failing hold uses
        if now - self.fix_last.get(ident, 0) < cooldown:
-            log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, fix_id, cooldown)
+            log.info("%s %s: %s ran less than %ds ago, skipped", verb, key_id, fix_id, cooldown)
            cur = self.notes.get(key_id)
            if not (cur and cur["level"] == "fixing" and now < cur["until"]):     # still running: keep saying so
-                self.note(key_id, "JUST DONE\nWAIT 1 MIN", "idle", now, 4)
+                self.note(key_id, "JUST DONE\n" + self.wait_text, "idle", now, 4)
            return None
        self.fix_last[ident] = now
        self.remember()
        fix = self.cfg["fixes"][fix_id]
-        log.info("FORCE %s: %s", key_id, fix.get("label", fix_id))
+        log.info("%s %s: %s", verb, key_id, fix.get("label", fix_id))
        self.note(key_id, "RESTARTING", "fixing", now, 180)
        def work():
@@ -1292,9 +1483,51 @@ class Engine:
                self.note(key_id, "FAILED", "failed", done, 30)
            else:
                self.note(key_id, "DONE", "ok", done, 20)
-        threading.Thread(target=work, daemon=True).start()
+        self.start_fix_thread(work, key_id, fix.get("short", fix_id.upper()))
        return fix_id
    def run_test(self, key_id, fix_id, which, snap, comp, now):
        """The settings page's "Run this fix now": one check's fix, run exactly as a hold on that key
        would run it (deck address, MIDI peer address, recording guard, cooldown), logged as TEST.
        `which` picks the check: its number on the key, or its identity. Refused for anything a hold
        would not do from that key, and for fixes that act on the switcher, a PoE port or Dante."""
        tile = self.tiles.get(key_id)
        if not tile:
            log.warning("TEST %s: no such key", key_id)
            return None
        fix = self.cfg["fixes"].get(fix_id)
        if not fix or fix.get("kind") in ("atem", "netgear_poe") or fix_id in (RESTART_CONNECTION, DANTE_AUTO):
            log.warning("TEST %s: %s is not a fix the page may run", key_id, fix_id)
            return None
        checks = tile.get("checks", [])
        chk = None
        for i, c in enumerate(checks):
            if which in (str(i), check_identity(c), urllib.parse.quote(check_identity(c), safe="")):
                chk = c
        if chk is None and tile.get("hold_always") == fix_id:
            chk = next((c for c in checks if c.get("conn")), {})
        if chk is None:
            log.warning("TEST %s: no check %r on it", key_id, which)
            return None
        own = chk.get("fix")
        own = set(own.values()) if isinstance(own, dict) else {own}
        if fix_id not in own and tile.get("hold_always") != fix_id:
            log.warning("TEST %s: %s is not attached to that check", key_id, fix_id)
            return None
        if chk.get("type") == "hyperdeck":
            if hyperdeck_recording(snap, chk["conn"]):
                log.info("TEST %s: %s is recording, left alone", key_id, chk["conn"])
                self.note(key_id, "RECORDING\nLEFT ALONE", "idle", now, 4)
                return None
            chk = dict(chk, ip=chk.get("ip") or snap.var(chk["conn"], "ip") or self.deck_ips.get(chk["conn"]))
        if fix.get("kind") == "midi_connect":
            address = snap.peer_addr.get((chk.get("host", ""), chk["session"], chk["peer"])) or chk.get("address")
            if not address:
                log.info("TEST %s: no address known for %s", key_id, chk["peer"])
                return None
            chk = dict(chk, address=address)
        return self.run_force(key_id, snap, comp, now, fix_id=fix_id, chk=chk, verb="TEST")
 def display_level(res, acked, service_on):
    if res.level == OFF:
@@ -1373,7 +1606,7 @@ def render(engine, service_on):
        elif down:                                    # flashes: this is the one to hold
            out[kid] = colored(key["label"] + "\nHOLD TO FIX", "err")
        else:
-            out[kid] = colored(key["label"] + "\nHOLD 2s", "idle")
+            out[kid] = colored(key["label"] + "\n" + engine.hold_text, "idle")
    for tile in engine.cfg["tiles"]:
        tid, kind = tile["id"], tile.get("kind")
        note = engine.notes.get(tid)
@@ -1616,13 +1849,18 @@ def playlist_tracks(playlist_uri):
    return tracks, False
 def spotify_playlist(cfg, chk=None):
    """The service playlist: "spotify" in config is the one place it is set; a check may override it."""
    return (chk or {}).get("playlist") or (cfg.get("spotify") or {}).get("playlist")
 def poll_spotify(snap, cfg):
    checks = [c for t in cfg["tiles"] for c in t.get("checks", []) if c["type"] == "spotify"]
    if not checks:
        return
    def refresh():
        try:
-            snap.spotify_tracks, snap.spotify_complete = playlist_tracks(checks[0]["playlist"])
+            snap.spotify_tracks, snap.spotify_complete = playlist_tracks(spotify_playlist(cfg, checks[0]))
        except (OSError, ValueError, KeyError, AttributeError, TypeError) as e:
            log.warning("Spotify playlist list: %s", e)
        snap.spotify_tracks_time = time.time()
@@ -1769,11 +2007,22 @@ def wake_on_lan(mac, broadcast=None, port=9):
    return packet
-def remote_script(cmds):
+def remote_script(cmds, shell="sh"):
-    """One shell line that runs every check and prints SP<n>=<exit code>."""
+    """One line that runs every check and prints SP<n>=<0 or 1>. On a Mac the shell evaluates $?
    after each command; cmd.exe expands %ERRORLEVEL% once before the line runs, so Windows gets a
    conditional echo per command instead (never a bare %ERRORLEVEL%)."""
    if shell == "cmd":
        return " & ".join("((%s) >nul 2>&1 && echo SP%d=0 || echo SP%d=1)" % (cmd, i, i) for i, cmd in enumerate(cmds))
    return "; ".join("(%s) >/dev/null 2>&1; echo SP%d=$?" % (cmd, i) for i, cmd in enumerate(cmds))
 def host_shell(cfg, host, chk=None):
    """Which shell a remote check's command line is written for: "cmd" on a Windows PC, else "sh"."""
    if chk and chk.get("shell"):
        return chk["shell"]
    return "cmd" if cfg.get("hosts", {}).get(host, {}).get("os") == "windows" else "sh"
 def parse_remote(out, n):
    codes = dict(re.findall(r"SP(\d+)=(\d+)", out or ""))
    return [None if str(i) not in codes else codes[str(i)] == "0" for i in range(n)]
@@ -1790,7 +2039,7 @@ def poll_remote(snap, cfg):
    result = dict(snap.remote)
    for host, cmds in by_host.items():
        try:
-            out = subprocess.run(ssh_cmd(cfg, host) + [remote_script(cmds)], capture_output=True,
+            out = subprocess.run(ssh_cmd(cfg, host) + [remote_script(cmds, host_shell(cfg, host))], capture_output=True,
                                 text=True, timeout=20).stdout
        except subprocess.TimeoutExpired:
            out = ""
@@ -1799,41 +2048,6 @@ def poll_remote(snap, cfg):
    snap.remote = result
 def hyperdeck_probe(ip, timeout=3, port=9993):
    """Ask a HyperDeck directly what it is doing, on its control port 9993.
    Returns its transport status ("record", "preview", "stopped", ...), "busy" if it
    refused another client, "answered" if it greeted but did not report a status,
    "hung" if it accepted the connection and said nothing at all, or "unreachable".
    Only "hung" justifies a reboot.
    """
    try:
        s = socket.create_connection((ip, port), timeout=timeout)
    except OSError:
        return "unreachable"
    buf = b""
    with s:
        s.settimeout(timeout)
        try:
            s.sendall(b"transport info\r\n")
            end = time.time() + timeout
            while time.time() < end:
                chunk = s.recv(4096)
                if not chunk:
                    break
                buf += chunk
                m = re.search(rb"208 transport info:.*?status: (\w+)", buf, re.S)
                if m:
                    return m.group(1).decode().lower()
                if b"120 connection rejected" in buf:
                    return "busy"
        except socket.timeout:
            pass
        except OSError:
            return "unreachable" if not buf else "answered"
    return "answered" if buf else "hung"
 # --- The self-updating Dante map: name -> IP -> MAC -> switch port -> PoE --------
 def dante_map_path(data_dir):
@@ -1950,6 +2164,8 @@ def execute_fix(cfg, fix_id, chk):
                ok = conn in ids and comp.restart_connection(ids[conn])
                log.info("FIX %s -> Companion connection %s restarted %s", label, conn, "ok" if ok else "FAILED")
            return r.returncode == 0
        if kind == "spotify":
            return play_spotify(cfg, fix, chk, label)
        if kind == "atem":
            ok = bool(ATEM_WATCHER) and ATEM_WATCHER.fix(fix["part"])
            log.info("FIX %s -> %s", label, "switcher matches the restore file again" if ok else "FAILED")
@@ -1990,6 +2206,27 @@ def execute_fix(cfg, fix_id, chk):
    return False
 def play_spotify(cfg, fix, chk, label, run=None):
    """Play the service playlist in Spotify on this Mac, opening Spotify first if it is closed."""
    run = run or subprocess.run
    sp_ = cfg.get("spotify") or {}
    playlist = spotify_playlist(cfg, chk) or fix.get("playlist")
    if not playlist:
        log.warning("FIX %s: no playlist set", label)
        return False
    if run(["/usr/bin/pgrep", "-xq", "Spotify"], timeout=5).returncode != 0:
        run(["/usr/bin/open", "-g", "-a", "Spotify"], timeout=15)
        time.sleep(sp_.get("open_wait", 8))
    run(["/usr/bin/osascript", "-e", 'tell application "Spotify" to play track "%s"' % playlist],
        capture_output=True, text=True, timeout=15)
    time.sleep(sp_.get("play_wait", 3))
    r = run(["/usr/bin/osascript", "-e", 'tell application "Spotify" to player state as text'],
            capture_output=True, text=True, timeout=10)
    ok = r.returncode == 0 and r.stdout.strip() == "playing"
    log.info("FIX %s -> %s", label, "playing" if ok else "not playing (%s)" % (r.stdout.strip() or r.stderr.strip()[-80:]))
    return ok
 def reconnect_and_verify(cfg, fix, label, sleep=None, conn=None, host=None, toggle=False):
    """Restart one Companion connection (toggle=True: switch it off and on again), then look again
    after "verify_seconds". Returns True if it came back; otherwise what to tell the operator:
@@ -2006,8 +2243,11 @@ def reconnect_and_verify(cfg, fix, label, sleep=None, conn=None, host=None, togg
        if conn.get("enabled") and not comp.disable_connection(conn["id"]):
            log.info("FIX %s -> Companion refused to switch %s off", label, fix["conn"])
            return False
        try:
            sleep(fix.get("off_seconds", 2))
-        if not comp.enable_connection(conn["id"]):
+        finally:                                   # whatever happens, it is switched back on
            on = comp.enable_connection(conn["id"])
        if not on:
            log.info("FIX %s -> Companion did not switch %s back on", label, fix["conn"])
            return "STILL OFF IN\nCOMPANION"
        log.info("FIX %s -> %s switched off and on", label, fix["conn"])
@@ -2058,10 +2298,14 @@ CCU_PARAMS = {
 }
-def atem_reference(xml_path):
+def atem_reference(xml_path, compare=None):
    """The restore file (.xml from ATEM Software Control) as what atem_watch.js compares against:
-    every Fairlight source's mix option and fader, and every camera's settings above."""
+    every Fairlight source's mix option and fader, and every camera's settings above. `compare`
    ("compare" in config "atem") narrows it: {"audio": ["mixOption", "faderGain"], "camera": [...]}."""
    import xml.etree.ElementTree as ET
    compare = compare or {}
    audio_fields = compare.get("audio", ["mixOption", "faderGain"])
    camera_params = compare.get("camera")                 # parameter names, e.g. ["ApertureFstop", ...]
    root = ET.parse(xml_path).getroot()
    audio = {}
    mixer = root.find("FairlightAudioMixer")
@@ -2074,13 +2318,16 @@ def atem_reference(xml_path):
                    gain = float(src.attrib.get("faderGain", "nan"))
                except ValueError:
                    gain = float("nan")
-                if math.isfinite(gain):                     # the file saves dB; the switcher uses 1/100 dB
+                if math.isfinite(gain) and "faderGain" in audio_fields:   # the file saves dB; the switcher 1/100 dB
                    want["faderGain"] = int(round(gain * 100))
                if "mixOption" in audio_fields or "faderGain" in want:
                    audio["%s/%s" % (inp.attrib["id"], src.attrib["id"])] = want
    camera = {}
    cc = root.find("CameraControl")
    for p in (cc if cc is not None else []):
        spec = CCU_PARAMS.get((p.attrib.get("category"), p.attrib.get("parameter")))
        if camera_params is not None and p.attrib.get("parameter") not in camera_params:
            continue
        if spec and all(f in p.attrib for f in spec[2]):
            camera["%s/%d/%d" % (p.attrib["device"], spec[0], spec[1])] = [float(p.attrib[f]) for f in spec[2]]
    return {"audio": audio, "camera": camera}
@@ -2100,7 +2347,7 @@ class AtemWatcher:
    def __init__(self, cfg, snap, data_dir, stop):
        a = cfg["atem"]
-        self.cfg, self.snap, self.stop = a, snap, stop
+        self.atem, self.snap, self.stop = a, snap, stop
        self.proc, self.lock, self.fixed = None, threading.Lock(), {}
        here = os.path.dirname(os.path.abspath(__file__))
        self.script = os.path.join(here, "atem_watch.js")
@@ -2112,12 +2359,12 @@ class AtemWatcher:
        if not os.path.isabs(xml):
            xml = os.path.join(here, xml)
        with open(self.ref_path, "w") as f:
-            json.dump(atem_reference(xml), f)
+            json.dump(atem_reference(xml, a.get("compare")), f)
    def run(self):
        while not self.stop.is_set():
            try:
-                self.proc = subprocess.Popen([self.node, self.script, self.cfg["host"], self.library, self.ref_path],
+                self.proc = subprocess.Popen([self.node, self.script, self.atem["host"], self.library, self.ref_path],
                                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                                             text=True, bufsize=1)
                errors = deque(maxlen=5)                # read all the time, so a chatty stderr never stalls it
@@ -2231,9 +2478,53 @@ class Periodic(threading.Thread):
 # --- Main --------------------------------------------------------------------
 def check_windows(windows, where):
    for w in windows or []:
        if not isinstance(w, dict) or not isinstance(w.get("days"), list) or not w["days"]:
            raise ValueError("%s: a window needs days, start and end" % where)
        for d in w["days"]:
            if d not in DAYS:
                raise ValueError("%s: unknown day %r" % (where, d))
        for k in ("start", "end"):
            if not re.fullmatch(r"([01]\d|2[0-3]):[0-5]\d", str(w.get(k, ""))):
                raise ValueError("%s: %s must be HH:MM" % (where, k))
 def check_host(cfg, host, where):
    """A machine the watchdog logs in to: listed in "hosts", with an ssh address, and not this Mac."""
    h = cfg.get("hosts", {}).get(host)
    if h is None:
        raise ValueError("%s: unknown host %s" % (where, host))
    if h.get("self"):
        raise ValueError("%s: %s is this Mac - use a local fix or check" % (where, host))
    if not h.get("ssh"):
        raise ValueError("%s: host %s has no ssh address" % (where, host))
 def exit_timeout(cfg):
    """How long launchd must let the watchdog finish a running fix before killing it: the longest
    fix plus its verify time plus a margin, never under a minute (install.sh writes it into the plist)."""
    longest = 0
    for fix in cfg.get("fixes", {}).values():
        longest = max(longest, fix.get("timeout", 60) + (fix.get("verify_seconds", 20)
                                                          if fix.get("kind") in ("reconnect", "hyperdeck") else 0))
    return max(60, longest + 10)
 def load_config(path):
    with open(path) as f:
        cfg = json.load(f)
    for fid, fix in cfg.get("fixes", {}).items():
        kind = fix.get("kind", "ssh")
        if kind not in FIX_KINDS:
            raise ValueError("fix %s: unknown kind %s" % (fid, kind))
        if kind == "ssh":
            check_host(cfg, fix.get("host"), "fix " + fid)
        if kind in ("ssh", "local") and not fix.get("run"):
            raise ValueError("fix %s: nothing to run" % fid)
    check_windows(cfg.get("service_windows"), "service_windows")
    check_windows((cfg.get("auto_fix") or {}).get("windows"), "auto_fix")
    check_windows((cfg.get("site") or {}).get("live_services"), "site.live_services")
    ids = set()
    for t in cfg["tiles"]:
        if t["id"] in ids:
@@ -2254,13 +2545,15 @@ def load_config(path):
                if (f and f not in (RESTART_CONNECTION, DANTE_AUTO) and cfg["fixes"][f].get("kind", "ssh") == "ssh"
                        and cfg["fixes"][f]["host"] not in cfg.get("hosts", {})):
                    raise ValueError("fix %s: unknown host" % f)
-            if c["type"] == "remote" and c["host"] not in cfg.get("hosts", {}):
+            if c["type"] == "remote" or (c["type"] == "midi_peer" and c.get("host")):
-                raise ValueError("tile %s: unknown host %s" % (t["id"], c["host"]))
+                check_host(cfg, c["host"], "tile " + t["id"])
        if t.get("hold_always") and t["hold_always"] not in cfg.get("fixes", {}):
            raise ValueError("tile %s: unknown hold_always fix %s" % (t["id"], t["hold_always"]))
    for f in (cfg.get("auto_fix") or {}).get("fixes", []):
        if f not in cfg.get("fixes", {}):
            raise ValueError("auto_fix: unknown fix %s" % f)
        if cfg["fixes"][f].get("never_auto"):
            raise ValueError("auto_fix: %s is marked never to run by itself" % f)
        if re.search(r"pkill|taskkill|killall", cfg["fixes"][f].get("run", "")) and not cfg["fixes"][f].get("restart_ok"):
            raise ValueError("auto_fix: %s closes a running program - that stays a hold" % f)
    for k in cfg.get("restart_page", {}).get("keys", []):
@@ -2372,22 +2665,57 @@ def handle_commands(text, engine, comp, snap, now):
                engine.run_fixes(arg, snap, comp, now)
            else:                                    # a Restart page key, held 2 seconds
                engine.run_force(arg, snap, comp, now)
        elif verb == "test":
            # The settings page: "test <key> <fix> <check number or identity> <unix time>".
            if len(words) < 5 or not words[4].isdigit() or now - int(words[4]) > HOLD_MAX_AGE:
                log.info("TEST %s: no time on it, or too long ago - ignored", arg)
            else:
                engine.run_test(arg, words[2], words[3], snap, comp, now)
        elif verb == "resurvey":                     # the settings page: look at the Dante switches now
            snap.dante_map_time = 0.0
            log.info("RESURVEY: Dante map survey requested")
        else:
            log.warning("unknown command %r", part)
-def run(cfg, data_dir, dry_run=False, stop=None):
+def run(cfg, data_dir, dry_run=False, stop=None, config_sha=None):
    stop = stop or threading.Event()
    names = cfg["variables"]
    comp = make_companion(cfg)
    snap = Snapshot()
-    engine = Engine(cfg, os.path.join(data_dir, "service-mode.json"))
+    engine = Engine(cfg, os.path.join(data_dir, "state.json"), data_dir)
    snap.dante_map = load_dante_map(dante_map_path(data_dir))
    site = cfg.get("site", {})
    prefix = site.get("alert_prefix") or site.get("ingest_name") or "Status panel"
    alert_after = cfg.get("timing", {}).get("alert_after_seconds", 30)
    def on_term(signum, frame):
        log.info("stop requested: %d fix(es) running will finish first", len(engine.running))
        engine.stopping = True
        stop.set()
    try:
        signal.signal(signal.SIGTERM, on_term)
    except ValueError:                                # not the main thread (tests)
        pass
    threads = start_pollers(cfg, snap, stop, data_dir)
    if not dry_run:
        try:
            # A hold pressed while the watchdog was down must not fire on a config it was not meant for.
            queued = comp.get_custom(names["command"])
            if queued and queued.strip():
                comp.set_custom(names["command"], "")
                log.info("dropped commands queued before the start: %s", queued.strip())
        except (OSError, http.client.HTTPException) as e:
            log.warning("could not read the command variable at start: %s", e)
    stop.wait(2.5)  # let the first polls land before judging anything
    last_json, last_write, last_eval = None, 0.0, 0.0
    down_since, alerted, warned_missing = None, False, set()
-    log.info("status panel running (%s)", "dry run" if dry_run else "live")
+    log.info("status panel running (code %s, %s)", VERSION, "dry run" if dry_run else "live")
    try:
        write_json(os.path.join(data_dir, "started.json"),
                   {"pid": os.getpid(), "started_at": time.time(), "config_sha": config_sha, "version": VERSION})
    except OSError as e:
        log.warning("started.json: %s", e)
    while not stop.is_set():
        now = time.time()
        try:
@@ -2428,18 +2756,27 @@ def run(cfg, data_dir, dry_run=False, stop=None):
            if down_since is not None:
                log.info("Companion reachable again")
                if alerted:
-                    post_webhook(cfg.get("alert_webhook"), "NA Ingest: Companion is answering again.")
+                    post_webhook(cfg.get("alert_webhook"), "%s: Companion is answering again." % prefix)
                down_since, alerted = None, False
        except (OSError, http.client.HTTPException) as e:
            if down_since is None:
                down_since = now
                log.warning("Companion unreachable: %s", e)
-            elif not alerted and now - down_since > 30:
+            elif not alerted and now - down_since > alert_after:
-                post_webhook(cfg.get("alert_webhook"), "NA Ingest: Companion has not answered for 30 seconds.")
+                post_webhook(cfg.get("alert_webhook"), "%s: Companion has not answered for %d seconds."
                             % (prefix, alert_after))
                alerted = True
        stop.wait(0.25)
    # A fix that is running finishes before the process goes (launchd waits exit_timeout for us).
    budget = time.time() + max(5, exit_timeout(cfg) - 5)
    for t in engine.fix_threads:
        t.join(timeout=max(0.1, budget - time.time()))
    if any(t.is_alive() for t in engine.fix_threads):
        log.warning("stopping with a fix still running")
    else:
        log.info("stopped")
    for t in threads:
-        t.join(timeout=10)
+        t.join(timeout=2)
 def run_once(cfg):
@@ -2483,8 +2820,16 @@ def main(argv=None):
    ap.add_argument("--once", action="store_true", help="poll once, print the tiles, write nothing")
    ap.add_argument("--dry-run", action="store_true", help="run but never write to Companion")
    ap.add_argument("--log", default=os.path.expanduser("~/Library/Logs/status-panel.log"))
    ap.add_argument("--check", action="store_true", help="load and validate the config, then exit")
    ap.add_argument("--exit-timeout", action="store_true", help="print the launchd exit timeout this config needs")
    args = ap.parse_args(argv)
    cfg = load_config(args.config)
    if args.check:
        print("config OK: %d keys, %d fixes" % (len(cfg["tiles"]), len(cfg.get("fixes", {}))))
        return
    if args.exit_timeout:
        print(exit_timeout(cfg))
        return
    if args.once:
        run_once(cfg)
        return
@@ -2494,7 +2839,11 @@ def main(argv=None):
    if sys.stderr.isatty():
        log.addHandler(logging.StreamHandler())
    log.setLevel(logging.INFO)
-    run(cfg, args.data_dir, dry_run=args.dry_run)
+    if not (3, 9) <= sys.version_info[:2] <= (3, 12):
        log.warning("python %s: the Dante switch reader (telnetlib) needs 3.9-3.12", sys.version.split()[0])
    with open(args.config, "rb") as f:
        config_sha = hashlib.sha256(f.read()).hexdigest()
    run(cfg, args.data_dir, dry_run=args.dry_run, config_sha=config_sha)
 if __name__ == "__main__":
@@ -0,0 +1,296 @@
 """schema.py must describe exactly what the code reads: every config key the watchdog and the page
 tools read is declared, every default in the code equals the schema's, every check type and fix
 kind is listed, every word a key can show is in the glossary, and today's config still builds
 today's pages."""
 import ast
 import gzip
 import json
 import os
 import re
 import sys
 import unittest
 ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.insert(0, ROOT)
 import make_page  # noqa: E402
 import schema  # noqa: E402
 import status_panel as sp  # noqa: E402
 WALKED = ["status_panel.py", "make_page.py", "push_page.py", "nav_keys.py", "netgear.py"]
 # What a variable holds, by name, when nothing in the function says otherwise (parameters mostly).
 NAMES = {"cfg": "top", "config": "top", "chk": "check", "c": "check", "fix": "fix", "f": "fix", "t": "tile",
         "tile": "tile", "k": "restart_key", "key": "restart_key", "rp": "restart_page", "w": "window",
         "names": "variables", "host": "host", "h": "host", "windows": "windows", "site": "site"}
 ATTRS = {"self.cfg": "top", "engine.cfg": "top", "self.net": "network", "self.auto": "auto", "self.site": "site",
         "self.atem": "atem",
         "self.tiles": "tiles", "engine.tiles": "tiles", "self.restart_keys": "restart_keys",
         "engine.restart_keys": "restart_keys"}
 # A key of a top-level dict that holds a record of another kind.
 TOP = {"companion": "companion", "network": "network", "variables": "variables", "auto_fix": "auto", "site": "site",
       "atem": "atem", "spotify": "spotify", "timing": "timing", "flap": "flap", "hosts": "hosts", "fixes": "fixes",
       "tiles": "tiles", "restart_page": "restart_page", "gear": "gears", "service_windows": "windows"}
 SUB = {("auto", "windows"): "windows", ("site", "live_services"): "windows", ("tile", "checks"): "checks",
       ("restart_page", "keys"): "restart_keys", ("tile", "members"): "names"}
 ELEMENT = {"tiles": "tile", "checks": "check", "restart_keys": "restart_key", "windows": "window", "fixes": "fix",
           "hosts": "host", "gears": "gear"}
 # Keys the code reads that are the watchdog's own working data, not settings.
 NOT_SETTINGS = {("check", "ip")}
 def dotted(node):
    if isinstance(node, ast.Name):
        return node.id
    if isinstance(node, ast.Attribute):
        base = dotted(node.value)
        return base + "." + node.attr if base else None
    return None
 def collect(path):
    """[(kind, key, default node or None, line)] for every config read in one file."""
    tree = ast.parse(open(path, encoding="utf-8").read())
    reads = []
    def kind_of(node, local):
        if isinstance(node, ast.Name):
            return local.get(node.id, NAMES.get(node.id))
        if isinstance(node, ast.Attribute):
            return ATTRS.get(dotted(node))
        if isinstance(node, ast.Subscript):
            base = kind_of(node.value, local)
            key = node.slice.value if isinstance(node.slice, ast.Constant) else None
            return step(base, key)
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "get" and node.args:
            base = kind_of(node.func.value, local)
            key = node.args[0].value if isinstance(node.args[0], ast.Constant) else None
            return step(base, key)
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in ("items", "values"):
            return kind_of(node.func.value, local)
        if isinstance(node, ast.Call) and dotted(node.func) == "json.load":
            return "jsonfile"                             # a config file when it lands in cfg / config
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in ("enumerate", "dict", "sorted", "list") and node.args:
            return kind_of(node.args[0], local)
        if isinstance(node, ast.BoolOp):
            return kind_of(node.values[0], local)
        if isinstance(node, ast.IfExp):
            return kind_of(node.body, local)
        return None
    def step(base, key):
        if base is None:
            return None
        if key is None:                                   # dict[variable]: an element
            return ELEMENT.get(base)
        if base == "top":
            return TOP.get(key)
        return SUB.get((base, key))
    def scan(body, local):
        for node in ast.walk(ast.Module(body=body, type_ignores=[])):
            if isinstance(node, ast.FunctionDef):
                continue
            if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
                kind = kind_of(node.value, local)
                if kind == "jsonfile":
                    kind = "top" if node.targets[0].id in ("cfg", "config") else None
                local[node.targets[0].id] = kind
            elif isinstance(node, (ast.For, ast.comprehension)):
                elem = ELEMENT.get(kind_of(node.iter, local))
                if isinstance(node.target, ast.Name):
                    local[node.target.id] = elem
                elif isinstance(node.target, ast.Tuple) and len(node.target.elts) == 2 \
                        and isinstance(node.target.elts[1], ast.Name):
                    local[node.target.elts[1].id] = elem
        for node in ast.walk(ast.Module(body=body, type_ignores=[])):
            if isinstance(node, ast.FunctionDef):
                continue
            if isinstance(node, ast.Subscript) and isinstance(node.slice, ast.Constant) and isinstance(node.slice.value, str):
                kind = kind_of(node.value, local)
                if kind:
                    reads.append((kind, node.slice.value, None, node.lineno))
            elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr == "get" \
                    and node.args and isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str):
                kind = kind_of(node.func.value, local)
                if kind:
                    reads.append((kind, node.args[0].value, node.args[1] if len(node.args) > 1 else None, node.lineno))
    def functions(node):
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                local = {a.arg: NAMES.get(a.arg) for a in child.args.args + child.args.kwonlyargs}
                # a parameter shadows the name map only when the map knows it; "page", "kind" etc. stay unknown
                scan(child.body, {k: v for k, v in local.items() if v})
                functions(child)
            elif isinstance(child, ast.ClassDef):
                functions(child)
    scan([n for n in tree.body if not isinstance(n, (ast.FunctionDef, ast.ClassDef))], {})
    functions(tree)
    return reads
 def all_reads():
    out = []
    for name in WALKED:
        out += [(name,) + r for r in collect(os.path.join(ROOT, name))]
    return out
 def path_of(kind, key):
    prefix = {"top": "", "check": "tiles[].checks[].", "tile": "tiles[].", "fix": "fixes.*.", "host": "hosts.*.",
              "restart_key": "restart_page.keys[].", "restart_page": "restart_page.", "auto": "auto_fix.",
              "network": "network.", "variables": "variables.", "site": "site.", "atem": "atem.", "spotify": "spotify.",
              "companion": "companion.", "timing": "timing.", "flap": "flap.", "gear": "gear.*.",
              "window": "auto_fix.windows[]."}.get(kind)
    return None if prefix is None else prefix + key
 class TestSchemaMatchesCode(unittest.TestCase):
    def test_every_config_read_is_declared(self):
        record = schema.key_record()
        missing = sorted({(kind, key) for _, kind, key, _, _ in all_reads()
                          if kind in record and key not in record[kind] and (kind, key) not in NOT_SETTINGS})
        self.assertEqual(missing, [], "config keys the code reads that schema.py does not declare")
        kinds = {kind for _, kind, _, _, _ in all_reads()}
        self.assertNotIn(None, kinds)
        unknown = sorted(k for k in kinds if k not in record and k not in ("names", "windows", "checks", "tiles",
                                                                             "fixes", "hosts", "restart_keys", "gears"))
        self.assertEqual(unknown, [])
    def test_defaults_in_code_match_the_schema(self):
        defaults = schema.defaults()
        wrong = []
        for name, kind, key, default, line in all_reads():
            if default is None or not isinstance(default, ast.Constant):
                continue
            path = path_of(kind, key)
            if path in defaults and defaults[path] != default.value:
                wrong.append("%s:%d %s is %r in the code, %r in the schema" % (name, line, path, default.value, defaults[path]))
        self.assertEqual(wrong, [])
    def test_every_declared_key_is_read_somewhere(self):
        """Dead settings are caught; keys only the settings page uses are listed here."""
        ui_only = {"site.id", "site.name", "site.ingest_hostname", "site.ingest_ips", "site.contacts", "site.editing_code",
                   "site.settings_port", "site.dvs_helper", "site.solo_dir", "hosts.*.name", "hosts.*.role",
                   "hosts.*.notes", "hosts.*.pingable", "hosts.*.runs", "hosts.*.networks", "hosts.*.manual_done",
                   "gear", "gear.*.location", "gear.*.notes", "atem.source_folder", "fixes.*.template", "fixes.*.params",
                   "tiles[].checks[].template", "tiles[].checks[].params", "spotify.name", "site", "timing",
                   "fixes", "hosts", "tiles", "flap", "auto_fix", "network", "variables", "companion", "atem", "spotify",
                   "restart_page", "service_windows", "restart_page.keys", "tiles[].checks", "auto_fix.windows",
                   "site.live_services", "tiles[].checks[].type", "fixes.*.playlist", "tiles[].checks[].playlist"}
        read = {path_of(kind, key) for _, kind, key, _, _ in all_reads()}
        read |= {"service_windows[].days", "service_windows[].start", "service_windows[].end",
                 "site.live_services[].days", "site.live_services[].start", "site.live_services[].end"}
        for w in ("days", "start", "end"):
            if "auto_fix.windows[]." + w in read:
                read |= {"service_windows[].%s" % w, "site.live_services[].%s" % w}
        dead = sorted(p for p in schema.F if p not in read and p not in ui_only)
        self.assertEqual(dead, [], "declared settings nothing reads")
    def test_check_types_fix_kinds_and_subs(self):
        self.assertEqual(set(schema.CHECKS), set(sp.CHECKS))
        self.assertEqual(tuple(schema.FIX_KINDS), sp.FIX_KINDS)
        for t, spec in schema.CHECKS.items():
            for f in spec["fields"] + spec["identity"]:
                self.assertIn("tiles[].checks[]." + f, schema.F, "%s field %s" % (t, f))
        for k in sp.FIX_KINDS:
            self.assertIn(k, schema.FIX_KIND_FIELDS)
    def test_glossary_covers_every_word_a_key_can_show(self):
        words = set()
        for group in schema.GLOSSARY.values():
            for phrase in group:
                words |= set(phrase.split())
        tree = ast.parse(open(os.path.join(ROOT, "status_panel.py"), encoding="utf-8").read())
        texts = []
        for node in ast.walk(tree):
            if isinstance(node, ast.Call):
                name = node.func.id if isinstance(node.func, ast.Name) else (node.func.attr if isinstance(node.func, ast.Attribute) else "")
                if name == "Result" and len(node.args) > 1 and isinstance(node.args[1], ast.Constant):
                    texts.append(node.args[1].value)
                elif name == "note" and len(node.args) > 1 and isinstance(node.args[1], ast.Constant):
                    texts.append(node.args[1].value)
                elif name == "note" and len(node.args) > 1 and isinstance(node.args[1], ast.BinOp):
                    for part in ast.walk(node.args[1]):
                        if isinstance(part, ast.Constant) and isinstance(part.value, str):
                            texts.append(part.value)
        unknown = set()
        for text in texts:
            for word in re.findall(r"[A-Z][A-Z-]+", text):
                if word not in words:
                    unknown.add(word)
        self.assertEqual(sorted(unknown), [], "words a key can show that the glossary does not explain")
    def test_todays_config_builds_todays_pages(self):
        cfg = sp.load_config(os.path.join(ROOT, "config.json"))
        def strip(export):
            for row in export["page"]["controls"].values():
                for ctl in row.values():
                    if ctl and "options" in ctl:
                        ctl["options"].pop("notes", None)
            return export
        for build, name in ((make_page.build, "status-page.companionconfig"), (make_page.build_restart, "restart-page.companionconfig")):
            on_disk = json.loads(gzip.open(os.path.join(ROOT, name)).read())
            self.assertEqual(strip(build(cfg)), strip(on_disk), name)
        self.assertNotIn("restart_page", {k for k in cfg if k == "nothing"})
        bare = dict(cfg)
        bare.pop("restart_page")
        export = make_page.build(bare)
        notes = [((ctl.get("options") or {}).get("notes") or "") for row in export["page"]["controls"].values()
                 for ctl in row.values() if ctl]
        self.assertFalse(any("STATUS jump key" in n for n in notes))    # no Restart page: no jump key
    def test_todays_config_passes_the_rules(self):
        cfg = sp.load_config(os.path.join(ROOT, "config.json"))
        errors, warnings = schema.validate(cfg)
        self.assertEqual(errors, [])
        self.assertTrue(all("only during a service" in w for w in warnings), warnings)
    def test_rules_catch_the_classic_mistakes(self):
        cfg = sp.load_config(os.path.join(ROOT, "config.json"))
        bad = json.loads(json.dumps(cfg))
        bad["tiles"][7]["pos"] = bad["tiles"][8]["pos"]                   # two keys on one cell
        bad["tiles"].append({"id": "all", "label": "X", "pos": [3, 8], "kind": "master"})
        bad["fixes"]["vista"]["host"] = "nowhere"
        bad["restart_page"]["keys"][0]["fix"] = "no-such-fix"
        bad["hosts"]["bibletv"]["mac"] = "not-a-mac"
        bad["auto_fix"]["fixes"].append("propresenter")                   # closes a program
        bad["fixes"]["ableton"]["never_auto"] = True                      # ...and ableton is in the auto list
        errors, _ = schema.validate(bad)
        text = "\n".join(errors)
        for expect in ("share row", "two keys with the same id", "exactly one ALL key", "unknown computer nowhere",
                       "unknown fix no-such-fix", "not a hardware", "closes a running program", "never to run by itself"):
            self.assertIn(expect, text)
        home = json.loads(json.dumps(cfg))
        home["hosts"]["ingest"] = {"self": True, "name": "NA Ingest"}
        home["fixes"]["vista"]["host"] = "ingest"
        self.assertIn("is this Mac", "\n".join(schema.validate(home)[0]))
    def test_live_warnings(self):
        cfg = sp.load_config(os.path.join(ROOT, "config.json"))
        live = {"connections": {"HyperDeck_2": "generic-tcp-udp"}}
        _, warnings = schema.validate(cfg, live)
        self.assertTrue(any("HyperDeck_2 uses module generic-tcp-udp" in w for w in warnings), warnings)
        self.assertTrue(any("no connection called HyperDeck_1" in w for w in warnings))
    def test_secrets_are_masked(self):
        masked = schema.mask_secrets({"alert_webhook": "https://hooks/abc", "site": {"editing_code": "apple bee", "name": "NA"},
                                      "fixes": {"x": {"run": "ls"}}})
        self.assertEqual(masked["alert_webhook"], "set (17 characters)")
        self.assertEqual(masked["site"]["editing_code"], "set (9 characters)")
        self.assertEqual(masked["site"]["name"], "NA")
        self.assertEqual(masked["fixes"]["x"]["run"], "ls")
    def test_phrases(self):
        cfg = sp.load_config(os.path.join(ROOT, "config.json"))
        p = schema.phrases(cfg)
        self.assertEqual(p["auto_fix"], "may run by itself on Sundays 06:30-13:30")
        self.assertEqual(p["live"], "changes wait while a HyperDeck is recording")
        self.assertEqual(p["hold"], "HOLD 2s")
        self.assertEqual(sp.windows_phrase([{"days": ["sat", "sun"], "start": "18:00", "end": "00:30"}]),
                         "on Saturdays and Sundays 18:00-00:30")
 if __name__ == "__main__":
    unittest.main()
@@ -130,7 +130,8 @@ class TestChecks(unittest.TestCase):
        s.spotify_recent.append("spotify:track:q")
        self.assertEqual(msg().msg, "WRONG PLAYLIST")                     # full list: one unknown is enough
        fix = CONFIG["fixes"][chk["fix"]]
-        self.assertIn(chk["playlist"], fix["run"])
+        self.assertEqual(fix["kind"], "spotify")
        self.assertEqual(sp.spotify_playlist(CONFIG, chk), CONFIG["spotify"]["playlist"])
        self.assertIn(chk["fix"], CONFIG["auto_fix"]["fixes"])
    def test_connection_states(self):
@@ -1352,93 +1353,145 @@ class TestFixes(unittest.TestCase):
        self.assertEqual(sp.parse_remote("SP0=0\nSP1=1\n", 2), [True, False])
        self.assertEqual(sp.parse_remote("", 2), [None, None])
    def test_windows_remote_checks_use_conditional_echoes(self):
        line = sp.remote_script(["tasklist | find /I \"Vista.exe\"", "sc query rtpMIDIService | find \"RUNNING\""], "cmd")
        self.assertNotIn("%ERRORLEVEL%", line)
        self.assertIn("&& echo SP0=0 || echo SP0=1", line)
        self.assertIn("&& echo SP1=0 || echo SP1=1", line)
        self.assertEqual(sp.parse_remote("SP0=0\r\nSP1=1\r\n", 2), [True, False])
        self.assertEqual(sp.host_shell(CONFIG, "lighting"), "cmd")
        self.assertEqual(sp.host_shell(CONFIG, "tracks"), "sh")
    def test_overnight_windows(self):
        sat_2350 = datetime(2026, 10, 10, 23, 50).timestamp()
        sun_0020 = datetime(2026, 10, 11, 0, 20).timestamp()
        sun_0100 = datetime(2026, 10, 11, 1, 0).timestamp()
        w = [{"days": ["sat"], "start": "18:00", "end": "00:30"}]
        self.assertTrue(sp.in_windows(w, sat_2350))
        self.assertTrue(sp.in_windows(w, sun_0020))                          # past midnight, still Saturday's service
        self.assertFalse(sp.in_windows(w, sun_0100))
        self.assertFalse(sp.in_windows([{"days": ["sun"], "start": "18:00", "end": "00:30"}], sat_2350))
    def test_check_identity_is_about_the_thing_not_the_position(self):
        midi = [c for c in CONFIG["tiles"][9]["checks"] if c["type"] == "midi_peer"]
        self.assertEqual(len({sp.check_identity(c) for c in midi}), len(midi))      # six peers, six identities
        self.assertEqual(sp.check_identity({"type": "midi_peer", "host": "tracks", "session": "NA Tracks", "peer": "Lighting PC"}),
                         "tracks/NA Tracks/Lighting PC")
        self.assertEqual(sp.check_identity({"type": "ping", "conn": "Axient_1", "host": "10.1.1.101"}), "Axient_1")
        self.assertEqual(sp.check_identity({"type": "conn", "conn": "ATEM"}), "ATEM")
    def test_acknowledgements_survive_a_restart(self):
        state = os.path.join(tempfile.mkdtemp(prefix="sp-state-"), "state.json")
        engine = sp.Engine(CONFIG, state)
        engine.executor = engine.poe_executor = lambda *a: True
        snap = healthy_snapshot(SUNDAY)
        snap.conns["ATEM"]["category"] = "error"
        run_steps(engine, snap, SUNDAY, 5)
        engine.ack("atem")
        self.assertEqual(level(run_steps(engine, snap, SUNDAY + 6, 1)[0]["atem"]), "ack")
        saved = json.load(open(state))
        self.assertIn("atem|ATEM|conn", saved["acks"])
        again = sp.Engine(CONFIG, state)                                     # the watchdog restarts
        again.executor = again.poe_executor = lambda *a: True
        tiles, _ = run_steps(again, snap, SUNDAY + 10, 5)
        self.assertEqual(level(tiles["atem"]), "ack")                        # still acknowledged, no new flashing
        snap.conns["ATEM"]["category"] = "good"
        run_steps(again, snap, SUNDAY + 20, 2)
        self.assertEqual(json.load(open(state))["acks"], [])                 # cleared: forgotten
    def test_run_this_fix_now_goes_through_the_hold_path(self):
        ran = []
        self.engine.executor = lambda cfg, fix_id, chk: ran.append((fix_id, chk.get("conn"), chk.get("ip"), chk.get("peer"))) or True
        snap = self.snap
        run_steps(self.engine, snap, SUNDAY, 2)
        comp = mock_comp()
        now = SUNDAY + 3
        self.engine.run_test("hd2", "hyperdeck", "0", snap, comp, now)        # the check's number...
        self.engine.run_test("midi", "midi_reconnect", "tracks/NA Tracks/Studio B Mac Studio", snap, comp, now)  # ...or its identity
        self.engine.run_test("midi", "midi_reconnect", "9", snap, comp, now)   # no such check
        self.engine.run_test("atem", "atem_audio", "1", snap, comp, now)       # never from the page
        self.engine.run_test("hd2", "propresenter", "0", snap, comp, now)      # not attached to that key
        for t in self.engine.fix_threads:
            t.join(2)
        self.assertEqual(ran, [("hyperdeck", "HyperDeck_2", "10.1.10.12", None),
                               ("midi_reconnect", None, None, "Studio B Mac Studio")])
        self.assertEqual(self.engine.notes["hd2"]["text"], "DONE")
    def test_a_restart_lets_the_running_fix_finish_and_starts_nothing_new(self):
        gate = threading.Event()
        self.engine.executor = lambda cfg, fix_id, chk: gate.wait(3) or True
        run_steps(self.engine, self.snap, SUNDAY, 2)
        self.engine.run_force("rs_bridge", self.snap, mock_comp(), SUNDAY + 3)
        self.assertEqual(len(self.engine.running), 1)                        # published for the settings page
        self.engine.stopping = True                                          # SIGTERM arrived
        self.engine.run_force("rs_captions", self.snap, mock_comp(), SUNDAY + 4)
        self.assertEqual(self.engine.notes["rs_captions"]["text"], "RESTARTING\nSOON")
        self.assertEqual(self.engine.run_fixes("ingest", self.snap, mock_comp(), SUNDAY + 4), [])
        gate.set()
        for t in self.engine.fix_threads:
            t.join(2)
        self.assertEqual(self.engine.running, {})
        self.assertEqual(sp.exit_timeout(CONFIG), 90)                        # the longest fix + its verify + 10
    def test_a_hyperdeck_switched_off_is_always_switched_back_on(self):
        calls = []
        class Comp:
            def connections(self):
                return [{"label": "HyperDeck_2", "id": "x", "enabled": True, "status": {"category": "good"}}]
            def disable_connection(self, cid):
                calls.append("off")
                return True
            def enable_connection(self, cid):
                calls.append("on")
                return True
        import unittest.mock as mock
        def boom(seconds):
            raise RuntimeError("killed mid-way")
        with mock.patch.object(sp, "Companion", lambda *a, **k: Comp()):
            with self.assertRaises(RuntimeError):
                sp.reconnect_and_verify(CONFIG, CONFIG["fixes"]["hyperdeck"], "x", sleep=boom, conn="HyperDeck_2", toggle=True)
        self.assertEqual(calls, ["off", "on"])
    def test_spotify_fix_opens_and_plays(self):
        calls = []
        class R:
            def __init__(self, code, out=""):
                self.returncode, self.stdout, self.stderr = code, out, ""
        def run(argv, **kw):
            calls.append(argv)
            if argv[1] == "-xq":
                return R(1)                                                  # Spotify is closed
            if "player state" in argv[-1]:
                return R(0, "playing\n")
            return R(0)
        import unittest.mock as mock
        with mock.patch.object(sp.time, "sleep", lambda s: None):
            ok = sp.play_spotify(CONFIG, CONFIG["fixes"]["spotify_play"], {}, "spotify", run=run)
        self.assertTrue(ok)
        self.assertEqual(calls[1][:3], ["/usr/bin/open", "-g", "-a"])
        self.assertIn(CONFIG["spotify"]["playlist"], calls[2][-1])
    def test_never_auto_fixes_stay_a_hold(self):
        cfg = json.loads(json.dumps(CONFIG))
        cfg["fixes"]["ableton"]["never_auto"] = True
        cfg["auto_fix"]["fixes"].remove("ableton")
        engine = sp.Engine(cfg, None)
        self.assertNotIn("ableton", {f for f in engine.auto.get("fixes", []) if not engine.cfg["fixes"][f].get("never_auto")})
        cfg["auto_fix"]["fixes"].append("ableton")
        path = os.path.join(tempfile.mkdtemp(prefix="sp-cfg-"), "config.json")
        json.dump(cfg, open(path, "w"))
        with self.assertRaises(ValueError):
            sp.load_config(path)
    def test_every_fix_is_valid(self):
        for f in CONFIG["fixes"].values():
-            self.assertIn(f.get("kind", "ssh"), ("ssh", "local", "hyperdeck", "midi_connect", "reconnect", "atem"))
+            self.assertIn(f.get("kind", "ssh"), sp.FIX_KINDS)
 class FakeDeck:
    """A pretend HyperDeck on a local port: answers like a real one, stays silent, or greets only."""
    def __init__(self, mode):
        import socket as _s
        self.mode, self.received = mode, b""
        self.srv = _s.socket()
        self.srv.bind(("127.0.0.1", 0))
        self.srv.listen(5)
        self.port = self.srv.getsockname()[1]
        threading.Thread(target=self.serve, daemon=True).start()
    def serve(self):
        while True:
            try:
                conn, _ = self.srv.accept()
            except OSError:
                return
            with conn:
                conn.settimeout(4)
                if self.mode != "silent":
                    conn.sendall(b"500 connection info:\r\nprotocol version: 1.11\r\nmodel: HyperDeck Studio 4K Pro\r\n\r\n")
                try:
                    data = conn.recv(1024)
                    self.received += data
                    if self.mode in ("record", "preview") and b"transport info" in data:
                        conn.sendall(b"208 transport info:\r\nstatus: %s\r\nspeed: 100\r\n\r\n" % self.mode.encode())
                    more = conn.recv(1024)
                    self.received += more
                except OSError:
                    pass
    def close(self):
        self.srv.close()
 class TestHyperDeckProbe(unittest.TestCase):
    def test_probe_answers(self):
        for mode, expected in (("record", "record"), ("preview", "preview"), ("greet", "answered"), ("silent", "hung")):
            deck = FakeDeck(mode)
            try:
                self.assertEqual(sp.hyperdeck_probe("127.0.0.1", timeout=1, port=deck.port), expected, mode)
            finally:
                deck.close()
        deck = FakeDeck("preview")
        port = deck.port
        deck.close()
        self.assertEqual(sp.hyperdeck_probe("127.0.0.1", timeout=1, port=port), "unreachable")
    def test_the_hold_switches_the_connection_off_and_on_and_never_reboots(self):
        from unittest import mock
        deck = FakeDeck("silent")                         # even a deck that would once have been rebooted
        try:
            comp = mock.Mock()
            comp.connections.side_effect = [
                [{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "error"}}],
                [{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "good"}}]]
            comp.disable_connection.return_value = comp.enable_connection.return_value = True
            chk = {"conn": "HyperDeck_1", "ip": "127.0.0.1", "port": deck.port}
            with mock.patch.object(sp, "Companion", return_value=comp), mock.patch.object(sp.time, "sleep"):
                self.assertTrue(sp.execute_fix(CONFIG, "hyperdeck", chk))
            self.assertEqual([c[0] for c in comp.method_calls if c[0].endswith("_connection")],
                             ["disable_connection", "enable_connection"])
            comp.restart_connection.assert_not_called()
            time.sleep(0.3)
            self.assertNotIn(b"reboot", deck.received)
        finally:
            deck.close()
    def test_a_deck_that_stays_offline_says_so(self):
        from unittest import mock
        fix = CONFIG["fixes"]["hyperdeck"]
        for pings, expect in ((True, "STILL OFFLINE\nHOLD AGAIN"), (False, "NO NETWORK\nCHECK POWER")):
            comp = mock.Mock()
            comp.connections.side_effect = [
                [{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}],
                [{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}]]
            with mock.patch.object(sp, "Companion", return_value=comp), \
                    mock.patch.object(sp, "ping_once", return_value=pings) as ping:
                self.assertEqual(sp.reconnect_and_verify(CONFIG, fix, "x", sleep=lambda s: None,
                                                         conn="HyperDeck_2", host="10.1.10.12", toggle=True), expect)
            ping.assert_called_once_with("10.1.10.12")     # the deck's own address, from Companion
 class TestNetgearParsing(unittest.TestCase):
@@ -1679,10 +1732,15 @@ class TestPage(unittest.TestCase):
        nav = {"0": {"0": {"type": "pageup"}}, "1": {"0": {"type": "button-layered",
                                                           "options": {"notes": "STATUS nav key (pagenum): x"}}}}
        self.assertTrue(push_page.safe_target({"page": {"name": "PAGE", "controls": nav}}, "23\\nRestart")[0])
-        busy = {"page": {"name": "PAGE", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}}
+        ours = {"page": {"name": "whatever", "controls": dict(nav, **{"3": {"7": {"type": "button-layered",
-        self.assertFalse(push_page.safe_target(busy, "23\\nRestart")[0])        # pages 22 and 97
+                 "options": {"notes": "STATUS tile 'led' - drawn by status_panel.py on NA Ingest."}},
-        busy["page"]["name"] = "23\\nRestart"
+                 "6": {"type": "button-layered", "options": {"notes": "STATUS jump key - Tap: open the Restart page."}}}})}}
-        self.assertTrue(push_page.safe_target(busy, "23\\nRestart")[0])         # our own page again
+        self.assertTrue(push_page.safe_target(ours, "23\\nRestart")[0])        # our keys, whatever the page is called
        busy = {"page": {"name": "23\\nRestart", "controls": dict(nav, **{"2": {"3": {"type": "button"}}})}}
        ok, why = push_page.safe_target(busy, "23\\nRestart")
        self.assertFalse(ok)                                                  # a key someone made by hand: refused
        self.assertIn("row 2 col 3", why)                                     # ...and named by its place
        self.assertIn("move or delete them first", why)
    def test_each_button_reads_its_own_tile(self):
        for t in CONFIG["tiles"]: