diff --git a/README.md b/README.md index fe656fe..987437f 100644 --- a/README.md +++ b/README.md @@ -278,7 +278,7 @@ The **page number key** on every page follows the ALL button. It's green when ev | Button | Checks | |---|---| -| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't; **RESI LIVE NOT REC** when Resi has been playing for a minute (`rec_watch`: resi-studio-cues `playback_state` PLAYING) and no deck records. Holding HD1–6 or REC then **starts recording** on every deck with a disk that is not recording (over the deck's own protocol, port 9993; a recording deck is sent nothing), named like the Director page's REC key: `CAM3-YYYYMMDD_HHMMSS-` (`rec_name` per deck) | +| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't; **RESI LIVE NOT REC** when Resi is live and no deck records (`rec_watch`: the Resi auto-record service's own verdict, Companion's `custom:autorec_status` saying LIVE, NOT RECORDING / RECORDING / STARTING for 20 s; not while it holds REC until 10 minutes before a service that Resi started early, and nothing while its heartbeat `autorec_hb` is stale). Holding HD1–6 or REC then **starts recording** on every deck with a disk that is not recording (over the deck's own protocol, port 9993; a recording deck is sent nothing), named like the Director page's REC key: `CAM3-YYYYMMDD_HHMMSS-` (`rec_name` per deck) | | | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. | | TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded.; pinged at the address on its Companion connection. | | X32, ARENA, OBS | Companion connection | @@ -292,7 +292,7 @@ The **page number key** on every page follows the ALL button. It's green when ev | VISTA | Vista on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise **SILENT** (Vista closed or crashed, or the Lighting PC is off the network). Hold 2 s starts Vista if it is not running; Vista then opens **Prop Takeover** from its show folder (`E:\media\Documents\Vista 3\Shows`) by itself, through its own autoload preference (`` in `E:\media\Documents\Vista 3\.preferences`, set 2026-10-08). Resolume is the ARENA button. | | RESI | decoder (RossTalk connection and ping, 10.1.10.203); **encoder** (ping and its web page, 10.1.10.202, LAOEncoder); resi-studio-cues; Resi cloud reachable. Only shows they are on the network, not that the encoder is streaming. | | NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21). A blip **flashes amber** for 5 minutes, then goes by itself; press NET to stop the flashing, hold NET 2 s to clear it now. | -| INGEST | the bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, **ProdCom** (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk | +| INGEST | **AUTO REC** (`resi_autorec.py`, LaunchAgent `org.waterschurch.resi-autorec`: presses the Director page's REC key 3/2/3 when Resi's encoder goes live and STOP a minute after it goes off air; its own key is 3/3/3): **AUTO REC STOPPED / NOT LOADED**, **AUTO REC STUCK** (heartbeat `custom:autorec_hb` over 90 s old), amber **AUTO REC NO RESI** (it cannot see Resi); a hold restarts it. The bridge (makes ProPresenter's stage screen count down to the next Resi cue), lobby captions server and caption screen, **ProdCom** (transcription, answering on :24480), ProdDash (running and its page on :24500); free disk | | SPOTIFY | Spotify on NA Ingest must be playing the **Christian EDM 2026** playlist (`spotify:playlist:2uTpJVYQLFnKF1WnSOuco8`): **SPOTIFY CLOSED**, **NOT PLAYING**, or **WRONG PLAYLIST** (it reads all of the playlist's songs from Spotify every hour, and again whenever a song plays that isn't on the list, since the playlist keeps getting new songs). Hold 2 s (and Sunday auto-fix) plays the playlist, opening Spotify first if needed. | | RF (group) | Shure connections; battery ≤1 bar red, 2 bars amber; RF interference (only while that transmitter is on); PSM RF muted; each receiver pinged every 3 s ("AXIENT 1 NO PING" in about 6–9 s; Companion's Shure modules take 18–27 s to notice). The address is the one set on its Companion connection, read from Companion's settings every minute, so changing it there is enough; `"host"` in `config.json` is only the fallback. | @@ -310,6 +310,21 @@ Fixes and the remote checks are built from fields (`settings/builders.py`): a bu The page reads what the watchdog writes next to its config (`snapshot.json` every 5 s, `started.json`, `live-now.json`, `fixes-running.json`) instead of asking the gear again. Its own look at the computers (`settings/probes.py`) runs fixed read-only commands over the same login key, fetches whole lists and matches them here, never while a fix runs. +## Wiring (settings page) + +**Wiring** on the settings page draws every computer and device the panel knows and what connects them, from what +the watchdog already sees (`/api/wiring`, refreshed every 5 s): **Companion control** (each connection, to the computer +or device at its address), **Network MIDI** (each session's peers), **Watchdog** (its SSH logins and pings: hidden at +first), **Web checks**, **Dante** (each device to its switch port, from the Dante map; a computer's Dante name joins +its box) and **Art-Net**. Click a type to hide or show it, untick single connections in the side panel, or click a +machine and **Only this one's connections**. Red dashes = that connection has a problem, grey dashes = can't see it +from here. Drag machines to arrange them; the wheel or +/− zooms, dragging the background pans, Fit shows everything. +Positions and what is hidden stay in that browser. +Each computer shows how many **scripts** run on it and which (side panel): our LaunchAgents (`org.waterschurch.*`, +linked to their script), any running `.py/.js/.sh/...` script, and on PCs the `\StatusPanel\` tasks (running while +their program runs). Only file names are kept, never the rest of a command line. **Check scripts now** looks +(read-only; the same probe as Computers › Check now). + ## Other locations: auto-setup A location sets itself up. All a person does is let the panel in: one line on the location's Mac that runs Companion, diff --git a/config.json b/config.json index 9fc4426..83b72aa 100644 --- a/config.json +++ b/config.json @@ -553,6 +553,18 @@ "run": "tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul || (schtasks /Run /TN \"\\StatusPanel\\Start Vista\" >nul && ping -n 9 127.0.0.1 >nul && tasklist /NH /FI \"IMAGENAME eq Vista.exe\" | find /I \"Vista.exe\" >nul)", "short": "VISTA", "template": "win_start" + }, + "autorec": { + "kind": "local", + "label": "restart the Resi auto-record", + "params": { + "labels": [ + "org.waterschurch.resi-autorec" + ] + }, + "run": "launchctl kickstart -k gui/$(id -u)/org.waterschurch.resi-autorec || launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.waterschurch.resi-autorec.plist", + "short": "AUTO REC", + "template": "helpers" } }, "flap": { @@ -2618,6 +2630,24 @@ "path": "/", "type": "disk", "warn_gb": 25 + }, + { + "fix": "autorec", + "label": "org.waterschurch.resi-autorec", + "name": "AUTO REC", + "type": "launchd" + }, + { + "type": "heartbeat", + "conn": "custom", + "var": "autorec_hb", + "max_age_seconds": 90, + "msg": "AUTO REC STUCK", + "status_var": "autorec_status", + "warn_if": "RESI ERROR", + "warn_msg": "AUTO REC NO RESI", + "debounce_seconds": 180, + "fix": "autorec" } ], "id": "ingest", @@ -2819,7 +2849,14 @@ "state": "sp_state" }, "rec_watch": { - "conn": "resi-studio-cues", - "after_seconds": 60 + "conn": "custom", + "var": "autorec_status", + "heartbeat": "autorec_hb", + "live": [ + "LIVE, NOT", + "RECORDING", + "STARTING" + ], + "after_seconds": 20 } } diff --git a/schema.py b/schema.py index 8e2a915..e77cd6d 100644 --- a/schema.py +++ b/schema.py @@ -104,9 +104,10 @@ field("service_windows[].end", "hhmm", None, "", "Schedules", required=True) field("resi_events", "dict", None, "Resi's scheduled events (studio.resi.io), so the checks marked quiet \"resi_events\" keep quiet " "while an event starts or stops: {\"weekly\": [{\"days\": [\"sun\"], \"start\": \"07:30\", \"end\": \"09:15\"}, ...], " "\"margin_minutes\": 10, \"pco_specials\": true (special services' times from Planning Center)}.", "Site") -field("rec_watch", "dict", None, "Every deck with a disk must record while Resi plays: {\"conn\": \"resi-studio-cues\", " - "\"after_seconds\": 60} (its playback_state PLAYING that long). A deck that does not shows RESI LIVE NOT REC; " - "a hold starts recording on the idle decks only.", "Site") +field("rec_watch", "dict", None, "Every deck with a disk must record while Resi is live: {\"conn\": \"custom\", \"var\": " + "\"autorec_status\", \"heartbeat\": \"autorec_hb\", \"live\": [\"LIVE, NOT\", \"RECORDING\", \"STARTING\"], " + "\"after_seconds\": 20} (the Resi auto-record service's own verdict; without a heartbeat a module's playback_state " + "PLAYING). A deck that does not record shows RESI LIVE NOT REC; a hold starts recording on the idle decks only.", "Site") field("debounce_seconds", "int", 3, "A fault must last this long before it shows (one bad reading is not an " "alarm).", "Schedules", advanced=True, lo=0, hi=60) field("flap", "dict", {}, "Repeated faults show amber 'xN'.", "Schedules", advanced=True) @@ -325,6 +326,11 @@ field("tiles[].checks[].name", "str", None, "The name the key shows.", "Keys") field("tiles[].checks[].msg", "str", None, "The words the key shows when it fails.", "Keys") field("tiles[].checks[].failures", "int", 2, "Pings missed before NO PING.", "Keys", advanced=True, lo=1, hi=10) field("tiles[].checks[].label", "str", None, "The launchd helper.", "Keys") +field("tiles[].checks[].var", "str", None, "The Companion variable with the service's heartbeat (unix time).", "Keys") +field("tiles[].checks[].max_age_seconds", "int", 90, "A heartbeat older than this: the service is stuck.", "Keys", advanced=True, lo=10, hi=3600) +field("tiles[].checks[].status_var", "str", None, "The service's status text (Companion variable).", "Keys", advanced=True) +field("tiles[].checks[].warn_if", "str", None, "Amber while the status text contains this.", "Keys", advanced=True) +field("tiles[].checks[].warn_msg", "str", None, "What the key says then.", "Keys", advanced=True) field("tiles[].checks[].match", "str", None, "The program (pgrep -f pattern).", "Keys") field("tiles[].checks[].url", "url", None, "The web address to fetch.", "Keys") field("tiles[].checks[].interval_seconds", "int", 30, "Look this often, in seconds: a web address every 30, a command " @@ -515,6 +521,9 @@ CHECKS = { "question": "Is a recorder OK?", "words": "The Tascam recorder reports no error or caution."}, "ping": {"fields": ["host", "conn", "name", "msg", "failures", "interval_seconds", "service_only"], "subs": ["ping"], "identity": ["conn", "host"], "module": None, "question": "Is a device answering?", "words": "An address answers ping."}, + "heartbeat": {"fields": ["conn", "var", "max_age_seconds", "msg", "status_var", "warn_if", "warn_msg"], "subs": ["hb"], + "identity": ["var"], "module": None, "question": "Is a service still writing its heartbeat?", + "words": "A service writes the time into a Companion variable (custom autorec_hb: the Resi auto-record); too old means it is stuck."}, "launchd": {"fields": ["label", "name"], "subs": ["svc"], "identity": ["label"], "module": None, "question": "Is a program running?", "words": "A helper (LaunchAgent) on this Mac is loaded and running."}, "process": {"fields": ["match", "name", "template", "params"], "subs": ["proc"], "identity": ["match"], "module": None, @@ -565,12 +574,12 @@ GLOSSARY = { "DISABLED", "OFFLINE", "NO SESSION", "NOT RUNNING", "DOWN", "NOT REC", "SILENT", "WARNING", "STARTING", "ATEM AUDIO WRONG", "ATEM CAMERA WRONG", "ATEM SW MIDI ON", "PT SOLO NOT AFL", "PROPRES NOT RUNNING", "ABLETON NOT RUNNING", "ABLESET NOT RUNNING", "TRANSLATE NOT RUNNING", "PROP TV DOWN", "BIBLE APP DOWN", "PRODCOM DOWN", "PRODDASH DOWN", - "ENCODER WEB DOWN", "PT SOLO CHECK OFF", "UNREACH", "DROPPED", "RESI LIVE NOT REC", + "ENCODER WEB DOWN", "PT SOLO CHECK OFF", "UNREACH", "DROPPED", "RESI LIVE NOT REC", "AUTO REC STOPPED", "AUTO REC STUCK", ], "check the device or the cable": [ "NO PING", "NO ANSWER", "UNREACHABLE", "LINK DOWN", "NO GATEWAY", "NO DISK", "FULL", "LOW", "BATT", "INTERF", "RF MUTE", "HUNG - POWER CYCLE IT", "NO NETWORK CHECK POWER", "STILL DOWN", "REFUSED", "LOGIN", "BLIP", - "NOT REACHABLE", "DISK", "RATE LIMITED", + "NOT REACHABLE", "DISK", "RATE LIMITED", "AUTO REC NO RESI", ], "the READY page: what is not ready yet (hold the key to fix it or look again)": [ "TO DO", "ALL DONE", "TAP WHEN DONE", "TAP WHEN SET", "NOT DONE", "DONE", "NOT SET UP", "CHECKING", diff --git a/settings/probes.py b/settings/probes.py index 76f47fb..e462328 100644 --- a/settings/probes.py +++ b/settings/probes.py @@ -27,7 +27,7 @@ MAC_PROBE = ( 'echo SP_AGENTS; launchctl list 2>/dev/null; echo SP_PROCS; ps -axo comm= 2>/dev/null; ' 'echo SP_APPS; ls /Applications 2>/dev/null; echo SP_HELPER; ls -l {helper} 2>&1; sudo -n -l {helper} 2>&1 | tail -1; ' 'echo SP_SOLO; cat {solo}/solo-mode.json 2>/dev/null; echo; stat -f %m {solo}/solo-mode.json 2>/dev/null; ' - 'echo SP_NOW; date +%s; echo SP_END' + 'echo SP_NOW; date +%s; echo SP_SCRIPTS; ps -axo pid=,args= 2>/dev/null | grep -E \'[.](py|js|mjs|sh|zsh|command|swift|rb|pl|scpt)( |$)\' | grep -v SP_SCRIPTS; echo SP_END' ) WIN_PROBE = ( 'echo SP_NAME=%COMPUTERNAME% & echo SP_TASKS & schtasks /Query /FO CSV /NH 2>nul & echo SP_PROCS & tasklist /FO CSV /NH 2>nul & ' @@ -35,8 +35,15 @@ WIN_PROBE = ( '"(Get-NetConnectionProfile | Select -First 1 -ExpandProperty NetworkCategory); ' '(Get-NetAdapterPowerManagement | Select -First 1 -ExpandProperty WakeOnMagicPacket); ' '((Get-NetFirewallRule -Name OpenSSH-Server-In-TCP -ErrorAction SilentlyContinue | Get-NetFirewallAddressFilter | ' - 'Select -ExpandProperty RemoteAddress) -join \',\')" 2>nul & echo SP_END' + 'Select -ExpandProperty RemoteAddress) -join \',\')" 2>nul & echo SP_SCRIPTS & powershell -NoProfile -Command ' + '"Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -match \'[.](ps1|py|js|bat|cmd|vbs)\' } | ' + 'ForEach-Object { [string]$_.ProcessId + \' \' + $_.CommandLine }" 2>nul & echo SP_END' ) +# A script: a file run by an interpreter (its name only is kept, never the rest of the command line). +SCRIPT_RE = r"[^ /\\]+[.](py|js|mjs|sh|zsh|command|swift|rb|pl|scpt|ps1|bat|cmd|vbs)( |\"|$)" +# Not the user's scripts: programs' own insides (Companion's modules, app bundles, the system). +NOT_SCRIPTS = (".app/Contents/", "/companion/modules/", "/System/", "/usr/libexec/", "/Library/Apple/", "\\Windows\\", + "/node_modules/", "Program Files") def shell_path(path): @@ -113,6 +120,38 @@ def parse_sc(lines): return services +def parse_scripts(lines, agents=None, tasks=None): + """The scripts a computer runs: [{"name", "pid", "by", "running"}]. From `ps` (pid, command line) lines, + our own launchd jobs (org.waterschurch.*, linked to their script by pid) and the \\StatusPanel\\ tasks + ({path: state}). Only a script's file name is kept, never the rest of its command line.""" + agents, out, seen_pids = agents or {}, [], set() + by_pid = {pid: label for label, pid in agents.items() if pid} + for line in lines: + line = line.strip() + m = re.match(r"^(\d+)\s+(.*)$", line) + if not m or "SP_SCRIPTS" in line: + continue + found = re.search(SCRIPT_RE, m.group(2)) + if not found: + continue + # The script's own path (from the last " /", ' "' or " C:\" before its name): the interpreter may sit + # inside an app (Apple's python3 is in Python.app) without the script being a program's insides. + head = m.group(2)[:found.end()] + cut = max([head.rfind(" /"), head.rfind(' "')] + [x.start() for x in re.finditer(r" [A-Za-z]:\\", head)]) + if any(x in head[cut + 1:] for x in NOT_SCRIPTS): + continue + name = re.split(r"[\\/]", found.group(0).strip().strip('"'))[-1] + pid = int(m.group(1)) + out.append({"name": name, "pid": pid, "by": by_pid.get(pid, ""), "running": True}) + seen_pids.add(pid) + for label, pid in sorted(agents.items()): + if label.startswith("org.waterschurch.") and pid not in seen_pids: + out.append({"name": label[len("org.waterschurch."):], "pid": pid, "by": label, "running": pid is not None}) + for path, st in sorted((tasks or {}).items()): + out.append({"name": path.split("\\")[-1], "pid": None, "by": path, "running": st.lower() == "running", "state": st}) + return sorted(out, key=lambda x: (not x["running"], x["name"].lower())) + + def light(ok, reason=""): return {"light": "ok" if ok else "problem", "reason": reason if not ok else ""} @@ -192,7 +231,7 @@ def mac_items(host, parsed, cfg): item.update({"light": "unknown", "reason": ""}) items.append(item) return items, {"agents": sorted(a for a in agents if not a.startswith(("com.apple.", "application."))), - "processes": sorted(proc_names)[:200]} + "processes": sorted(proc_names)[:200], "scripts": parse_scripts(parsed.get("SP_SCRIPTS", []), agents)} def windows_items(host, parsed): @@ -228,8 +267,13 @@ def windows_items(host, parsed): item.update({"light": "unknown", "reason": ""}) items.append(item) setup = {"profile": profile, "wake_on_lan": wol, "firewall_allow": allow} - return items, {"tasks": sorted(t for t in tasks if "StatusPanel" in t), "services": sorted(k for k, v in services.items() - if v == "RUNNING" and not k.lower().startswith(("w", "s", "d", "e", "n", "p", "t", "u", "c")))[:60]}, setup + # A task that starts a program is "Ready" once it did: it counts as running while that program runs. + up = {"\\StatusPanel\\" + run["name"] for run, item in zip(host.get("runs", []), items) + if run.get("kind") == "task" and run.get("exe") and item.get("light") == "ok"} + ours = {t: ("Running" if t in up else row[2] if len(row) > 2 else "") for t, row in tasks.items() if "StatusPanel" in t} + return items, {"tasks": sorted(ours), "services": sorted(k for k, v in services.items() + if v == "RUNNING" and not k.lower().startswith(("w", "s", "d", "e", "n", "p", "t", "u", "c")))[:60], + "scripts": parse_scripts(parsed.get("SP_SCRIPTS", []), tasks=ours)}, setup def probe_host(cfg, alias, host): @@ -245,7 +289,7 @@ def probe_host(cfg, alias, host): rec["error"] = err.strip() or "no answer" return rec rec["reachable"] = True - parsed = sections(out, ("SP_TASKS", "SP_PROCS", "SP_SVC", "SP_PS", "SP_END")) + parsed = sections(out, ("SP_TASKS", "SP_PROCS", "SP_SVC", "SP_PS", "SP_SCRIPTS", "SP_END")) rec["computer_name"] = values(out).get("NAME", "").strip() rec["items"], rec["found"], rec["setup"] = windows_items(host, parsed) rec["setup"]["key_ok"] = True @@ -257,7 +301,7 @@ def probe_host(cfg, alias, host): return rec rec["reachable"] = True v = values(out) - parsed = sections(out, ("SP_AGENTS", "SP_PROCS", "SP_APPS", "SP_HELPER", "SP_SOLO", "SP_NOW", "SP_END")) + parsed = sections(out, ("SP_AGENTS", "SP_PROCS", "SP_APPS", "SP_HELPER", "SP_SOLO", "SP_NOW", "SP_SCRIPTS", "SP_END")) rec["computer_name"], rec["os_version"], rec["python"] = v.get("NAME", ""), v.get("OSV", ""), v.get("PY", "") rec["items"], rec["found"] = mac_items(host, parsed, cfg) rec["setup"] = {"key_ok": (v.get("KEY") or "0").strip() not in ("", "0"), "python_ok": "3." in v.get("PY", "")} @@ -301,6 +345,7 @@ def probe_self(cfg, alias, host): try: agents_out = subprocess.run(["/bin/launchctl", "list"], capture_output=True, text=True, timeout=5).stdout procs_out = subprocess.run(["/bin/ps", "-axo", "comm="], capture_output=True, text=True, timeout=5).stdout + args_out = subprocess.run(["/bin/ps", "-axo", "pid=,args="], capture_output=True, text=True, timeout=5).stdout except (OSError, subprocess.TimeoutExpired) as e: rec["error"] = str(e) return rec @@ -308,7 +353,8 @@ def probe_self(cfg, alias, host): helper = subprocess.run(["/bin/zsh", "-c", "ls -l %s 2>&1; sudo -n -l %s 2>&1 | tail -1" % (paths["helper"], paths["helper"])], capture_output=True, text=True, timeout=6).stdout parsed = {"SP_AGENTS": agents_out.splitlines(), "SP_PROCS": procs_out.splitlines(), "SP_APPS": [], - "SP_HELPER": helper.splitlines(), "SP_SOLO": [], "SP_NOW": [str(int(time.time()))]} + "SP_HELPER": helper.splitlines(), "SP_SOLO": [], "SP_NOW": [str(int(time.time()))], + "SP_SCRIPTS": [l for l in args_out.splitlines() if re.search(SCRIPT_RE, l)]} rec["items"], rec["found"] = mac_items(host, parsed, cfg) for item, run in zip(rec["items"], host.get("runs", [])): if run.get("kind") == "folder": diff --git a/settings/server.py b/settings/server.py index f1b0111..448aacc 100644 --- a/settings/server.py +++ b/settings/server.py @@ -522,6 +522,134 @@ class Site: "the DVS helper and its sudo rule", "the Dante and switch-management ports are up", "Google Drive is signed in (saved switcher files)"], } + # The Wiring screen: every computer and device the panel knows, and what connects them (2026-10-27: + # "like a wiring diagram ... hide different connections ... how many and what scripts are running on each machine"). + WIRE_TYPES = [ + {"id": "companion", "label": "Companion control", "color": "#4f8cff", "hint": "Companion's connections: what the deck's buttons drive"}, + {"id": "midi", "label": "Network MIDI", "color": "#b36bff", "hint": "rtpMIDI / Audio MIDI Setup sessions and their peers"}, + {"id": "watch", "label": "Watchdog", "color": "#8a94a6", "hint": "What the status panel logs in to or pings"}, + {"id": "web", "label": "Web checks", "color": "#2bb3a3", "hint": "Web pages the panel asks (apps' own pages, Resi's cloud)"}, + {"id": "dante", "label": "Dante", "color": "#f0a020", "hint": "Dante devices and the switch port each is plugged into"}, + {"id": "artnet", "label": "Art-Net", "color": "#e8618c", "hint": "Lighting data the panel hears"}, + ] + + def wiring_view(self): + cfg = self.load() + snap = self.read_json("snapshot.json", {}) or {} + live = self.read_json("machines-live.json", {}) or {} + hosts = cfg.get("hosts", {}) + me = next((a for a, h in hosts.items() if h.get("self")), None) + lines = {l["alias"]: l for l in self.computer_lines(snap)} + word_light = {"OK": "ok", "Problem": "problem", "Off": "not_needed"} + nodes, edges, owner = {}, {}, {"127.0.0.1": me, "localhost": me} + + def norm(x): + return re.sub(r"[^a-z0-9]", "", str(x).lower()) + for alias, h in hosts.items(): + probe = live.get(alias) or {} + ip = (h.get("ssh") or "").split("@")[-1] + for a in [ip] + [n.get("ip") for n in h.get("networks", [])] + [i.get("ip") for i in (probe.get("found") or {}).get("interfaces", [])]: + if a: + owner[a] = alias + line = lines.get(alias, {}) + found = probe.get("found") or {} + nodes["h:" + alias] = {"id": "h:" + alias, "label": h.get("name", alias), "kind": "computer", "os": h.get("os", "mac"), + "self": bool(h.get("self")), "address": ip or ("this Mac" if h.get("self") else ""), + "role": h.get("role", ""), "light": word_light.get(line.get("word"), "unknown"), + "reason": line.get("reason", ""), "scripts": found.get("scripts"), "looked": probe.get("time")} + for m in (self.read_json("fleet.json", {}) or {}).values(): + alias = next((a for a, h in hosts.items() if norm(h.get("name")) == norm(m.get("name"))), None) + for a in m.get("ips") or [] if alias else []: + owner.setdefault(a, alias) + + def target(address, label, kind="device"): + """The node at an address: a computer when it is one of them, else a device (one per address).""" + host = (address or "").split("//")[-1].split("/")[0].split(":")[0] if "://" in (address or "") else (address or "").split(":")[0] + if host in owner and owner[host]: + return "h:" + owner[host] + nid = "d:" + (host or norm(label)) + node = nodes.setdefault(nid, {"id": nid, "label": label, "kind": kind, "address": host, "light": "unknown", "names": []}) + if label not in node["names"]: + node["names"].append(label) + return nid + + def edge(kind, a, b, label, light): + if not a or not b or a == b: + if a and a == b and a in nodes: + nodes[a].setdefault("local", []).append({"type": kind, "label": label, "light": light}) + return + key = "|".join([kind] + sorted([a, b]) + [label]) + edges[key] = {"id": key, "type": kind, "from": a, "to": b, "label": label, "light": light} + conns, conn_hosts = snap.get("conns") or {}, snap.get("conn_hosts") or {} + gear = cfg.get("gear") or {} + kinds = {"hyperdeck": "recorder", "tascam": "recorder", "shure": "wireless", "psm": "in-ears", "atem": "switcher"} + conn_kind = {c["conn"]: kinds.get(c["type"], "device") for t in cfg["tiles"] for c in t.get("checks", []) if c.get("conn")} + me_id = "h:" + me if me else None + for label, st in sorted(conns.items()): + address = conn_hosts.get(label) or (gear.get(label) or {}).get("address") + light = "ok" if st.get("category") == "good" else ("not_needed" if not st.get("enabled") else "problem") + if label == "audinate-dantecontroller": + continue # the Dante type draws the Dante network itself + node = target(address, label, conn_kind.get(label, "device")) if address else target(None, label, "service") + if node.startswith("d:"): + nodes[node]["light"] = light if nodes[node]["light"] in ("unknown", "ok") else nodes[node]["light"] + edge("companion", me_id, node, label, light) + sessions = {me: snap.get("midi") or {}} if me else {} + sessions.update(snap.get("remote_midi") or {}) + for alias, sess in sessions.items(): + for name, s in (sess or {}).items(): + for peer in s.get("peers", []): + other = target(peer.get("address"), peer.get("name") or peer.get("address"), "MIDI") + edge("midi", "h:" + alias, other, "%s <-> %s" % (name, peer.get("name")), + "ok" if peer.get("state") in ("running", "connected") else "problem") + pings = snap.get("ping") or {} + for alias, h in hosts.items(): + if h.get("self") or h.get("disabled"): + continue + probe = live.get(alias) or {} + ip = (h.get("ssh") or "").split("@")[-1] + ok = probe.get("reachable") if probe else (pings.get(ip) == 0 if ip in pings else None) + edge("watch", me_id, "h:" + alias, "logs in (SSH)", "unknown" if ok is None else ("ok" if ok else "problem")) + http = snap.get("http") or {} + for t in cfg["tiles"]: + for c in t.get("checks", []): + if c["type"] == "ping" and c.get("host"): + node = target(c["host"], c.get("name") or c["host"]) + fails = pings.get(c["host"]) + edge("watch", me_id, node, "ping", "unknown" if fails is None else ("ok" if fails == 0 else "problem")) + elif c["type"] == "http" and c.get("url"): + ok = http.get(c["url"]) + node = target(c["url"], c.get("name") or c.get("msg") or c["url"].split("//")[-1].split("/")[0], + "device" if re.search(r"//(10|192\.168|172|127)\.", c["url"]) else "cloud") + edge("web", me_id, node, c.get("name") or c.get("msg") or "web page", "unknown" if ok is None else ("ok" if ok else "problem")) + for dev, d in sorted((snap.get("dante_map") or {}).items()): + sw = "sw:" + str(d.get("switch")) + nodes.setdefault(sw, {"id": sw, "label": "Dante switch %s" % d.get("switch"), "kind": "switch", "address": d.get("switch"), + "light": "ok", "names": []}) + nd = norm(dev) + # A computer's Dante name ends with its own (NA-Tracks: tracks; NA-ProPresenter-Mac: ProPresenter Mac); + # STUDIO-B-MON is a monitor, not the Studio B Mac. A device already drawn joins up by name (SuperRack-LiveBox). + alias = next((a for a, h in hosts.items() if nd.endswith(norm(a)) or nd.endswith(norm(h.get("name", a)))), None) + same = next((n["id"] for n in nodes.values() if n["kind"] != "computer" and n["id"] != sw + and any(len(norm(x)) >= 6 and nd.endswith(norm(x)) for x in n.get("names", []))), None) + node = "h:" + alias if alias else same or target(None, dev, "Dante") + if node.startswith("d:") and dev not in nodes[node]["names"]: + nodes[node]["names"].append(dev) + fresh = time.time() - (d.get("seen") or 0) < 3600 + edge("dante", node, sw, "port %s%s" % (d.get("port"), " (PoE)" if d.get("poe") else ""), "ok" if fresh else "unknown") + for src, age in (snap.get("artnet") or {}).items(): + edge("artnet", target(src, "Art-Net from %s" % src), me_id, "Art-Net", "ok" if age < 10 else "problem") + rank = {"problem": 3, "ok": 1, "not_needed": 0, "unknown": -1} + for e in edges.values(): # a device is as good as what reaches it + for end in (e["from"], e["to"]): + n = nodes.get(end) + if n and n["kind"] != "computer" and rank.get(e["light"], -1) > rank.get(n["light"], -1): + n["light"] = e["light"] + used = {e["from"] for e in edges.values()} | {e["to"] for e in edges.values()} + out = [n for n in nodes.values() if n["kind"] == "computer" or n["id"] in used] + return {"nodes": out, "edges": list(edges.values()), "types": self.WIRE_TYPES, "me": me_id, "time": snap.get("time"), + "probing": bool(self.probing and self.probing.is_alive())} + def gear_view(self): cfg = self.load() snap = self.read_json("snapshot.json", {}) or {} @@ -769,6 +897,8 @@ class Handler(http.server.BaseHTTPRequestHandler): return self.send(200, site.companion_page(int(n))) except (OSError, ValueError) as e: return self.send(200, {"page": int(n), "buttons": [], "error": "Companion did not answer: %s" % e}) + if path == "/api/wiring": + return self.send(200, site.wiring_view()) if path == "/api/machines": return self.send(200, site.machines_view()) if path == "/api/gear": diff --git a/settings/ui/app.js b/settings/ui/app.js index 07ab6ef..150be31 100644 --- a/settings/ui/app.js +++ b/settings/ui/app.js @@ -233,7 +233,7 @@ search.onblur = () => setTimeout(() => { results.hidden = true; }, 200); // ---- routing --------------------------------------------------------------- - const screens = { overview, keys, computers, settings, logs, history, help, review }; + const screens = { overview, keys, computers, wiring, settings, logs, history, help, review }; let timer = null; async function route() { let [name, ...rest] = (location.hash || "#overview").slice(1).split("/"); @@ -247,7 +247,7 @@ await fn(rest); wireInputs(main); wireOnOff(main); - const refresh = name === "overview" || (name === "computers" && !["gear", "setup"].includes(rest[0])); + const refresh = name === "overview" || name === "wiring" || (name === "computers" && !["gear", "setup"].includes(rest[0])); if (refresh) timer = setInterval(() => fn(rest, true).then(() => wireInputs(main)).catch(() => {}), 5000); } catch (e) { main.innerHTML = `

Could not load this screen: ${esc(e.message)}

`; } } @@ -630,6 +630,237 @@ ${canRunNow(r.fix) && editing() ? `` : ""}
more
id
${esc(r.id)}
${r.watch && r.watch.length ? `
flashes while down
${esc(r.watch.join(", "))}
` : ""}
${editing() ? `

` : ""}`; } + // ---- Wiring ------------------------------------------------------------------ + // Every computer and device the panel knows and what connects them, like a wiring diagram (2026-10-27). Drag a + // machine to arrange it; click one to see its scripts and connections; hide connection types or single + // connections. Positions and what is hidden are kept in this browser only. + const W_W = 2000, W_H = 1300; + const wire = { data: null, pos: null, hidden: null, types: null, sel: null, focus: false, problems: false, drag: null, view: null }; + const wireRead = (k, d) => { try { return JSON.parse(store.get(k) || "null") || d; } catch (e) { return d; } }; + const wireSave = () => { store.set("sp-wire-pos", JSON.stringify(wire.pos)); store.set("sp-wire-hidden", JSON.stringify([...wire.hidden])); store.set("sp-wire-types", JSON.stringify([...wire.types])); }; + const wireType = (id) => (wire.data.types.find((t) => t.id === id) || { label: id, color: "#888" }); + const wireNode = (id) => wire.data.nodes.find((n) => n.id === id); + const wireName = (id) => (wireNode(id) || { label: id }).label; + async function wiring(rest, quiet) { + if (quiet && (wire.drag || document.getElementById("dialog").hidden === false)) return; + wire.data = await api("/api/wiring"); + if (!wire.pos) { wire.pos = wireRead("sp-wire-pos", {}); wire.hidden = new Set(wireRead("sp-wire-hidden", [])); wire.types = new Set(wireRead("sp-wire-types", ["watch"])); } // the watchdog's own lines: hidden at first + if (wire.sel && !(wire.sel.edge ? wire.data.edges.some((e) => e.id === wire.sel.id) : wireNode(wire.sel.id))) wire.sel = null; + wireLayout(); + if (!(quiet && document.getElementById("wiresvg"))) { + main.innerHTML = `

Wiring${hint("What is connected to what. Drag a machine to move it, click it for its scripts and connections. Colours: the kind of connection; red = that connection has a problem; dashed = can't see it from here.")}

+
`; + wireEvents(); + } + wireTools(); wireDraw(); wireSide(); + } + + function wireLayout(all) { + const d = wire.data, cx = W_W / 2, cy = W_H / 2; + const fresh = d.nodes.filter((n) => all || !wire.pos[n.id]).map((n) => n.id); + if (!fresh.length) return; + const nb = {}; + for (const e of d.edges) if (e.type !== "watch") { (nb[e.from] = nb[e.from] || []).push(e.to); (nb[e.to] = nb[e.to] || []).push(e.from); } + const move = new Set(fresh), hubs = d.nodes.filter((n) => (n.kind === "computer" || n.kind === "switch") && n.id !== d.me); + hubs.sort((a, b) => (a.kind === "switch") - (b.kind === "switch") || a.label.localeCompare(b.label)); + if (d.me && move.has(d.me)) wire.pos[d.me] = { x: cx, y: cy }; + const angle = {}; + hubs.forEach((n, i) => { angle[n.id] = 2 * Math.PI * i / hubs.length - Math.PI / 2; if (move.has(n.id)) wire.pos[n.id] = { x: cx + W_W * 0.24 * Math.cos(angle[n.id]), y: cy + W_H * 0.26 * Math.sin(angle[n.id]) }; }); + // Each device beside the machine it belongs to: its switch or computer with the most links, else Companion's Mac. + const anchor = {}, groups = {}; + for (const n of d.nodes) { + if (n.kind === "computer" || n.kind === "switch") continue; + const count = {}; + for (const o of nb[n.id] || []) if (angle[o] !== undefined) count[o] = (count[o] || 0) + 1; + const best = Object.keys(count).sort((a, b) => count[b] - count[a])[0] || d.me; + anchor[n.id] = best; (groups[best] = groups[best] || []).push(n); + } + for (const [hub, list] of Object.entries(groups)) { + list.sort((a, b) => String(a.kind).localeCompare(String(b.kind)) || String(a.label).localeCompare(String(b.label))); + list.forEach((n, i) => { + if (!move.has(n.id)) return; + if (hub === d.me || angle[hub] === undefined) { // the outer ring, kinds together + const a = 2 * Math.PI * i / list.length - Math.PI / 2 + 0.2; + wire.pos[n.id] = { x: cx + W_W * 0.43 * Math.cos(a), y: cy + W_H * 0.44 * Math.sin(a) }; + } else { // a fan outward from its machine + const a = angle[hub] + (i - (list.length - 1) / 2) * 0.12, r = 1.0 + 0.16 * (1 + (i % 2)); + wire.pos[n.id] = { x: cx + W_W * 0.24 * r * Math.cos(a), y: cy + W_H * 0.26 * r * Math.sin(a) }; + } + }); + } + // Push apart whatever overlaps (boxes, with a margin), each device held near its machine. + const ids = d.nodes.map((n) => n.id), size = {}; + for (const n of d.nodes) size[n.id] = nodeSize(n); + for (let it = 0; it < 260; it++) { + for (let i = 0; i < ids.length; i++) for (let j = i + 1; j < ids.length; j++) { + const a = ids[i], b = ids[j]; + if (!move.has(a) && !move.has(b)) continue; + const pa = wire.pos[a], pb = wire.pos[b]; + const ox = (size[a].w + size[b].w) / 2 + 14 - Math.abs(pa.x - pb.x), oy = (size[a].h + size[b].h) / 2 + 12 - Math.abs(pa.y - pb.y); + if (ox <= 0 || oy <= 0) continue; + const ma = move.has(a) ? (move.has(b) ? 0.5 : 1) : 0, mb = 1 - ma; + if (ox < oy) { const sx = (pa.x < pb.x ? -1 : 1) * ox; pa.x += sx * ma; pb.x -= sx * mb; } + else { const sy = (pa.y < pb.y ? -1 : 1) * oy; pa.y += sy * ma; pb.y -= sy * mb; } + } + for (const id of move) { + const h = anchor[id], p = wire.pos[id]; + if (h && h !== d.me && wire.pos[h]) { const q = wire.pos[h], dx = q.x - p.x, dy = q.y - p.y, dd = Math.hypot(dx, dy) || 1; if (dd > 230) { p.x += dx * 0.02; p.y += dy * 0.02; } } + p.x = Math.max(size[id].w / 2 + 6, Math.min(W_W - size[id].w / 2 - 6, p.x)); p.y = Math.max(20, Math.min(W_H - 20, p.y)); + } + } + wireSave(); + } + + function wireFit() { + const { nodes } = wireShown(); + if (!nodes.length) { wire.view = { x: 0, y: 0, w: W_W, h: W_H }; return; } + let x0 = Infinity, y0 = Infinity, x1 = -Infinity, y1 = -Infinity; + for (const n of nodes) { const p = wire.pos[n.id], { w, h } = nodeSize(n); x0 = Math.min(x0, p.x - w / 2); x1 = Math.max(x1, p.x + w / 2); y0 = Math.min(y0, p.y - h / 2); y1 = Math.max(y1, p.y + h / 2); } + const svg = document.getElementById("wiresvg"), box = svg && svg.getBoundingClientRect(); + const pad = 30, w = x1 - x0 + 2 * pad, h = y1 - y0 + 2 * pad, ratio = box && box.height ? box.width / box.height : W_W / W_H; + const vw = Math.max(w, h * ratio), vh = vw / ratio; + wire.view = { x: (x0 + x1) / 2 - vw / 2, y: (y0 + y1) / 2 - vh / 2, w: vw, h: vh }; + } + function wireZoom(f, px, py) { + const v = wire.view, nw = Math.max(250, Math.min(W_W * 2.5, v.w * f)), k = nw / v.w; + px = px == null ? v.x + v.w / 2 : px; py = py == null ? v.y + v.h / 2 : py; + wire.view = { x: px - (px - v.x) * k, y: py - (py - v.y) * k, w: nw, h: v.h * k }; + const svg = document.getElementById("wiresvg"); if (svg) svg.setAttribute("viewBox", `${wire.view.x} ${wire.view.y} ${wire.view.w} ${wire.view.h}`); + } + + function wireShown() { + const d = wire.data, sel = wire.sel && !wire.sel.edge ? wire.sel.id : null; + const edges = d.edges.filter((e) => !wire.types.has(e.type) && !wire.hidden.has(e.id) && (!wire.problems || e.light === "problem") + && (!wire.focus || !sel || e.from === sel || e.to === sel)); + const ends = new Set(); for (const e of edges) { ends.add(e.from); ends.add(e.to); } + const nodes = d.nodes.filter((n) => ends.has(n.id) || (n.kind === "computer" && !wire.focus && !wire.problems) || n.id === sel); + return { edges, nodes }; + } + const nodeSize = (n) => n.kind === "computer" ? { w: 168, h: 46 } : { w: Math.min(190, 26 + 6.6 * String(n.label).length), h: 26 }; + + function wireDraw() { + const svg = document.getElementById("wiresvg"); + if (!svg) return; + if (!wire.view) wireFit(); + svg.setAttribute("viewBox", `${wire.view.x} ${wire.view.y} ${wire.view.w} ${wire.view.h}`); + const { edges, nodes } = wireShown(), sel = wire.sel; + const near = new Set(); + if (sel && !sel.edge) { near.add(sel.id); for (const e of edges) if (e.from === sel.id || e.to === sel.id) { near.add(e.from); near.add(e.to); } } + if (sel && sel.edge) { const e = wire.data.edges.find((x) => x.id === sel.id); if (e) { near.add(e.from); near.add(e.to); } } + const pairs = {}; + for (const e of edges) { const k = [e.from, e.to].sort().join("|"); (pairs[k] = pairs[k] || []).push(e); } + let out = ""; + for (const list of Object.values(pairs)) list.forEach((e, i) => { + const a = wire.pos[e.from], b = wire.pos[e.to]; + if (!a || !b) return; + const off = (i - (list.length - 1) / 2) * 16, dx = b.x - a.x, dy = b.y - a.y, len = Math.hypot(dx, dy) || 1; + const mx = (a.x + b.x) / 2 - dy / len * off, my = (a.y + b.y) / 2 + dx / len * off; + const t = wireType(e.type), on = sel ? (sel.edge ? sel.id === e.id : e.from === sel.id || e.to === sel.id) : true; + out += `${esc(t.label)}: ${esc(wireName(e.from))} ↔ ${esc(wireName(e.to))} · ${esc(e.label)} · ${esc(lightWord[e.light] || e.light)}`; + }); + for (const n of nodes) { + const p = wire.pos[n.id], { w, h } = nodeSize(n), dim = sel && !near.has(n.id); + const scripts = n.scripts ? n.scripts.filter((x) => x.running).length : null; + const shape = n.kind === "computer" ? `` + : n.kind === "switch" ? `` + : ``; + const name = String(n.label).length > 26 ? String(n.label).slice(0, 25) + "…" : n.label; + out += `${esc(n.label)}${n.address ? " · " + esc(n.address) : ""}${n.reason ? " · " + esc(n.reason) : ""}${shape} + ${esc(name)} + ${n.kind === "computer" ? `${scripts == null ? "scripts: not looked yet" : scripts + " script" + (scripts === 1 ? "" : "s") + " running"}${n.self ? " · this Mac" : ""}` : ""}`; + } + svg.innerHTML = out; + } + + function wireTools() { + const el = document.getElementById("wiretools"); + if (!el) return; + const d = wire.data, count = {}; + for (const e of d.edges) count[e.type] = (count[e.type] || 0) + 1; + const bad = d.edges.filter((e) => e.light === "problem").length; + el.innerHTML = d.types.filter((t) => count[t.id]).map((t) => ``).join("") + + `` + + `` + + `` + + ``; + for (const b of el.querySelectorAll("[data-wtype]")) b.onclick = () => { const t = b.dataset.wtype; wire.types.has(t) ? wire.types.delete(t) : wire.types.add(t); wireSave(); wireTools(); wireDraw(); wireSide(); }; + el.querySelector("#wprob").onclick = () => { wire.problems = !wire.problems; wireTools(); wireDraw(); wireSide(); }; + el.querySelector("#wall").onclick = () => { wire.types.clear(); wire.hidden.clear(); wire.focus = false; wire.problems = false; wireSave(); wireTools(); wireDraw(); wireSide(); }; + el.querySelector("#wre").onclick = () => { wireLayout(true); wireFit(); wireDraw(); }; + el.querySelector("#wprobe").onclick = async (ev) => { ev.target.disabled = true; ev.target.textContent = "Looking…"; try { await post("/api/machines/probe"); } catch (e) { alertBox(e.message); } setTimeout(() => wiring([], true), 6000); }; + } + + function wireSide() { + const el = document.getElementById("wireside"); + if (!el) return; + const d = wire.data, sel = wire.sel; + const row = (e, other) => `
    • `; + let html; + if (sel && sel.edge) { + const e = d.edges.find((x) => x.id === sel.id), t = wireType(e.type); + html = `

    ${esc(t.label)}

    ${esc(wireName(e.from))} ↔ ${esc(wireName(e.to))}

    ${esc(e.label)} · ${lightDot(e.light)} ${esc(lightWord[e.light] || e.light)}

    +

    `; + } else if (sel) { + const n = wireNode(sel.id), mine = d.edges.filter((e) => e.from === n.id || e.to === n.id); + const byType = {}; + for (const e of mine) (byType[e.type] = byType[e.type] || []).push(e); + const scripts = n.kind !== "computer" ? "" : n.scripts == null + ? `

    Scripts

    Not looked yet: press Check scripts now.

    ` + : `

    Scripts ${n.scripts.filter((x) => x.running).length} running${n.scripts.some((x) => !x.running) ? `, ${n.scripts.filter((x) => !x.running).length} stopped` : ""}

    ${n.scripts.length ? `
      ${n.scripts.map((x) => `
    • ${lightDot(x.running ? "ok" : "not_needed", x.running ? "running" : (x.state || "not running"))}${esc(x.name)}${x.by && x.by !== x.name ? ` ${esc(x.by.replace(/^org\.waterschurch\./, ""))}` : ""}
      • `).join("")}
    ` : `

    None.

    `}${n.looked ? `

    looked ${esc(ago(Date.now() / 1000 - n.looked))}

    ` : ""}`; + html = `

    ${lightDot(n.light, lightWord[n.light])} ${esc(n.label)}

    ${esc(n.kind === "computer" ? (n.os === "windows" ? "Windows PC" : "Mac") + (n.self ? " · this Mac (the panel)" : "") : n.kind)}${n.address ? " · " + esc(n.address) : ""}${n.role ? " · " + esc(n.role) : ""}

    + ${n.reason ? `

    ${esc(n.reason)}

    ` : ""}${n.names && n.names.length > 1 ? `

    also called ${esc(n.names.filter((x) => x !== n.label).join(", "))}

    ` : ""} +

    ${scripts} +

    Connections ${mine.length}

    ${Object.entries(byType).map(([t, list]) => `

    ${esc(wireType(t).label)}

      ${list.map((e) => row(e, e.from === n.id ? e.to : e.from)).join("")}
    `).join("") || `

    None.

    `} + ${(n.local || []).length ? `

    On this machine itself: ${esc(n.local.map((x) => x.label).join(", "))}

    ` : ""}`; + } else { + const comps = d.nodes.filter((n) => n.kind === "computer"); + const bad = d.edges.filter((e) => e.light === "problem"); + html = `

    ${comps.length} computers, ${d.nodes.length - comps.length} devices

    ${d.edges.length} connections. Click a machine for its scripts and connections; untick a connection to hide it.

    +

    Scripts per computer

      ${comps.map((n) => `
    • ${esc(n.label)} ${n.scripts == null ? "not looked yet" : n.scripts.filter((x) => x.running).length + " running"}
      • `).join("")}
    + ${bad.length ? `

    Problems ${bad.length}

    ` : ""} + ${wire.hidden.size ? `

    Hidden ${wire.hidden.size}

      ${[...wire.hidden].map((id) => d.edges.find((e) => e.id === id)).filter(Boolean).map((e) => `
      • `).join("")}
    ` : ""}`; + } + el.innerHTML = html; + for (const c of el.querySelectorAll("[data-whide]")) c.onchange = () => { c.checked ? wire.hidden.delete(c.dataset.whide) : wire.hidden.add(c.dataset.whide); wireSave(); wireDraw(); wireSide(); }; + for (const b of el.querySelectorAll("[data-whide-one]")) b.onclick = () => { const id = b.dataset.whideOne; wire.hidden.has(id) ? wire.hidden.delete(id) : wire.hidden.add(id); wireSave(); wireDraw(); wireSide(); }; + for (const a of el.querySelectorAll("[data-wsel]")) a.onclick = (ev) => { ev.preventDefault(); wire.sel = { id: a.dataset.wsel }; wireDraw(); wireSide(); }; + for (const a of el.querySelectorAll("[data-wedge]")) a.onclick = (ev) => { ev.preventDefault(); wire.sel = { id: a.dataset.wedge, edge: true }; wireDraw(); wireSide(); }; + const f = el.querySelector("#wfocus"); if (f) f.onclick = () => { wire.focus = !wire.focus; wireDraw(); wireSide(); }; + } + + function wireEvents() { + const svg = document.getElementById("wiresvg"); + const at = (ev) => { const pt = svg.createSVGPoint(); pt.x = ev.clientX; pt.y = ev.clientY; return pt.matrixTransform(svg.getScreenCTM().inverse()); }; + document.getElementById("wzin").onclick = () => wireZoom(0.8); + document.getElementById("wzout").onclick = () => wireZoom(1.25); + document.getElementById("wfit").onclick = () => { wireFit(); wireDraw(); }; + svg.addEventListener("wheel", (ev) => { ev.preventDefault(); const p = at(ev); wireZoom(ev.deltaY > 0 ? 1.12 : 1 / 1.12, p.x, p.y); }, { passive: false }); + svg.addEventListener("pointerdown", (ev) => { + const g = ev.target.closest("[data-node]"), path = ev.target.closest("[data-edge]"); + const p = at(ev); + if (g) { const id = g.dataset.node; wire.drag = { id, dx: wire.pos[id].x - p.x, dy: wire.pos[id].y - p.y, x0: ev.clientX, y0: ev.clientY, moved: false }; } + else if (path) { wire.sel = { id: path.dataset.edge, edge: true }; wireDraw(); wireSide(); return; } + else wire.drag = { pan: true, x0: ev.clientX, y0: ev.clientY, vx: wire.view.x, vy: wire.view.y, k: wire.view.w / svg.getBoundingClientRect().width, moved: false }; + svg.setPointerCapture(ev.pointerId); ev.preventDefault(); + }); + svg.addEventListener("pointermove", (ev) => { + const dr = wire.drag; if (!dr) return; + if (!dr.moved && Math.hypot(ev.clientX - dr.x0, ev.clientY - dr.y0) < 4) return; + dr.moved = true; + if (dr.pan) { wire.view.x = dr.vx - (ev.clientX - dr.x0) * dr.k; wire.view.y = dr.vy - (ev.clientY - dr.y0) * dr.k; svg.setAttribute("viewBox", `${wire.view.x} ${wire.view.y} ${wire.view.w} ${wire.view.h}`); return; } + const p = at(ev); + wire.pos[dr.id] = { x: p.x + dr.dx, y: p.y + dr.dy }; + wireDraw(); + }); + const end = () => { + const dr = wire.drag; wire.drag = null; if (!dr) return; + if (dr.pan) { if (!dr.moved) { wire.sel = null; wire.focus = false; wireDraw(); wireSide(); } return; } + if (dr.moved) wireSave(); else { wire.sel = wire.sel && !wire.sel.edge && wire.sel.id === dr.id ? null : { id: dr.id }; if (!wire.sel) wire.focus = false; } + wireDraw(); wireSide(); + }; + svg.addEventListener("pointerup", end); svg.addEventListener("pointercancel", end); + } + // ---- Computers --------------------------------------------------------------- let compTab = "map"; async function computers(rest, quiet) { diff --git a/settings/ui/index.html b/settings/ui/index.html index bccd3cb..134bbd0 100644 --- a/settings/ui/index.html +++ b/settings/ui/index.html @@ -13,6 +13,7 @@ Overview Keys Computers + Wiring Settings Logs History diff --git a/settings/ui/style.css b/settings/ui/style.css index 6f693f3..3f5908e 100644 --- a/settings/ui/style.css +++ b/settings/ui/style.css @@ -153,3 +153,35 @@ details.script textarea { width: 100%; font-size: .8rem; } .onoff.on span { background: #1f8f4e; color: #fff; } .onoff input:disabled + span { opacity: .75; cursor: default; } tr.off td, .trig.off, .key.off { opacity: .55; } + +/* Wiring: the diagram and the panel beside it (one column on a phone) */ +.wire { display: grid; grid-template-columns: minmax(0, 1fr) 320px; gap: 12px; align-items: start; } +@media (max-width: 900px) { .wire { grid-template-columns: minmax(0, 1fr); } } +.wirecanvas { position: relative; background: var(--surface); border: 1px solid var(--line); border-radius: 10px; overflow: hidden; touch-action: none; } +.wzoom { position: absolute; top: 8px; left: 8px; display: flex; gap: 4px; z-index: 1; } .wzoom button { min-width: 30px; justify-content: center; } +.wirecanvas svg.canvas { cursor: move; } +.wirecanvas svg.canvas { width: 100%; height: calc(100vh - 175px); min-height: 420px; display: block; fill: none; stroke: none; stroke-width: 1; user-select: none; -webkit-user-select: none; } +.wedge { fill: none; stroke-width: 2.2; opacity: .85; cursor: pointer; } +.wedge:hover, .wedge.sel { stroke-width: 4.5; opacity: 1; } +.wedge.unknown { stroke-dasharray: 5 5; } .wedge.not_needed { opacity: .3; } +.wedge.problem { stroke-width: 3.4; stroke-dasharray: 9 5; animation: wflow 1s linear infinite; } +@keyframes wflow { to { stroke-dashoffset: -14; } } +.wedge.dim, .wnode.dim { opacity: .12; } +.wnode { cursor: grab; } .wnode:active { cursor: grabbing; } +.wnode rect { fill: var(--surface); stroke: var(--line); stroke-width: 1.5; } +.wnode.computer rect { stroke: var(--fg); stroke-width: 1.8; } +.wnode.switch rect { fill: var(--bg); } +.wnode.problem rect { stroke: var(--problem); stroke-width: 2.5; } +.wnode.sel rect { stroke: var(--accent); stroke-width: 3.5; } +.wnode text { fill: var(--fg); font: 600 13px var(--font); stroke: none; } +.wnode:not(.computer) text { font-weight: 500; font-size: 12px; } +.wnode text.wsub { fill: var(--muted); font: 11px var(--font); } +.wnode .wdot { fill: var(--unknown); stroke: none; } +.wnode.ok .wdot { fill: var(--ok); } .wnode.problem .wdot { fill: var(--problem); } .wnode.not_needed .wdot { fill: var(--none); } +.wtype i, .wsw { display: inline-block; width: 12px; height: 4px; border-radius: 2px; margin-right: 2px; vertical-align: middle; } +.wtype.off { opacity: .45; text-decoration: line-through; } .wtype.on { border-color: var(--line); } +.wireside { position: sticky; top: 70px; max-height: calc(100vh - 90px); overflow: auto; margin: 0; } +.wireside h2 { margin: 4px 0 6px; font-size: 1.1rem; } .wireside h3 { margin: 14px 0 4px; font-size: .95rem; } +.wlist { list-style: none; margin: 0; padding: 0; } .wlist li { padding: 3px 0; display: flex; gap: 6px; align-items: center; flex-wrap: wrap; } +.wrow { display: flex; gap: 6px; align-items: center; cursor: pointer; } .wtyp { margin: 8px 0 2px; } +main:has(.wire) { max-width: none; } diff --git a/status_panel.py b/status_panel.py index 57b0d9b..6bcc0a9 100644 --- a/status_panel.py +++ b/status_panel.py @@ -484,6 +484,25 @@ def check_launchd(chk, snap, ctx): return [("svc", Result(OK))] +def check_heartbeat(chk, snap, ctx): + """A service that writes the time into a Companion variable (the Resi auto-record: custom autorec_hb): + older than max_age_seconds is stuck. With status_var, a status containing warn_if is amber.""" + conn = chk.get("conn", "custom") + raw = snap.var(conn, chk["var"]) + if raw in (None, ""): + return [("hb", Result(OK))] # not read yet + try: + age = ctx["now"] - float(raw) + except ValueError: + age = None + if age is None or age > chk.get("max_age_seconds", 90): + return [("hb", Result(ERR, chk.get("msg", "STUCK"), named=True))] + status = (snap.var(conn, chk["status_var"]) or "") if chk.get("status_var") else "" + if chk.get("warn_if") and chk["warn_if"].upper() in status.upper(): + return [("hb", Result(WARN, chk.get("warn_msg", chk["warn_if"]), named=True))] + return [("hb", Result(OK))] + + def check_process(chk, snap, ctx): running = snap.procs.get(chk["match"]) if running is False: @@ -956,7 +975,7 @@ CHECKS = { "psm": check_psm, "tascam": check_tascam, "ping": check_ping, "launchd": check_launchd, "process": check_process, "http": check_http, "disk": check_disk, "midi_peer": check_midi_peer, "midi_wired": check_midi_wired, "pref": check_pref, "spotify": check_spotify, "mdns": check_mdns, "midi_rx": check_midi_rx, "remote": check_remote, "artnet": check_artnet, - "ready": check_ready, "ready_manual": check_ready_manual, "tune_rx": check_tune_rx, + "ready": check_ready, "ready_manual": check_ready_manual, "tune_rx": check_tune_rx, "heartbeat": check_heartbeat, } @@ -1040,6 +1059,14 @@ def needed_variables(cfg): rw = cfg.get("rec_watch") or {} if rw.get("conn"): fast.append((rw["conn"], rw.get("var", "playback_state"))) + if rw.get("heartbeat"): + fast.append((rw["conn"], rw["heartbeat"])) + for tile in all_tiles(cfg): + for chk in tile.get("checks", []): + if chk["type"] == "heartbeat": + fast.append((chk.get("conn", "custom"), chk["var"])) + if chk.get("status_var"): + fast.append((chk.get("conn", "custom"), chk["status_var"])) dedupe = lambda xs: list(dict.fromkeys(xs)) return dedupe(fast), dedupe(slow) @@ -1957,13 +1984,22 @@ class Engine: self.toggle_bypass(tid, comp, now) def resi_playing(self, snap, now): - """config "rec_watch": Resi's playback (resi-studio-cues "playback_state") has read PLAYING for - after_seconds, with the module itself connected. The decks must record then.""" + """config "rec_watch": Resi is live and the decks should record, for after_seconds. NA reads the Resi + auto-record service's own verdict (custom variable autorec_status: RECORDING / STARTING / LIVE, NOT + RECORDING; not while it holds REC until 10 minutes before a service), and nothing while its heartbeat + is stale. Without a heartbeat the connection must be up (a module's playback_state PLAYING).""" rw = self.cfg.get("rec_watch") or {} if not rw.get("conn"): return False - value = (snap.var(rw["conn"], rw.get("var", "playback_state")) or "").strip().upper() - if value != rw.get("playing", "PLAYING").upper() or conn_result(snap, rw["conn"], now).level != OK: + value = (snap.var(rw["conn"], rw.get("var", "playback_state")) or "").upper() + if rw.get("heartbeat"): + try: + fresh = now - float(snap.var(rw["conn"], rw["heartbeat"]) or 0) < rw.get("stale_seconds", 90) + except ValueError: + fresh = False + else: + fresh = conn_result(snap, rw["conn"], now).level == OK + if not fresh or not any(w.upper() in value for w in rw.get("live", ["PLAYING"])): self.resi_since = None return False self.resi_since = self.resi_since or now @@ -2865,7 +2901,11 @@ def companion_conn_hosts(path): inst = json.loads(value) except (TypeError, ValueError): continue - host = (inst.get("config") or {}).get("host") if isinstance(inst, dict) else None + conf = (inst.get("config") or {}) if isinstance(inst, dict) else {} + host = conf.get("host") + if not (isinstance(host, str) and host.strip()) and isinstance(conf.get("bonjourHost"), str): + host = conf["bonjourHost"].rsplit(":", 1)[0] if re.match(r"^[\d.]+:\d+$", conf["bonjourHost"].strip()) \ + else conf["bonjourHost"] # HyperDecks: "10.1.10.11:9993" if inst.get("label") and isinstance(host, str) and host.strip(): hosts[inst["label"]] = host.strip() return hosts diff --git a/tests/test_settings.py b/tests/test_settings.py index e022676..c490dd7 100644 --- a/tests/test_settings.py +++ b/tests/test_settings.py @@ -102,6 +102,40 @@ class TestServer(unittest.TestCase): self.assertEqual(web["light"], "ok") # from snapshot.json, not a probe self.assertIn("mac", o["setup_steps"]) + def test_wiring_shows_machines_their_connections_and_scripts(self): + # 2026-10-27: "like a wiring diagram ... how many and what scripts are running on each machine" + live = {"tracks": {"alias": "tracks", "reachable": True, "time": time.time(), "items": [], + "found": {"scripts": [{"name": "statuspanel.py", "pid": 5, "by": "", "running": True}]}}} + json.dump(live, open(os.path.join(self.data, "machines-live.json"), "w")) + snap = json.load(open(os.path.join(self.data, "snapshot.json"))) + snap.update({"conns": {"ATEM": {"category": "good", "enabled": True}, "x32": {"category": "error", "enabled": True}, + "HyperDeck_1": {"category": "good", "enabled": True}, "tcp-HyperDeck-1": {"category": "good", "enabled": True}}, + "conn_hosts": {"ATEM": "10.1.10.2", "x32": "10.1.1.112", "HyperDeck_1": "10.1.10.11", "tcp-HyperDeck-1": "10.1.10.11"}, + "remote_midi": {"tracks": {"NA Tracks": {"peers": [{"address": "10.1.13.140:5004", "name": "SuperRack-LiveBox", "state": "running"}]}}}, + "dante_map": {"NA-Tracks": {"switch": "10.66.1.100", "port": "0/6", "seen": time.time()}, + "FOH-SuperRack-LiveBox": {"switch": "10.66.1.102", "port": "0/9", "seen": time.time()}, + "STUDIO-B-MON": {"switch": "10.66.1.100", "port": "0/2", "seen": time.time()}}}) + path = os.path.join(self.data, "snapshot.json") + old = open(path).read() + try: + json.dump(snap, open(path, "w")) + w = json.loads(get(self.port, "/api/wiring")[1]) + finally: + open(path, "w").write(old) + os.remove(os.path.join(self.data, "machines-live.json")) + nodes = {n["id"]: n for n in w["nodes"]} + edges = [(e["type"], e["from"], e["to"], e["light"]) for e in w["edges"]] + self.assertEqual(w["me"], "h:ingest") + self.assertEqual(nodes["h:tracks"]["scripts"][0]["name"], "statuspanel.py") + self.assertIsNone(nodes["h:prop"]["scripts"]) # not looked yet + self.assertIn(("companion", "h:ingest", "d:10.1.1.112", "problem"), edges) + self.assertEqual(sorted(nodes["d:10.1.10.11"]["names"]), ["HyperDeck_1", "tcp-HyperDeck-1"]) # one deck, one box + self.assertIn(("midi", "h:tracks", "d:10.1.13.140", "ok"), edges) + self.assertIn(("dante", "h:tracks", "sw:10.66.1.100", "ok"), edges) # NA-Tracks is the Tracks Mac + self.assertIn("FOH-SuperRack-LiveBox", nodes["d:10.1.13.140"]["names"]) # same box as its MIDI peer + self.assertIn("d:studiobmon", nodes) # a monitor, not the Studio B Mac + self.assertEqual({t["id"] for t in w["types"]} >= {"companion", "midi", "watch", "dante"}, True) + def test_gear_and_keys_and_pages(self): gear = json.loads(get(self.port, "/api/gear")[1]) self.assertTrue(any(r["name"] == "HyperDeck_2" and "hd2" in r["watched_by"] for r in gear)) @@ -362,6 +396,25 @@ class TestBindingAndProbes(unittest.TestCase): self.assertEqual(by["sshd"]["light"], "ok") self.assertEqual((setup["profile"], setup["wake_on_lan"], setup["firewall_allow"]), ("Public", "Enabled", "10.1.1.1,10.6.3.222")) self.assertIn("\\StatusPanel\\Start Arena", found["tasks"]) + scripts = {x["name"]: x["running"] for x in found["scripts"]} + self.assertEqual((scripts["Start Vista"], scripts["Start rtpMIDI"]), (True, False)) # Vista.exe runs; rtpMIDI.exe does not + + def test_scripts_are_listed_by_file_name_only(self): + agents = {"org.waterschurch.prop-guard": 412, "org.waterschurch.status-panel": None, "com.apple.x": 9} + lines = ["412 /usr/bin/python3 /Users/m/Library/Application Support/status-panel/prop_guard.py /Users/m/cfg --token=SECRET", + "77 /bin/bash /Users/m/backup/backup-campuses.sh", "55 /Applications/Companion.app/Contents/Resources/node main.js", + "60 node /Users/m/Library/Application Support/companion/modules/x/main.js", "9 zsh -c echo SP_SCRIPTS", + "88 /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Resources/" + "Python.app/Contents/MacOS/Python /Users/m/Library/Application Support/resi-autorec/resi_autorec.py"] + got = probes.parse_scripts(lines, agents) + self.assertEqual([(x["name"], x["by"], x["running"]) for x in got], + [("backup-campuses.sh", "", True), ("prop_guard.py", "org.waterschurch.prop-guard", True), + ("resi_autorec.py", "", True), + ("status-panel", "org.waterschurch.status-panel", False)]) + self.assertNotIn("SECRET", json.dumps(got)) # never the rest of the line + win = probes.parse_scripts(['812 powershell.exe -File "C:\\ProgramData\\StatusPanel\\start-vista.ps1"'], + tasks={"\\StatusPanel\\Start Vista": "Running", "\\StatusPanel\\Start rtpMIDI": "Ready"}) + self.assertEqual([(x["name"], x["running"]) for x in win], [("Start Vista", True), ("start-vista.ps1", True), ("Start rtpMIDI", False)]) def test_probe_commands_never_carry_names_from_the_settings(self): cfg = json.loads(json.dumps(CONFIG)) diff --git a/tests/test_status_panel.py b/tests/test_status_panel.py index 9174126..95e680a 100644 --- a/tests/test_status_panel.py +++ b/tests/test_status_panel.py @@ -76,7 +76,8 @@ def healthy_snapshot(now): snap.ping = {"10.1.3.227": 0, "10.6.7.13": 0, "10.1.0.1": 0} snap.link = True snap.launchd = {"org.waterschurch.resi-propresenter": 884, "org.waterschurch.lobby-captions": 900, - "org.waterschurch.lobby-captions-display": 901, "org.waterschurch.digico-relay": 902} + "org.waterschurch.lobby-captions-display": 901, "org.waterschurch.digico-relay": 902, + "org.waterschurch.resi-autorec": 903} snap.launchd_time = now snap.procs = {"/Applications/ProdDash.app/Contents/MacOS/ProdDash": True} snap.remote = {"studiob\0pgrep -xq 'Pro Tools'": True} # the failover's check has a verdict @@ -488,12 +489,14 @@ class TestChecks(unittest.TestCase): db.execute("CREATE TABLE instances (id STRING, value STRING)") rows = [("a", {"label": "Axient_1", "config": {"host": "10.1.1.101", "port": 2202}}), ("b", {"label": "OBS-Youtube", "config": {"host": " 10.1.9.126 "}}), - ("c", {"label": "no-host", "config": {}}), ("d", "not json")] + ("c", {"label": "no-host", "config": {}}), ("d", "not json"), + ("e", {"label": "HyperDeck_1", "config": {"bonjourHost": "10.1.10.11:9993"}})] db.executemany("INSERT INTO instances VALUES (?, ?)", [(i, v if isinstance(v, str) else json.dumps(v)) for i, v in rows]) db.commit() db.close() - self.assertEqual(sp.companion_conn_hosts(path), {"Axient_1": "10.1.1.101", "OBS-Youtube": "10.1.9.126"}) + self.assertEqual(sp.companion_conn_hosts(path), {"Axient_1": "10.1.1.101", "OBS-Youtube": "10.1.9.126", + "HyperDeck_1": "10.1.10.11"}) self.assertEqual(sp.companion_conn_hosts(os.path.join(d, "missing.sqlite")), {}) def test_fx_lights_midi_on_the_tracks_mac(self): @@ -1517,43 +1520,58 @@ class TestFixes(unittest.TestCase): self.assertEqual(self.plan("rec"), [("enable", "HyperDeck_1")]) self.assertEqual(self.plan("x32"), [("enable", "x32")]) + def autorec(self, status, hb=None): + """What the Resi auto-record service publishes (custom variables).""" + self.snap.vars[("custom", "autorec_status")] = status + self.snap.vars[("custom", "autorec_hb")] = str(int(SUNDAY + 100000 if hb is None else hb)) + def test_resi_live_and_no_deck_recording_is_red_and_a_hold_starts_them(self): # 2026-10-27: "notify me if we are not recording on the hyperdecks, but resi is playing" - self.snap.vars[("resi-studio-cues", "playback_state")] = "PLAYING" + self.autorec("AUTO REC\nLIVE, NOT\nRECORDING") self.snap.vars[("HyperDeck_2", "slotId")] = "-" # optional deck, no disk: not in use self.snap.vars[("HyperDeck_2", "recordingTime")] = "--:--:--" - tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 30) - self.assertEqual(level(tiles["hd1"]), "ok") # not yet: Resi just started - tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 30, 40) + tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 10) + self.assertEqual(level(tiles["hd1"]), "ok") # not yet: give it a moment + tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 10, 30) self.assertEqual(tiles["hd1"]["x"], "HD1\nRESI LIVE NOT REC") self.assertEqual(level(tiles["rec"]), "err") self.assertEqual(level(tiles["hd2"]), "off") self.snap.vars[("HyperDeck_3", "status")] = "Record" # once one records, the others are NOT REC - tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 70, 4) + self.autorec("AUTO REC\nRECORDING\n9:30 AM") + tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 40, 4) self.assertEqual(tiles["hd1"]["x"], "HD1\nNOT REC") started = [] self.engine.record_executor = lambda ip, name: started.append((ip, name)) or True - self.engine.run_fixes("rec", self.snap, None, SUNDAY + 80) + self.engine.run_fixes("rec", self.snap, None, SUNDAY + 50) time.sleep(0.2) self.assertEqual([ip for ip, _ in started], ["10.1.10.11", "10.1.10.14", "10.1.10.15", "10.1.10.16"]) self.assertTrue(started[0][1].startswith("CAM3-") and started[0][1].endswith("-")) self.assertTrue(started[3][1].startswith("PGM4-")) self.assertEqual(self.engine.notes["rec"]["text"], "STARTING\nREC") - def test_resi_paused_or_its_module_down_is_not_live(self): - self.snap.vars[("resi-studio-cues", "playback_state")] = "PAUSED" - tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 70) + def test_not_live_while_auto_rec_waits_or_is_stuck(self): + self.autorec("AUTO REC\nREC AT\n9:20 AM") # live early: REC held until 10 min before + tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 40) self.assertEqual(level(tiles["hd1"]), "ok") - self.snap.vars[("resi-studio-cues", "playback_state")] = "PLAYING" - self.snap.conns["resi-studio-cues"]["category"] = "error" - run_steps(self.engine, self.snap, SUNDAY + 70, 70) + self.autorec("AUTO REC\nLIVE, NOT\nRECORDING", hb=SUNDAY - 600) # a stale verdict counts for nothing... + tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 40, 200) self.assertFalse(self.engine.resi_live) + self.assertEqual(tiles["ingest"]["x"], "INGEST\nAUTO REC STUCK") # ...and says the service is stuck ran = [] - self.engine.executor = lambda cfg, fix_id, chk: ran.append(chk.get("conn")) or True + self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True self.engine.record_executor = lambda ip, name: self.fail("must not record when Resi is not live") - self.engine.run_fixes("hd1", self.snap, None, SUNDAY + 150) # the usual hold: reconnect + self.engine.run_fixes("ingest", self.snap, None, SUNDAY + 250) # the hold restarts the service time.sleep(0.2) - self.assertEqual(ran, ["HyperDeck_1"]) + self.assertEqual(ran, ["autorec"]) + + def test_auto_rec_not_running_or_without_resi(self): + del self.snap.launchd["org.waterschurch.resi-autorec"] + tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 6) + self.assertEqual(tiles["ingest"]["x"], "INGEST\nAUTO REC NOT LOADED") + self.snap.launchd["org.waterschurch.resi-autorec"] = 903 + self.autorec("AUTO REC\nRESI ERROR") + tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 6, 200) # amber only once it lasts 3 minutes + self.assertEqual((tiles["ingest"]["x"], level(tiles["ingest"])), ("INGEST\nAUTO REC NO RESI", "warn")) def test_not_recording_is_not_fixed_by_a_reboot(self): for i in (1, 3):