ALARMS and AUTO-FIX as two keys; LIGHTS at Southcoast hands the lights to an operator

- kind autofix: its own switch on the auto-fix hours; ALARMS then only silences (NA keeps the old
  combined key until it has room for one)
- kind bypass: hold takes every network MIDI link to a peer out (here and on listed Macs), checks stop
  calling it lost, re-taken every 10 s; hold again puts back exactly what was taken plus every watched link
- a MIDI peer with no address is never called (it connected SC Tracks to itself)
- holds from NA's CAMPUS page flip switch keys; a rebuild keeps them; new campuses get AUTO-FIX
- key kind changes restart the watchdog; settings page has separate alarm / auto-fix buttons
- WS and SC configs: AUTO-FIX at 3,1; SC LIGHTS at 1,3; NA config synced from live

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 13:33:48 -04:00
co-authored by Claude Opus 5.5
parent 2f8e3d47a3
commit d0c755824b
15 changed files with 605 additions and 44 deletions
+4 -2
View File
@@ -68,7 +68,9 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking **and stop every auto-fix**, for example during maintenance; press it again to turn both back on. Both come back by themselves at midnight. The key says both states (`ALARMS OFF` / `AUTO-FIX OFF` / `TIL 12AM`). Holds still fix while muted; the Sunday stream-audio failover still switches by itself. The settings page's Overview has the same switch.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking; press it again to turn them back on; they come back by themselves at midnight. Holds still fix while muted; the Sunday stream-audio failover still switches by itself. The settings page's Overview has the same switch. On a page with no AUTO-FIX key (NA today), ALARMS also stops every auto-fix and the key says both (`ALARMS OFF` / `AUTO-FIX OFF` / `TIL 12AM`).
- **AUTO-FIX** (beside ALARMS on the campuses; `kind: autofix`): its own switch since 2026-10-21. On in the auto-fix hours. Press: pause it (`AUTO-FIX OFF / TIL SUN 5AM`: until the hours next start), or run it outside the hours until they next start or end; it goes back to its hours by itself. Holds always fix. With this key, ALARMS only silences. From NA's CAMPUS page a hold is the press. The Overview has a Pause auto-fix button.
- **LIGHTS** (Southcoast; `kind: bypass`, `"bypass": {"peers": ["Lighting"], "hosts": ["", "sc_tracks_new"]}`): hold 2 s to hand the lights to a lighting operator. Every network MIDI link to Lighting (ProPresenter's session, Tracks' session, and Companion's 56 Midi-Lighting buttons, which go out through ProPresenter's session) is taken out, the MIDI checks stop calling it lost and auto-fix stops putting it back; the key says `LIGHTS / OPERATOR / MIDI OFF` (steady blue). A link that comes back while the operator has it is taken out again within 10 s. Hold again: everything taken out is put back as it was, and every watched link to Lighting is reconnected (`LIGHTS / AUTO`). A peer that had called in (no address, ":5004") is not called back: it rejoins when Lighting calls again. Survives a restart; a hold from NA's CAMPUS page works too. `"conns"` can also switch Companion connections off for the operator.
## Auto-fix on Sundays (06:30–13:30)
@@ -270,7 +272,7 @@ once. Everything it makes is ordinary settings, changed on the page like North A
| Apollo Beach (standalone) | their own | their Companion Mac | 19 / 20 or the first free |
Every campus: alarms armed all day; auto-fix on NA's hours (Tue 14:00–22:00, Wed 09:00–21:00, Sun 05:00–14:00);
a hold on any key fixes it at any time, on that campus's own deck or from NA's CAMPUS page; ALARMS stops auto-fix.
a hold on any key fixes it at any time, on that campus's own deck or from NA's CAMPUS page; ALARMS silences and AUTO-FIX pauses auto-fix, separately.
`python3 -m settings.campus rebuild` (on a campus's Companion Mac, in the panel folder) makes its setup again from
what is found now, keeping its name, PIN and pages. `python3 -m settings.campus upgrade-companion` brings an older
Companion to 5.0.7 from NA's copy (settings come along; the old version stays installed).
+33
View File
@@ -266,6 +266,39 @@ if (mine.length && (stuck || (REPOINT && elsewhere))) {
"""
# Disconnect, for a Mac without python: what disconnect_peer does. Prints SPMIDI"asked" / "not connected".
JXA_DISCONNECT = r"""
ObjC.import('CoreMIDI'); ObjC.import('Foundation');
var SESSION = __SESSION__, PEER = __PEER__;
function str(o, k) { var r = Ref(); if ($.MIDIObjectGetStringProperty(o, k, r) !== 0) return null; return ObjC.unwrap(ObjC.castRefToObject(r[0])); }
function dict(o, k) { var r = Ref(); if ($.MIDIObjectGetDictionaryProperty(o, k, r) !== 0) return null; return ObjC.deepUnwrap(ObjC.castRefToObject(r[0])); }
var found = null, n = $.MIDIGetNumberOfDevices();
for (var i = 0; i < n && !found; i++) {
var dev = $.MIDIGetDevice(i), drv = str(dev, $('driver')) || '';
if (drv.indexOf('Network') < 0 && drv.toLowerCase().indexOf('rtp') < 0) continue;
for (var j = 0; j < $.MIDIDeviceGetNumberOfEntities(dev); j++) {
var ent = $.MIDIDeviceGetEntity(dev, j), s = dict(ent, $('apple.midirtp.session'));
if (s && (s['dns-sd-name'] || str(ent, $('name'))) === SESSION) { found = [ent, s]; break; }
}
}
if (!found) throw new Error('no network MIDI session named ' + SESSION + ' on this Mac');
var ent = found[0], s = found[1], peers = s.peers || [], result = 'not connected';
var left = peers.filter(function (p) { return p.name !== PEER; });
if (left.length !== peers.length) {
s.peers = left;
var e = $.MIDIObjectSetDictionaryProperty(ent, $('apple.midirtp.session'), $(s));
if (e !== 0) throw new Error('MIDIObjectSetDictionaryProperty failed: ' + e);
result = 'asked';
}
'SPMIDI' + JSON.stringify(result);
"""
def jxa_disconnect(session_name, peer_name):
import json as _json
return JXA_DISCONNECT.replace("__SESSION__", _json.dumps(session_name)).replace("__PEER__", _json.dumps(peer_name))
def jxa_connect(session_name, peer_name, address, repoint=False):
"""The JXA_CONNECT script for one peer (values go in as JSON, never as code)."""
import json as _json
+16
View File
@@ -198,9 +198,25 @@ def button(tid, tile, cfg):
press, hold = "service toggle", None
sw = cfg.get("service_windows") or []
always = len(sw) == 1 and set(sw[0].get("days", [])) == set(DAYS) and sw[0].get("start") == "00:00" and sw[0].get("end") in ("23:59", "00:00")
if any(t.get("kind") == "autofix" for t in cfg.get("tiles", [])):
note = ("Press: silence the alarms; press again to turn them back on. They come back by themselves %s. "
"Auto-fix is the AUTO-FIX key." % ("at midnight" if always else "when the alarm schedule next starts or ends"))
else:
note = ("Press: silence the alarms AND stop every auto-fix; press again to turn both back on. They come back by "
"themselves %s. Holds still fix. The stream-audio failover still switches by itself on Sundays."
% ("at midnight" if always else "when the alarm schedule next starts or ends"))
elif kind == "autofix":
stale_text = "'%s\\n?'" % label
press, hold = "autofix toggle", None
note = ("Press: pause auto-fix (alarms stay on); press again to turn it back on - or on outside its hours. It goes "
"back to its hours by itself when they next start or end. Holds always fix.")
elif kind == "bypass":
stale_text = "'%s\\n?'" % label
press, hold = "ack " + tid, "restart " + tid
b = tile.get("bypass") or {}
note = ("Hold %s s: hand %s to an operator - its MIDI links (%s) are taken out%s and the panel stops putting them "
"back. Hold again: everything is put back." % (secs, label, ", ".join(b.get("peers") or []),
" and Companion's %s switched off" % ", ".join(b["conns"]) if b.get("conns") else ""))
else:
stale_text = "'%s\\n?'" % label
press = "ack " + tid
+24 -5
View File
@@ -258,8 +258,12 @@ field("tiles[].label", "str", None, "The first line on the key.", "Keys", requir
field("tiles[].pos", "list", None, "[row, column]; absent = no key of its own (shown inside a group).", "Keys",
apply="pages")
field("tiles[].kind", "enum", None, "A special key: master (ALL), service (ALARMS), network (NET), ready (READY: "
"opens the READY page and sums up its checklist).", "Keys", advanced=True,
options=["master", "service", "network", "ready"], apply="pages")
"opens the READY page and sums up its checklist), autofix (AUTO-FIX: its own switch, so ALARMS only "
"silences), bypass (hold: hand a device to an operator - its MIDI links out, its Companion connections off).",
"Keys", advanced=True, options=["master", "service", "network", "ready", "autofix", "bypass"])
field("tiles[].bypass", "dict", None, "A bypass key: peers (MIDI peer or session names to take out, e.g. Lighting), "
"hosts (the Macs whose sessions to look in; \"\" is this Mac) and conns (Companion connections to switch off).",
"Keys", advanced=True)
field("tiles[].members", "list", None, "A group: the keys it sums up.", "Keys")
field("tiles[].checks", "list", [], "What the key watches.", "Keys")
field("tiles[].disabled", "bool", False, "Switched off: the key stays on the page (grey, OFF) but checks nothing "
@@ -491,7 +495,10 @@ GLOSSARY = {
"HOLD", "REC", "PLAYING", "OK", "OFF", "ON", "ALL OK", "WARN", "TRS", "GB", "NOT", "REACHABLE", "RECORDING",
"LEFT ALONE", "TO FIX", "TO CLEAR", "NO AUTO", "FIX", "AUTO", "IN", "COMPANION", "POWER", "CYCLE",
"STILL", "OFF IN", "DISABLED IN", "CONNECTING", "CHECK", "NETWORK", "NO", "PRESSING", "NO BUTTON THERE",
"ON FOH BACKUP",
"ON FOH BACKUP", "BUSY", "TAKING MIDI OUT", "PUTTING MIDI BACK", "TAKING", "PUTTING", "MIDI OUT", "MIDI BACK",
],
"a switch key (AUTO-FIX, or one like LIGHTS that hands a device to an operator)": [
"OPERATOR", "MIDI OFF", "OFF IN SETTINGS", "SETTINGS", "TIL",
],
}
@@ -583,7 +590,7 @@ def validate(cfg, live=None):
if errors:
return errors, warnings
ids, seen_pos = set(), {}
kinds = {"master": 0, "service": 0, "network": 0}
kinds = {"master": 0, "service": 0, "network": 0, "autofix": 0}
fixes, hosts = cfg.get("fixes", {}), cfg.get("hosts", {})
for t in cfg["tiles"]:
tid = t.get("id", "")
@@ -659,6 +666,16 @@ def validate(cfg, live=None):
errors.append("there must be exactly one ALARMS key (kind service); found %d" % kinds["service"])
if kinds["network"] > 1:
errors.append("at most one NET key (kind network); found %d" % kinds["network"])
if kinds["autofix"] > 1:
errors.append("at most one AUTO-FIX key (kind autofix); found %d" % kinds["autofix"])
for t in cfg["tiles"]:
if t.get("kind") == "bypass":
b = t.get("bypass") if isinstance(t.get("bypass"), dict) else {}
if not b.get("peers"):
errors.append("key %s: a bypass key needs bypass.peers (the MIDI peers it takes out)" % t.get("id"))
for h in b.get("hosts") or []:
if h and h not in cfg.get("hosts", {}):
errors.append("key %s: bypass host %s is not a computer in the settings" % (t.get("id"), h))
for fid, fix in fixes.items():
where = "fix %s" % fid
kind = fix.get("kind", "ssh")
@@ -917,7 +934,9 @@ def phrases(cfg):
"failover": "switches by itself %s" % sp.windows_phrase(fo_windows) if fo_windows else "never switches by itself",
"live": ("changes wait while a HyperDeck is recording" + (" or %s" % sp.windows_phrase(site["live_services"])
if site.get("live_services") else "")),
"alarms": "Alarms are on every day. The ALARMS key silences them and stops every auto-fix until midnight, then both come back by themselves.",
"alarms": ("Alarms are on every day. The ALARMS key silences them until midnight; AUTO-FIX pauses or resumes "
"auto-fix on its own. Both come back by themselves." if any(t.get("kind") == "autofix" for t in cfg.get("tiles", []))
else "Alarms are on every day. The ALARMS key silences them and stops every auto-fix until midnight, then both come back by themselves."),
"hold": "HOLD %gs" % (cfg.get("page_hold_ms", 2000) / 1000.0),
"ingest": site.get("ingest_name", "NA Ingest"),
}
+13
View File
@@ -570,6 +570,19 @@ def keep_from(new, old):
cfg = copy.deepcopy(new)
have = referenced(cfg)
kept = []
# Switch keys made for this campus by hand (Southcoast's LIGHTS) and an AUTO-FIX key: auto-setup makes neither.
for t in old.get("tiles", []):
if t.get("kind") not in ("bypass", "autofix") or t["id"] in {x["id"] for x in cfg["tiles"]}:
continue
if t["kind"] == "autofix" and any(x.get("kind") == "autofix" for x in cfg["tiles"]):
continue
used = {tuple(x["pos"]) for x in cfg["tiles"] if x.get("pos")} | (
{tuple(cfg["restart_page"]["jump_pos"])} if cfg.get("restart_page") else set())
if t.get("pos") and tuple(t["pos"]) not in used:
cfg["tiles"].append(copy.deepcopy(t))
else:
place_new(cfg, {k: v for k, v in copy.deepcopy(t).items() if k != "pos"})
kept.append(t["label"])
# Computers in the old setup that did not answer this time, and the keys that check them.
gone = {a: h for a, h in (old.get("hosts") or {}).items() if not h.get("self") and a not in cfg["hosts"]}
for alias, h in gone.items():
+8
View File
@@ -163,6 +163,14 @@ def key_guide(cfg):
parts.append("<p>Shows the worst of: %s. Hold: %s.</p>" % (", ".join(e(tiles[m]["label"]) for m in t["members"]), hold_words(t, cfg)))
if t.get("kind") == "service":
parts.append("<p>%s</p>" % e(ph["alarms"]))
if t.get("kind") == "autofix":
parts.append("<p>Press: pause auto-fix, or run it outside its hours (auto-fix %s). It goes back to its hours by itself "
"when they next start or end. Holds always fix.</p>" % e(ph["auto_fix"]))
if t.get("kind") == "bypass":
b = t.get("bypass") or {}
parts.append("<p>Hold: hand it to an operator - the MIDI links to %s are taken out%s, and the panel stops putting them "
"back. Hold again: everything is put back as it was.</p>" % (
e(", ".join(b.get("peers") or [])), (" and Companion's %s is switched off" % e(", ".join(b["conns"]))) if b.get("conns") else ""))
if t.get("kind") == "network":
n = cfg.get("network", {})
parts.append("<p>Watches this Mac's network port%s, and calls it a blip when %d connections from %d groups drop within %d s "
+2 -1
View File
@@ -276,6 +276,7 @@ class Site:
"live": live, "running": running, "service_on": snap.get("service_on") if fresh else None,
"alarms_override": snap.get("alarms_override") if fresh else None,
"auto_fix": {"phrase": schema.phrases(cfg)["auto_fix"], "state": snap.get("auto_fix")},
"bypass": (snap.get("bypass") or {}) if fresh else {},
"problems": sorted(problems, key=lambda p: (p["level"] != "err", p["since"] or 0)), "needs": needs,
"computers": self.computer_lines(snap), "pages": pages, "snapshot_age": (now - snap["time"]) if snap.get("time") else None,
"last_change": self.last_change(), "phrases": schema.phrases(cfg), "version": started.get("version"),
@@ -1030,7 +1031,7 @@ class Handler(http.server.BaseHTTPRequestHandler):
if path == "/api/command":
text = str(body.get("text", "")).strip()
verb = text.split(" ")[0].lower() if text else ""
if verb not in ("test", "resurvey", "service"):
if verb not in ("test", "resurvey", "service", "autofix"):
return self.send(400, {"error": "not a command the page may send"})
if verb == "test":
live, reason = site.applier.live()
+7 -2
View File
@@ -272,6 +272,7 @@
const live = isLive ? `<div class="notice">Live now: ${esc(o.live.reason)}. Changes wait.</div>` : "";
const comps = o.computers.map((c) => `<tr class="click" onclick="location.hash='#computers/${esc(c.alias)}'"><td>${wordOf(c.word)}</td><td>${esc(c.name)}</td><td class="muted small">${esc(c.reason)}</td></tr>`).join("");
const w = o.watchdog;
const af = (o.auto_fix && o.auto_fix.state) || {};
const dis = editing() ? "" : "disabled";
const chip = (ok, text, hintText) => `<span class="chip ${ok === true ? "ok" : ok === false ? "bad" : ""}" data-hint="${esc(hintText)}">${text}</span>`;
main.innerHTML = `
@@ -281,7 +282,8 @@
${chip(w.running, w.running ? "Watchdog ✓" : "Watchdog down", (w.running ? "Running since " + when(w.since) : "Not running") + (w.version ? " · code " + w.version : "") + (o.heartbeat_age != null ? " · heartbeat " + ago(o.heartbeat_age) : ""))}
${chip(o.companion_ok, o.companion_ok ? "Companion ✓" : "Companion down", o.pages && o.pages.status ? (o.pages.status.ok ? "Pages OK" : "Page " + o.pages.status.page + " is called " + (o.pages.status.live || "?")) : "")}
${chip(o.service_on == null ? null : o.service_on, o.service_on == null ? "Alarms ?" : o.service_on ? "Alarms on" : "Alarms silenced", o.phrases.alarms)}
${chip(null, "Auto-fix " + (o.auto_fix.state && o.auto_fix.state.on ? "active" : "waiting"), o.auto_fix.phrase + (o.auto_fix.state && o.auto_fix.state.paused ? " (waiting out a network blip)" : ""))}
${chip(null, "Auto-fix " + (af.override === false ? "paused" : af.on ? "active" : "waiting"), o.auto_fix.phrase + (af.override === false ? " (paused with the AUTO-FIX key)" : af.paused ? " (waiting out a network blip)" : ""))}
${Object.keys(o.bypass || {}).length ? chip(null, Object.keys(o.bypass).map((k) => ((cfg().tiles.find((t) => t.id === k) || {}).label || k)).join(", ") + ": operator", "Handed to an operator with its key: its MIDI is out until the key is held again") : ""}
${Object.keys(o.running || {}).length ? chip(null, "Fix running", Object.values(o.running).map((r) => r.what + " on " + r.key).join(", ")) : ""}
${o.last_change ? `<a class="chip" href="#history" data-hint="${esc(o.last_change.when + " · " + (o.last_change.by || "") + " · " + (o.last_change.note || ""))}">Last change ${esc(o.last_change.when || "")}</a>` : ""}
</div>
@@ -289,7 +291,9 @@
<div>${deckHtml(o.deck, cfg(), "status")}
<div class="row" style="margin-top:10px">
<button id="restartwd" ${dis} data-hint="${w.running ? "Restart the watchdog: the deck goes purple for about 5 s; a running fix finishes first" : "Start the watchdog"}">${w.running ? "Restart watchdog" : "Start watchdog"}</button>
<button id="alarms" ${dis} data-hint="${o.service_on === false ? "Alarms and auto-fix back on now" : "Same as the ALARMS key: no alarms and no auto-fix until midnight"}">${o.service_on === false ? "Alarms + auto-fix on" : "Silence + stop auto-fix"}</button>
${af.split ? `<button id="alarms" ${dis} data-hint="${o.service_on === false ? "Alarms back on now" : "Same as the ALARMS key: no alarms until midnight (auto-fix carries on)"}">${o.service_on === false ? "Alarms on" : "Silence alarms"}</button>
<button id="autofixbtn" ${dis} data-hint="${af.switch ? "Same as the AUTO-FIX key: nothing is fixed by itself until the hours next start or end (holds still fix)" : "Same as the AUTO-FIX key: gentle fixes run by themselves until the hours next start or end"}">${af.switch ? "Pause auto-fix" : "Auto-fix on"}</button>`
: `<button id="alarms" ${dis} data-hint="${o.service_on === false ? "Alarms and auto-fix back on now" : "Same as the ALARMS key: no alarms and no auto-fix until midnight"}">${o.service_on === false ? "Alarms + auto-fix on" : "Silence + stop auto-fix"}</button>`}
${w.running ? `<button id="stopwd" ${dis} data-hint="Stop the watchdog: every key purple, no alarms until it is started again">Stop</button>` : ""}
</div></div>
<div><div class="tablewrap"><table>${comps}</table></div></div>
@@ -297,6 +301,7 @@
main.querySelector("#restartwd").onclick = () => confirmBox(w.running ? "Restart the watchdog? The deck goes purple for about 5 seconds; a fix that is running finishes first." : "Start the watchdog?", w.running ? "Restart" : "Start", async () => { try { const r = await post(w.running ? "/api/watchdog/restart" : "/api/watchdog/start"); alertBox(r.ok ? "Done: " + r.detail : "Not done: " + r.detail); } catch (e) { alertBox(e.message); } });
const stop = main.querySelector("#stopwd"); if (stop) stop.onclick = () => confirmBox("Stop the watchdog? The deck goes purple and every alarm is off until it is started again.", "Stop", async () => { try { const r = await post("/api/watchdog/stop", { word: "STOP" }); alertBox(r.ok ? "Stopped." : "Not stopped: " + r.detail); } catch (e) { alertBox(e.message); } }, "STOP");
main.querySelector("#alarms").onclick = async () => { try { await post("/api/command", { text: o.service_on === false ? "service on" : "service off" }); setTimeout(() => overview(_, true), 1500); } catch (e) { alertBox(e.message); } };
const afb = main.querySelector("#autofixbtn"); if (afb) afb.onclick = async () => { try { await post("/api/command", { text: af.switch ? "autofix off" : "autofix on" }); setTimeout(() => overview(_, true), 1500); } catch (e) { alertBox(e.message); } };
for (const b of main.querySelectorAll("[data-run]")) b.onclick = () => runNow(...b.dataset.run.split("|"));
if (!quiet) window.scrollTo(0, 0);
}
+9
View File
@@ -48,6 +48,15 @@
],
"kind": "service"
},
{
"id": "autofix",
"label": "AUTO-FIX",
"pos": [
3,
1
],
"kind": "autofix"
},
{
"id": "net",
"label": "NET",
+99 -1
View File
@@ -63,7 +63,8 @@
"propresenter_ingest",
"prop_port_propresenter_api",
"hyperdeck",
"midi_reconnect"
"midi_reconnect",
"ableton_sc_tracks_new"
]
},
"tiles": [
@@ -216,6 +217,55 @@
1,
1
]
},
{
"id": "ableton",
"label": "ABLETON",
"checks": [
{
"type": "remote",
"template": "mac_program",
"params": {
"host": "sc_tracks_new",
"process": "Live"
},
"msg": "ABLETON NOT RUNNING",
"fix": "ableton_sc_tracks_new",
"host": "sc_tracks_new",
"cmd": "pgrep -xq Live"
}
],
"pos": [
1,
2
]
},
{
"id": "autofix",
"label": "AUTO-FIX",
"kind": "autofix",
"pos": [
3,
1
]
},
{
"id": "lights",
"label": "LIGHTS",
"kind": "bypass",
"pos": [
1,
3
],
"bypass": {
"peers": [
"Lighting"
],
"hosts": [
"",
"sc_tracks_new"
]
}
}
],
"fixes": {
@@ -271,6 +321,34 @@
"params": {},
"short": "RECONNECT",
"label": "reconnect the peer to the network MIDI session it dropped out of (like Connect in Audio MIDI Setup), at its wired address"
},
"ableton_sc_tracks_new": {
"template": "ableton",
"params": {
"host": "sc_tracks_new",
"app": "Ableton Live 11 Suite",
"restart": false,
"wait_seconds": 15
},
"label": "start Ableton on SC Tracks NEW if it is not running",
"short": "ABLETON",
"kind": "ssh",
"host": "sc_tracks_new",
"run": "pgrep -xq Live || { L=$(ls -td ~/Library/Preferences/Ableton/Live*/ | head -1); S=$(grep -a 'info: Loading document \"' \"${L}Log.txt\" | grep -v '/Templates/' | tail -1 | sed -E 's/^.*Loading document \"(.*)\"[^\"]*$/\\1/'); if [ -n \"$S\" ] && [ -f \"$S\" ]; then open -a 'Ableton Live 11 Suite' \"$S\"; else open -a 'Ableton Live 11 Suite'; fi; }"
},
"restart_ableton_sc_tracks_new": {
"template": "ableton",
"params": {
"host": "sc_tracks_new",
"app": "Ableton Live 11 Suite",
"restart": true,
"wait_seconds": 15
},
"label": "restart Ableton on SC Tracks NEW (closes it first)",
"short": "ABLETON",
"kind": "ssh",
"host": "sc_tracks_new",
"run": "L=$(ls -td ~/Library/Preferences/Ableton/Live*/ | head -1); S=$(grep -a 'info: Loading document \"' \"${L}Log.txt\" | grep -v '/Templates/' | tail -1 | sed -E 's/^.*Loading document \"(.*)\"[^\"]*$/\\1/'); pkill -TERM -x Live; i=0; while pgrep -xq Live && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq Live && pkill -KILL -x Live; sleep 1; if [ -n \"$S\" ] && [ -f \"$S\" ]; then open -a 'Ableton Live 11 Suite' \"$S\"; else open -a 'Ableton Live 11 Suite'; fi"
}
},
"hosts": {
@@ -298,6 +376,17 @@
"http": "http://127.0.0.1:8000/api/connections"
}
]
},
"sc_tracks_new": {
"name": "SC Tracks NEW",
"os": "mac",
"ssh": "tracks@10.5.3.72",
"role": "found by auto-setup",
"runs": [
{
"kind": "sshd"
}
]
}
},
"gear": {},
@@ -338,6 +427,15 @@
0,
1
]
},
{
"id": "rs_ableton",
"label": "ABLETON",
"fix": "restart_ableton_sc_tracks_new",
"pos": [
0,
2
]
}
]
},
+9
View File
@@ -575,6 +575,15 @@
1,
3
]
},
{
"id": "autofix",
"label": "AUTO-FIX",
"kind": "autofix",
"pos": [
3,
1
]
}
],
"timing": {
+230 -23
View File
@@ -494,6 +494,8 @@ def check_midi_peer(chk, snap, ctx):
sessions = None # that Mac stopped reporting: no verdict
if sessions is None:
return [("peer", Result(OK))]
if chk["peer"] in ctx.get("bypassed", ()) or chk["session"] in ctx.get("bypassed", ()):
return [("peer", Result(OK))] # an operator has it (LIGHTS key): taken out on purpose
sess = sessions.get(chk["session"])
if sess is None:
return [("peer", Result(ERR, "NO SESSION"))]
@@ -508,6 +510,101 @@ def check_midi_peer(chk, snap, ctx):
return [("peer", Result(ERR, "MIDI LOST"))]
def midi_on(cfg, host, call, jxa):
"""Run a coremidi call on another Mac (python there, else the JavaScript for Automation `jxa`):
(result, error text)."""
r = subprocess.run(ssh_cmd(cfg, host) + remote_python("-"), input=remote_midi_script(call),
capture_output=True, text=True, timeout=30)
if r.returncode == 97: # no python on that Mac
r = subprocess.run(ssh_cmd(cfg, host) + ["/usr/bin/osascript -l JavaScript -"], input=jxa,
capture_output=True, text=True, timeout=30)
result = parse_remote_midi(r.stdout) if r.returncode == 0 else None
return result, (None if result is not None else (r.stdout + r.stderr).strip().replace("\n", " ")[-200:])
def midi_peer(cfg, host, action, session, peer, address=None, repoint=False):
"""Connect ("connect") or drop ("disconnect") one peer of a network MIDI session, on this Mac
(host "") or another: (result word or None, error text)."""
import coremidi
if action == "connect" and not str(address or "").rsplit(":", 1)[0].strip("[]"):
# A peer that called in keeps no address (":5004"): adding it back connects the session to
# itself (SC Tracks, 2026-10-21). It comes back when it calls again.
return None, "no address to call (%r): it reconnects when %s calls in" % (address, peer)
if not host:
if action == "connect":
return coremidi.connect_peer(session, peer, address, repoint=repoint), None
return coremidi.disconnect_peer(session, peer), None
if action == "connect":
return midi_on(cfg, host, "ns['connect_peer'](%r, %r, %r, repoint=%r)" % (session, peer, address, repoint),
coremidi.jxa_connect(session, peer, address, repoint=repoint))
return midi_on(cfg, host, "ns['disconnect_peer'](%r, %r)" % (session, peer), coremidi.jxa_disconnect(session, peer))
def midi_sessions(cfg, host):
"""The network MIDI sessions on this Mac (host "") or another, or None."""
import coremidi
if not host:
return coremidi.sessions()
return midi_on(cfg, host, "ns['sessions']()", coremidi.JXA_SESSIONS)[0]
def bypass_apply(cfg, tile, on, entry, comp, names_ips, lock=None):
"""Hand a bypass key's device to an operator, or take it back. `entry["dropped"]` keeps every peer
taken out ({host, session, peer, address}), so taking it back puts exactly those back; the MIDI
checks naming it are reconnected too. Its Companion connections are switched off, then on again."""
b = tile.get("bypass") or {}
names, ips = set(b.get("peers") or []), set(names_ips[1]) if on else set()
hosts = b.get("hosts") or [""] + sorted({c["host"] for t in cfg["tiles"] for c in t.get("checks", [])
if c.get("type") == "midi_peer" and c.get("peer") in names and c.get("host")})
if not on:
for c in [x for t in cfg["tiles"] for x in t.get("checks", [])
if x.get("type") == "midi_peer" and x.get("peer") in names]:
ip = peer_ip(c.get("address"))
if ip:
ips.add(str(ip))
conns = {c["label"]: c for c in comp.connections()} if b.get("conns") else {}
for label in b.get("conns") or []:
conn = conns.get(label)
if not conn:
log.info("BYPASS %s: Companion has no connection %s", tile["label"], label)
elif on and conn.get("enabled"):
log.info("BYPASS %s: Companion %s off -> %s", tile["label"], label, comp.disable_connection(conn["id"]))
elif not on and not conn.get("enabled"):
log.info("BYPASS %s: Companion %s on -> %s", tile["label"], label, comp.enable_connection(conn["id"]))
if on:
for host in hosts:
try:
sessions = midi_sessions(cfg, host)
except Exception as e:
sessions, err = None, str(e)
if sessions is None:
continue # that Mac is off or unreachable: tried again in 10 s
for sname, sess in sessions.items():
for p in sess.get("peers") or []:
ip = peer_ip(p.get("address"))
if p.get("name") in names or (ip and str(ip) in ips):
res, err = midi_peer(cfg, host, "disconnect", sname, p["name"])
log.info("BYPASS %s: %s out of %s%s -> %s", tile["label"], p["name"], sname,
" on " + host if host else "", res or err)
if res:
rec = {"host": host, "session": sname, "peer": p["name"], "address": p.get("address") or ""}
with (lock or threading.Lock()):
if rec not in entry["dropped"]:
entry["dropped"].append(rec)
return
done = set()
for rec in entry.get("dropped") or []:
res, err = midi_peer(cfg, rec["host"], "connect", rec["session"], rec["peer"], rec["address"])
done.add((rec["host"], rec["session"], rec["peer"]))
log.info("BYPASS %s: %s back into %s%s -> %s", tile["label"], rec["peer"], rec["session"],
" on " + rec["host"] if rec["host"] else "", res or err)
for t in cfg["tiles"]:
for c in t.get("checks", []):
if c.get("type") == "midi_peer" and c.get("peer") in names and (c.get("host", ""), c["session"], c["peer"]) not in done:
res, err = midi_peer(cfg, c.get("host", ""), "connect", c["session"], c["peer"], c["address"])
log.info("BYPASS %s: %s back into %s -> %s", tile["label"], c["peer"], c["session"], res or err)
def peer_ip(address):
"""The IPv4 address of a MIDI peer ("10.1.1.1:5040"), or None for a Bonjour name or IPv6."""
host = str(address or "").strip().rsplit(":", 1)[0]
@@ -967,10 +1064,11 @@ class ServiceMode:
edge is worked out at the press and kept, so the override ends there even when the watchdog
was not running at that minute (a restart over 23:59 left the alarms muted for good)."""
def __init__(self, windows, state):
def __init__(self, windows, state, name="service"):
self.windows = windows
self.name = name # "service" (ALARMS) or "autofix" (AUTO-FIX)
self.state = state if isinstance(state, State) else State(state)
saved = self.state.get("service") or {}
saved = self.state.get(name) or {}
self.override, self.last_sched = saved.get("override"), saved.get("last_sched")
self.edge = saved.get("edge")
@@ -1027,7 +1125,18 @@ class ServiceMode:
self._save()
def _save(self):
self.state.set("service", {"override": self.override, "last_sched": self.last_sched, "edge": self.edge})
self.state.set(self.name, {"override": self.override, "last_sched": self.last_sched, "edge": self.edge})
def until(self, now):
"""When the state shown now next changes by itself (an override's end, else the schedule's next edge),
looked up once a minute."""
minute = int(now // 60)
cached = getattr(self, "_until", None)
if cached and cached[0] == (minute, self.override):
return cached[1]
found = self.back_at(now) if self.override is not None else self.next_change(now)
self._until = ((minute, self.override), found)
return found
# --- Engine ------------------------------------------------------------------
@@ -1050,6 +1159,16 @@ class Engine:
self.hold_ms = cfg.get("page_hold_ms", 2000) # how long a key is held to run its fix
self.state = State(state_path)
self.service = ServiceMode(cfg.get("service_windows", []), self.state)
# With an AUTO-FIX key, ALARMS and auto-fix are two switches (2026-10-21, the user: "seperatly
# silence alarms and auto fixes"); without one, ALARMS stops both, as before.
self.split = any(t.get("kind") == "autofix" for t in cfg["tiles"])
self.autofix = ServiceMode((cfg.get("auto_fix") or {}).get("windows", []), self.state, "autofix")
# Keys that hand a device to an operator (Southcoast's LIGHTS: a lighting op runs the desk, so
# nothing may send it MIDI): tile id -> {"since", "dropped": [peers taken out, to put back]}.
self.bypass = {k: v for k, v in (self.state.get("bypass") or {}).items() if k in self.tiles}
self.bypass_lock = threading.Lock()
self.bypass_busy = False
self.bypass_last = 0.0
self.data_dir = data_dir
self.stopping = False # SIGTERM arrived: finish what runs, start nothing new
self.running = {} # fix thread name -> {"key", "what", "since"} (published as fixes-running.json)
@@ -1184,7 +1303,8 @@ class Engine:
# raw results for every check key
def evaluate(self, snap, now, service_on):
ctx = {"now": now, "service": service_on, "any_recording": False, "dante_all": {},
"ready_due": in_windows(self.ready_cfg.get("due", []), now), "ready_done": self.ready_done(now)}
"ready_due": in_windows(self.ready_cfg.get("due", []), now), "ready_done": self.ready_done(now),
"bypassed": self.bypass_names()[0]}
for t in all_tiles(self.cfg):
for chk in t.get("checks", []):
if chk["type"] == "dante":
@@ -1500,11 +1620,77 @@ class Engine:
"""Show `text` on the button for a while (hold feedback), instead of its usual text."""
self.notes[tile_id] = {"text": text, "level": level, "until": now + seconds}
# --- operator bypass (a LIGHTS key) ---------------------------------------------------------
def bypass_names(self):
"""(peer and session names an operator has, their IPv4 addresses as the MIDI checks know them)."""
names, ips = set(), set()
for tid in self.bypass:
names |= set(((self.tiles.get(tid) or {}).get("bypass") or {}).get("peers") or [])
if names:
for t in self.cfg["tiles"]:
for c in t.get("checks", []):
if c.get("type") == "midi_peer" and (c.get("peer") in names):
ip = peer_ip(c.get("address"))
if ip:
ips.add(str(ip))
return names, ips
def toggle_bypass(self, tile_id, comp, now):
"""A hold on a bypass key: hand it to an operator (take its MIDI links out, switch its Companion
connections off), or take it back (put everything back, as it was)."""
if self.bypass_busy:
self.note(tile_id, "BUSY\n" + self.wait_text, "idle", now, 4)
return
tile = self.tiles[tile_id]
with self.bypass_lock:
if tile_id in self.bypass:
entry = self.bypass.pop(tile_id)
on = False
else:
entry = self.bypass[tile_id] = {"since": now, "dropped": []}
on = True
self.state.set("bypass", self.bypass)
log.info("BYPASS %s: %s", tile["label"], "handed to an operator" if on else "back to the panel")
self.note(tile_id, "TAKING\nMIDI OUT" if on else "PUTTING\nMIDI BACK", "fixing", now, 6)
self.bypass_last = now
self.start_bypass_job(tile_id, on, entry, comp)
def start_bypass_job(self, tile_id, on, entry, comp):
self.bypass_busy = True
def job():
try:
bypass_apply(self.cfg, self.tiles[tile_id], on, entry, comp, self.bypass_names(), self.bypass_lock)
if tile_id in self.bypass:
self.state.set("bypass", self.bypass) # the peers taken out, to put back later
except Exception:
log.exception("bypass %s failed", tile_id)
finally:
self.bypass_busy = False
threading.Thread(target=job, name="bypass-" + tile_id, daemon=True).start()
def bypass_tick(self, comp, now):
"""While an operator has it, every 10 s: a link that came back (rtpMIDI asking again) goes out again."""
if not self.bypass or self.bypass_busy or self.stopping or now - self.bypass_last < 10:
return
self.bypass_last = now
tid = next(iter(self.bypass))
self.start_bypass_job(tid, True, self.bypass[tid], comp)
def run_fixes(self, tile_id, snap, comp, now, cooldown=None):
cooldown = self.cooldown if cooldown is None else cooldown
if self.stopping:
self.note(tile_id, "RESTARTING\nSOON", "idle", now, 4)
return []
kind = self.tiles.get(tile_id, {}).get("kind")
if kind in ("service", "autofix"): # from North Attleboro's CAMPUS page: the hold is the press
mode = self.service if kind == "service" else self.autofix
mode.set(not mode.on(now), now)
log.info("%s %s (hold)", "SERVICE" if kind == "service" else "AUTO-FIX", "ON" if mode.on(now) else "OFF")
return []
if kind == "bypass":
self.toggle_bypass(tile_id, comp, now)
return []
if self.tiles.get(tile_id, {}).get("kind") == "network":
# NET: a hold clears a remembered network blip (link and gateway clear themselves).
if self.burst is None:
@@ -1631,6 +1817,12 @@ class Engine:
if not a or now - self.auto_last < 5 or self.stopping:
return []
self.auto_last = now
if self.split:
# The AUTO-FIX key: paused, nothing is fixed by itself (holds still work); pressed outside
# the hours, the gentle fixes run until the hours next start or end.
hours = self.autofix.on(now)
muted = self.autofix.override is False
else:
# The ALARMS key is the on-the-fly switch: muted, nothing is fixed by itself at all (holds
# still work). It comes back by itself with the alarms (2026-10-19: the user's choice).
muted = not self.service.on(now)
@@ -1641,7 +1833,7 @@ class Engine:
and not self.cfg["fixes"].get(f, {}).get("never_auto") and not self.cfg["fixes"].get(f, {}).get("disabled")}
if muted:
if self.autolog and (self.auto_on or (anytime and not self.auto_muted)):
self.autolog.write(now, "PANEL", "auto-fix off (ALARMS muted)")
self.autolog.write(now, "PANEL", "auto-fix off (%s)" % ("AUTO-FIX key" if self.split else "ALARMS muted"))
self.auto_on, self.auto_muted = False, True
self.auto_tries.clear()
self.auto_seen.clear()
@@ -2070,12 +2262,30 @@ def render(engine, service_on):
# they come back by themselves after a press.
word = "ON" if service_on else "OFF"
fix_word = "OFF" if (not service_on or (engine.cfg.get("auto_fix") or {}).get("disabled")) else "ON"
text = "%s %s\nAUTO-FIX %s" % (tile["label"], word, fix_word)
text = ("%s\n%s" % (tile["label"], word)) if engine.split else "%s %s\nAUTO-FIX %s" % (tile["label"], word, fix_word)
back = engine.service.back_at(engine.now)
if back:
text += "\nTIL " + clock12(back)
out[tid] = colored(text, "service" if service_on else "off")
continue
if kind == "autofix":
# Its own switch: on in the auto-fix hours, a press pauses it (or runs it outside them).
if (engine.cfg.get("auto_fix") or {}).get("disabled"):
out[tid] = colored(tile["label"] + "\nOFF IN\nSETTINGS", "off")
continue
on = engine.autofix.on(engine.now)
until = engine.autofix.until(engine.now)
text = "%s\n%s" % (tile["label"], "ON" if on else "OFF")
if until and (on or engine.autofix.override is not None):
same_day = datetime.fromtimestamp(until).date() == datetime.fromtimestamp(engine.now).date()
text += "\nTIL " + ("" if same_day else datetime.fromtimestamp(until).strftime("%a ").upper()) + clock12(until)
out[tid] = colored(text, "service" if on else "off")
continue
if kind == "bypass":
# Handed to an operator: steady blue, says so; otherwise the panel runs it.
on = tid in engine.bypass
out[tid] = colored("%s\n%s" % (tile["label"], "OPERATOR\nMIDI OFF" if on else "AUTO"), "service" if on else "ok")
continue
if kind == "master":
# Idle faults are expected outside service; they do not stop "ALL OK".
items = [("ok" if lv == "idle" else lv, res, since, engine.tiles[t]) for t in engine.detail_ids
@@ -2856,24 +3066,11 @@ def execute_fix(cfg, fix_id, chk):
return propresenter_port(cfg, fix, label)
if kind == "midi_connect":
# Like Connect in Audio MIDI Setup: add the peer back to the session it dropped out of.
args = (chk["session"], chk["peer"], chk["address"])
repoint = bool(chk.get("repoint"))
if chk.get("host"):
r = subprocess.run(ssh_cmd(cfg, chk["host"]) + remote_python("-"),
input=remote_midi_script("ns['connect_peer'](%r, %r, %r, repoint=%r)" % (args + (repoint,))),
capture_output=True, text=True, timeout=30)
if r.returncode == 97: # no python on that Mac: the same, in JavaScript for Automation
import coremidi
r = subprocess.run(ssh_cmd(cfg, chk["host"]) + ["/usr/bin/osascript -l JavaScript -"],
input=coremidi.jxa_connect(*args, repoint=repoint), capture_output=True, text=True, timeout=30)
result = parse_remote_midi(r.stdout) if r.returncode == 0 else None
result, err = midi_peer(cfg, chk.get("host", ""), "connect", chk["session"], chk["peer"], chk["address"],
repoint=bool(chk.get("repoint")))
if result is None:
log.info("FIX %s -> %s on %s failed: %s", label, chk["peer"], chk["host"],
(r.stdout + r.stderr).strip().replace("\n", " ")[-200:])
log.info("FIX %s -> %s on %s failed: %s", label, chk["peer"], chk.get("host"), err)
return False
else:
import coremidi
result = coremidi.connect_peer(*args, repoint=repoint)
log.info("FIX %s -> %s at %s, session %s%s: %s", label, chk["peer"], chk["address"], chk["session"],
(" on " + chk["host"]) if chk.get("host") else "", result)
return result in ("asked", "already", "kicked", "moved")
@@ -3401,7 +3598,9 @@ def snapshot_summary(snap, engine, now):
for name, l in snap.listeners.items()},
"service_on": engine.service.on(now),
"alarms_override": engine.service.override,
"auto_fix": {"on": engine.auto_on, "paused": engine.auto_paused},
"auto_fix": {"on": engine.auto_on, "paused": engine.auto_paused, "split": engine.split,
"switch": engine.autofix.on(now), "override": engine.autofix.override},
"bypass": {tid: {"since": b.get("since")} for tid, b in engine.bypass.items()},
"failover": dict(engine.failed),
"burst": engine.burst and {"start": engine.burst["start"], "conns": sorted(engine.burst["conns"]),
"cleared": engine.burst["cleared"]},
@@ -3446,6 +3645,10 @@ def handle_commands(text, engine, comp, snap, now):
target = {"on": True, "off": False}.get(arg, not engine.service.on(now))
engine.service.set(target, now)
log.info("SERVICE %s (button)", "ON" if target else "OFF")
elif verb == "autofix":
target = {"on": True, "off": False}.get(arg, not engine.autofix.on(now))
engine.autofix.set(target, now)
log.info("AUTO-FIX %s (button)", "ON" if target else "OFF")
elif verb in ("restart", "force"):
# Holds carry Companion's clock ("force rs_x 1791496225"). One pressed while the watchdog
# was down must not fire minutes later in a burst with the others.
@@ -3540,6 +3743,10 @@ def run(cfg, data_dir, dry_run=False, stop=None, config_sha=None):
engine.auto_fix(snap, comp, now)
except Exception: # a bug here must never take the panel down
log.exception("auto-fix failed; the panel carries on without it this round")
try:
engine.bypass_tick(comp, now)
except Exception:
log.exception("operator bypass failed; the panel carries on without it this round")
try:
engine.failover(snap, comp, now)
except Exception:
+15
View File
@@ -531,3 +531,18 @@ class TestStandalone(unittest.TestCase):
self.assertIn("--standalone", inst)
self.assertNotIn("10.1.1.1", inst)
self.assertNotIn("authorized_keys", inst) # no one else's key is let in
class TestSwitchKeys(unittest.TestCase):
def test_a_new_campus_gets_auto_fix_by_alarms_and_a_rebuild_keeps_its_own_switches(self):
new, _ = ac.build(BASE, small(), {"id": "sc", "name": "Southcoast"}, "1234")
af = [t for t in new["tiles"] if t.get("kind") == "autofix"]
self.assertEqual([t["pos"] for t in af], [[3, 1]]) # beside ALARMS
self.assertEqual(schema.validate(new)[0], [])
old = json.loads(json.dumps(new))
old["tiles"].append({"id": "lights", "label": "LIGHTS", "kind": "bypass", "pos": [2, 7], "bypass": {"peers": ["Lighting"]}})
again, _ = ac.build(BASE, small(), {"id": "sc", "name": "Southcoast"}, "1234")
kept_cfg, kept = ac.keep_from(again, old)
lights = [t for t in kept_cfg["tiles"] if t["id"] == "lights"]
self.assertEqual((len(lights), lights[0]["pos"], "LIGHTS" in kept), (1, [2, 7], True))
self.assertEqual(len([t for t in kept_cfg["tiles"] if t.get("kind") == "autofix"]), 1)
+1 -1
View File
@@ -85,7 +85,7 @@ class TestSites(unittest.TestCase):
errors, _ = schema.validate(cfg)
self.assertEqual(errors, [])
make_page.build(cfg)
self.assertEqual(sorted(t["kind"] for t in cfg["tiles"]), ["master", "network", "service"])
self.assertEqual(sorted(t["kind"] for t in cfg["tiles"]), ["autofix", "master", "network", "service"])
self.assertEqual([a for a, h in cfg["hosts"].items() if h.get("self")], ["ingest"])
self.assertEqual(cfg["hosts"]["ingest"]["networks"], [])
self.assertNotIn("Attleboro", json.dumps(cfg))
+126
View File
@@ -2451,3 +2451,129 @@ class TestDanteNewcomers(unittest.TestCase):
self.assertFalse(sp.watch_dante_newcomers(self.snap, CONFIG, self.comp, self.engine, now + 100))
self.engine.any_recording = False
self.assertTrue(sp.watch_dante_newcomers(self.snap, CONFIG, self.comp, self.engine, now + 200))
class TestSwitches(unittest.TestCase):
"""ALARMS and AUTO-FIX as two keys (2026-10-21), and a key that hands a device to an operator
(Southcoast's LIGHTS: a lighting op runs the desk, so nothing may send it MIDI)."""
def setUp(self):
self.snap = healthy_snapshot(SUNDAY)
self.cfg = json.loads(json.dumps(CONFIG))
arena = next(t for t in self.cfg["tiles"] if t.get("pos") == [3, 7])
arena.pop("pos") # NA's page is full: make room
self.cfg["tiles"].append({"id": "autofix", "label": "AUTO-FIX", "kind": "autofix", "pos": [3, 7]})
self.midi = next(c for t in self.cfg["tiles"] for c in t.get("checks", []) if c["type"] == "midi_peer")
self.cfg["tiles"].append({"id": "lights", "label": "LIGHTS", "kind": "bypass",
"bypass": {"peers": [self.midi["peer"]], "hosts": [""], "conns": ["x32"]}})
self.state = os.path.join(tempfile.mkdtemp(), "state.json")
self.engine = self.new_engine()
def new_engine(self):
e = sp.Engine(self.cfg, self.state)
e.executor = e.poe_executor = lambda *a: True
e.autolog = sp.AutoLog(tempfile.mkdtemp())
return e
def test_alarms_silence_only_and_auto_fix_has_its_own_key(self):
import schema
self.assertEqual(schema.validate(self.cfg)[0], [])
ran = []
self.engine.executor = lambda cfg, fix_id, chk: ran.append(fix_id) or True
self.snap.remote["tracks\0pgrep -xq Live"] = False # Ableton closed
tiles, _ = run_steps(self.engine, self.snap, SUNDAY, 4)
self.assertEqual(tiles["service"]["x"], "SERVICE\nON")
self.assertEqual(tiles["autofix"]["x"], "AUTO-FIX\nON\nTIL 1:30PM") # Sunday's hours end then
self.engine.service.set(False, SUNDAY + 5) # ALARMS: silence only
run_steps(self.engine, self.snap, SUNDAY + 5, 25)
self.engine.auto_fix(self.snap, mock_comp(), SUNDAY + 30)
time.sleep(0.2)
self.assertEqual(ran, ["ableton"]) # auto-fix carried on
self.engine.autofix.set(False, SUNDAY + 40) # AUTO-FIX pressed: paused
tiles = self.engine.step(self.snap, SUNDAY + 41)
self.assertEqual(tiles["autofix"]["x"], "AUTO-FIX\nOFF\nTIL WED 4PM") # back on with Wednesday's hours
self.assertEqual(level(tiles["autofix"]), "off")
self.assertEqual(self.engine.auto_fix(self.snap, mock_comp(), SUNDAY + 200), [])
self.assertFalse(self.new_engine().autofix.on(SUNDAY + 300)) # survives a restart
self.assertTrue(self.engine.autofix.on(datetime(2026, 10, 14, 16, 30).timestamp())) # Wednesday's hours: back by itself
weekday = SUNDAY - 3 * 86400 # Thursday: pressed = runs anyway
self.engine.autofix.set(True, weekday)
self.assertTrue(self.engine.autofix.on(weekday + 60))
page = make_page.build(self.cfg)["page"]["controls"]
notes = {(ctl.get("options") or {}).get("notes", "") for row in page.values() for ctl in row.values() if ctl}
self.assertTrue(any("Auto-fix is the AUTO-FIX key" in n for n in notes))
self.assertTrue(any("pause auto-fix" in n for n in notes))
def test_a_hold_from_north_attleboro_flips_a_switch(self):
self.engine.run_fixes("autofix", self.snap, mock_comp(), SUNDAY)
self.assertFalse(self.engine.autofix.on(SUNDAY + 1))
self.engine.run_fixes("autofix", self.snap, mock_comp(), SUNDAY + 2)
self.assertTrue(self.engine.autofix.on(SUNDAY + 3))
def test_the_operator_key_takes_the_midi_out_and_puts_it_back(self):
calls = []
here = {self.midi["session"]: {"port": 5004, "peers": [{"name": self.midi["peer"], "address": self.midi["address"]},
{"name": "Other", "address": "10.9.9.9:5004"}]}}
real = (sp.midi_sessions, sp.midi_peer)
sp.midi_sessions = lambda cfg, host: here
sp.midi_peer = lambda cfg, host, action, session, peer, address=None, repoint=False: calls.append((action, host, session, peer)) or ("asked", None)
class Comp:
def __init__(self):
self.on = True
self.log = []
def connections(self):
return [{"label": "x32", "id": "id-x32", "enabled": self.on}]
def disable_connection(self, cid):
self.on = False; self.log.append("off"); return True
def enable_connection(self, cid):
self.on = True; self.log.append("on"); return True
comp = Comp()
try:
if self.midi.get("host"):
self.snap.remote_midi[self.midi["host"]] = {} # its Mac reports: no peers
self.snap.remote_midi_time[self.midi["host"]] = SUNDAY + 100
else:
self.snap.midi, self.snap.midi_time = {self.midi["session"]: {"port": 5004, "peers": []}}, SUNDAY + 100
tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 100, 3)
self.assertEqual(level(tiles["lights"]), "ok")
self.engine.run_fixes("lights", self.snap, comp, SUNDAY + 104) # hold: an operator has it
for _ in range(50):
if not self.engine.bypass_busy:
break
time.sleep(0.05)
self.assertEqual(calls, [("disconnect", "", self.midi["session"], self.midi["peer"])]) # never "Other"
self.assertEqual(comp.log, ["off"])
tiles, _ = run_steps(self.engine, self.snap, SUNDAY + 105, 3)
midi_tile = next(t["id"] for t in self.cfg["tiles"] if self.midi in t.get("checks", []))
tag = self.midi.get("tag") or self.midi["peer"]
self.assertNotIn(tag + " MIDI", tiles[midi_tile]["x"]) # taken out on purpose: no alarm
self.assertEqual(tiles["lights"]["x"], "LIGHTS\nTAKING\nMIDI OUT")
tiles = self.engine.step(self.snap, SUNDAY + 120)
self.assertEqual(tiles["lights"]["x"], "LIGHTS\nOPERATOR\nMIDI OFF")
self.assertIn("lights", self.new_engine().bypass) # survives a restart
calls.clear()
self.engine.run_fixes("lights", self.snap, comp, SUNDAY + 130) # hold again: put back
for _ in range(50):
if not self.engine.bypass_busy:
break
time.sleep(0.05)
self.assertEqual(calls[0], ("connect", "", self.midi["session"], self.midi["peer"])) # what was taken out
self.assertEqual({c[0] for c in calls}, {"connect"}) # and every link the checks watch
self.assertEqual(comp.log, ["off", "on"])
self.assertNotIn("lights", self.engine.bypass)
finally:
sp.midi_sessions, sp.midi_peer = real
def test_a_peer_without_an_address_is_never_called(self):
res, why = sp.midi_peer(self.cfg, "", "connect", "SC Tracks NEW", "Lighting", ":5004")
self.assertIsNone(res)
self.assertIn("calls in", why)
def test_a_bypass_key_needs_its_peers(self):
import schema
bad = json.loads(json.dumps(self.cfg))
next(t for t in bad["tiles"] if t["id"] == "lights")["bypass"] = {"hosts": ["nowhere"]}
errors = schema.validate(bad)[0]
self.assertTrue(any("bypass.peers" in e for e in errors))
self.assertTrue(any("nowhere" in e for e in errors))