christian/status-indicators
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ALARMS key stops every auto-fix too, and says so

Browse Source 
Muted, nothing fixes itself at all: no Spotify exception (while_muted is gone) and no any-hour
READY fix; holds still fix and the Sunday stream-audio failover still switches. The key reads
ALARMS OFF / AUTO-FIX OFF / TIL 12AM, and its note says both. The settings page's switch says the same.

The mute's end is worked out at the press and kept: a watchdog that was not running at 23:59 no
longer leaves the alarms muted until the next schedule change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-19 14:46:20 -04:00
co-authored by Claude Opus 5.5
parent 5d918d60af
commit b41b8a49d2
8 changed files with 121 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -3
View File
@@ -68,7 +68,7 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking **and stop every auto-fix**, for example during maintenance; press it again to turn both back on. Both come back by themselves at midnight. The key says both states (`ALARMS OFF` / `AUTO-FIX OFF` / `TIL 12AM`). Holds still fix while muted; the Sunday stream-audio failover still switches by itself. The settings page's Overview has the same switch.
## Auto-fix on Sundays (06:30–13:30)
@@ -78,8 +78,8 @@ During that window, with ALARMS on, the panel fixes a problem by itself once it
- It waits while the network is having a blip, and does nothing to Dante when several devices vanish at once (that is NA Ingest's view or the Dante network, not the Macs).
- Never automatic: restarting a running ProPresenter, Bible app, ProdCom (or DVS on NA Ingest, which restarts ProdCom), captions or the bridge, power-cycling a switch port, or turning on a connection someone switched off. `load_config` refuses an auto fix that closes a program.
- **Daily log:** everything auto-fix does is written as it happens to `~/Desktop/Auto-Fix Logs/<date> <weekday> Auto-Fix.log` on NA Ingest (e.g. `2026-10-11 Sunday Auto-Fix.log`): the problem, what it did, whether it worked, when the fault cleared, when it gave up or held back. Double-click it to open it in Console, which shows new lines live. One file per day; each is deleted a calendar month after its date (the 11 Oct file on 11 Nov).
- Press ALARMS to mute the panel for maintenance: that also stops auto-fix. The window and the list are `"auto_fix"` in `config.json`.
- **READY page fixes** ticked under Settings › Auto-fix act on the checklist's gaps too, amber or red. A fix marked `"anytime": true` (only a READY fix may be; today the LTC write, `ready_ltc`) runs at any hour, not only in the auto-fix hours: once the weekend's set is on Tracks, every song's timecode is written into Planning Center for all four campuses by itself, even while ALARMS is muted (nothing in the room notices a note in Planning Center).
- **Stopping it:** press ALARMS (on the fly; nothing at all fixes itself until you press it again or midnight), or switch auto-fix off as a whole under Settings › Auto-fix (stays off until switched on). The window and the list are `"auto_fix"` in `config.json`.
- **READY page fixes** ticked under Settings › Auto-fix act on the checklist's gaps too, amber or red. A fix marked `"anytime": true` (only a READY fix may be; today the LTC write, `ready_ltc`) runs at any hour, not only in the auto-fix hours: once the weekend's set is on Tracks, every song's timecode is written into Planning Center for all four campuses by itself. ALARMS muted stops it like every other auto-fix; it catches up once they are back on.
## Audio failover (Studio B -> FOH Backup)
config.json
-3
View File
@@ -33,9 +33,6 @@
"pause_after_network_seconds": 60,
"restart_connections": true,
"retry_seconds": 90,
"while_muted": [
"spotify_play"
],
"windows": [
{
"days": [
make_page.py
+6 -2
View File
@@ -29,7 +29,7 @@ import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
from status_panel import RESTARTABLE, load_config, ready_keys # noqa: E402
from status_panel import DAYS, RESTARTABLE, load_config, ready_keys # noqa: E402
# Defaults; "companion", "page_grid", "page_stale_seconds" and "page_hold_ms" in config.json override them.
EXPORT_VERSION = 12 # Companion 5.0.7's export format
@@ -196,7 +196,11 @@ def button(tid, tile, cfg):
elif kind == "service":
stale_text = "'%s\\n?'" % label
press, hold = "service toggle", None
note = "Press: turn service mode on/off until the next scheduled window starts or ends."
sw = cfg.get("service_windows") or []
always = len(sw) == 1 and set(sw[0].get("days", [])) == set(DAYS) and sw[0].get("start") == "00:00" and sw[0].get("end") in ("23:59", "00:00")
note = ("Press: silence the alarms AND stop every auto-fix; press again to turn both back on. They come back by "
"themselves %s. Holds still fix. The stream-audio failover still switches by itself on Sundays."
% ("at midnight" if always else "when the alarm schedule next starts or ends"))
else:
stale_text = "'%s\\n?'" % label
press = "ack " + tid
schema.py
+3 -8
View File
@@ -90,7 +90,7 @@ field("page_hold_ms", "int", 2000, "How long a key is held to run its fix.", "Sc
# -- schedules and timing --
field("service_windows", "list", [{"days": list(sp.DAYS), "start": "00:00", "end": "23:59"}],
"When alarms are armed: always (the ALARMS key silences them until midnight).", "Schedules", advanced=True)
"When alarms are armed: always (the ALARMS key silences them and stops auto-fix until midnight).", "Schedules", advanced=True)
field("service_windows[].days", "days", None, "", "Schedules", required=True)
field("service_windows[].start", "hhmm", None, "", "Schedules", required=True)
field("service_windows[].end", "hhmm", None, "", "Schedules", required=True)
@@ -111,8 +111,6 @@ field("auto_fix.windows", "list", [], "When auto-fix may act.", "Schedules")
field("auto_fix.windows[].days", "days", None, "", "Schedules", required=True)
field("auto_fix.windows[].start", "hhmm", None, "", "Schedules", required=True)
field("auto_fix.windows[].end", "hhmm", None, "", "Schedules", required=True)
field("auto_fix.while_muted", "list", [], "Auto-fixes that keep running inside the window even while ALARMS is "
"muted (harmless ones, like playing the Spotify playlist).", "Schedules")
field("auto_fix.disabled", "bool", False, "Auto-fix switched off as a whole: nothing is fixed by itself (holds "
"still work).", "Schedules")
field("auto_fix.after_seconds", "int", 20, "A fault must last this long before auto-fix acts.", "Schedules",
@@ -218,7 +216,7 @@ field("fixes.*.reconnect", "bool", False, "Reconnect the Companion connection af
field("fixes.*.conn", "str", None, "The Companion connection it reconnects.", "Fixes")
field("fixes.*.restart_ok", "bool", False, "May close a running program (and still run by itself).", "Fixes")
field("fixes.*.never_auto", "bool", False, "Never runs by itself.", "Fixes")
field("fixes.*.anytime", "bool", False, "Ticked for auto-fix, runs by itself at any hour, not only in the auto-fix hours, muted or not (a READY fix that only writes to Planning Center).", "Fixes")
field("fixes.*.anytime", "bool", False, "Ticked for auto-fix, runs by itself at any hour, not only in the auto-fix hours, unless ALARMS is muted (a READY fix that only writes to Planning Center).", "Fixes")
field("fixes.*.part", "enum", None, "Which part of the switcher.", "Fixes", options=["audio", "camera"])
field("fixes.*.verify_seconds", "int", 20, "Wait this long before judging a reconnect.", "Fixes", advanced=True,
lo=5, hi=120)
@@ -713,9 +711,6 @@ def validate(cfg, live=None):
errors.append("auto-fix: %s closes a running program; tick 'may close a running program' or take it out" % f)
elif fixes[f].get("failover"):
warnings.append("auto-fix: %s is a failover - it already runs by itself as one" % f)
for f in a.get("while_muted", []):
if f not in a.get("fixes", []):
errors.append("auto-fix: %s runs while ALARMS is muted but is not an auto-fix" % f)
_windows(cfg.get("service_windows"), "alarms window", errors)
_windows(a.get("windows"), "auto-fix window", errors)
_windows((cfg.get("failover") or {}).get("windows"), "failover window", errors)
@@ -881,7 +876,7 @@ def phrases(cfg):
"failover": "switches by itself %s" % sp.windows_phrase(fo_windows) if fo_windows else "never switches by itself",
"live": ("changes wait while a HyperDeck is recording" + (" or %s" % sp.windows_phrase(site["live_services"])
if site.get("live_services") else "")),
"alarms": "Alarms are on every day. The ALARMS key silences them until midnight, then they come back by themselves.",
"alarms": "Alarms are on every day. The ALARMS key silences them and stops every auto-fix until midnight, then both come back by themselves.",
"hold": "HOLD %gs" % (cfg.get("page_hold_ms", 2000) / 1000.0),
"ingest": site.get("ingest_name", "NA Ingest"),
}
settings/ui/app.js
+1 -1
View File
@@ -289,7 +289,7 @@
<div>${deckHtml(o.deck, cfg(), "status")}
<div class="row" style="margin-top:10px">
<button id="restartwd" ${dis} data-hint="${w.running ? "Restart the watchdog: the deck goes purple for about 5 s; a running fix finishes first" : "Start the watchdog"}">${w.running ? "Restart watchdog" : "Start watchdog"}</button>
<button id="alarms" ${dis} data-hint="${o.service_on === false ? "Alarms back on now" : "Same as the ALARMS key: quiet until midnight"}">${o.service_on === false ? "Alarms on" : "Silence alarms"}</button>
<button id="alarms" ${dis} data-hint="${o.service_on === false ? "Alarms and auto-fix back on now" : "Same as the ALARMS key: no alarms and no auto-fix until midnight"}">${o.service_on === false ? "Alarms + auto-fix on" : "Silence + stop auto-fix"}</button>
${w.running ? `<button id="stopwd" ${dis} data-hint="Stop the watchdog: every key purple, no alarms until it is started again">Stop</button>` : ""}
</div></div>
<div><div class="tablewrap"><table>${comps}</table></div></div>
status_panel.py
+78 -20
View File
@@ -814,6 +814,13 @@ def in_windows(windows, now):
return any(window_holds(w, day, hm, DAYS[dt.weekday() - 1]) for w in windows)
def clock12(t):
"""12AM, 6:30PM: a time on a key, the way people say it."""
dt = datetime.fromtimestamp(t)
h = dt.hour % 12 or 12
return ("%d" % h if dt.minute == 0 else "%d:%02d" % (h, dt.minute)) + ("AM" if dt.hour < 12 else "PM")
def windows_phrase(windows):
"""Windows in words, for log headers and notes: "on Sundays 06:30-13:30"."""
parts = []
@@ -886,32 +893,71 @@ class State:
class ServiceMode:
"""On inside a scheduled window; a button press overrides until the next window edge."""
"""On inside a scheduled window; a button press overrides until the next window edge. The
edge is worked out at the press and kept, so the override ends there even when the watchdog
was not running at that minute (a restart over 23:59 left the alarms muted for good)."""
def __init__(self, windows, state):
self.windows = windows
self.state = state if isinstance(state, State) else State(state)
saved = self.state.get("service") or {}
self.override, self.last_sched = saved.get("override"), saved.get("last_sched")
self.edge = saved.get("edge")
def scheduled(self, now):
return in_windows(self.windows, now)
def next_change(self, now):
"""The first minute after `now` whose schedule differs from now's, or None (within 8 days)."""
sched, t = self.scheduled(now), int(now // 60) * 60 + 60
while t < now + 8 * 86400:
if self.scheduled(t) != sched:
return t
t += 60
return None
def on(self, now):
sched = self.scheduled(now)
if sched != self.last_sched:
self.last_sched, self.override = sched, None
if self.override is not None and self.edge is None:
self.edge = self.next_change(now) # an override saved before edges were kept
self._save()
if sched != self.last_sched or (self.override is not None and self.edge and now >= self.edge):
self.last_sched, self.override, self.edge = sched, None, None
self._save()
return self.override if self.override is not None else sched
def back_at(self, now):
"""When a manual override (ALARMS pressed) ends by itself, as a timestamp, or None. The
override holds until the schedule next changes; from then on the schedule rules, so the
state flips there, or at the change after (at NA: cleared 23:59, alarms back 00:00).
Looked up once a minute."""
if self.override is None:
return None
minute = int(now // 60)
cached = getattr(self, "_back", None)
if cached and cached[0] == (minute, self.override):
return cached[1]
sched, t, found = self.scheduled(now), minute * 60 + 60, None
while t < now + 8 * 86400:
s = self.scheduled(t)
if s != sched:
sched = s
if s != self.override:
found = t
break
t += 60
self._back = ((minute, self.override), found)
return found
def set(self, value, now):
sched = self.scheduled(now)
self.last_sched = sched
self.override = None if value == sched else value
self.edge = self.next_change(now) if self.override is not None else None
self._save()
def _save(self):
self.state.set("service", {"override": self.override, "last_sched": self.last_sched})
self.state.set("service", {"override": self.override, "last_sched": self.last_sched, "edge": self.edge})
# --- Engine ------------------------------------------------------------------
@@ -984,7 +1030,7 @@ class Engine:
self.auto_watch = {} # "BUTTON FAULT" auto-fix went after -> when (to log when it clears)
self.auto_paused = False
self.auto_on = False # inside the window (for the "auto-fix on" line)
self.auto_muted = False # ...with ALARMS muted, so only the "while_muted" fixes run
self.auto_muted = False # ALARMS muted last round (nothing fixes itself then)
self.auto_last = 0.0
self.deck_ips = {} # HyperDeck connection -> last address Companion reported
self.now = 0.0
@@ -1498,18 +1544,27 @@ class Engine:
if not a or now - self.auto_last < 5 or self.stopping:
return []
self.auto_last = now
# ALARMS muted (maintenance) stops auto-fix, except the harmless fixes in "while_muted"
# (playing the Spotify playlist): those keep running inside the window.
# The ALARMS key is the on-the-fly switch: muted, nothing is fixed by itself at all (holds
# still work). It comes back by itself with the alarms (2026-10-19: the user's choice).
muted = not self.service.on(now)
hours = in_windows(a.get("windows", []), now)
# A fix marked "anytime" (writing the timecode into Planning Center: nothing in the room
# notices) runs by itself at any hour, not only in the auto-fix hours, muted or not.
# A fix marked "anytime" (writing the timecode into Planning Center) runs by itself at any
# hour, not only in the auto-fix hours - but never while muted, like every other.
anytime = {f for f in a.get("fixes", []) if self.cfg["fixes"].get(f, {}).get("anytime")
and not self.cfg["fixes"].get(f, {}).get("never_auto") and not self.cfg["fixes"].get(f, {}).get("disabled")}
keep = (set(a.get("while_muted", [])) | anytime) if muted else None
if not hours or (muted and not keep):
if muted:
if self.autolog and (self.auto_on or (anytime and not self.auto_muted)):
self.autolog.write(now, "PANEL", "auto-fix off (ALARMS muted)")
self.auto_on, self.auto_muted = False, True
self.auto_tries.clear()
self.auto_seen.clear()
return []
if self.auto_muted and self.autolog and not hours and anytime:
self.autolog.write(now, "PANEL", "ALARMS back on: the any-hour fixes (%s) run again" % ", ".join(sorted(anytime)))
self.auto_muted = False
if not hours:
if self.auto_on and self.autolog:
self.autolog.write(now, "PANEL", "auto-fix off (%s)" % ("ALARMS muted" if hours else "end of the window"))
self.autolog.write(now, "PANEL", "auto-fix off (end of the window)")
self.auto_on = False
for ident in [i for i in self.auto_tries if not (i[0] == "fix" and i[1] in anytime)]:
del self.auto_tries[ident] # the window's fixes start fresh next time
@@ -1517,11 +1572,10 @@ class Engine:
if not anytime:
return []
else:
if (not self.auto_on or self.auto_muted != muted) and self.autolog: # opens today's file, so it can be watched from the start
if not self.auto_on and self.autolog: # opens today's file, so it can be watched from the start
ends = [w["end"] for w in a.get("windows", []) if in_windows([w], now)]
self.autolog.write(now, "PANEL", "auto-fix on%s%s" % (" until " + ends[0] if ends else "",
" (ALARMS muted: only %s)" % ", ".join(sorted(keep)) if muted else ""))
self.auto_on, self.auto_muted = True, muted
self.autolog.write(now, "PANEL", "auto-fix on%s" % (" until " + ends[0] if ends else ""))
self.auto_on = True
net_down = any(self.trackers.get("net:" + k) and self.trackers["net:" + k].confirmed
and self.trackers["net:" + k].result.level == ERR for k in ("link", "gateway"))
if net_down or (self.burst and (self.burst["cleared"] is None
@@ -1560,8 +1614,6 @@ class Engine:
for ident, kind, what, chk in self.fix_plan(t["id"], snap, now, a.get("after_seconds", 20), True, levels):
if not ((kind == "fix" and what in gentle) or (kind == "restart" and restarts)):
continue # not gentle: stays a hold
if keep is not None and not (kind == "fix" and what in keep):
continue # ALARMS muted: only the "while_muted" fixes
self.auto_seen[ident] = now
tries, last = self.auto_tries.get(ident, (0, 0.0))
if tries >= a.get("max_tries", 2):
@@ -1926,8 +1978,14 @@ def render(engine, service_on):
out[tid] = colored(tile["label"] + "\n" + note["text"], note["level"])
continue
if kind == "service":
manual = engine.service.override is not None
text = tile["label"] + "\n" + ("ON" if service_on else "OFF") + ("\nmanual" if manual else "")
# One key, two switches: ALARMS muted also stops every auto-fix. Says both, and when
# they come back by themselves after a press.
word = "ON" if service_on else "OFF"
fix_word = "OFF" if (not service_on or (engine.cfg.get("auto_fix") or {}).get("disabled")) else "ON"
text = "%s %s\nAUTO-FIX %s" % (tile["label"], word, fix_word)
back = engine.service.back_at(engine.now)
if back:
text += "\nTIL " + clock12(back)
out[tid] = colored(text, "service" if service_on else "off")
continue
if kind == "master":
tests/test_readiness.py
+9 -6
View File
@@ -522,8 +522,8 @@ class TestFixes(unittest.TestCase):
self.assertFalse(any("/clips/" in path for _, path in calls)) # never plays a clip by itself
class TestSpotifyWhileMuted(unittest.TestCase):
def test_spotify_still_plays_with_alarms_muted(self):
class TestSpotifyAndMute(unittest.TestCase):
def test_alarms_muted_stops_spotify_too(self):
cfg = json.loads(json.dumps(CONFIG))
cfg["auto_fix"]["log_dir"] = __import__("tempfile").mkdtemp(prefix="sp-autolog-")
engine, snap = sp.Engine(cfg, None), sp.Snapshot()
@@ -538,14 +538,17 @@ class TestSpotifyWhileMuted(unittest.TestCase):
engine.auto_fix(snap, None, wed + 40)
for th in engine.fix_threads:
th.join(2)
self.assertEqual(ran, ["spotify_play"])
cfg["auto_fix"]["while_muted"] = [] # without it: nothing while muted
self.assertEqual(ran, []) # muted: nothing, not even Spotify
engine2 = sp.Engine(cfg, None)
engine2.service.set(False, wed)
engine2.service.set(True, wed) # alarms on: it plays
engine2.executor = engine.executor
snap.remote = {}
for t in range(0, 40, 2):
engine2.step(snap, wed + t)
self.assertEqual(engine2.auto_fix(snap, None, wed + 40), [])
engine2.auto_fix(snap, None, wed + 40)
for th in engine2.fix_threads:
th.join(2)
self.assertEqual(ran, ["spotify_play"])
if __name__ == "__main__":
tests/test_status_panel.py
+21 -4
View File
@@ -610,7 +610,7 @@ class TestBehaviour(unittest.TestCase):
self.assertEqual(level(tiles["x32"]), "quiet")
self.assertEqual(tiles["all"]["x"], "1 DOWN\nX32 OFFLINE")
self.assertEqual(level(tiles["all"]), "quiet")
self.assertEqual(tiles["service"]["x"], "SERVICE\nOFF")
self.assertEqual(tiles["service"]["x"], "SERVICE OFF\nAUTO-FIX OFF")
for t in tiles.values():
self.assertEqual((t["b"], t["f"]), (t["b2"], t["f2"]), t["x"]) # no blinking anywhere
@@ -994,11 +994,12 @@ class TestFixes(unittest.TestCase):
time.sleep(0.2)
self.assertEqual((done, ran), (["rd_timecode"], ["ready_ltc"])) # never "ableton"
self.assertEqual(self.engine.auto_fix(self.snap, comp, thursday + 60), []) # 90 s between tries
self.engine.service.set(False, thursday + 61) # ALARMS muted: it still writes
self.assertEqual(self.engine.auto_fix(self.snap, comp, thursday + 200), ["rd_timecode"])
self.engine.service.set(False, thursday + 61) # ALARMS muted: nothing at all
self.assertEqual(self.engine.auto_fix(self.snap, comp, thursday + 200), [])
self.engine.service.set(True, thursday + 201) # back on: it catches up
self.assertEqual(self.engine.auto_fix(self.snap, comp, thursday + 210), ["rd_timecode"])
time.sleep(0.2)
self.assertEqual(ran, ["ready_ltc", "ready_ltc"]) # and still never "ableton"
self.engine.service.set(True, thursday + 201)
del cfg["fixes"]["ready_ltc"]["anytime"] # not marked: the window only
self.engine = sp.Engine(cfg)
self.engine.autolog = sp.AutoLog(tempfile.mkdtemp())
@@ -1016,6 +1017,22 @@ class TestFixes(unittest.TestCase):
bad["fixes"]["ableton"]["anytime"] = True
self.assertTrue(any("anytime" in e for e in schema.validate(bad)[0])) # only a READY fix may
def test_alarms_key_says_alarms_and_auto_fix_and_when_they_come_back(self):
cfg = json.loads(json.dumps(CONFIG))
cfg["service_windows"] = [{"days": list(sp.DAYS), "start": "00:00", "end": "23:59"}] # NA: armed all day
engine = sp.Engine(cfg)
monday = datetime(2026, 10, 19, 14, 30).timestamp()
tiles = engine.step(self.snap, monday)
self.assertEqual(tiles["service"]["x"], "SERVICE ON\nAUTO-FIX ON")
engine.service.set(False, monday) # pressed
tiles = engine.step(self.snap, monday + 1)
self.assertEqual(tiles["service"]["x"], "SERVICE OFF\nAUTO-FIX OFF\nTIL 12AM")
self.assertTrue(engine.service.on(datetime(2026, 10, 20, 0, 0, 30).timestamp())) # back by itself
note = [ctl for row in make_page.build(cfg)["page"]["controls"].values() for ctl in row.values()
if ctl and "'service'" in (ctl.get("options") or {}).get("notes", "")][0]["options"]["notes"]
self.assertIn("stop every auto-fix", note)
self.assertIn("at midnight", note)
def test_restart_keys_flash_when_their_thing_is_down(self):
devices = json.loads(self.snap.vars[("audinate-dantecontroller", "devices")])
devices.remove("NA-Tracks") # DVS on the Tracks Mac gone