christian/status-indicators
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

False alarms between services at NA: RESI AUDIO SILENT only while recording; TRACKS MIDI waits 2 min

Browse Source 
2026-10-25 09:00 between services NA showed 2 DOWN while everything was fine: the Resi send is quiet
between services (the silence check now judges only while a HyperDeck records), and the NA Tracks
Bonjour advert blips for 10-70 s (24 times on record) while the real MIDI links stay up (its check now
waits 120 s). Applied on NA between services together with the queued RESI VOX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-25 09:06:55 -04:00
co-authored by Claude Opus 5.5
parent b6221e0d63
commit c471ac2c92
5 changed files with 24 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -253,7 +253,7 @@ The **page number key** on every page follows the ALL button. It's green when ev
| X32, ARENA, OBS | Companion connection |
| DIGICO (inside AUDIO) | Companion's `DiGiCo_FOH` connection; the relay helper `org.waterschurch.digico-relay` on NA Ingest (`~/digico-bridge/relay.py`: Companion and the DiGiCo tablets share the console's one External Control slot through it), **DIGICO RELAY STOPPED** / **NOT LOADED**, hold AUDIO restarts it; the console itself answers ping at 10.1.60.100, **DIGICO NO PING**. Companion's OSC connection reads "good" even with the console off, so the ping is what catches that. |
| STB AUDIO (inside AUDIO) | the Studio B audio source for the stream: Studio B answers ping (10.1.13.175, every second, **STUDIO B NO PING** after 2 misses); Pro Tools running on the Studio B Mac Studio (over SSH every 2 s, **PRO TOOLS NOT RUNNING**); the Dante device STUDIO-B-RedNet-TNX (**STB TNX GONE**). Hold 2 s presses FOH Backup; on Sundays that press happens by itself (see Audio failover). |
| ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). **RESI AUDIO SILENT**: input 16 (Resi - RX1, `atem.silence`) has read the meter floor (-100 dB, nothing arriving at all) for 30 s, in auto-fix's hours or while a HyperDeck records; a quiet room or a pause is far above that (2026-10-25: the Resi AJA passed no Dante audio and the panel stayed green, because DANTE only sees that the card is on the network). What differs (audio, camera, silence) is written to the log when it changes. `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. |
| ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). **RESI AUDIO SILENT**: input 16 (Resi - RX1, `atem.silence`) has read the meter floor (-100 dB, nothing arriving at all) for 30 s while a HyperDeck records (a service is on; between services the Resi send is quiet on purpose); a quiet room or a pause is far above that (2026-10-25: the Resi AJA passed no Dante audio and the panel stayed green, because DANTE only sees that the card is on the network). What differs (audio, camera, silence) is written to the log when it changes. `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". |
| MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. |
config.json
+2 -1
View File
@@ -2225,7 +2225,8 @@
{
"instance": "NA Tracks",
"name": "TRACKS MIDI",
"type": "mdns"
"type": "mdns",
"debounce_seconds": 120
},
{
"name": "TRACKS IN",
schema.py
+2 -2
View File
@@ -188,8 +188,8 @@ field("atem.reference", "path", "atem-reference.xml", "The saved switcher file t
field("atem.compare", "dict", None, "What to keep in line: audio fields and camera settings; blank = everything.",
"ATEM", advanced=True)
field("atem.silence", "dict", None, "Inputs whose audio must never be silent, metered on the switcher: "
"{\"inputs\": {\"16\": \"Resi - RX1\"}, \"seconds\": 30}. A silent one turns its check red in auto-fix's "
"hours or while a HyperDeck records.", "ATEM")
"{\"inputs\": {\"16\": \"Resi - RX1\"}, \"seconds\": 30}. A silent one turns its check red while a "
"HyperDeck records (a service is on); between services the send is quiet on purpose.", "ATEM")
field("atem.source_folder", "path", "", "Where ATEM Software Control saves its files (for 'pick the newest').",
"ATEM", apply="none")
status_panel.py
+3 -3
View File
@@ -3471,13 +3471,13 @@ ATEM_WATCHER = None
def check_atem(chk, snap, ctx):
"""chk["part"] "audio" or "camera": the switcher must match the restore file. "silence": a metered input
(Resi) has had no audio at all for a while; only in auto-fix's hours or while a
HyperDeck records - at 4 AM the console may simply be off."""
(Resi) has had no audio at all for a while; only while a HyperDeck records (a service is on): between services
the Resi send is quiet on purpose (2026-10-25 09:00, a false alarm at NA)."""
a = snap.atem
if not a or not a.get("connected") or ctx["now"] - a.get("time", 0) > 30:
return [(chk["part"], Result(OK))] # watcher not reporting: no verdict
diffs = a.get(chk["part"]) or []
if chk["part"] == "silence" and diffs and not (ctx.get("any_recording") or in_windows(ctx.get("autofix_windows") or [], ctx["now"])):
if chk["part"] == "silence" and diffs and not ctx.get("any_recording"):
return [(chk["part"], Result(OK))]
if diffs:
return [(chk["part"], Result(ERR, chk["msg"], info="; ".join(diffs[:3]), named=True))]
tests/test_status_panel.py
+16 -7
View File
@@ -350,15 +350,24 @@ class TestChecks(unittest.TestCase):
def test_resi_audio_silent(self):
"""An input stuck at the meter floor (2026-10-25: the Resi AJA passed no Dante audio and nothing said so):
red in auto-fix's hours or while a deck records, left alone at 4 AM when the console may be off."""
red while a deck records (a service is on); between services the Resi send is quiet on purpose."""
self.assertEqual(sp.atem_levels(CONFIG["atem"]["silence"]), {"levels": {"16": "Resi - RX1"}, "silence_seconds": 30})
engine = sp.Engine(CONFIG)
sunday_7am = datetime(2026, 10, 25, 7, 0).timestamp() # the tests' auto-fix hours start 06:30
quiet = ["input 16 Resi - RX1: no audio for 30 s or more"]
decks = [c["conn"] for t in CONFIG["tiles"] for c in t.get("checks", []) if c["type"] == "hyperdeck"]
for d in decks:
self.snap.vars[(d, "status")] = "stopped"
self.snap.atem = {"connected": True, "audio": [], "camera": [], "silence": quiet, "time": sunday_7am}
self.assertEqual(run_steps(engine, self.snap, sunday_7am, 4)[0]["atem"]["x"], "ATEM\nRESI AUDIO SILENT")
self.assertEqual(run_steps(engine, self.snap, sunday_7am, 4)[0]["atem"]["x"], "ATEM\nOK") # between services
recording = [c for t in CONFIG["tiles"] for c in t.get("checks", []) if c["type"] == "hyperdeck"][0]["conn"]
self.snap.conns[recording] = {"id": "x", "enabled": True, "category": "good", "message": ""}
self.snap.vars[(recording, "status")] = "record"
engine = sp.Engine(CONFIG)
self.assertEqual(run_steps(engine, self.snap, sunday_7am + 10, 4)[0]["atem"]["x"], "ATEM\nRESI AUDIO SILENT")
thursday_4am = datetime(2026, 10, 22, 4, 0).timestamp()
engine = sp.Engine(CONFIG)
self.snap.vars[(recording, "status")] = "stopped" # nothing recording: no verdict on silence
self.snap.atem = dict(self.snap.atem, time=thursday_4am)
self.assertEqual(run_steps(engine, self.snap, thursday_4am, 4)[0]["atem"]["x"], "ATEM\nOK")
@@ -520,12 +529,12 @@ class TestChecks(unittest.TestCase):
if p["name"] != "Lighting PC"]
self.snap.mdns.discard("NA Tracks")
tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, self.now, 4)
self.assertEqual(tiles["midi"]["x"], "MIDI\nLIGHT PC MIDI LOST\n+1")
self.assertEqual(tiles["show"]["x"], "SHOW\nLIGHT PC MIDI LOST\n+1") # named, no "MIDI" prefix
self.assertEqual(tiles["midi"]["x"], "MIDI\nLIGHT PC MIDI LOST") # the Bonjour blip waits 2 min (2026-10-25)
self.assertEqual(tiles["show"]["x"], "SHOW\nLIGHT PC MIDI LOST") # named, no "MIDI" prefix
self.assertNotIn("MIDI TRACKS", tiles["show"]["x"])
self.snap.midi["NA Ingest"]["peers"].append({"name": "Lighting PC", "address": "", "state": "running"})
tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, self.now, 4)
self.assertEqual(tiles["midi"]["x"], "MIDI\nTRACKS MIDI GONE")
tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, self.now, 125)
self.assertEqual(tiles["midi"]["x"], "MIDI\nTRACKS MIDI GONE") # still gone after 2 minutes
def test_bible_tv_app_down(self):
self.snap.http["http://10.6.7.13:8080/"] = False
@@ -1520,7 +1529,7 @@ class TestFixes(unittest.TestCase):
self.assertEqual(self.plan("dante"), [("fix", "dvs_ingest")])
# A failure with no fix must not claim there is nothing wrong.
self.snap.mdns = self.snap.mdns - {"NA Tracks"} # an advert gone: nothing to reconnect
_, t = run_steps(self.engine, self.snap, SUNDAY, 4)
_, t = run_steps(self.engine, self.snap, SUNDAY, 125)
self.engine.run_fixes("midi", self.snap, None, t)
self.assertEqual(self.engine.notes["midi"]["text"], "NO AUTO\nFIX")