diff --git a/README.md b/README.md index 5d8bcdf..0fc2557 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,7 @@ The **page number key** on every page follows the ALL button. It's green when ev | X32, ARENA, OBS | Companion connection | | DIGICO (inside AUDIO) | Companion's `DiGiCo_FOH` connection; the relay helper `org.waterschurch.digico-relay` on NA Ingest (`~/digico-bridge/relay.py`: Companion and the DiGiCo tablets share the console's one External Control slot through it), **DIGICO RELAY STOPPED** / **NOT LOADED**, hold AUDIO restarts it; the console itself answers ping at 10.1.60.100, **DIGICO NO PING**. Companion's OSC connection reads "good" even with the console off, so the ping is what catches that. | | STB AUDIO (inside AUDIO) | the Studio B audio source for the stream: Studio B answers ping (10.1.13.175, every second, **STUDIO B NO PING** after 2 misses); Pro Tools running on the Studio B Mac Studio (over SSH every 2 s, **PRO TOOLS NOT RUNNING**); the Dante device STUDIO-B-RedNet-TNX (**STB TNX GONE**). Hold 2 s presses FOH Backup; on Sundays that press happens by itself (see Audio failover). | -| ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. | +| ATEM | Companion connection; the switcher's **audio** (every Fairlight input's on/off/AFV and fader, e.g. Resi on input 16 ON, TRS off) and **camera settings** (iris, white balance, shutter, gain, detail, ND, lift/gamma/gain/offset, contrast, saturation) must match the restore file `atem-reference.xml` (a copy of `4k plus 2026-10-07 14-59-55.xml` from Google Drive Backup/NA/ATEM): **ATEM AUDIO WRONG** / **ATEM CAMERA WRONG**; ATEM Software Control's *Enable MIDI control* must be off: **ATEM SW MIDI ON**. Hold 2 s (and Sunday auto-fix) sends back only what differs; MIDI is fixed by quitting the app, switching it off and reopening it (the switcher is not affected). **RESI AUDIO SILENT**: input 16 (Resi - RX1, `atem.silence`) has read the meter floor (-100 dB, nothing arriving at all) for 30 s, in auto-fix's hours or while a HyperDeck records; a quiet room or a pause is far above that (2026-10-25: the Resi AJA passed no Dante audio and the panel stayed green, because DANTE only sees that the card is on the network). What differs (audio, camera, silence) is written to the log when it changes. `atem_watch.js` keeps one connection to the switcher. New restore file: copy it over `atem-reference.xml` and re-run install.sh. | | LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. | | DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". | | MIDI | All six network MIDI links on the FOH / Studio B checklist, each named when it drops: **LIGHT PC** (Ingest → Lighting), **SUPERRACK** (Ingest → Livebox), **FX LIGHTS** (Tracks → Lighting, the FX lighting cues), **TRK LIVEBOX** (Tracks → Livebox), **TRK STUDIO B** (Tracks → Studio B), **PROP LIGHTS** (ProPresenter → Lighting). NA Ingest's own session is read locally; NA Tracks' and NA ProP's over SSH every 5 s. Also: the Lighting PC and NA Tracks MIDI sessions still announced on the network. A clean disconnect shows within about 10 s; a crash or pulled cable takes about a minute (macOS's own timeout, measured). Hold 2 s reconnects whichever links dropped. | diff --git a/atem_watch.js b/atem_watch.js index dd4fcc2..a5bfd29 100644 --- a/atem_watch.js +++ b/atem_watch.js @@ -5,11 +5,14 @@ // // reference.json (made by status_panel.py from the ATEM Software Control restore .xml): // {"audio": {"16/-65280": {"mixOption": 2, "faderGain": 0}, ...}, -// "camera": {"2/1/2": [4500, 0], ...}} // camera/category/parameter -> values +// "camera": {"2/1/2": [4500, 0], ...}, // camera/category/parameter -> values +// "levels": {"16": "Resi - RX1"}, "silence_seconds": 30} // inputs whose audio must never go silent // Prints a line "ATEM {json}" when the differences change (and every 10 s): -// {"connected": true, "audio": ["16 Resi..: Off, should be On", ...], "camera": [...]} +// {"connected": true, "audio": ["16 Resi..: Off, should be On", ...], "camera": [...], +// "silence": ["input 16 Resi - RX1: no audio for 30 s"]} // Reads lines on stdin: "fix audio" / "fix camera" sends back only the settings that differ. -// Nothing else is ever sent to the switcher. +// Besides those fixes, the only thing ever sent is the request for the audio meters (as ATEM Software +// Control's mixer does); nothing on the switcher changes. const [ip, lib, refPath] = process.argv.slice(2) const fs = require('fs') const readline = require('readline') @@ -18,6 +21,13 @@ const { Atem, Commands } = require(lib) const ref = JSON.parse(fs.readFileSync(refPath, 'utf8')) const MIX = { 1: 'Off', 2: 'On', 4: 'AFV' } const ccu = {} // "camera/category/parameter" -> {type, values} +// Audio meters: the switcher reads -100.00 dB (-10000) on an input with nothing on it at all; a quiet room, a +// pause or an open mic is far above that. So an input stuck at the floor means its audio is not arriving +// (2026-10-25: the Resi AJA stopped passing Dante audio and nothing said so). +const LEVELS = ref.levels || {} +const FLOOR = -9000 +const heard = {} // input -> when it last had any audio +let metersAt = 0 // when the switcher last sent meters (none: no verdict) const atem = new Atem({ disableMultithreaded: true }) let connected = false @@ -32,7 +42,17 @@ atem.on('receivedCommands', (cmds) => { } } }) -atem.on('connected', () => { connected = true }) +atem.on('levelChanged', (l) => { + if (l.system !== 'fairlight' || l.type !== 'source') return + metersAt = Date.now() + const k = String(l.index) + if (k in LEVELS && Math.max(l.levels.inputLeftLevel, l.levels.inputRightLevel) > FLOOR) heard[k] = Date.now() +}) +atem.on('connected', () => { + connected = true + for (const k of Object.keys(LEVELS)) heard[k] = Date.now() // a fresh start gets the full grace time + if (Object.keys(LEVELS).length) atem.startFairlightMixerSendLevels().catch(() => {}) +}) atem.on('disconnected', () => { connected = false }) atem.on('error', () => {}) @@ -68,10 +88,18 @@ function cameraDiffs() { return out } +function silence() { + const now = Date.now() + if (!connected || now - metersAt > 5000) return [] + const secs = ref.silence_seconds || 30 + return Object.entries(LEVELS).filter(([k]) => now - (heard[k] || now) >= secs * 1000) + .map(([k, name]) => `input ${k} ${name}: no audio for ${secs} s or more`) +} + let last = '', lastAt = 0 function report() { const state = { connected, audio: connected ? audioDiffs().map((d) => d.text) : [], camera: connected ? cameraDiffs().map((d) => d.text) : [], - camerasSeen: Object.keys(ccu).length } + camerasSeen: Object.keys(ccu).length, silence: silence() } const s = JSON.stringify(state) if (s !== last || Date.now() - lastAt > 10000) { process.stdout.write('ATEM ' + s + '\n') diff --git a/config.json b/config.json index ba283ed..9ae466f 100644 --- a/config.json +++ b/config.json @@ -5,7 +5,13 @@ "library": "~/Library/Application Support/companion/modules/bmd-atem-*/node_modules/atem-connection", "node": "/Applications/Companion [0-9]*.app/Contents/Resources/node-runtimes/node22/bin/node", "reference": "atem-reference.xml", - "source_folder": "~/Library/CloudStorage/GoogleDrive-media@waterschurch.org/Shared drives/Waters Creative/Production Team/Backup/NA/ATEM" + "source_folder": "~/Library/CloudStorage/GoogleDrive-media@waterschurch.org/Shared drives/Waters Creative/Production Team/Backup/NA/ATEM", + "silence": { + "inputs": { + "16": "Resi - RX1" + }, + "seconds": 30 + } }, "auto_fix": { "after_seconds": 20, @@ -2326,6 +2332,12 @@ "part": "audio", "type": "atem" }, + { + "type": "atem", + "part": "silence", + "msg": "RESI AUDIO SILENT", + "fix": "atem_audio" + }, { "fix": "atem_camera", "msg": "ATEM CAMERA WRONG", diff --git a/schema.py b/schema.py index 9871336..1f213e3 100644 --- a/schema.py +++ b/schema.py @@ -187,6 +187,9 @@ field("atem.library", "path", "~/Library/Application Support/companion/modules/b field("atem.reference", "path", "atem-reference.xml", "The saved switcher file to keep in line with.", "ATEM") field("atem.compare", "dict", None, "What to keep in line: audio fields and camera settings; blank = everything.", "ATEM", advanced=True) +field("atem.silence", "dict", None, "Inputs whose audio must never be silent, metered on the switcher: " + "{\"inputs\": {\"16\": \"Resi - RX1\"}, \"seconds\": 30}. A silent one turns its check red in auto-fix's " + "hours or while a HyperDeck records.", "ATEM") field("atem.source_folder", "path", "", "Where ATEM Software Control saves its files (for 'pick the newest').", "ATEM", apply="none") @@ -328,7 +331,8 @@ field("tiles[].checks[].max_silence_seconds", "int", None, "Amber/red after this field("tiles[].checks[].cmd", "str", None, "The command run on that computer.", "Keys", advanced=True) field("tiles[].checks[].shell", "enum", None, "sh (Macs) or cmd (PCs); blank = from the computer's kind.", "Keys", advanced=True, options=["sh", "cmd"]) -field("tiles[].checks[].part", "enum", None, "Which part of the switcher.", "Keys", options=["audio", "camera"]) +field("tiles[].checks[].part", "enum", None, "Which part of the switcher (silence: a metered input has no audio).", "Keys", + options=["audio", "camera", "silence"]) field("tiles[].checks[].template", "str", None, "The builder this check was made with.", "Keys", advanced=True, apply="none") field("tiles[].checks[].item", "str", None, "Which line of the weekly checklist (READY page).", "Ready") field("tiles[].checks[].params", "dict", None, "The builder's fields.", "Keys", advanced=True, apply="none") @@ -508,7 +512,7 @@ CHECKS = { "module": None, "question": "Is a program running?", "words": "A command on another computer succeeds."}, "artnet": {"fields": ["source", "max_silence_seconds", "msg"], "subs": ["artnet"], "identity": ["source"], "module": None, "question": "Is the lighting console sending?", "words": "Art-Net is arriving from the lighting program."}, - "atem": {"fields": ["part", "msg"], "subs": ["audio", "camera"], "identity": ["part"], "module": "bmd-atem", + "atem": {"fields": ["part", "msg"], "subs": ["audio", "camera", "silence"], "identity": ["part"], "module": "bmd-atem", "question": "Is a setting right?", "words": "The switcher matches the saved file."}, "ready": {"fields": ["item"], "subs": ["ready", ""], "identity": ["item"], "module": None, "question": "Is the weekend ready?", "words": "A line of the weekly checklist, checked against Planning Center."}, diff --git a/status_panel.py b/status_panel.py index 3ecf36d..43cb1ae 100644 --- a/status_panel.py +++ b/status_panel.py @@ -1407,7 +1407,7 @@ class Engine: def evaluate(self, snap, now, service_on): ctx = {"now": now, "service": service_on, "any_recording": False, "dante_all": {}, "ready_due": in_windows(self.ready_cfg.get("due", []), now), "ready_done": self.ready_done(now), - "bypassed": self.bypass_names()[0]} + "bypassed": self.bypass_names()[0], "autofix_windows": (self.cfg.get("auto_fix") or {}).get("windows", [])} for t in all_tiles(self.cfg): for chk in t.get("checks", []): if chk["type"] == "dante": @@ -3354,6 +3354,12 @@ def atem_reference(xml_path, compare=None): return {"audio": audio, "camera": camera} +def atem_levels(sil): + """The inputs whose audio is metered for silence ("silence" in config "atem"), for atem_watch.js.""" + sil = sil or {} + return {"levels": {str(k): v for k, v in (sil.get("inputs") or {}).items()}, "silence_seconds": sil.get("seconds", 30)} + + def newest(pattern): """The path matching a pattern like ".../bmd-atem-*/..." with the highest version number.""" import glob @@ -3380,7 +3386,8 @@ class AtemWatcher: if not os.path.isabs(xml): xml = os.path.join(here, xml) with open(self.ref_path, "w") as f: - json.dump(atem_reference(xml, a.get("compare")), f) + json.dump(dict(atem_reference(xml, a.get("compare")), **atem_levels(self.atem.get("silence"))), f) + self.said = {} # what each part last said, so a change is logged once def run(self): while not self.stop.is_set(): @@ -3396,6 +3403,11 @@ class AtemWatcher: data = json.loads(line[5:]) data["time"] = time.time() self.snap.atem = data + for part in ("audio", "camera", "silence"): # say what differs, not only that it does + now_says = data.get(part) or [] + if data.get("connected") and now_says != self.said.get(part, []): + log.info("ATEM %s: %s", part, "; ".join(now_says) or "as it should be") + self.said[part] = now_says elif line.startswith("FIXED "): r = json.loads(line[6:]) self.fixed[r.get("part")] = r @@ -3436,11 +3448,15 @@ ATEM_WATCHER = None def check_atem(chk, snap, ctx): - """chk["part"] "audio" or "camera": the switcher must match the restore file.""" + """chk["part"] "audio" or "camera": the switcher must match the restore file. "silence": a metered input + (Resi) has had no audio at all for a while; only in auto-fix's hours or while a + HyperDeck records - at 4 AM the console may simply be off.""" a = snap.atem if not a or not a.get("connected") or ctx["now"] - a.get("time", 0) > 30: return [(chk["part"], Result(OK))] # watcher not reporting: no verdict diffs = a.get(chk["part"]) or [] + if chk["part"] == "silence" and diffs and not (ctx.get("any_recording") or in_windows(ctx.get("autofix_windows") or [], ctx["now"])): + return [(chk["part"], Result(OK))] if diffs: return [(chk["part"], Result(ERR, chk["msg"], info="; ".join(diffs[:3]), named=True))] return [(chk["part"], Result(OK))] diff --git a/tests/test_status_panel.py b/tests/test_status_panel.py index 409478d..ee87bd8 100644 --- a/tests/test_status_panel.py +++ b/tests/test_status_panel.py @@ -348,6 +348,20 @@ class TestChecks(unittest.TestCase): self.assertIn(f, CONFIG["auto_fix"]["fixes"]) self.assertIn("midiEnabled -bool false", CONFIG["fixes"]["atem_midi"]["run"]) + def test_resi_audio_silent(self): + """An input stuck at the meter floor (2026-10-25: the Resi AJA passed no Dante audio and nothing said so): + red in auto-fix's hours or while a deck records, left alone at 4 AM when the console may be off.""" + self.assertEqual(sp.atem_levels(CONFIG["atem"]["silence"]), {"levels": {"16": "Resi - RX1"}, "silence_seconds": 30}) + engine = sp.Engine(CONFIG) + sunday_7am = datetime(2026, 10, 25, 7, 0).timestamp() # the tests' auto-fix hours start 06:30 + quiet = ["input 16 Resi - RX1: no audio for 30 s or more"] + self.snap.atem = {"connected": True, "audio": [], "camera": [], "silence": quiet, "time": sunday_7am} + self.assertEqual(run_steps(engine, self.snap, sunday_7am, 4)[0]["atem"]["x"], "ATEM\nRESI AUDIO SILENT") + thursday_4am = datetime(2026, 10, 22, 4, 0).timestamp() + engine = sp.Engine(CONFIG) + self.snap.atem = dict(self.snap.atem, time=thursday_4am) + self.assertEqual(run_steps(engine, self.snap, thursday_4am, 4)[0]["atem"]["x"], "ATEM\nOK") + def test_atem_fix_goes_through_the_watcher(self): class W: def __init__(self): self.parts = []