HyperDeck IPs from Companion, Vista Art-Net heartbeat, quiet Tascam module

- The hold-to-fix deck probe takes each HyperDeck's IP from its Companion
  connection's "ip" variable, so the config no longer repeats them.
- APPS shows "VISTA SILENT" when the Lighting PC's Art-Net poll (every 0.5 s)
  is not heard for 5 s: a crashed or closed Vista shows in seconds, not the
  minute macOS takes to drop a network MIDI peer.
- tascam-quiet/ builds the Tascam DA-6400 module 1.3.1-quiet: the recorder's
  "Illegal Status" warning is logged once a minute, not 5 times a second.
- README: hold-to-fix table fixed and updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 13:46:49 -04:00
co-authored by Claude Opus 5.5
parent d06798adf0
commit 77daf7424d
5 changed files with 186 additions and 17 deletions
+19 -6
View File
@@ -49,11 +49,12 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
| DANTE: PROP MAC / TRACKS / TRANSL / INGEST GONE | starts Dante Virtual Soundcard's engine on that Mac (`remote-access/mac-dvs-helper.sh`) |
| DANTE: any device powered by its switch port (today the four Studio B boxes) | power-cycles the port it is on **now**, found by its MAC on the Netgear Dante switches |
| DANTE: any other device GONE | restarts Companion's Dante module (hardware: if still gone, it is off or unplugged) |
| INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest |
| any button whose Companion connection is **disabled** | turns the connection back on (nothing else) |
| HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. |
| anything else with a Companion connection | restarts that Companion connection |
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
| INGEST: BRIDGE / CAPTIONS / PRODDASH | restarts that service on NA Ingest |
| HD1–6 lost by Companion | reboots that deck. Never one that is recording, and never for NO DISK or NOT REC |
| anything else with a Companion connection | restarts that Companion connection |
It checks over SSH every 10 s that ProPresenter, Ableton and Church Translate are running, using NA Ingest's key `~/.ssh/id_status_panel`. That key was authorized on the Macs with `remote-access/mac-authorize.sh`.
- **ALARMS:** alarms are armed all the time (`service_windows` in `config.json`: every day, 00:00–23:59). Press ALARMS to mute the blinking, for example during maintenance. It re-arms itself at midnight.
@@ -74,12 +75,12 @@ The **page number key** on every page follows the ALL button. It's green when ev
|---|---|
| HD1–HD6 | Companion connection; **NO DISK**; record time **LOW** (<30 min, amber) / **FULL** (<5 min); **NOT REC** when another deck is recording and this one isn't |
| | HD2 and HD4 are `"optional": true` in `config.json`: with no disk they sit grey with no alarm, and once a disk is in they are watched like the others. Add or remove `"optional": true` on any deck to change that. |
| TASCAM | connection; the recorder's own Error / Caution |
| TASCAM | connection; the recorder's own Error / Caution. Use the **1.3.1-quiet** Tascam module (see below) so Companion's log isn't flooded. |
| X32, ATEM, ARENA, OBS | Companion connection |
| LED | VX1000 and UHD Jr: Companion connection **and** each one answers ping (the NovaStar module keeps saying "Connected" after a processor is switched off). "UHDJR NO PING" = processor off or unreachable; "UHDJR OFFLINE" = Companion's connection failed; "UHDJR DISABLED" (amber) = connection turned off in Companion. |
| DANTE | All 19 Dante devices ("19/19"), named when one drops: "WING GONE", "TRACKS GONE", "PROP MAC GONE". |
| MIDI | Lighting PC and SuperRack still joined to NA Ingest's network MIDI session; the Lighting PC and NA Tracks MIDI sessions still on the network. A clean disconnect shows within seconds; a crash or pulled cable takes about a minute (macOS's own timeout, measured). |
| APPS | ProPresenter API; Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. |
| APPS | ProPresenter API; Ableton OSC; tracks Mac pings; Bible TV app answering on :8080 (every 5 s); Bible TV OSC; **Vista** on the Lighting PC: its Art-Net poll (sent every 0.5 s) must be heard within 5 s, otherwise "VISTA SILENT" (Vista closed, crashed, or the Lighting PC is off the network). Resolume is checked by the ARENA button. Messages name the app ("PROPRES OFFLINE", "BIBLE APP DOWN"). Hold 2 s restarts the ProPresenter, Ableton and Bible TV Companion connections. |
| RESI | decoder (RossTalk connection and ping); resi-studio-cues; Resi cloud reachable |
| NET | NA Ingest's link; gateway 10.1.0.1 pings; **4+ connections from 3+ groups dropping within 10 s** (what happened Sunday 10/5 at 10:21) |
| INGEST | Resi bridge and lobby captions services running; ProdDash running; free disk |
@@ -101,7 +102,7 @@ A problem has to last 3 seconds before it shows, so one bad reading doesn't trig
This folder is the source of truth. Edit `config.json` here, then copy the folder to NA Ingest:
```bash
scp status_panel.py coremidi.py config.json install.sh checklist-ingest:status-panel-install/
scp status_panel.py coremidi.py netgear.py config.json install.sh checklist-ingest:status-panel-install/
```
Then run the installer there. It backs up the old config and restarts the watchdog:
@@ -146,11 +147,21 @@ The watchdog picks it up within about 10 seconds. Until then that line just stay
The **LIGHT PC** tile can do the same in the other direction. If the lighting software can send a heartbeat note back to NA Ingest every few seconds, set `"max_silence_seconds": 10` on the LIGHT PC `midi_rx` check. The tile then goes amber if the lighting PC stops answering during service.
## Quiet Tascam module
The stock Tascam DA-6400 module (1.3.0) polls the recorder 5 times a second, the recorder rejects one of those polls, and the module logs a warning every time: about 430,000 lines a day in Companion's log. `tascam-quiet/make_quiet_tascam.py` builds **1.3.1-quiet** from the stock module: the same module, but that warning is logged at most once a minute and names the commands that were just sent.
1. In Companion: **Modules → Import module package**, choose `tascam-da-6400-1.3.1-quiet.tgz` (on NA Ingest's Desktop).
2. **Connections → Tascam → edit**, set the module version to **1.3.1-quiet**, save.
To go back, pick 1.3.0 on the connection again.
## Files
| File | Purpose |
|---|---|
| `status_panel.py` | the watchdog |
| `netgear.py` | reads the Netgear Dante switches (MAC tables, PoE) and power-cycles one PoE port |
| `coremidi.py` | reads network MIDI sessions and listens to MIDI sources (ctypes, no extra installs) |
| `config.json` | tiles, positions, checks, service windows |
| `make_page.py` | builds `status-page.companionconfig` from `config.json` |
@@ -158,4 +169,6 @@ The **LIGHT PC** tile can do the same in the other direction. If the lighting so
| `push_page.py` | imports the status page into page 19 over Companion's API |
| `nav_keys.py` | colours (or restores) the page nav keys on every page |
| `nav_templates.json` | Companion's own nav key designs, coloured by `make_page.py` |
| `remote-access/` | one-time setup scripts for SSH to the Macs and Windows PCs |
| `tascam-quiet/` | builds the quiet Tascam module |
| `tests/` | offline tests, including a fake Companion and a MIDI loopback |
+7 -6
View File
@@ -62,12 +62,12 @@
{ "id": "video", "label": "VIDEO", "pos": [0, 6], "members": ["atem", "led", "arena"] },
{ "id": "stream", "label": "STREAM", "pos": [0, 7], "members": ["obs", "resi"] },
{ "id": "hd1", "label": "HD1", "pos": [1, 1], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_1", "ip": "10.1.10.11", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd2", "label": "HD2", "pos": [1, 2], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_2", "optional": true, "ip": "10.1.10.12", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd3", "label": "HD3", "pos": [1, 3], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_3", "ip": "10.1.10.13", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "ip": "10.1.10.14", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "ip": "10.1.10.15", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "ip": "10.1.10.16", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd1", "label": "HD1", "pos": [1, 1], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_1", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd2", "label": "HD2", "pos": [1, 2], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_2", "optional": true, "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd3", "label": "HD3", "pos": [1, 3], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_3", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] },
{ "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] },
{ "id": "tascam", "label": "TASCAM", "pos": [1, 7], "checks": [{ "type": "tascam", "conn": "tascam" }] },
{ "id": "x32", "label": "X32", "pos": [2, 1], "checks": [{ "type": "conn", "conn": "x32" }] },
@@ -98,6 +98,7 @@
{ "type": "remote", "host": "tracks", "cmd": "pgrep -xq Live", "msg": "ABLETON NOT RUNNING", "fix": "ableton" },
{ "type": "remote", "host": "translation", "cmd": "pgrep -xq 'Church Translate'", "msg": "TRANSLATE NOT RUNNING", "fix": "translate" },
{ "type": "ping", "host": "10.1.3.227", "name": "TRACKS MAC" },
{ "type": "artnet", "source": "10.1.10.206", "max_silence_seconds": 5, "msg": "VISTA SILENT" },
{ "type": "http", "url": "http://10.6.7.13:8080/", "msg": "BIBLE APP DOWN", "interval_seconds": 5 },
{ "type": "conn", "conn": "BibleTV", "tag": "BIBLE OSC" }
] },
+55 -3
View File
@@ -147,6 +147,8 @@ class Snapshot:
self.remote = {} # (host, command) -> True / False / None (could not ask)
self.dante_map = {} # Dante device name -> {"ip", "mac", "switch", "port", "shared", "poe", "seen"}
self.dante_map_time = 0.0 # last successful survey of the Dante switches
self.artnet_last = {} # source IP -> time its last Art-Net packet arrived
self.artnet_since = None # when the Art-Net listener started (None: not listening)
def var(self, label, name):
return self.vars.get((label, name))
@@ -434,6 +436,18 @@ def check_midi_rx(chk, snap, ctx):
return [("rx", Result(OK, info="%s %s" % (name, fmt_age(age))))]
def check_artnet(chk, snap, ctx):
"""A lighting program's Art-Net traffic (Vista polls for nodes every 0.5 s): its heartbeat."""
limit = chk.get("max_silence_seconds", 5)
since = snap.artnet_since
if since is None or ctx["now"] - since < limit: # not listening, or only just started
return [("artnet", Result(OK))]
last = snap.artnet_last.get(chk["source"])
if last is None or ctx["now"] - last > limit:
return [("artnet", Result(ERR, chk.get("msg", "ART-NET SILENT"), named=True))]
return [("artnet", Result(OK))]
def remote_key(chk):
return "%s\0%s" % (chk["host"], chk["cmd"])
@@ -449,7 +463,7 @@ CHECKS = {
"conn": check_conn, "hyperdeck": check_hyperdeck, "dante": check_dante, "shure": check_shure,
"psm": check_psm, "tascam": check_tascam, "ping": check_ping, "launchd": check_launchd,
"process": check_process, "http": check_http, "disk": check_disk, "midi_peer": check_midi_peer,
"mdns": check_mdns, "midi_rx": check_midi_rx, "remote": check_remote,
"mdns": check_mdns, "midi_rx": check_midi_rx, "remote": check_remote, "artnet": check_artnet,
}
# Checks whose connection a long press may restart (Dante is shared, so not that one).
@@ -471,6 +485,7 @@ def needed_variables(cfg):
t, label = chk["type"], chk.get("conn")
if t == "hyperdeck":
fast += [(label, "status"), (label, "slotId"), (label, "recordingTime")]
slow.append((label, "ip")) # the deck's address, from Companion
elif t == "dante":
fast.append((label, "devices"))
elif t == "tascam":
@@ -607,6 +622,7 @@ class Engine:
self.executor = execute_fix
self.poe_executor = execute_poe
self.notes = {} # tile id -> hold feedback {"text", "level", "until"}
self.deck_ips = {} # HyperDeck connection -> last address Companion reported
self.now = 0.0
# raw results for every check key
@@ -779,9 +795,16 @@ class Engine:
if fix_id:
if chk["type"] == "hyperdeck" and hyperdeck_recording(snap, chk["conn"]):
continue
# One run per fix, except a HyperDeck reboot, which is per deck.
# One run per fix, except a HyperDeck fix, which is per deck. The deck's
# address comes from its Companion connection (remembered if it goes away).
per_deck = self.cfg["fixes"][fix_id].get("kind") == "hyperdeck"
ident = ("fix", fix_id, chk.get("ip", "") if per_deck else "")
if per_deck:
ip = chk.get("ip") or snap.var(chk["conn"], "ip") or self.deck_ips.get(chk["conn"])
if not ip:
continue
self.deck_ips[chk["conn"]] = ip
chk = dict(chk, ip=ip)
ident = ("fix", fix_id, chk["conn"] if per_deck else "")
if ident not in seen:
seen.add(ident)
plan.append((ident, "fix", fix_id, chk))
@@ -1311,6 +1334,33 @@ def execute_fix(cfg, fix_id, chk):
return False
def start_artnet_listener(snap, stop, port=6454):
"""Note when each source last sent Art-Net. Shares the port (SO_REUSEPORT) so it never
keeps another program from listening too. Passive: nothing is ever sent."""
import select as _select
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if hasattr(socket, "SO_REUSEPORT"):
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
try:
s.bind(("", port))
except OSError as e:
log.warning("Art-Net listener could not start on port %d: %s", port, e)
return None
def loop():
snap.artnet_since = time.time()
while not stop.is_set():
if _select.select([s], [], [], 0.5)[0]:
data, (src, _) = s.recvfrom(2048)
if data[:8] == b"Art-Net\x00":
snap.artnet_last = dict(snap.artnet_last, **{src: time.time()})
s.close()
t = threading.Thread(target=loop, name="artnet", daemon=True)
t.start()
return t
class Periodic(threading.Thread):
def __init__(self, name, interval, fn, stop):
super().__init__(name=name, daemon=True)
@@ -1400,6 +1450,8 @@ def start_pollers(cfg, snap, stop, data_dir=None):
]
for t in threads:
t.start()
if any(c["type"] == "artnet" for t in cfg["tiles"] for c in t.get("checks", [])):
start_artnet_listener(snap, stop)
return threads
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Build the "quiet" Tascam DA-6400 Companion module (1.3.1-quiet) from the stock 1.3.0.
The stock module polls the recorder every 200 ms. The recorder rejects one of those polls
with an "F2 Illegal Status" reply, and the module logs a warning for every reply: about
430,000 warnings a day in Companion's log. This build logs it at most once a minute and
names the commands that were just sent. Nothing else changes.
On NA Ingest the stock module is in
~/Library/Application Support/companion/modules/tascam-da-6400-1.3.0
python3 make_quiet_tascam.py <stock module folder> <output folder>
Writes <output folder>/tascam-da-6400-1.3.1-quiet.tgz. Import it in Companion under
Modules -> Import module package, then pick 1.3.1-quiet on the Tascam connection.
"""
import json
import os
import shutil
import sys
import tarfile
import tempfile
VERSION = "1.3.1-quiet"
PATCHES = [
# remember the last two commands sent, so the warning can name the rejected one
("return this.socket.send(e+c),!0",
"return this.spSent=[(this.spSent||[]).slice(-1)[0],e].filter(Boolean),this.socket.send(e+c),!0"),
# log "Illegal Status" at most once a minute, with a count
('case n.illegalStatus:this.log("warn",`Illegal Status: Invalid Command ${t.substr(3)}`);break;',
'case n.illegalStatus:{const q=Date.now();this.spIllegal=(this.spIllegal||0)+1;'
'if(!this.spIllegalAt||q-this.spIllegalAt>6e4){this.log("warn",`Illegal Status: Invalid Command '
'${t.substr(3)} (last commands sent: ${(this.spSent||[]).join(", ")}; ${this.spIllegal} rejected '
'since the last report - reported at most once a minute)`);this.spIllegalAt=q;this.spIllegal=0}}break;'),
]
def build(src, out_dir):
with tempfile.TemporaryDirectory() as tmp:
dst = os.path.join(tmp, "tascam-da-6400")
shutil.copytree(src, dst)
main = os.path.join(dst, "main.js")
code = open(main, encoding="utf-8").read()
for old, new in PATCHES:
if code.count(old) != 1:
sys.exit("main.js does not look like the stock 1.3.0 module (patch anchor not found)")
code = code.replace(old, new)
open(main, "w", encoding="utf-8").write(code)
for name in ("companion/manifest.json", "package.json"):
path = os.path.join(dst, name)
data = json.load(open(path))
data["version"] = VERSION
json.dump(data, open(path, "w"), indent="\t")
with open(os.path.join(dst, "companion/HELP.md"), "a") as f:
f.write("\n\n## %s (Waters Church build)\n\nSame as 1.3.0, except the recorder's "
"\"Illegal Status\" reply is logged at most once a minute, with the commands "
"that were just sent, instead of 5 times a second.\n" % VERSION)
out = os.path.join(out_dir, "tascam-da-6400-%s.tgz" % VERSION)
with tarfile.open(out, "w:gz") as tar:
tar.add(dst, arcname="tascam-da-6400")
print(out)
if __name__ == "__main__":
if len(sys.argv) != 3:
sys.exit(__doc__)
build(sys.argv[1], sys.argv[2])
+37 -2
View File
@@ -38,6 +38,7 @@ def healthy_snapshot(now):
snap.conns = {l: {"id": "id-" + l, "enabled": True, "category": "good", "message": None} for l in labels}
snap.conns_time = now
for i in range(1, 7):
snap.vars[("HyperDeck_%d" % i, "ip")] = "10.1.10.1%d" % i
snap.vars[("HyperDeck_%d" % i, "status")] = "Preview"
snap.vars[("HyperDeck_%d" % i, "slotId")] = "1"
snap.vars[("HyperDeck_%d" % i, "recordingTime")] = "28:00:00"
@@ -213,6 +214,36 @@ class TestChecks(unittest.TestCase):
"19:21:09.001 Rmv 1 27 local. _apple-midi._udp. Gone Soon\n")
self.assertEqual(sp.parse_dns_sd(out), {"NA Tracks", "Lighting PC"})
def test_vista_heartbeat(self):
self.snap.artnet_since = self.now - 60
self.snap.artnet_last = {"10.1.10.206": self.now - 1}
chk = {"source": "10.1.10.206", "max_silence_seconds": 5, "msg": "VISTA SILENT"}
self.assertEqual(sp.check_artnet(chk, self.snap, self.ctx)[0][1].level, sp.OK)
self.snap.artnet_last = {"10.1.10.206": self.now - 8}
self.assertEqual(sp.check_artnet(chk, self.snap, self.ctx)[0][1].msg, "VISTA SILENT")
self.snap.artnet_since = self.now - 2 # just started listening: no verdict
self.assertEqual(sp.check_artnet(chk, self.snap, self.ctx)[0][1].level, sp.OK)
self.snap.artnet_since = None # could not listen: no verdict
self.assertEqual(sp.check_artnet(chk, self.snap, self.ctx)[0][1].level, sp.OK)
def test_artnet_listener_hears_a_poll(self):
import socket as _s
snap, stop = sp.Snapshot(), threading.Event()
probe = _s.socket(_s.AF_INET, _s.SOCK_DGRAM)
probe.bind(("127.0.0.1", 0))
port = probe.getsockname()[1]
probe.close()
try:
self.assertIsNotNone(sp.start_artnet_listener(snap, stop, port=port))
time.sleep(0.2)
tx = _s.socket(_s.AF_INET, _s.SOCK_DGRAM)
tx.sendto(b"Art-Net\x00\x00\x20\x00\x0e\x00\x00", ("127.0.0.1", port))
tx.sendto(b"not art-net", ("127.0.0.1", port))
time.sleep(0.6)
self.assertIn("127.0.0.1", snap.artnet_last)
finally:
stop.set()
def test_midi_peer_joining_is_not_an_error(self):
for p in self.snap.midi["NA Ingest"]["peers"]:
p["state"] = "connected" # joined, clock sync not finished yet
@@ -470,8 +501,12 @@ class TestFixes(unittest.TestCase):
self.snap.conns["HyperDeck_%d" % i]["category"] = "error"
self.snap.vars[("HyperDeck_3", "status")] = "Record" # last known state: recording
self.assertEqual(self.plan("rec"), [("fix", "hyperdeck"), ("fix", "hyperdeck")])
decks = [chk["conn"] for _, _, _, chk in self.engine.fix_plan("rec", self.snap)]
self.assertEqual(decks, ["HyperDeck_2", "HyperDeck_5"])
decks = [(chk["conn"], chk["ip"]) for _, _, _, chk in self.engine.fix_plan("rec", self.snap)]
self.assertEqual(decks, [("HyperDeck_2", "10.1.10.12"), ("HyperDeck_5", "10.1.10.15")])
# the address comes from Companion and is remembered if the connection loses it
self.snap.vars[("HyperDeck_2", "ip")] = None
decks = [(chk["conn"], chk["ip"]) for _, _, _, chk in self.engine.fix_plan("rec", self.snap)]
self.assertEqual(decks[0], ("HyperDeck_2", "10.1.10.12"))
self.assertEqual(self.plan("hd3"), [])
def test_disabled_connection_is_switched_on_never_rebooted(self):