HyperDeck hold switches its Companion connection off and on
After a deck restarts, Companion's HyperDeck connection does not come back by itself and a plain restart of the connection does not help; switching it off and on does. Each HD key's hold now does exactly that (even when it looks green), waits 15 s and checks: DONE, STILL OFFLINE (deck pings) or NO NETWORK. Decks are no longer rebooted by the panel; a recording deck is left alone. Tested live on HD2 (off/on, reconnected in 17 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -58,7 +58,7 @@ Row 0 holds the group buttons. Each one shows the first problem in its group, pl
|
||||
| INGEST: BRIDGE / CAPTIONS / CAPTION SCREEN / PRODCOM / PRODDASH | restarts that one on NA Ingest |
|
||||
| TASCAM (always, even when it looks fine) | reconnects Companion to the recorder and looks again after 20 s. **HUNG - POWER CYCLE IT**: the recorder answers ping but its control port has hung (only a power-cycle at the rack fixes that; it has no network reboot). **NO NETWORK / CHECK POWER**: it does not answer at all. |
|
||||
| any button whose Companion connection is **disabled** | turns the connection back on (nothing else) |
|
||||
| HD1–6 lost by Companion | asks the deck itself first: if it answers, Companion's connection is restarted; only a deck that is reachable but hung is rebooted. Never one that is recording, and never for NO DISK or NOT REC. The deck's IP comes from its Companion connection, so changing an IP in Companion needs no config change. |
|
||||
| HD1–6 (always, even when it looks fine) | switches Companion's connection to that deck off and on, which is what gets a restarted deck green, and checks again after 15 s. Never reboots the deck, and leaves a recording deck alone. |
|
||||
| anything else with a Companion connection | restarts that Companion connection |
|
||||
|
||||
**The Dante map is automatic.** Every 10 minutes the watchdog maps each Dante device: its name and IP from Companion, its MAC from NA Ingest's Dante port, and its switch port and PoE from the Netgear switches listed in `"dante_switches"`. It saves the map to `dante-map.json` in its data folder. Re-patching, adding or renaming a device needs no config change. A device that has been dead for a while ages out of the switch tables, so its last known port is used, but only if no other device has appeared there. A port carrying more than 2 devices (an uplink) is never power-cycled. The switch login is in NA Ingest's login keychain (service `netgear-status-panel`).
|
||||
|
||||
+8
-7
@@ -74,7 +74,8 @@
|
||||
"run": "pg=$(ps -o pgid= -p \"$(pgrep -x 'Church Translate' | head -1)\" 2>/dev/null | tr -d ' '); pkill -TERM -x 'Church Translate'; i=0; while pgrep -xq 'Church Translate' && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'Church Translate' && pkill -KILL -x 'Church Translate'; [ -n \"$pg\" ] && [ \"$pg\" != \"$(ps -o pgid= -p $$ | tr -d ' ')\" ] && kill -KILL -- -\"$pg\" 2>/dev/null; sleep 1; open -a 'Church Translate'" },
|
||||
"tascam_reconnect": { "kind": "reconnect", "conn": "tascam", "short": "TASCAM", "verify_seconds": 20, "hung_note": "HUNG - POWER\nCYCLE IT",
|
||||
"label": "reconnect Companion to the Tascam; if it still does not answer but pings, its control port has hung: power-cycle it at the rack" },
|
||||
"hyperdeck": { "kind": "hyperdeck", "short": "HYPERDECK", "label": "reconnect Companion to the HyperDeck, or reboot the deck only if it is hung" }
|
||||
"hyperdeck": { "kind": "hyperdeck", "short": "HYPERDECK", "verify_seconds": 15, "hung_note": "STILL OFFLINE\nHOLD AGAIN",
|
||||
"label": "turn Companion's connection to that HyperDeck off and on (what gets a restarted deck green); never reboots the deck" }
|
||||
},
|
||||
|
||||
"restart_page": {
|
||||
@@ -104,12 +105,12 @@
|
||||
{ "id": "video", "label": "VIDEO", "pos": [0, 6], "members": ["atem", "led", "arena"] },
|
||||
{ "id": "stream", "label": "STREAM", "pos": [0, 7], "members": ["obs", "resi"] },
|
||||
|
||||
{ "id": "hd1", "label": "HD1", "pos": [1, 1], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_1", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd2", "label": "HD2", "pos": [1, 2], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_2", "optional": true, "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd3", "label": "HD3", "pos": [1, 3], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_3", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd4", "label": "HD4", "pos": [1, 4], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd5", "label": "HD5", "pos": [1, 5], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd6", "label": "HD6", "pos": [1, 6], "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd1", "label": "HD1", "pos": [1, 1], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_1", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd2", "label": "HD2", "pos": [1, 2], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_2", "optional": true, "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd3", "label": "HD3", "pos": [1, 3], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_3", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd4", "label": "HD4", "pos": [1, 4], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_4", "optional": true, "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd5", "label": "HD5", "pos": [1, 5], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_5", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "hd6", "label": "HD6", "pos": [1, 6], "hold_always": "hyperdeck", "checks": [{ "type": "hyperdeck", "conn": "HyperDeck_6", "fix": { "conn": "hyperdeck" } }] },
|
||||
{ "id": "tascam", "label": "TASCAM", "pos": [1, 7], "hold_always": "tascam_reconnect", "checks": [
|
||||
{ "type": "tascam", "conn": "tascam", "fix": "tascam_reconnect" },
|
||||
{ "type": "ping", "conn": "tascam", "host": "10.1.11.74", "name": "TASCAM" }
|
||||
|
||||
+38
-29
@@ -127,6 +127,10 @@ class Companion:
|
||||
status, _ = self._request("POST", "/api/connections/%s/restart" % urllib.parse.quote(conn_id, safe=""))
|
||||
return status == 200
|
||||
|
||||
def disable_connection(self, conn_id):
|
||||
status, _ = self._request("POST", "/api/connections/%s/disable" % urllib.parse.quote(conn_id, safe=""))
|
||||
return status == 200
|
||||
|
||||
def enable_connection(self, conn_id):
|
||||
status, _ = self._request("POST", "/api/connections/%s/enable" % urllib.parse.quote(conn_id, safe=""))
|
||||
return status == 200
|
||||
@@ -906,8 +910,15 @@ class Engine:
|
||||
plan = self.fix_plan(tile_id, snap)
|
||||
always = self.tiles.get(tile_id, {}).get("hold_always")
|
||||
if not plan and always:
|
||||
# e.g. TASCAM: its hold reconnects even when it looks fine (its control port hangs).
|
||||
return self.run_force(tile_id, snap, comp, now, cooldown, fix_id=always)
|
||||
# e.g. TASCAM, HD1-6: the hold reconnects even when the button looks fine.
|
||||
chk = next((c for c in self.tiles[tile_id].get("checks", []) if c.get("conn")), {})
|
||||
if self.cfg["fixes"][always].get("kind") == "hyperdeck":
|
||||
if hyperdeck_recording(snap, chk["conn"]):
|
||||
log.info("HOLD %s: %s is recording, left alone", tile_id, chk["conn"])
|
||||
self.note(tile_id, "RECORDING\nLEFT ALONE", "idle", now, 4)
|
||||
return []
|
||||
chk = dict(chk, ip=chk.get("ip") or snap.var(chk["conn"], "ip") or self.deck_ips.get(chk["conn"]))
|
||||
return self.run_force(tile_id, snap, comp, now, cooldown, fix_id=always, chk=chk)
|
||||
if not plan:
|
||||
if self.failing(tile_id):
|
||||
log.info("HOLD %s: failing, but there is no automatic fix for it", tile_id)
|
||||
@@ -976,7 +987,7 @@ class Engine:
|
||||
threading.Thread(target=work, args=(fn, args, short), daemon=True).start()
|
||||
return plan
|
||||
|
||||
def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None):
|
||||
def run_force(self, key_id, snap, comp, now, cooldown=60, fix_id=None, chk=None):
|
||||
"""Run one fix whatever the status says: a Restart page key, or a button's "hold_always".
|
||||
Shares the 60 s guard with holds, so a hold and a restart key never double up."""
|
||||
if fix_id is None:
|
||||
@@ -985,7 +996,8 @@ class Engine:
|
||||
log.warning("FORCE %s: no such restart key", key_id)
|
||||
return None
|
||||
fix_id = key["fix"]
|
||||
ident = ("fix", fix_id, "")
|
||||
per_deck = self.cfg["fixes"][fix_id].get("kind") == "hyperdeck" and chk and chk.get("conn")
|
||||
ident = ("fix", fix_id, chk["conn"] if per_deck else "") # the same guard a failing hold uses
|
||||
if now - self.fix_last.get(ident, 0) < cooldown:
|
||||
log.info("FORCE %s: %s ran less than %ds ago, skipped", key_id, fix_id, cooldown)
|
||||
cur = self.notes.get(key_id)
|
||||
@@ -998,7 +1010,7 @@ class Engine:
|
||||
self.note(key_id, "RESTARTING", "fixing", now, 180)
|
||||
|
||||
def work():
|
||||
r = self.executor(self.cfg, fix_id, {})
|
||||
r = self.executor(self.cfg, fix_id, chk or {})
|
||||
done = time.time()
|
||||
if r == WAKING:
|
||||
self.note(key_id, "WAKING PC", "fixing", done, 120)
|
||||
@@ -1578,40 +1590,37 @@ def execute_fix(cfg, fix_id, chk):
|
||||
log.info("FIX %s -> switch %s port %s power-cycled", label, fix["switch"], fix["port"])
|
||||
return True
|
||||
if kind == "hyperdeck":
|
||||
port = chk.get("port", 9993)
|
||||
verdict = hyperdeck_probe(chk["ip"], port=port)
|
||||
if verdict == "hung":
|
||||
with socket.create_connection((chk["ip"], port), timeout=3) as s:
|
||||
s.sendall(b"reboot\r\n")
|
||||
log.info("FIX %s %s -> deck accepted a connection but did not answer: reboot sent",
|
||||
label, chk["conn"])
|
||||
return True
|
||||
if verdict == "unreachable":
|
||||
log.info("FIX %s %s -> deck does not answer on the network: nothing sent", label, chk["conn"])
|
||||
return False
|
||||
# The deck answers (and may be recording): leave it alone, reconnect Companion instead.
|
||||
comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000))
|
||||
ids = {c["label"]: c["id"] for c in comp.connections()}
|
||||
ok = chk["conn"] in ids and comp.restart_connection(ids[chk["conn"]])
|
||||
log.info("FIX %s %s -> deck is fine (%s): Companion connection restarted %s",
|
||||
label, chk["conn"], verdict, "ok" if ok else "FAILED")
|
||||
return bool(ok)
|
||||
# Turn Companion's connection to the deck off and on: after a deck restarts, that is what
|
||||
# brings it back green (a plain reconnect does not, 2026-10-11). The deck is never rebooted.
|
||||
return reconnect_and_verify(cfg, fix, label, conn=chk["conn"], host=chk.get("ip"), toggle=True)
|
||||
log.warning("FIX %s: unknown kind %r", label, kind)
|
||||
except Exception as e:
|
||||
log.warning("FIX %s failed: %s", label, e)
|
||||
return False
|
||||
|
||||
|
||||
def reconnect_and_verify(cfg, fix, label, sleep=time.sleep):
|
||||
"""Restart one Companion connection, then look again after "verify_seconds". Returns True if it
|
||||
came back; otherwise what to tell the operator: "hung_note" when the device still answers ping
|
||||
(its control side has hung - only a power-cycle helps), else that it is not on the network."""
|
||||
def reconnect_and_verify(cfg, fix, label, sleep=None, conn=None, host=None, toggle=False):
|
||||
"""Restart one Companion connection (toggle=True: switch it off and on again), then look again
|
||||
after "verify_seconds". Returns True if it came back; otherwise what to tell the operator:
|
||||
"hung_note" when the device still answers ping, else that it is not on the network."""
|
||||
sleep = sleep or time.sleep
|
||||
label_conn = conn or fix["conn"]
|
||||
fix = dict(fix, conn=label_conn)
|
||||
comp = Companion(cfg["companion"].get("host", "127.0.0.1"), cfg["companion"].get("port", 8000))
|
||||
conn = {c["label"]: c for c in comp.connections()}.get(fix["conn"])
|
||||
if conn is None:
|
||||
log.info("FIX %s -> no Companion connection %s", label, fix["conn"])
|
||||
return False
|
||||
if not conn.get("enabled"):
|
||||
if toggle:
|
||||
if conn.get("enabled") and not comp.disable_connection(conn["id"]):
|
||||
log.info("FIX %s -> Companion refused to switch %s off", label, fix["conn"])
|
||||
return False
|
||||
sleep(fix.get("off_seconds", 2))
|
||||
if not comp.enable_connection(conn["id"]):
|
||||
log.info("FIX %s -> Companion did not switch %s back on", label, fix["conn"])
|
||||
return "STILL OFF IN\nCOMPANION"
|
||||
log.info("FIX %s -> %s switched off and on", label, fix["conn"])
|
||||
elif not conn.get("enabled"):
|
||||
# Switched off in Companion: turn it on (never call that a hung device).
|
||||
ok = comp.enable_connection(conn["id"])
|
||||
log.info("FIX %s -> %s was disabled in Companion: turned on %s", label, fix["conn"], "ok" if ok else "FAILED")
|
||||
@@ -1629,7 +1638,7 @@ def reconnect_and_verify(cfg, fix, label, sleep=time.sleep):
|
||||
if status not in ("warning", "error"): # still starting up, or unknown: no verdict yet
|
||||
log.info("FIX %s -> %s is %s after %ss", label, fix["conn"], status, fix.get("verify_seconds", 20))
|
||||
return "STILL\nCONNECTING"
|
||||
host = companion_conn_hosts(companion_db_path(cfg)).get(fix["conn"])
|
||||
host = host or companion_conn_hosts(companion_db_path(cfg)).get(fix["conn"])
|
||||
if host and ping_once(host):
|
||||
log.info("FIX %s -> %s still %s but %s answers ping: control port hung", label, fix["conn"], status, host)
|
||||
return fix.get("hung_note", "STILL DOWN")
|
||||
|
||||
+43
-16
@@ -893,6 +893,21 @@ class TestFixes(unittest.TestCase):
|
||||
self.assertEqual(decks[0], ("HyperDeck_2", "10.1.10.12"))
|
||||
self.assertEqual(self.plan("hd3"), [])
|
||||
|
||||
def test_hd_hold_works_even_when_it_looks_green(self):
|
||||
ran = []
|
||||
self.engine.executor = lambda cfg, fix_id, chk: ran.append((fix_id, chk.get("conn"), chk.get("ip"))) or True
|
||||
run_steps(self.engine, self.snap, SUNDAY, 4) # every deck fine
|
||||
self.engine.run_fixes("hd4", self.snap, None, SUNDAY + 5)
|
||||
self.snap.vars[("HyperDeck_3", "status")] = "Record"
|
||||
self.engine.run_fixes("hd3", self.snap, None, SUNDAY + 5) # recording: left alone
|
||||
time.sleep(0.2)
|
||||
self.assertEqual(ran, [("hyperdeck", "HyperDeck_4", "10.1.10.14")])
|
||||
self.assertEqual(self.engine.notes["hd3"]["text"], "RECORDING\nLEFT ALONE")
|
||||
self.engine.run_fixes("hd5", self.snap, None, SUNDAY + 6) # another deck: its own guard
|
||||
self.engine.run_fixes("hd4", self.snap, None, SUNDAY + 20) # same deck within a minute
|
||||
time.sleep(0.2)
|
||||
self.assertEqual([r[1] for r in ran], ["HyperDeck_4", "HyperDeck_5"])
|
||||
|
||||
def test_disabled_connection_is_switched_on_never_rebooted(self):
|
||||
# 2026-10-08: HyperDeck_1 disabled in Companion, a hold sent the deck a reboot.
|
||||
self.snap.conns["HyperDeck_1"]["enabled"] = False
|
||||
@@ -1082,28 +1097,40 @@ class TestHyperDeckProbe(unittest.TestCase):
|
||||
deck.close()
|
||||
self.assertEqual(sp.hyperdeck_probe("127.0.0.1", timeout=1, port=port), "unreachable")
|
||||
|
||||
def test_only_a_silent_deck_is_rebooted(self):
|
||||
deck = FakeDeck("silent")
|
||||
def test_the_hold_switches_the_connection_off_and_on_and_never_reboots(self):
|
||||
from unittest import mock
|
||||
deck = FakeDeck("silent") # even a deck that would once have been rebooted
|
||||
try:
|
||||
comp = mock.Mock()
|
||||
comp.connections.side_effect = [
|
||||
[{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "error"}}],
|
||||
[{"label": "HyperDeck_1", "id": "h1", "enabled": True, "status": {"category": "good"}}]]
|
||||
comp.disable_connection.return_value = comp.enable_connection.return_value = True
|
||||
chk = {"conn": "HyperDeck_1", "ip": "127.0.0.1", "port": deck.port}
|
||||
with mock.patch.object(sp, "Companion", return_value=comp), mock.patch.object(sp.time, "sleep"):
|
||||
self.assertTrue(sp.execute_fix(CONFIG, "hyperdeck", chk))
|
||||
time.sleep(0.3)
|
||||
self.assertIn(b"reboot", deck.received)
|
||||
finally:
|
||||
deck.close()
|
||||
|
||||
def test_a_recording_deck_is_never_rebooted(self):
|
||||
deck = FakeDeck("record")
|
||||
try:
|
||||
cfg = json.loads(json.dumps(CONFIG))
|
||||
cfg["companion"]["port"] = 1 # no Companion here: the reconnect just fails
|
||||
chk = {"conn": "HyperDeck_1", "ip": "127.0.0.1", "port": deck.port}
|
||||
sp.execute_fix(cfg, "hyperdeck", chk)
|
||||
self.assertEqual([c[0] for c in comp.method_calls if c[0].endswith("_connection")],
|
||||
["disable_connection", "enable_connection"])
|
||||
comp.restart_connection.assert_not_called()
|
||||
time.sleep(0.3)
|
||||
self.assertNotIn(b"reboot", deck.received)
|
||||
finally:
|
||||
deck.close()
|
||||
|
||||
def test_a_deck_that_stays_offline_says_so(self):
|
||||
from unittest import mock
|
||||
fix = CONFIG["fixes"]["hyperdeck"]
|
||||
for pings, expect in ((True, "STILL OFFLINE\nHOLD AGAIN"), (False, "NO NETWORK\nCHECK POWER")):
|
||||
comp = mock.Mock()
|
||||
comp.connections.side_effect = [
|
||||
[{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}],
|
||||
[{"label": "HyperDeck_2", "id": "h2", "enabled": True, "status": {"category": "error"}}]]
|
||||
with mock.patch.object(sp, "Companion", return_value=comp), \
|
||||
mock.patch.object(sp, "ping_once", return_value=pings) as ping:
|
||||
self.assertEqual(sp.reconnect_and_verify(CONFIG, fix, "x", sleep=lambda s: None,
|
||||
conn="HyperDeck_2", host="10.1.10.12", toggle=True), expect)
|
||||
ping.assert_called_once_with("10.1.10.12") # the deck's own address, from Companion
|
||||
|
||||
|
||||
class TestNetgearParsing(unittest.TestCase):
|
||||
BIG = """ VLAN ID MAC Address Interface IfIndex Status
|
||||
@@ -1233,8 +1260,8 @@ class TestAgainstFakeCompanion(unittest.TestCase):
|
||||
sp.poll_connections(comp, snap)
|
||||
run_steps(engine, snap, now, 4)
|
||||
before = len(store["restarts"])
|
||||
# hd3 is green, so its hold does nothing; LED is failing on VX1000, so that restarts.
|
||||
sp.handle_commands(";restart hd3;restart led;service off;bogus", engine, comp, snap, now + 5)
|
||||
# x32 is green, so its hold does nothing; LED is failing on VX1000, so that restarts.
|
||||
sp.handle_commands(";restart x32;restart led;service off;bogus", engine, comp, snap, now + 5)
|
||||
self.assertEqual(store["restarts"][before:], ["id-VX1000"])
|
||||
finally:
|
||||
store["conns"]["VX1000"]["category"] = "good"
|
||||
|
||||
Reference in New Issue
Block a user