It flickers to "warning" every 15-20 minutes and made APPS blink for
nothing. "ignore_warning" on a conn check keeps offline, error and
disabled as alarms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A power-off test today took 18-27 s to show on RF (Companion's Shure
modules' own timeout). Each Axient, ULXD and PSM receiver is now pinged
every 3 s: "AXIENT 1 NO PING" in about 6-9 s. A ping check may name a
Companion "conn"; its address is read from Companion's settings database
(read-only, every 60 s), with "host" as the fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found in today's power-off test: a Shure/PSM check reports "conn" while its
connection is down and per-channel results once it is back, so the
"AXIENT 1 OFFLINE" tracker never saw OK. The button looked green, but no
CLEAR was logged and a hold on RF would still have restarted the Shure
connections. A tracker missing from a round now gets an OK update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Art-Net heartbeat moves from APPS to a VISTA tile ("VISTA / SILENT";
"VISTA SILENT" on SHOW and ALL), hold starts Vista if it is not running.
SHOW now groups MIDI, APPS and VISTA. Tested live before the move: closing
Vista and Resolume alarmed, and each hold brought them back (Vista 5 s,
Arena 9 s after the hold).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS's MIDINetworkSession API only reaches the default session (blank and
disabled on NA Ingest and NA Tracks), so coremidi.connect_peer writes the
named session's "apple.midirtp.session" settings with the peer added, as
Audio MIDI Setup's Connect does. Existing peers are passed back unchanged.
The watchdog remembers each peer's last address (config "address" is the
fallback) and runs it locally for NA Ingest's session or over SSH on the
Tracks Mac for NA Tracks' (FX lights). Checked live: both Macs answer
"already" while the Lighting PC is connected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tracks sends the FX lighting cues to the Lighting PC through NA Tracks'
network MIDI session. A midi_peer check may now name a "host": the watchdog
reads that Mac's sessions over SSH every 5 s, sending this folder's
coremidi.py over stdin (nothing installed there). Unreachable = no verdict;
the Tracks Mac's own ping check covers that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tested live: restarting rtpMIDIService brought the Lighting PC's Bonjour
advert back but dropped it from the NA Ingest and NA Tracks sessions, and it
did not rejoin them. macOS only exposes the default network session to
scripts, not NA Ingest's named one, so there is no safe automatic reconnect.
MIDI now has no hold fix; the README says how to reconnect by hand.
ssh fixes run with LogLevel=ERROR so warnings stay out of the log.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- OBS: start OBS on the LiveStream PC if it is not running, then reconnect
Companion. If the PC does not answer, send Wake-on-LAN and show WAKING PC.
- ARENA: start Resolume Arena on the Lighting PC if it is not running, then
reconnect Companion. VISTA SILENT: start Vista if it is not running.
A running OBS, Arena or Vista is never closed.
- MIDI (LIGHT PC): restart the rtpMIDI service; it reconnects to its saved
peers (NA Ingest, NA ProP, NA Tracks).
- Fixes may set "reconnect" (restart the check's Companion connection after
the command succeeds); hosts may set "mac"/"wake_broadcast" for Wake-on-LAN.
- windows-setup.ps1 finds OBS on any drive and Vista, and leaves Wake-on-LAN
alone when it is already on (changing it can reset the adapter).
- Tests stub the fix executors so a hold in a test never reaches a machine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The hold-to-fix deck probe takes each HyperDeck's IP from its Companion
connection's "ip" variable, so the config no longer repeats them.
- APPS shows "VISTA SILENT" when the Lighting PC's Art-Net poll (every 0.5 s)
is not heard for 5 s: a crashed or closed Vista shows in seconds, not the
minute macOS takes to drop a network MIDI peer.
- tascam-quiet/ builds the Tascam DA-6400 module 1.3.1-quiet: the recorder's
"Illegal Status" warning is logged once a minute, not 5 times a second.
- README: hold-to-fix table fixed and updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every 10 minutes the watchdog maps each Dante device: name and IP from
Companion, MAC from NA Ingest's ARP table, switch port and PoE state from
the Netgear switches in dante_switches, saved to dante-map.json. A DANTE hold
on a missing PoE-powered device power-cycles the port it is on now (found
live by MAC, or its last known port if it has aged out and nothing else is
there); never a port with more than 2 devices. Re-patching, adding or
renaming devices needs no config change. First live map: 19 devices, the 4
Studio B boxes on PoE.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STUDIO-B-SOLO-MON, MON, TB and TB-MON map to ports 0/14-0/17 on the Netgear
M4250 Dante switch 10.66.1.102. netgear.py poe_cycle first checks the port's
MAC table and refuses if a different device is plugged in there, then runs
poe reset. Tested 2026-10-08 13:10 on SOLO-MON: dropped off Dante and was
back about 17 s later. install.sh now copies netgear.py too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dante Virtual Soundcard on the four Macs is restarted directly; for hardware
(consoles, racks, cards, AVIOs) nothing can be restarted from here, so a hold
re-checks via a fresh Dante module connection (harmless to audio and routing).
If the device is still gone it is really off or unplugged. A check's fix can
now be "restart-connection", and a fix map can use "*" for every other sub-check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-08 a hold on HD1, whose Companion connection had been disabled,
sent the deck a reboot. Now a disabled connection on any button is fixed by
enabling it, before any other fix is considered. The HyperDeck fix asks the
deck itself (transport info on port 9993): if it answers - recording or not -
only Companion's connection is restarted; only a deck that accepts a
connection and says nothing at all is rebooted; an unreachable deck is left
alone and the button shows FIX FAILED. Tested against a fake deck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STUDIO-A-Ingest now maps to a local dvs_ingest fix (same helper as the other
Macs, which NA Ingest also needs). A hold on a failing button that has no
automatic fix now shows NO AUTO FIX instead of NOTHING TO FIX.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hold now turns the button blue with FIXING <what> while the fix runs, then
CHECKING for 20 s (green if it worked, back to the alarm if not), orange FIX
FAILED if the command failed, and grey NOTHING TO FIX or JUST TRIED / WAIT 1
MIN so a hold always visibly registers. The page redraws as soon as a command
arrives instead of on the next 1 s tick.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes come from config.json only and run over SSH with NA Ingest's
status-panel key (ProPresenter restart, Ableton / Church Translate start,
reopen DVS), locally (Resi bridge, captions, ProdDash) or as a HyperDeck
reboot. A green button does nothing, each fix waits 60 s before it can run
again, and a recording deck is never rebooted. New remote checks ask the Macs
every 10 s whether their apps are running. Group buttons and INGEST now have a
hold too; tests never run a real fix or open an SSH connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ping failures now read NO PING, Companion connection errors OFFLINE, and a
watched connection that is disabled in Companion shows amber DISABLED instead
of going quietly grey.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DANTE now alarms for all 19 devices by name. MIDI covers the Lighting PC and
SuperRack peers plus the Lighting PC and NA Tracks sessions; APPS covers
ProPresenter, Ableton, the tracks Mac and the Bible TV app. Five buttons
became three, and a check's tag names the device in the message.
dns-sd pads hours before 10 AM with a space, which the Bonjour parser did not
allow for, so every session read as gone each morning. Parsing is now its
own function with a test for both cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It now reads 19/19 to match Dante Controller. NA-Tracks, SuperRack and the
ProPresenter Mac still alarm on their own buttons, so a drop is not flagged
twice; DANTE just shows 18/19 while one of them is missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
status_panel.py runs on NA Ingest as a LaunchAgent. Every second it reads
Companion's connection statuses and module variables, adds what Companion
cannot see (network MIDI peers, Bonjour sessions, pings, launchd services,
disk, Resi cloud, the Bible TV app), and writes one JSON custom variable that
page 19 renders. Buttons acknowledge alarms, mute them, and restart a stuck
Companion connection on a two second hold. If the watchdog stops, the page
turns purple instead of freezing on stale green.
make_page.py builds the page export from config.json; tests/ runs offline
against a fake Companion and a CoreMIDI loopback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
