MIDI: STUDIO B KEY - the Waves Tune key the tracks send really arrives in Studio B

User 2026-10-26: make sure the key Ableton sends changes Waves Tune at FOH and in Studio B. Studio B's Pro
Tools can't report the plugin's key, so midi_witness.py (LaunchAgent there) logs every program change on
'Network Tracks Midi' and the watchdog checks it against the song ProPresenter shows (the program that
song's Waves Tune clip sends, from the READY look): wrong program, or none after 45 s of a running song,
is red. FOH's LiveBox has no network interface: a witness on NA Ingest logs whether it sends anything back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -13,7 +13,8 @@ There are two parts:
 |--------|-------|-------|-------|-------|-------|-------|-------|-------|
 | row 0  | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM |
 | row 1  | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM |
-| row 2  | DOWN | X32 | DANTE | MIDI | APPS | VISTA | RESTART PAGE | **READY** |
+| row 2  | DOWN | X32 | DANTE | MIDI | MIDI: STUDIO B KEY | the Waves Tune key the tracks send really arrives in Studio B: `midi_witness.py` (LaunchAgent `org.waterschurch.midi-witness` on Studio B) writes down every program change on `Network Tracks Midi` (Pro Tools keeps its own copy of the input), and the watchdog compares it with the song ProPresenter shows: during a song the program received must be the one that song's Waves Tune clip sends (READY W TUNE's map), and once a song has run 45 s with its slides moving one must have come. **STUDIO B KEY WRONG** / **STUDIO B GOT NO KEY**; only in auto-fix hours or while recording. FOH's LiveBox has no network interface: only its MIDI link is checked; a witness on NA Ingest (`Network NA Ingest`, where the LiveBox is a peer) logs whether SuperRack sends anything back. | looks again |
+| APPS | VISTA | RESTART PAGE | **READY** |
 | row 3  | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA |
 
 Row 0 holds the group buttons. Each one shows the first problem in its group, plus "+N" if there are more:

config.json

@@ -2230,6 +2230,11 @@
             "10.1.0.0/16"
           ],
           "type": "midi_wired"
+        },
+        {
+          "type": "tune_rx",
+          "host": "studiob",
+          "name": "STUDIO B"
         }
       ],
       "id": "midi",

midi_witness.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""MIDI witness: writes down the program changes (Waves Tune keys) and other channel messages a Mac really
+receives on its network MIDI inputs, so the status panel can check that what the tracks sent arrived.
+
+Runs on the receiving Mac as a LaunchAgent (org.waterschurch.midi-witness), next to Pro Tools: CoreMIDI
+gives every input port its own copy, so nothing is taken away from the app that uses the input.
+
+  midi_witness.py "Network Tracks Midi" [...]      (Studio B: what the NA Tracks Mac sends)
+
+Writes ~/Library/Application Support/status-panel/midi-witness.json once a second when something changed:
+  {"sources": {"Network Tracks Midi": {"connected": true, "last": 1793..., "count": 12}},
+   "pcs": [[time, source, channel, program], ...]}    (the last 50 program changes)
+Built 2026-10-26 for NA: the key Ableton sends must really change Waves Tune at FOH and in Studio B.
+"""
+import ctypes
+import json
+import os
+import sys
+import time
+from collections import deque
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import coremidi  # noqa: E402
+
+OUT = os.path.expanduser("~/Library/Application Support/status-panel/midi-witness.json")
+LENGTHS = {0x80: 3, 0x90: 3, 0xA0: 3, 0xB0: 3, 0xC0: 2, 0xD0: 2, 0xE0: 3}
+
+
+def messages(data):
+    """Channel messages in a packet's bytes: [(status, data bytes)]; system real-time and sysex skipped."""
+    out, i = [], 0
+    while i < len(data):
+        b = data[i]
+        if b >= 0xF8:                       # clock, active sensing...
+            i += 1
+        elif b == 0xF0:                     # sysex: to its end
+            j = data.find(b"\xf7", i)
+            i = len(data) if j < 0 else j + 1
+        elif b >= 0xF0:
+            i += {0xF1: 2, 0xF2: 3, 0xF3: 2}.get(b, 1)
+        elif b & 0x80:
+            n = LENGTHS[b & 0xF0]
+            out.append((b, data[i + 1:i + n]))
+            i += n
+        else:
+            i += 1                          # a stray data byte
+    return out
+
+
+class Witness:
+    def __init__(self, names, out=OUT, now=time.time):
+        self.names, self.out, self.now = names, out, now
+        self.pcs = deque(maxlen=50)
+        self.sources = {n: {"connected": False, "last": None, "count": 0} for n in names}
+        self.endpoints, self.port, self.dirty = {}, None, True
+        self.proc = coremidi._READPROC(self.on_packets)            # keep a reference
+        self.by_ref = {}
+
+    def take(self, name, data):
+        for status, body in messages(data):
+            src = self.sources[name]
+            src["last"], src["count"] = self.now(), src["count"] + 1
+            if status & 0xF0 == 0xC0 and body:
+                self.pcs.append([round(self.now(), 3), name, (status & 0x0F) + 1, body[0]])
+            self.dirty = True
+
+    def on_packets(self, pktlist, _ref, src_ref):
+        try:
+            name = self.by_ref.get(src_ref or 0)
+            if name is None:
+                return
+            pl = pktlist.contents
+            addr = ctypes.addressof(pl.packet[0])
+            for _ in range(pl.numPackets):
+                pkt = coremidi._MIDIPacket.from_address(addr)
+                n = min(pkt.length, 256)
+                self.take(name, bytes(pkt.data[:n]))
+                addr = (addr + 10 + pkt.length + 3) & ~3                 # MIDIPacketNext
+        except Exception:  # noqa: BLE001 - never raise into CoreMIDI's thread
+            pass
+
+    def poll(self):
+        """(Re)connect each named source; a network session that comes and goes is picked up again."""
+        with coremidi._lock:
+            if self.port is None:
+                ref = coremidi._U32()
+                cname = coremidi._cfstr("midi witness")
+                err = coremidi._cm.MIDIInputPortCreate(coremidi._get_client(), cname, self.proc, None, ctypes.byref(ref))
+                coremidi._cf.CFRelease(cname)
+                if err:
+                    raise OSError("MIDIInputPortCreate failed: %d" % err)
+                self.port = ref.value
+            found = {}
+            for i in range(coremidi._cm.MIDIGetNumberOfSources()):
+                ep = coremidi._cm.MIDIGetSource(i)
+                for n in self.names:
+                    if n in (coremidi._string_prop(ep, "displayName"), coremidi._string_prop(ep, "name")):
+                        found[n] = ep
+            for n in self.names:
+                ep = found.get(n)
+                if ep == self.endpoints.get(n):
+                    continue
+                if self.endpoints.get(n) is not None:
+                    coremidi._cm.MIDIPortDisconnectSource(self.port, self.endpoints[n])
+                self.endpoints[n] = None
+                if ep is not None:
+                    ref = len(self.by_ref) + 1
+                    self.by_ref[ref] = n
+                    if coremidi._cm.MIDIPortConnectSource(self.port, ep, ctypes.c_void_p(ref)) == 0:
+                        self.endpoints[n] = ep
+                self.sources[n]["connected"] = self.endpoints[n] is not None
+                self.dirty = True
+
+    def save(self):
+        if not self.dirty:
+            return
+        os.makedirs(os.path.dirname(self.out), exist_ok=True)
+        with open(self.out + ".tmp", "w") as f:
+            json.dump({"time": self.now(), "sources": self.sources, "pcs": list(self.pcs)}, f)
+        os.replace(self.out + ".tmp", self.out)
+        self.dirty = False
+
+
+def main():
+    names = sys.argv[1:] or ["Network Tracks Midi"]
+    w, polled = Witness(names), 0.0
+    while True:
+        if time.time() - polled > 5:
+            try:
+                w.poll()
+            except OSError:
+                pass
+            polled = time.time()
+        w.save()
+        time.sleep(1)
+
+
+if __name__ == "__main__":
+    main()

readiness.py

@@ -1208,7 +1208,7 @@ def same_tune(clip_key, want):
     return bool(kc and kw) and kw[1] and not kc[1] and (kw[0] + 3) % 12 == kc[0]
 
 
-def eval_tune(rc, plan, look):
+def eval_tune(rc, plan, look, sent=None):
     """The Waves Tune clip at each song's start is the song's PCO key (and the program it sends is that
     key's program in the clip library, so a renamed clip is caught)."""
     line = Line()
@@ -1240,6 +1240,8 @@ def eval_tune(rc, plan, look):
             line.fault(ERR, "NO TUNE " + name, "no %s clip at or before %s, where %s starts" % (track, bar, gname))
             continue
         clip = at[-1]
+        if sent is not None:                           # the key the tracks send for this song (the witness checks)
+            sent[norm(title)] = {"clip": clip[2], "program": clip[3], "title": title}
         have = tune_key(clip[2])
         lib = next((n for n in library if tune_key(n) and same_tune(tune_key(n), want) and same_key(tune_key(n), have or "")), None)
         right = next((n for n in library if tune_key(n) and same_tune(tune_key(n), want)), "%smaj" % want)
@@ -1809,7 +1811,9 @@ def look(cfg, now=None, data_dir=None, sources=None):
                 al = src["ableton"](plan)
                 lines.update(eval_ableton(rc, plan, al))
                 lines["keys"] = eval_keys(rc, plan, al)
-                lines["tune"] = eval_tune(rc, plan, al)
+                sent = {}
+                lines["tune"] = eval_tune(rc, plan, al, sent)
+                LAST["tune_sent"] = sent
                 if rc.get("light_timing_check", True):
                     try:
                         hits = src["hits"](al) if al.get("set") else []

schema.py

@@ -521,6 +521,8 @@ CHECKS = {
                   "question": "Is Spotify playing?", "words": "Spotify is playing the service playlist."},
     "mdns":      {"fields": ["instance", "name"], "subs": ["mdns"], "identity": ["instance"], "module": None,
                   "question": "Is a MIDI link up?", "words": "A network MIDI session is advertised on Bonjour."},
+    "tune_rx":   {"fields": ["host", "name"], "subs": ["tune"], "identity": ["host"], "module": None,
+                  "question": "Did the key arrive?", "words": "The Waves Tune key the tracks send arrives on a Mac (midi_witness.py there)."},
     "midi_rx":   {"fields": ["source", "name", "max_silence_seconds"], "subs": ["rx"], "identity": ["source"], "module": None,
                   "question": "Is a MIDI link up?", "words": "MIDI is arriving from a source."},
     "remote":    {"fields": ["host", "cmd", "msg", "shell", "interval_seconds", "template", "params"], "subs": ["remote"], "identity": ["host", "cmd"],
@@ -565,7 +567,7 @@ GLOSSARY = {
         "NOT ON PLAN", "DECLINED", "WRONG DECK", "NOTHING ON", "NOT SENT", "COPYING TO", "NO PCO LOGIN",
         "PCO UNREACHABLE", "NO PLAN IN PCO", "PROP UNREACHABLE", "LINEUP UNREACHABLE", "RESOLUME UNREACHABLE",
         "UNREACHABLE", "CHECK FAILED", "TURNED OFF", "COL", "COLUMNS OK", "NO COLUMNS USED", "LTC WRONG", "PCO WRITE FAILED", "IN PCO", "NO SET TO COPY", "SYNC SLOW", "NO PLAN YET", "VOX",
-        "RESI VOX", "RESI", "NOT VOX", "MIC WRONG", "NO RESI SONG", "NO LEAD", "NOT ON LINEUP", "LEADS", "LEAD",
+        "KEY WRONG", "GOT NO KEY", "KEY", "RESI VOX", "RESI", "NOT VOX", "MIC WRONG", "NO RESI SONG", "NO LEAD", "NOT ON LINEUP", "LEADS", "LEAD",
         "NO BUTTON", "AJA NO ANSWER", "START", "START FAILED", "BUILDING", "SAVING", "COPYING", "CONSOLIDATE", "NEW SONG", "NO SERVICES",
         "NO SERVICES IN PCO", "SERVICE ON", "ALREADY BUILDING", "DID NOT START", "NOTHING TO DO",
     ],

status_panel.py

@@ -188,6 +188,8 @@ class Snapshot:
         self.http = {}           # url -> reachable
         self.disk = {}           # path -> free GB
         self.listeners = {}      # MIDI source name -> coremidi.Listener
+        self.witness = {}        # host ("" here) -> midi_witness.py's file, with "read": when it was read
+        self.prop_song = None    # {"name", "since", "changes"}: the song ProPresenter shows, for the key checks
         self.remote = {}         # (host, command) -> True / False / None (could not ask)
         self.dante_map = {}      # Dante device name -> {"ip", "mac", "switch", "port", "shared", "poe", "seen"}
         self.dante_map_time = 0.0  # last successful survey of the Dante switches
@@ -473,6 +475,78 @@ def check_spotify(chk, snap, ctx):
     return [("spotify", Result(OK, info="PLAYING"))]
 
 
+WITNESS_FILE = "Library/Application Support/status-panel/midi-witness.json"
+
+
+def poll_tune(snap, cfg):
+    """What each witness Mac received (midi_witness.py) and the song ProPresenter shows now."""
+    hosts = {c.get("host", "") for t in all_tiles(cfg) for c in t.get("checks", []) if c["type"] == "tune_rx"}
+    for host in hosts:
+        try:
+            if host:
+                r = subprocess.run(ssh_cmd(cfg, host) + ["cat '%s'" % WITNESS_FILE], capture_output=True, text=True, timeout=8)
+                data = json.loads(r.stdout) if r.returncode == 0 else None
+            else:
+                with open(os.path.expanduser("~/" + WITNESS_FILE)) as f:
+                    data = json.load(f)
+        except (OSError, ValueError, subprocess.SubprocessError):
+            data = None
+        if data is not None:
+            data["read"] = time.time()
+            snap.witness = dict(snap.witness, **{host: data})
+    url = ((cfg.get("ready") or {}).get("prop_url") or "").rstrip("/")
+    if not url:
+        return
+    try:
+        with urllib.request.urlopen(url + "/v1/presentation/active", timeout=3) as r:
+            pres = (json.loads(r.read() or b"{}") or {}).get("presentation")
+        with urllib.request.urlopen(url + "/v1/presentation/slide_index", timeout=3) as r:
+            idx = ((json.loads(r.read() or b"{}") or {}).get("presentation_index") or {}).get("index")
+    except (OSError, ValueError):
+        return
+    name = ((pres or {}).get("id") or {}).get("name")
+    song = snap.prop_song
+    if not name:
+        snap.prop_song = None
+    elif not song or song["name"] != name:
+        snap.prop_song = {"name": name, "since": time.time(), "changes": 0, "slide": idx}
+    elif idx != song.get("slide"):
+        snap.prop_song = dict(song, changes=song["changes"] + 1, slide=idx)
+
+
+def check_tune_rx(chk, snap, ctx):
+    """The key the tracks send (the Waves Tune clip's program at the song's start) really arrived on a Mac (Studio
+    B's Pro Tools), judged against the song ProPresenter shows: a program change received during the song must be
+    the one that song's clip sends; and once a song has run 45 s with its slides moving, one must have come.
+    Only in auto-fix's hours or while a HyperDeck records (2026-10-26)."""
+    name = chk.get("name", "KEY")
+    w = (snap.witness or {}).get(chk.get("host", ""))
+    song = snap.prop_song
+    if not w or ctx["now"] - (w.get("read") or 0) > 30 or not song:
+        return [("tune", Result(OK))]
+    if not (ctx.get("any_recording") or in_windows(ctx.get("autofix_windows") or [], ctx["now"])):
+        return [("tune", Result(OK))]
+    try:
+        import readiness
+        sent = readiness.LAST.get("tune_sent") or {}
+        exp = next((v for k, v in sent.items() if readiness.same(k, song["name"])), None)
+    except Exception:  # noqa: BLE001 - no READY page here
+        exp = None
+    if not exp or exp.get("program") is None:
+        return [("tune", Result(OK))]
+    pcs = [p for p in w.get("pcs") or [] if p[0] >= song["since"] - 5]
+    if pcs:
+        got = pcs[-1][3]
+        if got != exp["program"]:
+            return [("tune", Result(ERR, "%s KEY WRONG" % name, info="%s got program %d; %s's clip %s sends %d" % (
+                name, got, song["name"], exp.get("clip"), exp["program"]), named=True))]
+        return [("tune", Result(OK, info="%s KEY %s" % (name, (exp.get("clip") or "").upper())))]
+    if ctx["now"] - song["since"] >= 45 and song.get("changes", 0) >= 2:
+        return [("tune", Result(ERR, "%s GOT NO KEY" % name, info="%s has run %d s and %s received no key (%s)" % (
+            song["name"], ctx["now"] - song["since"], name, exp.get("clip")), named=True))]
+    return [("tune", Result(OK))]
+
+
 def check_pref(chk, snap, ctx):
     """An app setting on NA Ingest, e.g. ATEM Software Control's "Enable MIDI control" must stay off."""
     value = snap.prefs.get((chk["domain"], chk["key"]))
@@ -838,7 +912,7 @@ CHECKS = {
     "psm": check_psm, "tascam": check_tascam, "ping": check_ping, "launchd": check_launchd,
     "process": check_process, "http": check_http, "disk": check_disk, "midi_peer": check_midi_peer, "midi_wired": check_midi_wired, "pref": check_pref, "spotify": check_spotify,
     "mdns": check_mdns, "midi_rx": check_midi_rx, "remote": check_remote, "artnet": check_artnet,
-    "ready": check_ready, "ready_manual": check_ready_manual,
+    "ready": check_ready, "ready_manual": check_ready_manual, "tune_rx": check_tune_rx,
 }
 
 
@@ -3818,6 +3892,8 @@ def start_pollers(cfg, snap, stop, data_dir=None):
         Periodic("remote", 1.0, lambda: poll_remote(snap, cfg, last=_remote_last), stop),   # runs the groups that are due
         Periodic("dante-map", 30.0, lambda: poll_dante_map(snap, cfg, c4, dante_map_path(data_dir)), stop),
     ]
+    if any(c["type"] == "tune_rx" for t in all_tiles(cfg) for c in t.get("checks", [])):
+        threads.append(Periodic("tune-witness", 2.0, lambda: poll_tune(snap, cfg), stop))
     for t in threads:
         t.start()
     threads += start_remote_midi_streams(snap, cfg, stop)

tests/test_midi_witness.py

@@ -0,0 +1,40 @@
+"""midi_witness.py (2026-10-26, NA): the key changes Studio B really receives, through real CoreMIDI."""
+import json
+import os
+import sys
+import tempfile
+import time
+import unittest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import coremidi  # noqa: E402
+import midi_witness as mw  # noqa: E402
+
+
+class TestWitness(unittest.TestCase):
+    def test_parses_a_packet(self):
+        self.assertEqual(mw.messages(bytes([0xC0, 7, 0xF8, 0x90, 60, 100, 0xF0, 1, 2, 0xF7, 0xC1, 4])),
+                         [(0xC0, b"\x07"), (0x90, b"<d"), (0xC1, b"\x04")])
+
+    def test_writes_down_program_changes_it_really_receives(self):
+        src = coremidi.VirtualSource("witness-test")
+        out = os.path.join(tempfile.mkdtemp(), "w.json")
+        try:
+            w = mw.Witness(["witness-test"], out=out)
+            w.poll()
+            time.sleep(0.3)
+            src.send(bytes([0xC0, 7]))          # Ebmaj
+            src.send(bytes([0x90, 60, 100]))
+            src.send(bytes([0xC2, 11]))
+            time.sleep(0.5)
+            w.save()
+            d = json.load(open(out))
+            self.assertTrue(d["sources"]["witness-test"]["connected"])
+            self.assertEqual([p[2:] for p in d["pcs"]], [[1, 7], [3, 11]])
+            self.assertEqual(d["sources"]["witness-test"]["count"], 3)
+        finally:
+            src.close()
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_status_panel.py

@@ -377,6 +377,30 @@ class TestChecks(unittest.TestCase):
         tiles, _ = run_steps(sp.Engine(CONFIG), self.snap, sunday + 38 * 60, 4)
         self.assertEqual(tiles["resi"]["x"], "RESI\nEVENT CHANGE")
 
+    def test_studio_b_got_the_key_the_tracks_sent(self):
+        """2026-10-26: the key Ableton sends must really change Waves Tune in Studio B (midi_witness.py there)."""
+        import readiness
+        readiness.LAST["tune_sent"] = {"OWE YOU PRAISE": {"clip": "Emaj", "program": 4, "title": "Owe You Praise"}}
+        sunday = datetime(2026, 10, 25, 9, 40).timestamp()                    # in the auto-fix hours
+        chk = {"type": "tune_rx", "host": "studiob", "name": "STUDIO B"}
+        ctx = {"now": sunday, "any_recording": False, "autofix_windows": CONFIG["auto_fix"]["windows"]}
+        try:
+            self.snap.prop_song = {"name": "Owe You Praise - [ Full ]", "since": sunday - 20, "changes": 3}
+            self.snap.witness = {"studiob": {"read": sunday, "pcs": [[sunday - 18, "Network Tracks Midi", 1, 4]]}}
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].info, "STUDIO B KEY EMAJ")
+            self.snap.witness["studiob"]["pcs"] = [[sunday - 18, "Network Tracks Midi", 1, 7]]          # Ebmaj arrived
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].msg, "STUDIO B KEY WRONG")
+            self.snap.witness["studiob"]["pcs"] = [[sunday - 600, "Network Tracks Midi", 1, 4]]         # nothing this song
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].level, sp.OK)                 # only 20 s in
+            self.snap.prop_song["since"] = sunday - 60
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].msg, "STUDIO B GOT NO KEY")
+            self.snap.prop_song["changes"] = 0                                                            # slides still: tracks off
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].level, sp.OK)
+            monday = datetime(2026, 10, 26, 12, 0).timestamp()                                            # outside the hours
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, dict(ctx, now=monday))[0][1].level, sp.OK)
+        finally:
+            readiness.LAST.pop("tune_sent", None)
+
     def test_resi_audio_silent(self):
         """An input stuck at the meter floor (2026-10-25: the Resi AJA passed no Dante audio and nothing said so):
         red while a deck records (a service is on); between services the Resi send is quiet on purpose."""