Tune key at every location: the key the tracks send must reach each location's FOH (SuperRack) in that location's own Planning Center key

NA works out each location's expected Waves Tune programs (campus_tune) and serves them at /api/tune/<id>; locations
fetch their list, read ProPresenter through Companion's connection, and compare with a MIDI witness on the FOH Mac
(SC FOH now; WS FOH needs the Command Line Tools first). FOH KEY WRONG / FOH GOT NO KEY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -13,7 +13,7 @@ There are two parts:
 |--------|-------|-------|-------|-------|-------|-------|-------|-------|
 | row 0  | UP | **ALL** | REC | AUDIO | RF | SHOW | VIDEO | STREAM |
 | row 1  | page | HD1 | HD2 | HD3 | HD4 | HD5 | HD6 | TASCAM |
-| row 2  | DOWN | X32 | DANTE | MIDI | MIDI: STUDIO B KEY | the Waves Tune key the tracks send really arrives in Studio B: `midi_witness.py` (LaunchAgent `org.waterschurch.midi-witness` on Studio B) writes down every program change on `Network Tracks Midi` (Pro Tools keeps its own copy of the input), and the watchdog compares it with the song ProPresenter shows: during a song the program received must be the one that song's Waves Tune clip sends (READY W TUNE's map), and once a song has run 45 s with its slides moving one must have come. **STUDIO B KEY WRONG** / **STUDIO B GOT NO KEY**; only in auto-fix hours or while recording. FOH's LiveBox has no network interface: only its MIDI link is checked; a witness on NA Ingest (`Network NA Ingest`, where the LiveBox is a peer) logs whether SuperRack sends anything back. | looks again |
+| row 2  | DOWN | X32 | DANTE | MIDI | MIDI: STUDIO B KEY | the Waves Tune key the tracks send really arrives in Studio B: `midi_witness.py` (LaunchAgent `org.waterschurch.midi-witness` on Studio B) writes down every program change on `Network Tracks Midi` (Pro Tools keeps its own copy of the input), and the watchdog compares it with the song ProPresenter shows: during a song the program received must be the one that song's Waves Tune clip sends (READY W TUNE's map), and once a song has run 45 s with its slides moving one must have come. **STUDIO B KEY WRONG** / **STUDIO B GOT NO KEY**; only in auto-fix hours or while recording. FOH's LiveBox has no network interface: only its MIDI link is checked; a witness on NA Ingest (`Network NA Ingest`, where the LiveBox is a peer) logs whether SuperRack sends anything back. **Other locations (2026-10-28): FOH KEY WRONG / FOH GOT NO KEY** on their MIDI key: their tracks' Tune key reaches the FOH Mac mini that runs SuperRack (a witness there: SC FOH on `Network SC FOH`; WS FOH once it has the Command Line Tools), judged against that location's own Planning Center keys: the hub works out each location's programs (`readiness.campus_tune`, the Tune library's clip for each song's key, a minor key may sit on its relative major) and serves them at `/api/tune/<id>`; the location fetches its list every minute (`"expect": "hub"`) and reads its ProPresenter through Companion's connection (`"prop_conn"`). AB is standalone: no hub, no list. | looks again |
 | APPS | VISTA | RESTART PAGE | **READY** |
 | row 3  | ALARMS | ATEM | OBS | RESI | NET | INGEST | LED | ARENA |
 

readiness.py

@@ -1267,6 +1267,21 @@ def eval_keys(rc, plan, look):
     return line
 
 
+def campus_tune(keys, library):
+    """The Waves Tune programs a location's songs must reach its FOH with (2026-10-28, every location): each song's own
+    Planning Center key -> the programs of the clips in the Tune library that fit it (a minor song may sit on its
+    relative major). keys: [{"title", "key"}] from that location's plan; library: {clip name: program}."""
+    out = {}
+    for it in keys or []:
+        want = it.get("key") or ""
+        if not key_of(want):
+            continue
+        fits = sorted(n for n in library if tune_key(n) and same_tune(tune_key(n), want) and library[n] is not None)
+        if fits:
+            out[norm(it["title"])] = {"title": it["title"], "key": want, "clips": fits, "programs": sorted({library[n] for n in fits})}
+    return out
+
+
 def tune_key(name):
     """The key a Waves Tune clip sets: "Ebmaj" -> "Eb", "C#min" -> "C#m"; None for "OFF | Chromatic" or "LOOP CUE"."""
     m = re.match(r"^\s*([A-G](?:#|b)?)\s*(maj|min|m)?\b", name or "")
@@ -1936,6 +1951,9 @@ def look(cfg, now=None, data_dir=None, sources=None):
                             cplan = src["pco_campus"](campus["service_type"])
                             LAST.setdefault("campus_keys", {})[campus.get("short")] = [
                                 {"title": it["title"], "key": it["key"]} for it in plan_songs(rc, cplan)]
+                            LAST.setdefault("tune_expect", {})[campus.get("short")] = dict(
+                                campus_tune(LAST["campus_keys"][campus.get("short")], al.get("tune_library") or {}),
+                                _plan={"id": cplan.get("id"), "dates": cplan.get("dates"), "time": time.time()})
                             res = campus_ltc(rc, plan, na_songs, campus, cplan)
                             eval_campus_ltc(rc, res, lines["timecode"])
                             others.append(dict(res, songs=[dict(s, item={"id": s["item"]["id"], "note_ids": s["item"]["note_ids"]}) for s in res["songs"]]))

schema.py

@@ -327,6 +327,8 @@ field("tiles[].checks[].name", "str", None, "The name the key shows.", "Keys")
 field("tiles[].checks[].msg", "str", None, "The words the key shows when it fails.", "Keys")
 field("tiles[].checks[].failures", "int", 2, "Pings missed before NO PING.", "Keys", advanced=True, lo=1, hi=10)
 field("tiles[].checks[].label", "str", None, "The launchd helper.", "Keys")
+field("tiles[].checks[].expect", "enum", None, "A location: the keys its songs must arrive in come from the hub (its own Planning Center keys).", "Keys", advanced=True, options=["hub"])
+field("tiles[].checks[].prop_conn", "str", None, "The Companion connection that reaches this location's ProPresenter (for the song on screen).", "Keys", advanced=True)
 field("tiles[].checks[].address", "str", None, "RossTalk: the address of the machine that must be connected (\"local\": a program on this Mac).", "Keys")
 field("tiles[].checks[].process", "str", None, "RossTalk from this Mac: the program's name (ProPresenter).", "Keys", advanced=True)
 field("tiles[].checks[].off_msg", "str", None, "What the key says when Companion's RossTalk listener is off.", "Keys", advanced=True)
@@ -549,7 +551,7 @@ CHECKS = {
                   "question": "Is Spotify playing?", "words": "Spotify is playing the service playlist."},
     "mdns":      {"fields": ["instance", "name"], "subs": ["mdns"], "identity": ["instance"], "module": None,
                   "question": "Is a MIDI link up?", "words": "A network MIDI session is advertised on Bonjour."},
-    "tune_rx":   {"fields": ["host", "name"], "subs": ["tune"], "identity": ["host"], "module": None,
+    "tune_rx":   {"fields": ["host", "name", "expect", "prop_conn"], "subs": ["tune"], "identity": ["host"], "module": None,
                   "question": "Did the key arrive?", "words": "The Waves Tune key the tracks send arrives on a Mac (midi_witness.py there)."},
     "midi_rx":   {"fields": ["source", "name", "max_silence_seconds"], "subs": ["rx"], "identity": ["source"], "module": None,
                   "question": "Is a MIDI link up?", "words": "MIDI is arriving from a source."},

settings/server.py

@@ -923,6 +923,12 @@ class Handler(http.server.BaseHTTPRequestHandler):
                     return self.send(200, site.companion_page(int(n)))
                 except (OSError, ValueError) as e:
                     return self.send(200, {"page": int(n), "buttons": [], "error": "Companion did not answer: %s" % e})
+            if path.startswith("/api/tune/"):
+                # A location asks the hub what Waves Tune program each of its songs must reach its FOH with.
+                want = path[len("/api/tune/"):].upper()
+                lists = (site.read_json("snapshot.json", {}) or {}).get("tune_expect") or {}
+                songs = lists.get(want)
+                return self.send(200 if songs is not None else 404, {"location": want, "songs": songs})
             if path == "/api/wiring":
                 return self.send(200, site.wiring_view())
             if path == "/api/machines":

sites/sc/config.json

@@ -269,6 +269,13 @@
             "10.5.0.0/16"
           ],
           "fix": "midi_reconnect"
+        },
+        {
+          "type": "tune_rx",
+          "host": "sc_foh",
+          "name": "FOH",
+          "expect": "hub",
+          "prop_conn": "Propresenter-API"
         }
       ],
       "pos": [

sites/ws/config.json

@@ -558,6 +558,13 @@
             "10.2.0.0/16"
           ],
           "type": "midi_wired"
+        },
+        {
+          "type": "tune_rx",
+          "host": "ws_foh",
+          "name": "FOH",
+          "expect": "hub",
+          "prop_conn": "Propresenter-API"
         }
       ],
       "id": "midi",

status_panel.py

@@ -191,6 +191,8 @@ class Snapshot:
         self.disk = {}           # path -> free GB
         self.listeners = {}      # MIDI source name -> coremidi.Listener
         self.witness = {}        # host ("" here) -> midi_witness.py's file, with "read": when it was read
+        self.tune_expect = None  # a location: {song: {"programs", "key", "clips"}} fetched from the hub (/api/tune/<id>)
+        self.tune_expect_time = 0.0
         self.prop_song = None    # {"name", "since", "changes"}: the song ProPresenter shows, for the key checks
         self.remote = {}         # (host, command) -> True / False / None (could not ask)
         self.dante_map = {}      # Dante device name -> {"ip", "mac", "switch", "port", "shared", "poe", "seen"}
@@ -579,7 +581,8 @@ def poll_tune(snap, cfg):
         if data is not None:
             data["read"] = time.time()
             snap.witness = dict(snap.witness, **{host: data})
-    url = ((cfg.get("ready") or {}).get("prop_url") or "").rstrip("/")
+    chk = next((c for t in all_tiles(cfg) for c in t.get("checks", []) if c["type"] == "tune_rx"), {})
+    url = ((cfg.get("ready") or {}).get("prop_url") or "").rstrip("/") or tune_location_inputs(snap, chk, (cfg.get("site") or {}).get("id") or "")
     if not url:
         return
     try:
@@ -599,6 +602,26 @@ def poll_tune(snap, cfg):
         snap.prop_song = dict(song, changes=song["changes"] + 1, slide=idx)
 
 
+def tune_location_inputs(snap, chk, site_id):
+    """A location's Tune check (2026-10-28): fetch from the hub, once a minute, the programs its songs must arrive
+    in ("expect": "hub"), and return its ProPresenter's address as Companion reaches it ("prop_conn")."""
+    if chk.get("expect") == "hub" and time.time() - snap.tune_expect_time > 60:
+        snap.tune_expect_time = time.time()
+        try:
+            here = os.path.dirname(os.path.abspath(__file__))     # installed beside campuses.json
+            campuses = json.load(open(os.path.join(here, "campuses.json"))) or {}
+            home = (campuses.get("home") or "").rstrip("/")
+            with urllib.request.urlopen("%s/api/tune/%s" % (home, urllib.parse.quote(site_id)), timeout=5) as r:
+                snap.tune_expect = json.loads(r.read()).get("songs")
+        except (OSError, ValueError, AttributeError):
+            pass                                          # the hub not answering: keep the last list
+    if not chk.get("prop_conn"):
+        return ""
+    host = snap.conn_hosts.get(chk["prop_conn"])
+    port = (snap.conn_info.get(chk["prop_conn"]) or {}).get("port")
+    return "http://%s:%s" % (host, port) if host and port else ""
+
+
 def check_tune_rx(chk, snap, ctx):
     """The key the tracks send (the Waves Tune clip's program at the song's start) really arrived on a Mac (Studio
     B's Pro Tools), judged against the song ProPresenter shows: a program change received during the song must be
@@ -613,22 +636,27 @@ def check_tune_rx(chk, snap, ctx):
         return [("tune", Result(OK))]
     try:
         import readiness
-        sent = readiness.LAST.get("tune_sent") or {}
-        exp = next((v for k, v in sent.items() if readiness.same(k, song["name"])), None)
+        if snap.tune_expect is not None:                  # a location: its own plan's keys, from the hub
+            exp = next((v for k, v in snap.tune_expect.items() if not k.startswith("_") and readiness.same(k, song["name"])), None)
+        else:                                             # the hub: what its own set sends
+            sent = readiness.LAST.get("tune_sent") or {}
+            exp = next((v for k, v in sent.items() if readiness.same(k, song["name"])), None)
+            exp = dict(exp, programs=[exp["program"]]) if exp and exp.get("program") is not None else None
     except Exception:  # noqa: BLE001 - no READY page here
         exp = None
-    if not exp or exp.get("program") is None:
+    if not exp or not exp.get("programs"):
         return [("tune", Result(OK))]
+    what = exp.get("clip") or "/".join(exp.get("clips") or []) or exp.get("key", "")
     pcs = [p for p in w.get("pcs") or [] if p[0] >= song["since"] - 5]
     if pcs:
         got = pcs[-1][3]
-        if got != exp["program"]:
-            return [("tune", Result(ERR, "%s KEY WRONG" % name, info="%s got program %d; %s's clip %s sends %d" % (
-                name, got, song["name"], exp.get("clip"), exp["program"]), named=True))]
-        return [("tune", Result(OK, info="%s KEY %s" % (name, (exp.get("clip") or "").upper())))]
+        if got not in exp["programs"]:
+            return [("tune", Result(ERR, "%s KEY WRONG" % name, info="%s got program %d; %s should get %s (program %s)" % (
+                name, got, song["name"], what, "/".join(str(x) for x in exp["programs"])), named=True))]
+        return [("tune", Result(OK, info="%s KEY %s" % (name, (exp.get("clip") or exp.get("key") or "").upper())))]
     if ctx["now"] - song["since"] >= 45 and song.get("changes", 0) >= 2:
         return [("tune", Result(ERR, "%s GOT NO KEY" % name, info="%s has run %d s and %s received no key (%s)" % (
-            song["name"], ctx["now"] - song["since"], name, exp.get("clip")), named=True))]
+            song["name"], ctx["now"] - song["since"], name, what), named=True))]
     return [("tune", Result(OK))]
 
 
@@ -4138,6 +4166,18 @@ def start_pollers(cfg, snap, stop, data_dir=None):
     return threads
 
 
+def tune_expect_now(snap):
+    """The hub: each location's expected Tune programs (readiness works them out from that location's plan);
+    a location: its own list, as fetched from the hub."""
+    if snap.tune_expect is not None:
+        return {"here": snap.tune_expect}
+    try:
+        import readiness
+        return readiness.LAST.get("tune_expect")
+    except Exception:  # noqa: BLE001 - no READY page here
+        return None
+
+
 def snapshot_summary(snap, engine, now):
     """Everything the watchdog knows right now, in plain JSON (the settings page reads it instead of
     asking the gear again): what Companion reports, pings, MIDI sessions here and on the other Macs,
@@ -4182,6 +4222,7 @@ def snapshot_summary(snap, engine, now):
                                    "cleared": engine.burst["cleared"]},
         "live": list(engine.live_now(now)),
         "ready": snap.ready,
+        "tune_expect": tune_expect_now(snap),
         "faults": {},
     }
     for key, tr in engine.trackers.items():

tests/test_settings.py

@@ -143,6 +143,21 @@ class TestServer(unittest.TestCase):
         self.assertIn(("rosstalk", "h:prop", "h:ingest", "ok"), edges)                     # ProPresenter's cues into Companion
         self.assertEqual({t["id"] for t in w["types"]} >= {"companion", "midi", "watch", "dante"}, True)
 
+    def test_the_hub_hands_each_location_its_tune_keys(self):
+        path = os.path.join(self.data, "snapshot.json")
+        old = open(path).read()
+        snap = json.loads(old)
+        snap["tune_expect"] = {"SC": {"owe you praise": {"title": "Owe You Praise", "key": "Eb", "clips": ["Ebmaj"], "programs": [7]}}}
+        try:
+            with open(path, "w") as f:
+                json.dump(snap, f)
+            st, body = get(self.port, "/api/tune/sc")
+            self.assertEqual((st, json.loads(body)["songs"]["owe you praise"]["programs"]), (200, [7]))
+            self.assertEqual(get(self.port, "/api/tune/ws")[0], 404)                       # no list for WS yet
+        finally:
+            with open(path, "w") as f:
+                f.write(old)
+
     def test_gear_and_keys_and_pages(self):
         gear = json.loads(get(self.port, "/api/gear")[1])
         self.assertTrue(any(r["name"] == "HyperDeck_2" and "hd2" in r["watched_by"] for r in gear))

tests/test_status_panel.py

@@ -405,6 +405,39 @@ class TestChecks(unittest.TestCase):
         finally:
             readiness.LAST.pop("tune_sent", None)
 
+    def test_a_location_foh_gets_the_key_its_own_plan_has(self):
+        """2026-10-28, every location: the key its tracks send must reach its FOH Mac (SuperRack) - the programs its
+        songs need come from the hub, worked out from that location's own Planning Center keys."""
+        import readiness
+        lib = {"Emaj": 4, "Ebmaj": 7, "Gmaj": 11, "Emin": 16, "OFF | Chromatic": 0}
+        exp = readiness.campus_tune([{"title": "Owe You Praise", "key": "Eb"}, {"title": "Firm Foundation", "key": "Em"},
+                                     {"title": "No key", "key": ""}], lib)
+        self.assertEqual(exp[readiness.norm("Owe You Praise")]["programs"], [7])
+        self.assertEqual(exp[readiness.norm("Firm Foundation")]["programs"], [11, 16])        # Emin, or G major (relative)
+        self.assertEqual(len(exp), 2)
+        sunday = datetime(2026, 10, 25, 9, 40).timestamp()
+        chk = {"type": "tune_rx", "host": "sc_foh", "name": "FOH", "expect": "hub", "prop_conn": "Propresenter-API"}
+        ctx = {"now": sunday, "any_recording": False, "autofix_windows": CONFIG["auto_fix"]["windows"]}
+        readiness.LAST["tune_sent"] = {"OWE YOU PRAISE": {"clip": "Emaj", "program": 4, "title": "Owe You Praise"}}   # NA's own key
+        try:
+            self.snap.tune_expect = dict(exp, _plan={"id": 1})
+            self.snap.prop_song = {"name": "Owe You Praise", "since": sunday - 20, "changes": 3}
+            self.snap.witness = {"sc_foh": {"read": sunday, "pcs": [[sunday - 18, "Network SC FOH", 1, 7]]}}
+            self.assertEqual(sp.check_tune_rx(chk, self.snap, ctx)[0][1].info, "FOH KEY EB")           # SC sings it in Eb
+            self.snap.witness["sc_foh"]["pcs"] = [[sunday - 18, "Network SC FOH", 1, 4]]             # NA's E arrived
+            r = sp.check_tune_rx(chk, self.snap, ctx)[0][1]
+            self.assertEqual(r.msg, "FOH KEY WRONG")
+            self.assertIn("should get Ebmaj (program 7)", r.info)
+        finally:
+            readiness.LAST.pop("tune_sent", None)
+            self.snap.tune_expect = None
+
+    def test_a_location_reaches_its_prop_through_companion(self):
+        self.snap.conn_hosts["Propresenter-API"] = "127.0.0.1"
+        self.snap.conn_info["Propresenter-API"] = {"module": "renewedvision-propresenter-api", "port": 49285}
+        self.assertEqual(sp.tune_location_inputs(self.snap, {"prop_conn": "Propresenter-API"}, "ws"), "http://127.0.0.1:49285")
+        self.assertEqual(sp.tune_location_inputs(self.snap, {}, "ws"), "")
+
     def test_resi_audio_silent(self):
         """An input stuck at the meter floor (2026-10-25: the Resi AJA passed no Dante audio and nothing said so):
         red while a deck records (a service is on); between services the Resi send is quiet on purpose."""