ROUTING learns stems the template does not name from the set (routing-learned.json); amber ones are fixed by hold/auto-fix from what the name says and earlier sets; START routes learned stems too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -2,3 +2,4 @@ __pycache__/
 *.pyc
 .DS_Store
 dist/
+routing-learned.json

README.md

@@ -155,7 +155,7 @@ everything on the page at once.
 | LIGHT TIMING | every blinder flash and FX cue (notes 30-39 on `LIGHTING CUES`: Blinder Flash 1-3, Blinder FX 1-5, named from `Ableton/LIGHTING CUES MIDI`) lands on the hit in the `Drums (Live)` stem under it, even when that clip is deactivated because the drummer plays live. Each cue's moment in the stem comes from the tempo map; the Tracks Mac listens around it (less than half a 16th note either way) for the strongest hit. **3 CUES OFF** (amber): a strong hit more than 25 ms before or after the cue while the cue itself sits on a quiet spot; the details say `JEHOVAH bar 329.2.1: Blinder Flash 2 is 37 ms before the hit` (bars as Live shows them, 4/4). Listened to once per saved set. | looks again (move the note in Live by hand: Live's control script cannot edit notes in the arrangement) |
 | SONG KEYS | every song is in the key PCO gives it: the set's locator for it (`3- JEHOVAH - B`), the pad clip under it on `AMBIENT PADS` (`B - WARM - CHURCHFRONT PADS`) and a `Finished Songs` / `Finished Custom Songs` folder (`JEHOVAH [B]`). Eb and D# are the same key. **JEHOVAH B NOT Db** (red): the locator is in another key. **JEHOVAH PAD D NOT B** (red): the wrong pad. Finished Songs having the song only in another key is fine (START transposes it). A song that is not in Finished Songs at all is only noted. | looks again |
 | W TUNE | the `WAVES TUNE` clip that starts each song (a clip from `Ableton/Waves Tune - midi Program Change`, sent to Waves Tune as a program change) is the song's PCO key; a minor song is fine on its relative major. **JEHOVAH TUNE Db** names the song; the details say which clip to put at which bar. **TUNE PGM**: a renamed clip that sends another key's program. | looks again (Live's control script cannot change a clip in the arrangement: swap the clip by hand) |
-| ROUTING | read **live** from Ableton (AbletonOSC, so tracks just dragged in count before the set is saved): every stem goes where `ready.routing` says (Piano, Keys, Organ → `Ext. Out 5/6 Keys`; AG, EG, Bass → `7/8 GTRS`; Drums, Perc, FX, Synth Bass → `9/10 PERC / FX`; BGVS, Choir, Vox FX → `11/12 VOX`). The table was taken from 261025 NA Project and is edited on the settings page; "EG 13" goes where every other EG goes. **PIANO TO MASTER** (red) names the stem; **NEW STEM BANJO** (amber): a stem name the table does not know. | puts every stem back where the table says (and IN EARS' sends to 0 dB). Runs by itself in the auto-fix hours. |
+| ROUTING | read **live** from Ableton (AbletonOSC, so tracks just dragged in count before the set is saved): every stem goes where `ready.routing` says (Piano, Keys, Organ → `Ext. Out 5/6 Keys`; AG, EG, Bass → `7/8 GTRS`; Drums, Perc, FX, Synth Bass → `9/10 PERC / FX`; BGVS, Choir, Vox FX → `11/12 VOX`). The table was taken from 261025 NA Project and is edited on the settings page; "EG 13" goes where every other EG goes. **PIANO TO MASTER** (red) names the stem; A stem the table does not name is judged by what its name says it is (Synth, Arps, Bells → keys; Banjo, Mandolin → guitars; Loop, Shaker, Tambourine → perc / FX; Vox, Choir → vox) and by where earlier sets had it (`routing-learned.json` beside the code, learned from every set as it is routed): **BANJO TO 5/6 KEYS** (amber) when it is somewhere else. A name that says nothing goes where earlier sets had it, and the first time is learned as routed. | puts every stem back where the table, or for amber its name and earlier sets, says (and IN EARS' sends to 0 dB). Runs by itself in the auto-fix hours. |
 | IN EARS | read live: no stem in a song group has its in-ear send (send A, to `A-IN EAR MIX`) at -inf, as songs dragged in from another set do. Levels someone set are left alone; `RESI TRACKS` and `MIDI GROUP` (`ready.ears_skip_groups`) are skipped. | sets those sends to 0 dB. Runs by itself in the auto-fix hours. |
 | MIDI | the Tracks Mac's MIDI links (FX LIGHTS, TRK LIVEBOX, TRK STUDIO B) and Ableton running | reconnects the MIDI link that dropped |
 | PROP | the ProPresenter playlist for the plan's date has every PCO item. **MC n/m SLIDES**: PCO lists more announcements than there are slides. **MC SLIDES OLD?**: the slide notes still list last week's announcements. **BAPTISM: NO SAMANTHA**: a person PCO lists has no slide | looks again |

readiness.py

@@ -819,6 +819,47 @@ def template_for(routing, name):
     return list(fam.pop()) if len(fam) == 1 else None
 
 
+# Stems the routing template does not name are learned from the set, as routed there (user, 2026-10-27: "where it's
+# routed right now is correct ... build your knowledge"). Kept beside the code, not in config.json: the settings'
+# template still wins, and a learned route is only ever asked about, never changed by a hold.
+LEARNED_ROUTING = os.path.join(os.path.dirname(os.path.abspath(__file__)), "routing-learned.json")
+
+
+def learned_routing():
+    try:
+        with open(LEARNED_ROUTING) as f:
+            return json.load(f)
+    except (OSError, ValueError):
+        return {}
+
+
+def save_learned_routing(learned):
+    tmp = LEARNED_ROUTING + ".tmp"
+    with open(tmp, "w") as f:
+        json.dump(learned, f, indent=1, sort_keys=True)
+    os.replace(tmp, LEARNED_ROUTING)
+
+
+# What a stem is, from its name (the workflow's routing table: keys sounds 5/6, guitars and regular bass 7/8, drums,
+# percussion, aux drums, sub bass and FX 9/10, vocal effects, BGVs and choir 11/12), in this order: "VOX FX" is
+# vocals, "SYNTH BASS" the low end, "SYNTH LOOP" keys, a bare "LOOP" drums. The output comes from the template's own
+# entry for that kind of stem.
+KINDS = (("BGVS", r"\b(VOX|VOCALS?|BGVS?|CHOIR|VOICES?|HARMON(Y|IES)|SINGERS?)\b"),
+         ("SYNTH BASS", r"\b(SUB|SYNTH BASS|808)\b"),
+         ("PIANO", r"\b(KEYS?|PIANO|SYNTHS?|ORGAN|PADS?|STRINGS?|ARPS?|BELLS?|RHODES|WURLI|PLUCKS?|CELLO|VIOLINS?|BRASS|HORNS?|ORCH\w*|CELESTE|GLOCK\w*)\b"),
+         ("EG 1", r"\b(GTRS?|GUITARS?|AG|EG|ACOUSTIC|ELECTRIC|MANDOLIN|BANJO|BASS)\b"),
+         ("PERC", r"\b(DRUMS?|PERC\w*|SHAKERS?|TAMB\w*|CYMBALS?|KICK|SNARE|HATS?|CLAPS?|LOOPS?|HITS?|IMPACTS?|RISERS?|FX|TOMS|TIMPANI)\b"))
+
+
+def stem_kind(table, name):
+    """Where a stem's kind goes ([type, channel]), by its name; None when the name says nothing."""
+    n = stem_name(name)
+    for rep_, pattern in KINDS:
+        if re.search(pattern, n) and rep_ in table:
+            return list(table[rep_])
+    return None
+
+
 def mix_plan(rc, live):
     """(routing Line, ears Line, changes): what is routed away from the template, which song stems have
     their in-ear send at -inf, and the changes that put both right."""
@@ -828,8 +869,10 @@ def mix_plan(rc, live):
             line.fault(WARN, live["error"], "%s: AbletonOSC on the Tracks Mac did not answer" % live["error"])
         return routing, ears, []
     table = {stem_name(k): v for k, v in (rc.get("routing") or {}).items()}
+    learned = learned_routing()
+    learned_before = dict(learned)
     skip = {norm(g) for g in rc.get("ears_skip_groups", [])}
-    changes, top, n_route, n_ears, new = {}, None, 0, 0, []
+    changes, top, n_route, n_ears = {}, None, 0, 0
     for t in live.get("tracks") or []:
         if t["group"] and not t["grouped"]:
             top = t["name"]
@@ -847,16 +890,33 @@ def mix_plan(rc, live):
         elif want:
             n_route += 1
         elif table and top and norm(top) not in skip:
-            new.append(t["name"])
+            # Not in the template: what earlier sets taught (learned) and what its name says (kind) decide;
+            # when they disagree the name wins, and a set that agrees with the name teaches it.
+            cur = [t["type"], t["channel"] or ""]
+            had, kind = template_for(learned, t["name"]), stem_kind(table, t["name"])
+            fix = kind if kind and cur != kind and had != cur else (had if had and not kind and cur != list(had) else None)
+            if fix:
+                why = ("its name says %s" % stem_name(t["name"]).split()[0]) if fix == kind else "earlier sets had it there"
+                changes.setdefault(t["index"], {"index": t["index"], "name": t["name"]}).update(type=fix[0], channel=fix[1])
+                routing.fault(WARN, "%s TO %s" % (short(stem_name(t["name"]), 12), (t["channel"] or t["type"]).upper()[:10]),
+                              "%s (%s) goes to %s %s; it belongs at %s %s (%s): a hold moves it" % (
+                                  t["name"], top, t["type"], t["channel"], fix[0], fix[1], why))
+            else:
+                if had is None or list(had) != cur:
+                    learned[stem_name(t["name"])] = cur
+                    routing.notes.append("learned: %s goes to %s %s (%s)" % (stem_name(t["name"]), t["type"], t["channel"], top))
+                n_route += 1
         if top and norm(top) not in skip and t.get("send") is not None:
             if t["send"] <= 0.0:
                 changes.setdefault(t["index"], {"index": t["index"], "name": t["name"]})["send"] = 1.0
                 ears.fault(ERR, "EARS OFF " + short(top, 12), "%s in %s: send A is at -inf" % (t["name"], top))
             else:
                 n_ears += 1
-    if new:
-        routing.fault(WARN, "NEW STEM " + short(stem_name(new[0]), 12), "not in the routing template (add them in the "
-                      "settings): " + ", ".join(new[:12]))
+    if learned != learned_before:
+        try:
+            save_learned_routing(learned)
+        except OSError:
+            pass
     routing.info = "%d ROUTED" % n_route if table else "NO TEMPLATE"
     ears.info = "%d SENDS ON" % n_ears
     if live.get("playing"):
@@ -2172,7 +2232,7 @@ def start_params(rc, services, mode):
     return {"mode": mode, "projects": projects, "site_folder": rc.get("site_folder", ""),
             "folder_format": rc.get("folder_format", "{date} {loc} Project"), "finished_folders": rc.get("finished_folders", []),
             "pads_folder": rc.get("pads_folder") or os.path.join(os.path.dirname(projects.rstrip("/")), "PADS"),
-            "tune_library": rc.get("tune_library", ""), "routing": rc.get("routing") or {},
+            "tune_library": rc.get("tune_library", ""), "routing": dict(learned_routing(), **(rc.get("routing") or {})),
             "keep_groups": rc.get("start_keep_groups") or rc.get("ears_skip_groups") or ["MIDI GROUP", "RESI TRACKS"],
             "pad_track": rc.get("pad_track", "AMBIENT PADS"), "tune_track": rc.get("tune_track", "WAVES TUNE"),
             "lighting_track": rc.get("lighting_track", "LIGHTING CUES"), "sections_track": rc.get("sections_track", "SECTIONS"),

tests/test_readiness.py

@@ -4,6 +4,8 @@ watchdog and the page builder show them. Run:  python3 -m unittest discover -s t
 import json
 import logging
 import os
+import shutil
+import tempfile
 import sys
 import unittest
 from datetime import datetime
@@ -13,6 +15,7 @@ sys.path.insert(0, ROOT)
 
 import make_page  # noqa: E402
 import readiness as rd  # noqa: E402
+rd.LEARNED_ROUTING = os.path.join(tempfile.mkdtemp(prefix="sp-learned-"), "routing-learned.json")   # never the repo's
 import status_panel as sp  # noqa: E402
 
 sp.log.addHandler(logging.NullHandler())
@@ -362,9 +365,49 @@ def live_set(**over):
 
 
 class TestAbletonLive(unittest.TestCase):
+    def setUp(self):
+        d = tempfile.mkdtemp()
+        self.addCleanup(shutil.rmtree, d)
+        orig = rd.LEARNED_ROUTING
+        rd.LEARNED_ROUTING = os.path.join(d, "routing-learned.json")
+        self.addCleanup(setattr, rd, "LEARNED_ROUTING", orig)
+
+    def test_a_new_stem_is_learned_from_the_set(self):
+        """User, 2026-10-27: the set's routing is right; learn the stems the template does not name."""
+        routing, _, changes = rd.mix_plan(RC, live_set())
+        self.assertNotIn("BANJO", " ".join(msgs(routing)))
+        self.assertIn("learned: BANJO goes to Ext. Out 7/8 GTRS (JEHOVAH)", routing.notes)
+        self.assertEqual(rd.learned_routing(), {"BANJO": ["Ext. Out", "7/8 GTRS"]})
+        self.assertEqual(rd.start_params(RC, [], "run")["routing"]["BANJO"], ["Ext. Out", "7/8 GTRS"])
+        # Next week the banjo lands on Keys: amber, and a hold (or auto-fix) moves it back (a banjo is a guitar).
+        over = live_set()
+        over["tracks"][-1] = dict(over["tracks"][-1], channel="5/6 Keys")
+        routing, _, changes = rd.mix_plan(RC, over)
+        self.assertIn("BANJO TO 5/6 KEYS", msgs(routing))
+        self.assertEqual([lv for lv, m, _ in routing.faults if m.startswith("BANJO")], [rd.WARN])
+        self.assertIn({"index": 16, "name": "70-Banjo [2026-10-21 132128]", "type": "Ext. Out", "channel": "7/8 GTRS"}, changes)
+        self.assertEqual(rd.learned_routing(), {"BANJO": ["Ext. Out", "7/8 GTRS"]})
+
+    def test_what_a_stem_is_by_its_name(self):
+        t = {rd.stem_name(k): v for k, v in RC["routing"].items()}
+        keys, gtrs, perc, vox = (list(t[k]) for k in ("PIANO", "EG 1", "PERC", "BGVS"))
+        for name, want in (("29-Synth FX", keys), ("30-Synth Loop", keys), ("33-Arps 1", keys), ("35-Bells", keys),
+                           ("49-Loop 1", perc), ("74-Shaker [2026-10-26 114326]", perc), ("75-Tambourine 1", perc),
+                           ("99-Drums [2026-10-26 114524]", perc), ("Sub Bass", list(t["SYNTH BASS"])), ("Mandolin", gtrs),
+                           ("Vox FX 2", vox), ("Choir 4", vox)):
+            self.assertEqual(rd.stem_kind(t, name), want, name)
+        self.assertIsNone(rd.stem_kind(t, "Click"))
+        # An unknown name goes where earlier sets had it; a new one that agrees with its name is learned.
+        rd.save_learned_routing({"WHISTLE": ["Ext. Out", "11/12 VOX"]})
+        over = live_set()
+        over["tracks"].append(dict(over["tracks"][-1], name="71-Whistle", index=17, channel="5/6 Keys"))
+        routing, _, changes = rd.mix_plan(RC, over)
+        self.assertIn("WHISTLE TO 5/6 KEYS", msgs(routing))
+        self.assertIn({"index": 17, "name": "71-Whistle", "type": "Ext. Out", "channel": "11/12 VOX"}, changes)
+
     def test_routing_and_in_ears(self):
         routing, ears, changes = rd.mix_plan(RC, live_set())
-        self.assertEqual(msgs(routing), ["BASS TO MASTER", "NEW STEM BANJO"])
+        self.assertEqual(msgs(routing), ["BASS TO MASTER"])
         self.assertEqual(msgs(ears), ["EARS OFF JEHOVAH"])
         self.assertIn("50-Keys 1 [2026-10-21 132128] in JEHOVAH: send A is at -inf", ears.as_dict()["details"])
         # EG 13 is not in the template, but every EG goes to 7/8 GTRS; RESI and top-level tracks keep their -inf sends