Every campus matches NA, tailored: Restart page, Companion-Mac programs, NA's auto-fix hours, nightly backups, restart from NA

- Auto-setup: programs on the Companion Mac itself (ProPresenter at Woonsocket) get a local process
  check and local start/restart fixes; every campus gets a Restart page (keys arrive as computers
  join); NA's auto-fix hours (Tue, Wed, Sun); a program found after its Companion connection joins
  that key instead of making a second one. settings.campus rebuild remakes a campus from what is
  found now, keeping its name, PIN and pages; upgrade-companion brings Companion to 5.0.7 from NA's
  copy (/campus/companion.zip), as Woonsocket was; setup stops with that advice on an older one.
- NA's CAMPUSES page has a RESTART key per campus; the CAMPUS page shows that campus's Restart page
  and a hold there restarts it at that campus.
- Backups: backup/backup-campuses.sh, nightly 04:30 on NA (installed everywhere, acts on the hub),
  into Drive > Waters Creative > Apps > Status Panel Backups/<campus>, 30 kept, alert email via the
  checklist's alert.js. Campuses report the account to back up with. sites/ws/config.json in the repo.
- Woonsocket rebuilt: 9 keys (PROP with ProPresenter on its Mac), Restart page 20 (SUPERRACK, VISTA,
  ABLETON, PROP), auto-fix on NA's hours; all OK. Both campuses backed up to Drive by the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -260,6 +260,28 @@ device or MIDI link into its key). Never during the auto-fix hours or a recordin
 was there before and was taken off by someone is not added back (`discover-seen.json`). "Look now" adds what is new at
 once. Everything it makes is ordinary settings, changed on the page like North Attleboro's.
 
+## The campuses at a glance
+
+| Campus | Settings page (bookmark these) | Companion Mac | Status / Restart pages |
+|---|---|---|---|
+| North Attleboro (hub) | http://10.1.1.1:8020/ | NA Ingest | 19 / 20, READY 24, CAMPUSES 98, CAMPUS 99 |
+| Woonsocket | http://10.2.2.210:8020/ | WS ProPresenter Mac (wsprop) | 19 / 20 |
+
+Every campus: alarms armed all day; auto-fix on NA's hours (Tue 14:00–22:00, Wed 09:00–21:00, Sun 05:00–14:00);
+a hold on any key fixes it at any time, on that campus's own deck or from NA's CAMPUS page; ALARMS stops auto-fix.
+`python3 -m settings.campus rebuild` (on a campus's Companion Mac, in the panel folder) makes its setup again from
+what is found now, keeping its name, PIN and pages. `python3 -m settings.campus upgrade-companion` brings an older
+Companion to 5.0.7 from NA's copy (settings come along; the old version stays installed).
+
+## Backups
+
+Nightly at 04:30 NA's LaunchAgent `org.waterschurch.status-backup` (`backup/backup-campuses.sh`) copies every
+campus's settings, Apply history and page-key record into Google Drive: Shared drives > Waters Creative > Apps >
+Status Panel Backups > <campus> > <campus>-<date>.tgz, newest 30 kept, the same pattern and alert email as the
+checklist's backup-to-drive.sh. Other campuses are reached over SSH as the account they report. The repo keeps each
+campus's settings too: `config.json` (NA) and `sites/<id>/config.json`; a reinstall with `--site <id>` starts from it.
+Smart-plug keys (outlets.json) are never copied.
+
 ## Every campus on North Attleboro's deck (NA only)
 
 Each campus set up from NA's install line tells NA's settings page where it is (`/api/campus/hello`) and learns
@@ -267,8 +289,8 @@ nothing back: no campus ever shows another, and only NA (the hub: no "home" in `
 (`settings/campuses.py`). NA's settings page reads each campus's key states every two seconds (`/api/state`) and
 keeps two pages in NA's Companion, added at the end when no page is empty (today 98 and 99):
 
-- **CAMPUSES (98):** one key per campus, its ALL key with its name ("WS / ALL OK"); purple "WS ?" when it stops answering. Tap NA: NA's status page. Tap another: the CAMPUS page showing it.
+- **CAMPUSES (98):** row 0, one key per campus, its ALL key with its name ("WS / ALL OK"); purple "WS ?" when it stops answering. Row 1, its RESTART key. Tap NA's: NA's own pages. Tap another's: the CAMPUS page showing it.
-- **CAMPUS (99):** the tapped campus's status page, same places, colours and words, live; view only (holds stay on that campus's own deck). Bottom-left goes back to CAMPUSES.
+- **CAMPUS (99):** the tapped campus's status (or Restart) page, same places, colours and words, live. Tap = acknowledge, hold = fix, at that campus, exactly as on its own deck (a Restart key's hold restarts it there). Bottom-left ("WS / ALL SITES") goes back.
 
 ## SSH from the admin laptop to every machine
 

backup/backup-campuses.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+# backup-campuses.sh - every campus's status panel settings, off-site in Google Drive, nightly.
+#
+# The same pattern as the checklist's backup-to-drive.sh on this Mac: a LaunchAgent runs it at 04:30,
+# it copies into the shared Drive (Waters Creative > Apps > Status Panel Backups > <campus>), keeps
+# the newest 30 per campus, and emails an alert (through the checklist's alert.js) when something
+# fails, because a backup nobody hears about failing is no backup.
+#
+# Only the hub (North Attleboro: no "home" in campuses.json) does anything; elsewhere it exits.
+# Each campus is reached over SSH with the panel's own key, as the account it reported. What is
+# copied: config.json (the settings), history/ (every Apply), the page-key record and the campus
+# lists. Never outlets.json (the smart-plug keys stay on their Mac).
+
+set -uo pipefail
+DEST="$HOME/Library/Application Support/status-panel"
+LOG="$HOME/Library/Logs/status-panel-backup.log"
+KEEP=30
+KEY="$HOME/.ssh/id_status_panel"
+FILES="config.json history nav_locations.json campuses.json computers.json discover-seen.json"
+log() { printf '%s  %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" >> "$LOG"; echo "$*"; }
+alert() {
+  log "ALERT: $1"
+  local js="$HOME/checklist-app/scripts/alert.js"
+  [ -f "$js" ] && node "$js" "Status panel backup: $1" "${2:-}$(printf '\n\nLog: %s' "$LOG")" >> "$LOG" 2>&1 || true
+}
+
+/usr/bin/python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("list") and not d.get("home") else 1)' \
+  "$DEST/campuses.json" 2>/dev/null || { log "not the hub: nothing to back up from here"; exit 0; }
+
+DRIVE=""
+for acct in "$HOME/Library/CloudStorage"/GoogleDrive-*; do
+  [ -d "$acct/Shared drives/Waters Creative/Apps" ] && { DRIVE="$acct/Shared drives/Waters Creative/Apps/Status Panel Backups"; break; }
+done
+if [ -z "$DRIVE" ]; then
+  alert "tonight's backup did not run - the Google Drive folder is missing" \
+        "Shared drives > Waters Creative > Apps could not be found on this Mac. Usually Google Drive is not running or signed out."
+  exit 1
+fi
+mkdir -p "$DRIVE" || { alert "could not create $DRIVE"; exit 1; }
+log "── run start ── into $DRIVE"
+
+STAMP="$(date '+%Y-%m-%d')"
+FAILED=""
+while IFS=$'\t' read -r id user host; do
+  [ -z "$id" ] && continue
+  mkdir -p "$DRIVE/$id"
+  OUT="$DRIVE/$id/$id-$STAMP.tgz"
+  TMP="$(mktemp -t status-backup)"
+  if [ "$host" = "self" ]; then
+    (cd "$DEST" && tar -czf "$TMP" $(ls -d $FILES 2>/dev/null)) 2>>"$LOG"
+  else
+    ssh -n -i "$KEY" -o BatchMode=yes -o ConnectTimeout=15 -o StrictHostKeyChecking=accept-new "$user@$host" \
+      "cd 'Library/Application Support/status-panel' && tar -czf - \$(ls -d $FILES 2>/dev/null)" > "$TMP" 2>>"$LOG"
+  fi
+  # Listing it proves it whole (not gzip -t: macOS tar pads a piped archive, which gzip calls "trailing garbage").
+  if [ -s "$TMP" ] && tar -tzf "$TMP" 2>/dev/null | grep -qx 'config.json'; then
+    cp "$TMP" "$OUT.part" && mv "$OUT.part" "$OUT" && log "✓ $id -> $(basename "$OUT") ($(du -k "$OUT" | cut -f1) KB)"
+    ls -1t "$DRIVE/$id"/"$id"-*.tgz 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do rm -f "$old"; done
+  else
+    FAILED="$FAILED $id"
+    log "✗ $id: nothing usable came back from $user@$host"
+  fi
+  rm -f "$TMP"
+done < <(/usr/bin/python3 - "$DEST/campuses.json" "$DEST/config.json" <<'PY'
+import json, sys
+d = json.load(open(sys.argv[1])); me = (json.load(open(sys.argv[2])).get("site") or {}).get("id")
+for cid, e in sorted((d.get("list") or {}).items()):
+    if cid == me:
+        print("%s\t-\tself" % cid)
+    elif e.get("user") and e.get("url"):
+        print("%s\t%s\t%s" % (cid, e["user"], e["url"].split("//")[1].split(":")[0]))
+    else:
+        print("%s\t?\t?" % cid)
+PY
+)
+if [ -n "$FAILED" ]; then
+  alert "no backup tonight for:$FAILED" "The other campuses were backed up. A campus missing here is usually off, off the network, or no longer letting this Mac log in."
+  exit 1
+fi
+log "── done ──"

install.sh

@@ -217,6 +217,28 @@ rm -rf "$DEST/kit" && mkdir -p "$DEST/kit/sites"
 cp "$SRC/install.sh" "$DEST/kit/"
 [[ -f "$SRC/README.md" ]] && cp "$SRC/README.md" "$DEST/kit/"
 [[ -d "$SRC/sites/_template" ]] && cp -R "$SRC/sites/_template" "$DEST/kit/sites/"
+# The nightly off-site backup of every campus's settings (does anything only on the hub, NA).
+if [[ -d "$SRC/backup" ]]; then
+  rm -rf "$DEST/backup" && cp -R "$SRC/backup" "$DEST/" && chmod +x "$DEST/backup/"*.sh
+  BACKUP_LABEL=org.waterschurch.status-backup
+  BACKUP_PLIST="$HOME/Library/LaunchAgents/$BACKUP_LABEL.plist"
+  cat > "$BACKUP_PLIST" <<EOF2
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key><string>$BACKUP_LABEL</string>
+  <key>ProgramArguments</key><array><string>/bin/bash</string><string>$DEST/backup/backup-campuses.sh</string></array>
+  <key>EnvironmentVariables</key><dict><key>PATH</key><string>/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string></dict>
+  <key>StartCalendarInterval</key><dict><key>Hour</key><integer>4</integer><key>Minute</key><integer>30</integer></dict>
+  <key>StandardErrorPath</key><string>$HOME/Library/Logs/status-panel-backup.err.log</string>
+  <key>StandardOutPath</key><string>$HOME/Library/Logs/status-panel-backup.out.log</string>
+</dict>
+</plist>
+EOF2
+  launchctl bootout "$DOMAIN/$BACKUP_LABEL" 2>/dev/null || true
+  launchctl bootstrap "$DOMAIN" "$BACKUP_PLIST" 2>/dev/null || true
+fi
 # The panel's own login key for the other computers (made once, never replaced).
 if [[ ! -f "$HOME/.ssh/id_status_panel" ]]; then
   mkdir -p "$HOME/.ssh" && chmod 700 "$HOME/.ssh"

settings/autoconf.py

@@ -86,8 +86,11 @@ SHARED_FIXES = {
 }
 
 STATUS_ROWS, COLS = 4, 8
+# North Attleboro's auto-fix hours, for every campus: Tuesday and Wednesday rehearsals, Sunday services.
 DEFAULT_AUTO_FIX = {"after_seconds": 20, "max_tries": 2, "retry_seconds": 90, "pause_after_network_seconds": 60,
-                    "restart_connections": True, "windows": [{"days": ["sun"], "start": "05:00", "end": "14:00"}]}
+                    "restart_connections": True, "windows": [{"days": ["tue"], "start": "14:00", "end": "22:00"},
+                                                             {"days": ["wed"], "start": "09:00", "end": "21:00"},
+                                                             {"days": ["sun"], "start": "05:00", "end": "14:00"}]}
 
 
 # --- small helpers ---------------------------------------------------------------------
@@ -218,6 +221,27 @@ def items(inv, nets):
             apps_at[(a["app"], c.get("ip"))] = item
             for ip in c.get("ips") or []:
                 apps_at.setdefault((a["app"], ip), item)
+    # Programs on this Mac itself (the Companion Mac): a local process check and local start/restart fixes.
+    ing = inv.get("ingest") or {}
+    for a in MAC_APPS:
+        exe = (ing.get("exes") or {}).get(a["process"])
+        app_path = (ing.get("paths") or {}).get(a["process"])
+        if not exe or not app_path:
+            continue
+        fid = slug("%s_ingest" % a["app"])
+        pretty = a["label"].title() if a["app"] not in ("propresenter", "ableton") else {"propresenter": "ProPresenter", "ableton": "Ableton"}[a["app"]]
+        start = {"template": "mac_start", "params": {"host": "ingest", "process": a["process"], "app": app_path},
+                 "label": "start %s on this Mac if it is not running" % pretty, "short": a["label"]}
+        again = {"template": "mac_restart", "params": {"host": "ingest", "process": a["process"], "app": app_path, "wait_seconds": 10, "force": True},
+                 "label": "restart %s on this Mac (closes it first)" % pretty, "short": a["label"]}
+        item = {"ident": "app:ingest|%s" % a["process"], "group": a.get("group") or ("VIDEO" if a["app"] == "arena" else "STREAM" if a["app"] == "obs" else "SHOW"),
+                "label": a["label"], "checks": [{"type": "process", "match": exe, "name": a["label"], "fix": fid}],
+                "fixes": {fid: start, "restart_" + fid: again}, "restart": [{"label": a["label"], "fix": "restart_" + fid}] if a.get("restart") else [],
+                "also": [], "fold": a.get("fold")}
+        out.append(item)
+        for ip in [(ing.get("wired") or {}).get("ip")] + list(ing.get("ips") or []) + ["127.0.0.1"]:
+            if ip:
+                apps_at.setdefault((a["app"], ip), item)
     # Companion connections.
     counts = {}
     for conn in inv.get("connections") or []:
@@ -343,7 +367,7 @@ def layout(cfg, new_tiles, restart_items):
     """Put the keys on the Status page: everything fits, so no group keys; otherwise group keys along
     the top row and the members after them (a member with no room left shows inside its group only)."""
     fixed = {tuple(t["pos"]) for t in cfg["tiles"] if t.get("pos")}
-    if restart_items:
+    if cfg.get("restart_page"):
         fixed.add(tuple(cfg["restart_page"]["jump_pos"]))
     cells = free_cells(fixed)
     groups = [g for g in GROUPS if any(t["_group"] == g for t in new_tiles)]
@@ -386,10 +410,10 @@ def place_new(cfg, tile):
 
 
 def restart_layout(cfg, restart_items):
-    if not restart_items:
+    """The Restart page always exists (a campus set up before its computers joined gets their keys later)."""
-        cfg.pop("restart_page", None)
+    rp = cfg.get("restart_page")
+    if not rp:
         return
-    rp = cfg["restart_page"]
     taken = {tuple(rp["back_pos"])} | {tuple(k["pos"]) for k in rp.get("keys", [])}
     cells = free_cells(taken)
     ids = {k["id"] for k in rp.get("keys", [])}
@@ -474,6 +498,18 @@ def extend(cfg, inv, seen):
             continue
         if any(c.get("params", {}).get("host") not in cfg["hosts"] for c in it["checks"] if c.get("template")):
             continue
+        # A program whose Companion connection already has a key (Woonsocket's PROP got a second key):
+        # its own checks join that key.
+        home = next((t for t in cfg["tiles"] if any(("conn:" + c.get("conn", "")) in it["also"] for c in t.get("checks", []) if c.get("conn"))), None)
+        if home is not None:
+            for fid, f in it["fixes"].items():
+                cfg["fixes"].setdefault(fid, f)
+            home["checks"] += [c for c in it["checks"] if not c.get("conn")]
+            if it["restart"] and cfg.get("restart_page"):
+                restart_layout(cfg, it["restart"])
+            added.append("%s checks on the %s key" % (key_text(it["label"]), home["label"]))
+            now_seen += [it["ident"]] + it["also"]
+            continue
         tid = slug(it["label"], taken)
         taken.add(tid)
         for fid, f in it["fixes"].items():

settings/campus.py

@@ -38,7 +38,7 @@ SEEN, LAST = "discover-seen.json", "discover-last.json"
 # What the program kit carries (no settings, keys or history of the campus that serves it).
 KIT_FILES = ("status_panel.py", "readiness.py", "coremidi.py", "netgear.py", "schema.py", "make_page.py", "push_page.py",
              "nav_keys.py", "atem_watch.js", "atem_state.js", "nav_templates.json")
-KIT_DIRS = ("settings", "remote-access", "companion-objects")
+KIT_DIRS = ("settings", "remote-access", "companion-objects", "backup")
 
 
 # --- Companion: the panel's variables, its blink clock, and two free pages ---------------
@@ -153,6 +153,23 @@ def wait_applied(applier, limit=240):
     return applier.status or {}
 
 
+def companion_version_problem(cfg):
+    """None when this Companion can take the panel (5.x), else what to tell the person."""
+    try:
+        discover.companion_get(cfg, "/api/connections", 5)
+        return None
+    except OSError:
+        pass
+    try:
+        discover.companion_get(cfg, "/socket.io/?EIO=4&transport=polling", 5)
+        return ("This Companion is older than version 5 (Woonsocket ran 4.0.3). The panel needs 5.0.7, as North "
+                "Attleboro runs. Upgrade it from NA's copy (its settings come along; the old version stays installed):\n"
+                "  cd \"$HOME/Library/Application Support/status-panel\" && /usr/bin/python3 -m settings.campus upgrade-companion\n"
+                "then paste the install line again.")
+    except OSError:
+        return "Companion does not answer on port 8000: open Companion (and its HTTP API, Settings > Protocols), then paste the line again."
+
+
 def setup(data_dir, cid, name=None, say=print, home=None):
     """The first setup of a campus (or, when it is set up already, a look for what is new)."""
     from settings import server, campuses
@@ -166,6 +183,9 @@ def setup(data_dir, cid, name=None, say=print, home=None):
         say("%s is set up already: looking for what is new." % (cfg.get("site") or {}).get("name", "This campus"))
         return look_and_add(site, say=say, force=True)
     name = name or CAMPUSES.get(cid) or cid.upper()
+    problem = companion_version_problem(cfg)
+    if problem:
+        raise RuntimeError(problem)
     for line in companion_prepare(cfg):
         say("Companion: added %s" % line)
     pages = pick_pages(cfg)
@@ -191,6 +211,83 @@ def setup(data_dir, cid, name=None, say=print, home=None):
     return new
 
 
+def rebuild(data_dir, say=print):
+    """A campus's setup made again from what is found now (today's recipes), keeping its name, PIN,
+    pages and site details. Applied like any Apply (it waits for a recording; history keeps the old one)."""
+    from settings import server
+    site = server.Site(data_dir)
+    cfg = site.load()
+    s = cfg.get("site") or {}
+    inv = discover.look(cfg, data_dir)
+    if inv.get("companion_error"):
+        raise RuntimeError("Companion does not answer: %s" % inv["companion_error"])
+    rp = (cfg.get("restart_page") or {}).get("page")
+    if not rp:
+        rp = next(p for p in pick_pages(cfg, 3) if p != cfg["page"])
+    new, seen = autoconf.build(template(data_dir), inv, {"id": s.get("id"), "name": s.get("name")}, s.get("editing_code"), (cfg["page"], rp))
+    for k in ("contacts", "live_services", "ingest_name", "settings_port"):
+        if k in s:
+            new["site"][k] = s[k]
+    new["discover"] = dict(new.get("discover") or {}, **(cfg.get("discover") or {}))
+    errors, _ = schema.validate(new)
+    if errors:
+        raise RuntimeError("the rebuilt config does not pass the rules: %s" % "; ".join(errors[:4]))
+    r = site.applier.start(new, "auto-setup: rebuilt with today's recipes", "auto-setup", "127.0.0.1")
+    if not r.get("ok"):
+        raise RuntimeError(r.get("error") or "the Apply did not start")
+    st = wait_applied(site.applier)
+    save_seen(data_dir, load_seen(data_dir) | set(seen))
+    say("%s: %s" % (st.get("state"), st.get("outcome", "")))
+    say("Keys: %s" % ", ".join(t["label"] for t in new["tiles"] if not t.get("kind")))
+    say("Restart page %s: %s" % (rp, ", ".join(k["label"] for k in new.get("restart_page", {}).get("keys", [])) or "no keys yet"))
+    return new
+
+
+def upgrade_companion(base, say=print):
+    """Companion 5.0.7 from `base` (North Attleboro's settings page), as Woonsocket was upgraded on 2026-10-20:
+    the running Companion is quit, the newest settings folder is copied for 5.0 (and marked newest, or the
+    launcher stops on "Config version mismatch"), 5.0.7 is installed beside the old one and started."""
+    import glob
+    import shutil
+    import subprocess
+    import urllib.request
+    root = os.path.expanduser("~/Library/Application Support/companion")
+    app = "/Applications/Companion 5.0.7.app"
+    if not os.path.exists(app):
+        say("Downloading Companion 5.0.7 from %s ..." % base)
+        tmp = "/tmp/companion-5.0.7.zip"
+        with urllib.request.urlopen(base.rstrip("/") + "/campus/companion.zip", timeout=600) as r, open(tmp, "wb") as f:
+            shutil.copyfileobj(r, f)
+        subprocess.run(["/usr/bin/ditto", "-x", "-k", tmp, "/Applications/"], check=True)
+        os.remove(tmp)
+    subprocess.run(["/usr/bin/codesign", "--verify", "--deep", "--strict", app], check=True)
+    dbs = sorted(glob.glob(os.path.join(root, "v*", "db.sqlite")), key=os.path.getmtime)
+    if not os.path.exists(os.path.join(root, "v5.0", "db.sqlite")):
+        if not dbs:
+            raise RuntimeError("no Companion settings found to bring along")
+        newest = os.path.dirname(dbs[-1])
+        subprocess.run(["/usr/bin/pkill", "-TERM", "-f", "Companion.*\\.app/Contents/MacOS/Companion"])
+        for _ in range(40):
+            if subprocess.run(["/usr/bin/pgrep", "-f", "Companion.*\\.app/Contents/MacOS/Companion"], capture_output=True).returncode:
+                break
+            time.sleep(0.5)
+        shutil.copytree(newest, os.path.join(root, "v5.0"))
+        say("Settings copied from %s to v5.0 (the old folder stays as it was)" % os.path.basename(newest))
+    for f in ("db.sqlite", "cache.sqlite"):
+        p = os.path.join(root, "v5.0", f)
+        if os.path.exists(p):
+            os.utime(p)
+    subprocess.run(["/usr/bin/open", "-g", app], check=True)
+    for _ in range(90):
+        try:
+            discover.companion_get({"companion": {}}, "/api/connections", 2)
+            say("Companion 5.0.7 is running. In System Settings > General > Login Items, replace the old Companion with 5.0.7.")
+            return True
+        except OSError:
+            time.sleep(1)
+    raise RuntimeError("Companion 5.0.7 did not start within 90 s: look at its window on this Mac")
+
+
 def summary(inv):
     known = set(autoconf.COMPANION)
     return {"connections": len(inv.get("connections") or []),
@@ -274,6 +371,14 @@ def loop(site, stop, poke):
             stop.wait(20)                                  # a computer just joined: give its SSH a moment
 
 
+def load_campuses_home(data_dir):
+    try:
+        with open(os.path.join(data_dir, "campuses.json")) as f:
+            return json.load(f).get("home")
+    except (OSError, ValueError):
+        return None
+
+
 def load_last(data_dir):
     try:
         with open(os.path.join(data_dir, LAST)) as f:
@@ -411,7 +516,7 @@ try {
 
 def main(argv=None):
     ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    ap.add_argument("what", choices=("setup", "look", "add"))
+    ap.add_argument("what", choices=("setup", "look", "add", "rebuild", "upgrade-companion"))
     ap.add_argument("--id", default="new")
     ap.add_argument("--name")
     ap.add_argument("--data-dir", default=PARENT)
@@ -419,6 +524,10 @@ def main(argv=None):
     a = ap.parse_args(argv)
     if a.what == "setup":
         setup(a.data_dir, a.id, a.name, home=a.home)
+    elif a.what == "rebuild":
+        rebuild(a.data_dir)
+    elif a.what == "upgrade-companion":
+        upgrade_companion(a.home or (load_campuses_home(a.data_dir) or "http://10.1.1.1:8020"))
     elif a.what == "look":
         cfg = sp.load_config(os.path.join(a.data_dir, "config.json"))
         inv = discover.look(cfg, a.data_dir)

settings/campuses.py

@@ -63,8 +63,11 @@ def my_url(cfg):
 
 
 def me(cfg, url=None):
+    import pwd
     site = cfg.get("site") or {}
-    return {"id": site.get("id") or "new", "name": site.get("name") or "This campus", "url": url or my_url(cfg)}
+    # The account this panel runs as: the hub logs in with it for the nightly backup.
+    return {"id": site.get("id") or "new", "name": site.get("name") or "This campus", "url": url or my_url(cfg),
+            "user": pwd.getpwuid(os.getuid()).pw_name}
 
 
 def layout_sha(cfg):
@@ -94,6 +97,7 @@ def http_json(url, body=None, timeout=3):
 VIEW = {"state": "sp_state_view", "heartbeat": "sp_hb_view"}
 SELECT = "sp_view"
 VIEW_CMD = "sp_view_cmd"       # a tap or hold on the CAMPUS page, passed on to that campus's own watchdog
+MODE = "sp_view_mode"          # "" = that campus's status page, "RESTART" = its Restart page
 CELLS = [(r, c) for r in range(4) for c in range(1, 8)]
 
 
@@ -132,14 +136,16 @@ def build_view(local_cfg, page, overview_page):
                                    "Tap: back to the CAMPUSES page (page %d)." % overview_page, "STATUS jump key")
     for layer in back["style"]["layers"]:
         if layer["type"] == "text":                          # which campus this page shows, then the way back
-            layer["text"] = make_page.prop("`${$(custom:%s)}\\nALL SITES`" % SELECT, True)
+            layer["text"] = make_page.prop("`${$(custom:%s)} ${$(custom:%s)}\\nALL SITES`" % (SELECT, MODE), True)
     make_page.place(controls, (3, 0), back)
     return make_page.page_export("%d\\nCampus" % page, page, controls, local_cfg)
 
 
-def select_action(cid):
+def select_action(cid, mode=""):
-    return {"type": "action", "id": make_page.stable_id("campus_" + cid, "select"), "definitionId": "custom_variable_set_value",
+    return [{"type": "action", "id": make_page.stable_id("campus_" + cid + mode, "select"), "definitionId": "custom_variable_set_value",
-            "connectionId": "internal", "options": {"name": make_page.prop(SELECT), "value": make_page.prop(cid.upper())}, "children": {}}
+             "connectionId": "internal", "options": {"name": make_page.prop(SELECT), "value": make_page.prop(cid.upper())}, "children": {}},
+            {"type": "action", "id": make_page.stable_id("campus_" + cid + mode, "mode"), "definitionId": "custom_variable_set_value",
+             "connectionId": "internal", "options": {"name": make_page.prop(MODE), "value": make_page.prop(mode)}, "children": {}}]
 
 
 def campus_key(cid, name, names, actions, note, local_cfg):
@@ -160,18 +166,29 @@ def campus_key(cid, name, names, actions, note, local_cfg):
 
 
 def build_overview(local_cfg, campuses, page, view_page):
+    """Row 0: each campus's ALL key (tap: its status page). Row 1: its RESTART key (tap: its Restart page)."""
     controls = make_page.nav_controls(local_cfg)
     mine = (local_cfg.get("site") or {}).get("id")
-    for cid, cell in zip(sorted(campuses, key=lambda c: (c != mine, c)), CELLS):
+    rp = (local_cfg.get("restart_page") or {}).get("page")
+    for col, cid in enumerate(sorted(campuses, key=lambda c: (c != mine, c))[:7], start=1):
         name = campuses[cid].get("name", cid)
+        tag = short(cid, name)
         if cid == mine:
             key = campus_key(cid, name, local_cfg["variables"], [make_page.set_page_action("campus_" + cid, "go", local_cfg["page"])],
                              "Tap: its status page (page %d)." % local_cfg["page"], local_cfg)
+            restart = make_page.static_button("campus_rs_" + cid, "%s\\nRESTART" % tag, 0x37474F, 0xFFFFFF, rp,
+                                              "Tap: this campus's Restart page (page %s)." % rp, "STATUS jump key") if rp else None
         else:
             key = campus_key(cid, name, {"state": "sp_state_" + cid, "heartbeat": "sp_hb_" + cid},
-                             [select_action(cid), make_page.set_page_action("campus_" + cid, "go", view_page)],
+                             select_action(cid) + [make_page.set_page_action("campus_" + cid, "go", view_page)],
                              "Tap: its status page, live (page %d)." % view_page, local_cfg)
-        make_page.place(controls, cell, key)
+            restart = make_page.static_button("campus_rs_" + cid, "%s\\nRESTART" % tag, 0x37474F, 0xFFFFFF, view_page,
+                                              "Tap: %s's Restart page, live (page %d); hold a key there to restart it at %s." % (name, view_page, name),
+                                              "STATUS jump key")
+            restart["steps"]["0"]["action_sets"]["up"] = select_action(cid, "RESTART") + restart["steps"]["0"]["action_sets"]["up"]
+        make_page.place(controls, (0, col), key)
+        if restart:
+            make_page.place(controls, (1, col), restart)
     return make_page.page_export("%d\\nCampuses" % page, page, controls, local_cfg)
 
 
@@ -179,7 +196,7 @@ def ensure_variables(cfg, ids):
     from settings import campus
     have, _ = campus.companion_state(cfg)
     want = [n for cid in ids for n in ("sp_state_" + cid, "sp_hb_" + cid) if n not in have]
-    want += [n for n in (VIEW["state"], VIEW["heartbeat"], SELECT, VIEW_CMD) if n not in have]
+    want += [n for n in (VIEW["state"], VIEW["heartbeat"], SELECT, VIEW_CMD, MODE) if n not in have]
     if not want:
         return
     comp = cfg.get("companion") or {}
@@ -250,6 +267,7 @@ class Mirror:
         self.site = site
         self.remote = {}           # id -> last /api/state answer
         self.keys = {}             # id -> [{"id", "pos"}] of its status page (from its /api/config)
+        self.rkeys = {}            # id -> the same for its Restart page
 
     @staticmethod
     def is_hub(data):
@@ -294,6 +312,7 @@ class Mirror:
             if st and (layouts.get(cid) != st.get("layout") or cid not in self.keys):
                 remote_cfg = http_json(e["url"].rstrip("/") + "/api/config", timeout=8)["config"]
                 self.keys[cid] = [{"id": t["id"], "pos": t["pos"]} for t in remote_cfg.get("tiles", []) if t.get("pos")]
+                self.rkeys[cid] = [{"id": k["id"], "pos": k["pos"]} for k in (remote_cfg.get("restart_page") or {}).get("keys", []) if k.get("pos")]
                 layouts[cid] = st.get("layout")
         ensure_variables(cfg, list(others))
         taken = self.taken_pages(cfg) | {p for p in pages.values() if p}
@@ -313,13 +332,14 @@ class Mirror:
         save(self.site.data_dir, data)
         return data
 
-    def pass_on(self, pick, others):
+    def pass_on(self, pick, others, restart=False):
         """A tap or hold on the CAMPUS page goes to the shown campus's watchdog, through its Companion."""
         text = (self.site.comp.get_custom(VIEW_CMD) or "").strip()
         if not text:
             return
         self.site.comp.set_custom(VIEW_CMD, "")
-        cmds = campus_command(self.keys.get(pick), text, time.time()) if pick else []
+        keys = (self.rkeys if restart else self.keys).get(pick)
+        cmds = campus_command(keys, text, time.time(), restart) if pick else []
         if not cmds:
             return
         host = urllib.parse.urlparse(others[pick]["url"]).hostname
@@ -362,20 +382,23 @@ class Mirror:
             # Nothing tapped yet (or a campus that left): show the first one rather than a purple page.
             pick = sorted(others)[0]
             self.site.comp.set_custom(SELECT, pick.upper())
-        self.pass_on(pick if pick in others else None, others)
+        restart = (self.site.comp.get_custom(MODE) or "").strip().upper() == "RESTART"
+        self.pass_on(pick if pick in others else None, others, restart)
         st = self.remote.get(pick)
-        if st and pick in self.keys and (data.get("pages") or {}).get("view"):
+        keys = (self.rkeys if restart else self.keys).get(pick)
+        if st and keys is not None and (data.get("pages") or {}).get("view"):
             try:
                 state = json.loads(st.get("state") or "{}")
             except ValueError:
                 state = {}
-            self.site.comp.set_custom(VIEW["state"], json.dumps(view_state(state, self.keys[pick]), separators=(",", ":")))
+            self.site.comp.set_custom(VIEW["state"], json.dumps(view_state(state, keys), separators=(",", ":")))
             self.site.comp.set_custom(VIEW["heartbeat"], str(st.get("hb") or "0"))
 
 
-def campus_command(keys, text, now):
+def campus_command(keys, text, now, restart=False):
     """The CAMPUS page's taps and holds ("tap 1 3 <time>;hold 0 2 <time>") as that campus's own
-    commands ("ack hd1", "restart hd1 <time>"), for the keys at those cells. Stale ones are dropped."""
+    commands ("ack hd1", "restart hd1 <time>"; on its Restart page a hold is "force rs_x <time>" and a
+    tap does nothing, as on its own deck), for the keys at those cells. Stale ones are dropped."""
     at = {tuple(k["pos"]): k["id"] for k in keys or []}
     out = []
     for part in (text or "").split(";"):
@@ -385,6 +408,10 @@ def campus_command(keys, text, now):
         tid = at.get((int(w[1]), int(w[2])))
         if not tid or now - int(w[3]) > 15:
             continue
+        if restart:
+            if w[0] == "hold":
+                out.append("force %s %s" % (tid, w[3]))
+            continue
         out.append("ack %s" % tid if w[0] == "tap" else "restart %s %s" % (tid, w[3]))
     return out
 

settings/discover.py

@@ -136,6 +136,11 @@ def this_mac():
     info = {"name": run(["/usr/sbin/scutil", "--get", "ComputerName"]).strip(),
             "hostname": (run(["/usr/sbin/scutil", "--get", "LocalHostName"]).strip() + ".local").lstrip("."),
             "ips": [ip for n, (ip, _) in sorted(ifs.items()) if not ip.startswith("127.")]}
+    # What runs on this Mac itself (ProPresenter on Woonsocket's Companion Mac), with each app's path.
+    ps = run(["/bin/ps", "-axo", "comm="]).splitlines()
+    info["procs"] = sorted({os.path.basename(p.strip()) for p in ps if p.strip()})
+    info["paths"] = {os.path.basename(p.strip()): p.strip().split(".app/")[0] + ".app" for p in ps if ".app/Contents/MacOS/" in p}
+    info["exes"] = {os.path.basename(p.strip()): p.strip() for p in ps if ".app/Contents/MacOS/" in p}
     if wired:
         ip, net = ifs[wired]
         # A /24 or smaller on a campus usually sits inside a /16 the rest of the gear shares: use what is configured.

settings/server.py

@@ -692,6 +692,27 @@ class Handler(http.server.BaseHTTPRequestHandler):
                 return self.send(200, campus.install_script(self.base_url(), q.get("id", ["new"])[0], key), "text/plain; charset=utf-8")
             if path == "/campus/program.tgz":
                 return self.send(200, campus.program_kit(site.data_dir), "application/gzip")
+            if path == "/campus/companion.zip":
+                # Companion 5.0.7 as this Mac has it, for `settings.campus upgrade-companion` on a campus
+                # still on an older one (packed once, kept beside the config, streamed).
+                app = "/Applications/Companion 5.0.7.app"
+                pack = os.path.join(site.data_dir, "companion-5.0.7.zip")
+                if not os.path.exists(app):
+                    return self.send(404, {"error": "no Companion 5.0.7 on this Mac"})
+                if not os.path.exists(pack):
+                    subprocess.run(["/usr/bin/ditto", "-c", "-k", "--keepParent", app, pack + ".tmp"], check=True, timeout=900)
+                    os.replace(pack + ".tmp", pack)
+                self.send_response(200)
+                self.send_header("Content-Type", "application/zip")
+                self.send_header("Content-Length", str(os.path.getsize(pack)))
+                self.end_headers()
+                with open(pack, "rb") as f:
+                    while True:
+                        chunk = f.read(1 << 20)
+                        if not chunk:
+                            break
+                        self.wfile.write(chunk)
+                return
             if path == "/join":
                 return self.send(200, campus.join_mac(site.load(), self.base_url()), "text/plain; charset=utf-8")
             if path == "/join.ps1":
@@ -893,7 +914,9 @@ class Handler(http.server.BaseHTTPRequestHandler):
                 lst = data.setdefault("list", {})
                 for other in [k for k, e in lst.items() if k != cid and (e or {}).get("url") == url]:
                     lst.pop(other)                             # the same Mac under its old name (before its first setup)
-                lst[cid] = {"id": cid, "name": str(body.get("name") or cid)[:40], "url": url}
+                user = str(body.get("user") or "")
+                lst[cid] = dict({"id": cid, "name": str(body.get("name") or cid)[:40], "url": url},
+                                **({"user": user} if re.fullmatch(r"[A-Za-z0-9._-]{1,40}", user) else {}))
                 campuses.save(site.data_dir, data)
                 return self.send(200, {"ok": True})
             if path == "/api/fleet/hello":

sites/ws/config.json

@@ -0,0 +1,581 @@
+{
+  "site": {
+    "id": "ws",
+    "name": "Woonsocket",
+    "ingest_name": "WS ProPresenter",
+    "ingest_hostname": "WS-ProPresenter-3.local",
+    "ingest_ips": [
+      "10.2.2.210",
+      "169.254.104.16"
+    ],
+    "contacts": "",
+    "live_services": [],
+    "editing_code": "9248"
+  },
+  "companion": {
+    "host": "127.0.0.1",
+    "port": 8000
+  },
+  "page": 19,
+  "page_name": "19\\nStatus",
+  "page_hold_ms": 2000,
+  "variables": {
+    "state": "sp_state",
+    "heartbeat": "sp_hb",
+    "command": "sp_cmd"
+  },
+  "timing": {
+    "fix_cooldown_seconds": 60,
+    "alert_after_seconds": 30
+  },
+  "auto_fix": {
+    "after_seconds": 20,
+    "max_tries": 2,
+    "retry_seconds": 90,
+    "pause_after_network_seconds": 60,
+    "restart_connections": true,
+    "windows": [
+      {
+        "days": [
+          "tue"
+        ],
+        "start": "14:00",
+        "end": "22:00"
+      },
+      {
+        "days": [
+          "wed"
+        ],
+        "start": "09:00",
+        "end": "21:00"
+      },
+      {
+        "days": [
+          "sun"
+        ],
+        "start": "05:00",
+        "end": "14:00"
+      }
+    ],
+    "fixes": [
+      "superrack_ws_foh",
+      "x32edit_ws_foh",
+      "vista_ws_lights",
+      "ableton_ws_tracks",
+      "propresenter_ingest",
+      "midi_reconnect"
+    ]
+  },
+  "tiles": [
+    {
+      "id": "all",
+      "label": "ALL",
+      "pos": [
+        0,
+        1
+      ],
+      "kind": "master"
+    },
+    {
+      "id": "service",
+      "label": "ALARMS",
+      "pos": [
+        3,
+        0
+      ],
+      "kind": "service"
+    },
+    {
+      "id": "net",
+      "label": "NET",
+      "pos": [
+        3,
+        7
+      ],
+      "kind": "network"
+    },
+    {
+      "id": "superrack",
+      "label": "SUPERRACK",
+      "checks": [
+        {
+          "type": "remote",
+          "template": "mac_program",
+          "params": {
+            "host": "ws_foh",
+            "process": "SuperRack SoundGrid"
+          },
+          "msg": "SUPERRACK NOT RUNNING",
+          "fix": "superrack_ws_foh",
+          "host": "ws_foh",
+          "cmd": "pgrep -xq 'SuperRack SoundGrid'"
+        },
+        {
+          "type": "remote",
+          "template": "soundgrid",
+          "params": {
+            "host": "ws_foh"
+          },
+          "msg": "SOUNDGRID LOST",
+          "host": "ws_foh",
+          "cmd": "lsof -nP -iTCP -sTCP:ESTABLISHED -a -c SuperRack 2>/dev/null | grep -q ':19000 (ESTABLISHED)'"
+        }
+      ],
+      "pos": [
+        0,
+        2
+      ]
+    },
+    {
+      "id": "x32_edit",
+      "label": "X32 EDIT",
+      "checks": [
+        {
+          "type": "remote",
+          "template": "mac_program",
+          "params": {
+            "host": "ws_foh",
+            "process": "X32-Edit"
+          },
+          "msg": "X32 EDIT NOT RUNNING",
+          "fix": "x32edit_ws_foh",
+          "host": "ws_foh",
+          "cmd": "pgrep -xq X32-Edit"
+        }
+      ],
+      "pos": [
+        0,
+        3
+      ]
+    },
+    {
+      "id": "m32",
+      "label": "M32",
+      "checks": [
+        {
+          "type": "conn",
+          "conn": "m32"
+        }
+      ],
+      "pos": [
+        0,
+        4
+      ]
+    },
+    {
+      "id": "vista",
+      "label": "VISTA",
+      "checks": [
+        {
+          "type": "remote",
+          "template": "mac_program",
+          "params": {
+            "host": "ws_lights",
+            "process": "Vista"
+          },
+          "msg": "VISTA NOT RUNNING",
+          "fix": "vista_ws_lights",
+          "host": "ws_lights",
+          "cmd": "pgrep -xq Vista"
+        }
+      ],
+      "pos": [
+        0,
+        5
+      ]
+    },
+    {
+      "id": "ableton",
+      "label": "ABLETON",
+      "checks": [
+        {
+          "type": "remote",
+          "template": "mac_program",
+          "params": {
+            "host": "ws_tracks",
+            "process": "Live"
+          },
+          "msg": "ABLETON NOT RUNNING",
+          "fix": "ableton_ws_tracks",
+          "host": "ws_tracks",
+          "cmd": "pgrep -xq Live"
+        }
+      ],
+      "pos": [
+        0,
+        6
+      ]
+    },
+    {
+      "id": "prop",
+      "label": "PROP",
+      "checks": [
+        {
+          "type": "conn",
+          "conn": "Propresenter-API"
+        },
+        {
+          "type": "process",
+          "match": "/Applications/ProPresenter.app/Contents/MacOS/ProPresenter",
+          "name": "PROP",
+          "fix": "propresenter_ingest"
+        }
+      ],
+      "pos": [
+        0,
+        7
+      ]
+    },
+    {
+      "id": "midi",
+      "label": "MIDI",
+      "checks": [
+        {
+          "type": "midi_peer",
+          "session": "WS FOH Mac mini",
+          "peer": "WS Tracks",
+          "address": "10.2.1.219:5004",
+          "fix": "midi_reconnect",
+          "tag": "WS TRACKS",
+          "host": "ws_foh"
+        },
+        {
+          "type": "midi_peer",
+          "session": "Lighting",
+          "peer": "WS Tracks",
+          "address": "10.2.1.219:5004",
+          "fix": "midi_reconnect",
+          "tag": "WS TRACKS",
+          "host": "ws_lights"
+        },
+        {
+          "type": "midi_wired",
+          "nets": [
+            "10.2.0.0/16"
+          ],
+          "fix": "midi_reconnect"
+        }
+      ],
+      "pos": [
+        1,
+        1
+      ]
+    },
+    {
+      "id": "atem",
+      "label": "ATEM",
+      "checks": [
+        {
+          "type": "conn",
+          "conn": "ATEM"
+        }
+      ],
+      "pos": [
+        1,
+        2
+      ]
+    },
+    {
+      "id": "resi",
+      "label": "RESI",
+      "checks": [
+        {
+          "type": "conn",
+          "conn": "Resi"
+        },
+        {
+          "type": "ping",
+          "conn": "Resi"
+        }
+      ],
+      "pos": [
+        1,
+        3
+      ]
+    }
+  ],
+  "fixes": {
+    "superrack_ws_foh": {
+      "template": "mac_start",
+      "params": {
+        "host": "ws_foh",
+        "process": "SuperRack SoundGrid",
+        "app": "/Applications/Waves/SuperRack SoundGrid/SuperRack SoundGrid.app"
+      },
+      "label": "start SuperRack on WS FOH if it is not running",
+      "short": "SUPERRACK",
+      "kind": "ssh",
+      "host": "ws_foh",
+      "run": "pgrep -xq 'SuperRack SoundGrid' || open '/Applications/Waves/SuperRack SoundGrid/SuperRack SoundGrid.app'"
+    },
+    "restart_superrack_ws_foh": {
+      "template": "mac_restart",
+      "params": {
+        "host": "ws_foh",
+        "process": "SuperRack SoundGrid",
+        "app": "/Applications/Waves/SuperRack SoundGrid/SuperRack SoundGrid.app",
+        "wait_seconds": 10,
+        "force": true,
+        "background": false,
+        "stop_children": false,
+        "newest_of": ""
+      },
+      "label": "restart SuperRack on WS FOH (closes it first)",
+      "short": "SUPERRACK",
+      "kind": "ssh",
+      "host": "ws_foh",
+      "run": "pkill -TERM -x 'SuperRack SoundGrid'; i=0; while pgrep -xq 'SuperRack SoundGrid' && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq 'SuperRack SoundGrid' && pkill -KILL -x 'SuperRack SoundGrid'; sleep 1; open '/Applications/Waves/SuperRack SoundGrid/SuperRack SoundGrid.app'"
+    },
+    "x32edit_ws_foh": {
+      "template": "mac_start",
+      "params": {
+        "host": "ws_foh",
+        "process": "X32-Edit",
+        "app": "/Applications/X32-Edit.app"
+      },
+      "label": "start X32-Edit on WS FOH if it is not running",
+      "short": "X32 EDIT",
+      "kind": "ssh",
+      "host": "ws_foh",
+      "run": "pgrep -xq X32-Edit || open /Applications/X32-Edit.app"
+    },
+    "vista_ws_lights": {
+      "template": "mac_start",
+      "params": {
+        "host": "ws_lights",
+        "process": "Vista",
+        "app": "/Applications/Vista.app"
+      },
+      "label": "start Vista on WS Lights if it is not running",
+      "short": "VISTA",
+      "kind": "ssh",
+      "host": "ws_lights",
+      "run": "pgrep -xq Vista || open /Applications/Vista.app"
+    },
+    "restart_vista_ws_lights": {
+      "template": "mac_restart",
+      "params": {
+        "host": "ws_lights",
+        "process": "Vista",
+        "app": "/Applications/Vista.app",
+        "wait_seconds": 10,
+        "force": true,
+        "background": false,
+        "stop_children": false,
+        "newest_of": ""
+      },
+      "label": "restart Vista on WS Lights (closes it first)",
+      "short": "VISTA",
+      "kind": "ssh",
+      "host": "ws_lights",
+      "run": "pkill -TERM -x Vista; i=0; while pgrep -xq Vista && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq Vista && pkill -KILL -x Vista; sleep 1; open /Applications/Vista.app"
+    },
+    "ableton_ws_tracks": {
+      "template": "ableton",
+      "params": {
+        "host": "ws_tracks",
+        "app": "Ableton Live 11 Suite",
+        "restart": false,
+        "wait_seconds": 15
+      },
+      "label": "start Ableton on WS Tracks if it is not running",
+      "short": "ABLETON",
+      "kind": "ssh",
+      "host": "ws_tracks",
+      "run": "pgrep -xq Live || { L=$(ls -td ~/Library/Preferences/Ableton/Live*/ | head -1); S=$(grep -a 'info: Loading document \"' \"${L}Log.txt\" | grep -v '/Templates/' | tail -1 | sed -E 's/^.*Loading document \"(.*)\"[^\"]*$/\\1/'); if [ -n \"$S\" ] && [ -f \"$S\" ]; then open -a 'Ableton Live 11 Suite' \"$S\"; else open -a 'Ableton Live 11 Suite'; fi; }"
+    },
+    "restart_ableton_ws_tracks": {
+      "template": "ableton",
+      "params": {
+        "host": "ws_tracks",
+        "app": "Ableton Live 11 Suite",
+        "restart": true,
+        "wait_seconds": 15
+      },
+      "label": "restart Ableton on WS Tracks (closes it first)",
+      "short": "ABLETON",
+      "kind": "ssh",
+      "host": "ws_tracks",
+      "run": "L=$(ls -td ~/Library/Preferences/Ableton/Live*/ | head -1); S=$(grep -a 'info: Loading document \"' \"${L}Log.txt\" | grep -v '/Templates/' | tail -1 | sed -E 's/^.*Loading document \"(.*)\"[^\"]*$/\\1/'); pkill -TERM -x Live; i=0; while pgrep -xq Live && [ $i -lt 30 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq Live && pkill -KILL -x Live; sleep 1; if [ -n \"$S\" ] && [ -f \"$S\" ]; then open -a 'Ableton Live 11 Suite' \"$S\"; else open -a 'Ableton Live 11 Suite'; fi"
+    },
+    "propresenter_ingest": {
+      "template": "mac_start",
+      "params": {
+        "host": "ingest",
+        "process": "ProPresenter",
+        "app": "/Applications/ProPresenter.app"
+      },
+      "label": "start ProPresenter on this Mac if it is not running",
+      "short": "PROP",
+      "kind": "local",
+      "run": "pgrep -xq ProPresenter || open /Applications/ProPresenter.app"
+    },
+    "restart_propresenter_ingest": {
+      "template": "mac_restart",
+      "params": {
+        "host": "ingest",
+        "process": "ProPresenter",
+        "app": "/Applications/ProPresenter.app",
+        "wait_seconds": 10,
+        "force": true,
+        "background": false,
+        "stop_children": false,
+        "newest_of": ""
+      },
+      "label": "restart ProPresenter on this Mac (closes it first)",
+      "short": "PROP",
+      "kind": "local",
+      "run": "pkill -TERM -x ProPresenter; i=0; while pgrep -xq ProPresenter && [ $i -lt 20 ]; do sleep 0.5; i=$((i+1)); done; pgrep -xq ProPresenter && pkill -KILL -x ProPresenter; sleep 1; open /Applications/ProPresenter.app"
+    },
+    "midi_reconnect": {
+      "kind": "midi_connect",
+      "template": "midi_connect",
+      "params": {},
+      "short": "RECONNECT",
+      "label": "reconnect the peer to the network MIDI session it dropped out of (like Connect in Audio MIDI Setup), at its wired address"
+    }
+  },
+  "hosts": {
+    "ingest": {
+      "self": true,
+      "name": "WS ProPresenter",
+      "role": "the watchdog, Companion, the settings page",
+      "os": "mac",
+      "networks": [],
+      "runs": [
+        {
+          "kind": "launchagent",
+          "label": "org.waterschurch.status-panel",
+          "name": "the watchdog"
+        },
+        {
+          "kind": "launchagent",
+          "label": "org.waterschurch.status-settings",
+          "name": "the settings page"
+        },
+        {
+          "kind": "app",
+          "process": "Companion",
+          "name": "Companion",
+          "http": "http://127.0.0.1:8000/api/connections"
+        }
+      ]
+    },
+    "ws_foh": {
+      "name": "WS FOH",
+      "os": "mac",
+      "ssh": "watersmedia@10.2.1.12",
+      "role": "found by auto-setup",
+      "runs": [
+        {
+          "kind": "sshd"
+        }
+      ]
+    },
+    "ws_lights": {
+      "name": "WS Lights",
+      "os": "mac",
+      "ssh": "wslighting@10.2.2.2",
+      "role": "found by auto-setup",
+      "runs": [
+        {
+          "kind": "sshd"
+        }
+      ]
+    },
+    "ws_tracks": {
+      "name": "WS Tracks",
+      "os": "mac",
+      "ssh": "wstracks@10.2.1.219",
+      "role": "found by auto-setup",
+      "runs": [
+        {
+          "kind": "sshd"
+        }
+      ]
+    }
+  },
+  "gear": {},
+  "debounce_seconds": 3,
+  "network": {
+    "interface": "en14",
+    "gateway": "10.2.0.1",
+    "burst_connections": 4,
+    "burst_groups": 3,
+    "burst_window_seconds": 10,
+    "remember_seconds": 300
+  },
+  "nav_keys": [
+    "pagenum"
+  ],
+  "nav_hold_ms": 1000,
+  "nav_tap_page": 1,
+  "discover": {
+    "auto": true
+  },
+  "restart_page": {
+    "page": 20,
+    "page_name": "20\\nRestart",
+    "jump_pos": [
+      3,
+      6
+    ],
+    "back_pos": [
+      3,
+      0
+    ],
+    "keys": [
+      {
+        "id": "rs_superrack",
+        "label": "SUPERRACK",
+        "fix": "restart_superrack_ws_foh",
+        "pos": [
+          0,
+          1
+        ]
+      },
+      {
+        "id": "rs_vista",
+        "label": "VISTA",
+        "fix": "restart_vista_ws_lights",
+        "pos": [
+          0,
+          2
+        ]
+      },
+      {
+        "id": "rs_ableton",
+        "label": "ABLETON",
+        "fix": "restart_ableton_ws_tracks",
+        "pos": [
+          0,
+          3
+        ]
+      },
+      {
+        "id": "rs_prop",
+        "label": "PROP",
+        "fix": "restart_propresenter_ingest",
+        "pos": [
+          0,
+          4
+        ]
+      }
+    ]
+  },
+  "service_windows": [
+    {
+      "days": [
+        "mon",
+        "tue",
+        "wed",
+        "thu",
+        "fri",
+        "sat",
+        "sun"
+      ],
+      "start": "00:00",
+      "end": "23:59"
+    }
+  ]
+}

tests/test_campus.py

@@ -142,6 +142,34 @@ class TestBuild(unittest.TestCase):
         chk = {"type": "midi_peer", "host": "tracks", "session": "WS Tracks", "peer": "Lighting"}
         self.assertEqual(sp.check_midi_peer(chk, snap, {"now": time.time()})[0][1].level, sp.OK)   # no state: older macOS, configured
 
+    def test_a_program_on_the_companion_mac_gets_a_local_check_and_fixes(self):
+        """Woonsocket: ProPresenter runs on the Mac that runs Companion."""
+        inv = small()
+        inv["computers"] = {}
+        exe = "/Applications/ProPresenter.app/Contents/MacOS/ProPresenter"
+        inv["ingest"].update(procs=["ProPresenter"], paths={"ProPresenter": "/Applications/ProPresenter.app"}, exes={"ProPresenter": exe})
+        inv["connections"] = [dict(c, host="10.4.1.1") if c["label"] == "ProP" else c for c in inv["connections"]]
+        cfg, _ = ac.build(BASE, inv, {"id": "ws", "name": "Woonsocket"}, "1234")
+        self.assertEqual(schema.validate(cfg)[0], [])
+        prop = next(t for t in cfg["tiles"] if t["label"] == "PROP")
+        self.assertEqual([(c["type"], c.get("match") or c.get("conn")) for c in prop["checks"]], [("conn", "ProP"), ("process", exe)])
+        fix = cfg["fixes"][prop["checks"][1]["fix"]]
+        self.assertEqual((fix["kind"], fix["run"]), ("local", "pgrep -xq ProPresenter || open /Applications/ProPresenter.app"))
+        self.assertIn(prop["checks"][1]["fix"], cfg["auto_fix"]["fixes"])
+        self.assertEqual([k["label"] for k in cfg["restart_page"]["keys"]], ["PROP"])
+        self.assertEqual(cfg["fixes"][cfg["restart_page"]["keys"][0]["fix"]]["kind"], "local")
+
+    def test_every_campus_gets_a_restart_page_and_north_attleboros_auto_fix_hours(self):
+        inv = small()
+        inv["computers"] = {}
+        cfg, _ = ac.build(BASE, inv, {"id": "sc", "name": "Southcoast"}, "1234")
+        self.assertEqual(cfg["restart_page"]["keys"], [])                            # computers join later: their keys come then
+        na = json.load(open(os.path.join(ROOT, "config.json")))
+        self.assertEqual(cfg["auto_fix"]["windows"], na["auto_fix"]["windows"])
+        self.assertEqual(cfg["service_windows"], na["service_windows"])
+        self.assertEqual(cfg["nav_keys"], na["nav_keys"])
+        make_page.build_restart(cfg)
+
     def test_an_m32_is_called_m32(self):
         inv = small()
         inv["connections"] = [c for c in inv["connections"] if c["label"] != "x32"] + [
@@ -188,6 +216,21 @@ class TestExtend(unittest.TestCase):
         again, added2, _ = ac.extend(new, inv, set(self.seen) | set(now_seen))
         self.assertEqual(added2, [])
 
+    def test_a_program_found_later_joins_its_connections_key(self):
+        """Woonsocket got a second PROP key when ProPresenter on the Companion Mac was found after the connection."""
+        inv = small()
+        inv["computers"] = {}
+        cfg, seen = ac.build(BASE, inv, {"id": "ws", "name": "Woonsocket"}, "1234")
+        exe = "/Applications/ProPresenter.app/Contents/MacOS/ProPresenter"
+        inv["ingest"].update(procs=["ProPresenter"], paths={"ProPresenter": "/Applications/ProPresenter.app"}, exes={"ProPresenter": exe})
+        inv["connections"] = [dict(c, host="127.0.0.1") if c["label"] == "ProP" else c for c in inv["connections"]]
+        new, added, _ = ac.extend(cfg, inv, seen)
+        self.assertEqual([t["label"] for t in new["tiles"]].count("PROP"), 1)
+        prop = next(t for t in new["tiles"] if t["label"] == "PROP")
+        self.assertEqual([c["type"] for c in prop["checks"]], ["conn", "process"])
+        self.assertIn("PROP checks on the PROP key", added)
+        self.assertEqual(schema.validate(new)[0], [])
+
     def test_a_key_someone_took_off_is_not_added_back(self):
         cfg = json.loads(json.dumps(self.cfg))
         cfg["tiles"] = [t for t in cfg["tiles"] if t["label"] != "X32"]
@@ -338,12 +381,17 @@ class TestCampusesView(unittest.TestCase):
         export = cv.build_overview(na, roster, 98, 99)
         keys = [(r, c, ctl) for r, row in export["page"]["controls"].items() for c, ctl in row.items() if "'campus " in (ctl.get("options") or {}).get("notes", "")]
         self.assertEqual([(r, c) for r, c, _ in keys], [("0", "1"), ("0", "2"), ("0", "3")])
+        rs = export["page"]["controls"]["1"]["3"]                                   # WS's RESTART key, under its ALL key
+        acts = rs["steps"]["0"]["action_sets"]["up"]
+        self.assertEqual([a["options"].get("value", {}).get("value") for a in acts[:2]] + [acts[2]["options"]["page"]["value"]], ["WS", "RESTART", 99])
+        self.assertEqual(cv.campus_command([{"id": "rs_prop", "pos": [0, 1]}], "tap 0 1 1000;hold 0 1 1000", 1001, restart=True),
+                         ["force rs_prop 1000"])                                     # on its Restart page a hold forces, a tap does nothing
         self.assertIn("'campus na'", keys[0][2]["options"]["notes"])
         self.assertIn("page 19", keys[0][2]["options"]["notes"])
         ws_key = next(k for _, _, k in keys if "'campus ws'" in k["options"]["notes"])
         acts = ws_key["steps"]["0"]["action_sets"]["up"]
-        self.assertEqual([a["definitionId"] for a in acts], ["custom_variable_set_value", "set_page"])
+        self.assertEqual([a["definitionId"] for a in acts], ["custom_variable_set_value", "custom_variable_set_value", "set_page"])
-        self.assertEqual((acts[0]["options"]["value"]["value"], acts[1]["options"]["page"]["value"]), ("WS", 99))
+        self.assertEqual((acts[0]["options"]["value"]["value"], acts[1]["options"]["value"]["value"], acts[2]["options"]["page"]["value"]), ("WS", "", 99))
 
     def test_a_campus_tells_the_hub_and_learns_nothing_back(self):
         from settings import campuses as cv
@@ -365,7 +413,10 @@ class TestCampusesView(unittest.TestCase):
             m = cv.Mirror(FakeSite())
             data = m.sync_registry()
             m.poll()
-            self.assertEqual(m.refresh_pages()["list"], {"ws": {"id": "ws", "name": "Woonsocket", "url": "http://10.5.1.1:8020"}})
+            listed = m.refresh_pages()["list"]
+            self.assertEqual({k: {f: v for f, v in e.items() if f != "user"} for k, e in listed.items()},
+                             {"ws": {"id": "ws", "name": "Woonsocket", "url": "http://10.5.1.1:8020"}})
+            self.assertTrue(listed["ws"]["user"])                                        # the account the hub backs it up with
         finally:
             cv.http_json, cv.my_url = real, real_url
         self.assertEqual(list(data["list"]), ["ws"])                                 # the others are forgotten