christian/church-translate
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 11 Wiki Activity

One voice per listener, and the voice process stops stalling itself

Browse Source 
Build 127. Reported from a live test: part of the translation came out in a
different voice on an Android phone, and iPhones were skipping and repeating.
Three investigations against the campus Mac's own log and transcript, one
test harness driving the voice process directly, and the answer is not what
it sounded like.

WHAT THE MACHINE DID. One voice process ran all afternoon. Every language
kept a single resident engine from its first sentence (14:54:26) to STOP
(15:23:18); 2,800 clips, 560 per language, every one filed under its own
language and spoken by its own bundle. No synthesis-side voice change was
possible with that configuration. What the log shows at the time of the
complaint instead:

  - Someone on the English stream heard the ROOM MICROPHONE for 21 s
    (15:09:10-32). English was the first button on every phone's list, forced
    there by the page whatever the operator had chosen (fixed in build 125).
  - A listener tapped Chinese at 15:11:17. The 121 MB zh model was fetched
    and unpacked THREE TIMES AT ONCE on the voice process's only thread.
    Every stream stalled for about a minute, 26 sentences were dropped across
    all five languages, and the catch-up rushed the Spanish voices to their
    +16% cap. Same voice, sounding wrong.

WHAT WAS WRONG, AND IS NOW FIXED.

The live path filed every engine as an "audition". speak() resolved the
language's default voice, which always has a bundle, and passed it as
`explicit`, so the key was 'audition:'+bundle for every ordinary sentence
while the preload warmed copies under the language name that no sentence
ever read. Five languages asked for nine engines against a cap of six; every
service's first sentence built its real engines on the spot (1.3-2 s); the
operator's voice save flushed nothing that mattered. Engines are now keyed
by bundle and shared by every language, variant and audition that uses one,
auditions take at most two slots of their own and can never evict a live
engine, and the cap is 8, sized to the largest campus from measurement
(~165 MB per engine once it has spoken, 1.3 GB for eight).

Voice installs ran on the thread that answers every sentence, and were not
de-duplicated. They now run in a worker thread (voice-install.js), one per
bundle however many ask, staged and renamed into place whole so a half-
written bundle can never pass isInstalled().

Listener rows are { code, count } and engine.js read { lang, n } in five
places, so the set of "languages someone is on" was always empty — and a late
sentence keeps speech only for those. Under load the app dropped EVERY
language's speech instead of only the unchosen ones. Late became silent.

HLS segment numbers restarted at 1 for every new stream object, under a
'max-age=60, immutable' header. A phone back from the lock screen could be
answered from its cache with the previous stream's audio under the new
number: an earlier sentence, replayed. Numbers are seeded from the clock.

In clips mode a clip still downloading when the listener switched language
landed and played in the voice they had left; the frame now carries its
language and voice and the page checks both when the bytes land. In stream
mode the old playlist stayed loaded through a switch and three things could
restart it; it is stopped, not paused. A listener's alternate voice no longer
loses a round trip to the default on the original-text toggle.

One genuine wrong-voice path, dormant today because natural voices are off on
this campus: a variant waiting on another variant's premium synthesis
inherited that variant's Piper FALLBACK, in that variant's bundle — spa_ES
received the byte-identical Mexican clip and its own voice never loaded. A
follower now shares a fallback only when it is its own voice.

An audition (the Voices card's play button) played on the system default
output, which on the campus Mac is the Dante card feeding the hardware
channels. It now plays on another output, or refuses and says why.

VERIFIED: a harness drives the voice process through the baseline, a batch
of four auditions during live sentences, a voices save mid-stream, and nine
live languages against the cache — every clip spoken by its own bundle, and
auditions of a live voice now cost 78-311 ms with nothing rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-15 16:05:15 -04:00
co-authored by Claude Fable 5.1
parent 95d1a7b9ec
commit 81da657094
7 changed files with 426 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/engine.js
+73 -9
View File
@@ -195,6 +195,23 @@ const state = {
degradedAt: 0
};
/*
* THE SHAPE OF A LISTENER ROW. The server sends { code, count } — see
* listenerCounts() in server.js — and this file read { lang, n } in five
* places, so every set built from the rows was EMPTY: no language ever counted
* as being listened to. Mostly harmless, except in the one place it mattered.
* When a sentence arrives late, speech is kept only for the languages someone
* is actually on, and "someone is on" was never true for any of them — so
* under load the app dropped the speech of EVERY language, for everyone,
* instead of only the languages nobody had chosen. Late became silent.
* MEASURED on a campus Mac, 2026-10-15 15:11-15:12: 26 sentences dropped
* across five languages during a stall, every one of them with listeners
* present. Both spellings are read, so a server one build older or newer
* still counts.
*/
function rowLang(r) { return r ? (r.code || r.lang || '') : ''; }
function rowCount(r) { return r ? (Number(r.count != null ? r.count : r.n) || 0) : 0; }
/*
* FOLLOW THE BOTTOM ONLY WHILE THE OPERATOR IS AT THE BOTTOM.
*
@@ -1070,10 +1087,15 @@ function preloadNewVoices() {
const want = state.targets.filter((l) =>
state.spokenLanguages.indexOf(l) !== -1 && !preloadedVoices.has(l));
if (!want.length) return;
want.forEach((l) => preloadedVoices.add(l));
/* One at a time in the child anyway; cap the request so a burst of joins
cannot queue up a re-download marathon. */
window.engineHost.ttsPreload(want.slice(0, 4));
cannot queue up a re-download marathon.
Only what is SENT counts as warmed. Every wanted language used to be
marked and only the first four sent, so a fifth was never preloaded and
built its engine on its first sentence instead — por_PT, at all five
starts on the Mini on 2026-10-15. Six is the voice process's own cap. */
const batch = want.slice(0, 6);
batch.forEach((l) => preloadedVoices.add(l));
window.engineHost.ttsPreload(batch);
}
/* The translation model reports its download from the main process. */
@@ -1084,7 +1106,10 @@ if (window.engineHost && window.engineHost.onTtsReady) {
state.spokenDefaults = info.defaults || state.spokenDefaults || {};
log('built-in voices ready — ' + state.spokenLanguages.length + ' languages', 'ok');
/* Warm whatever people are already reading, so the first sentence of a
service is not the one that pays for the download. */
service is not the one that pays for the download. A voice process
that has just (re)started holds no engines, whatever this page warmed
in the last one — so forget that first. */
preloadedVoices.clear();
preloadNewVoices();
});
window.engineHost.onTtsProgress((p) => log('voice ' + p.bundle + ': ' + p.status));
@@ -1778,9 +1803,9 @@ async function pumpPartial() {
* still gets a preview.
*/
const heard = (state.byLanguage || [])
.filter((r) => r && r.lang && r.lang !== state.source && (r.n || 0) > 0)
.filter((r) => r && rowLang(r) && rowLang(r) !== state.source && rowCount(r) > 0)
.sort((a, b) => (b.n || 0) - (a.n || 0))
.map((r) => r.lang)
.map(rowLang)
.filter((l) => state.targets.indexOf(l) !== -1);
const want = (heard.length ? heard : orderedTargets()).slice(0, PARTIAL_MAX_LANGS);
/* Never queue behind a finished sentence — a final in flight is worth more
@@ -1806,7 +1831,7 @@ async function pumpPartial() {
*/
function orderedTargets() {
const counts = {};
for (const row of state.byLanguage || []) counts[row.lang] = row.n || 1;
for (const row of state.byLanguage || []) counts[rowLang(row)] = rowCount(row) || 1;
return state.targets
.filter((t) => t !== state.source)
.sort((a, b) => (counts[b] || 0) - (counts[a] || 0));
@@ -1969,7 +1994,7 @@ async function pump() {
/* Late, but nothing is behind it: say so, and translate it anyway. */
log('audio ' + ageSec.toFixed(0) + 's late, but nothing is queued — translating it anyway', 'warn');
} else if (ageSec > 6) {
const listening = new Set((state.byLanguage || []).map((r) => r.lang));
const listening = new Set((state.byLanguage || []).map(rowLang));
const before = targets.length;
targets = targets.filter((t) => listening.has(t));
if (targets.length < before) { state.degradedFinals++; state.degradedAt = Date.now(); }
@@ -2169,7 +2194,7 @@ setInterval(() => {
}, 30000);
function translateSentence(cid, sentence, targets, gainDb, silent) {
const listening = new Set((state.byLanguage || []).map((r) => r.lang));
const listening = new Set((state.byLanguage || []).map(rowLang));
for (const lang of targets) {
if ((mtDepth[lang] || 0) > 6 && !listening.has(lang)) {
state.degradedFinals++;
@@ -4643,6 +4668,40 @@ function renderVoiceList() {
}
}
/*
* WHERE AN AUDITION PLAYS: never on the card that feeds the room.
*
* An <audio> element with no sink plays on the system default output, and on
* the production Mini that IS the Dante Virtual Soundcard (system_profiler:
* "Default Output Device: Yes") — the same card that carries Spanish on its
* third channel (outputMap spa_Latn -> [2]). So ▶ there put the sample line
* onto the Dante network, service or no service. The phones were never
* exposed: an audition comes back to this page as bytes and never becomes a
* clip the server can see.
*
* Which of the card's channels it reached is Chromium's mono up-mix and the
* Dante routing, neither of which this page controls, so it does not try to
* pick a "safe" channel. It plays on another output, or not at all, and says
* why.
*/
async function auditionSink() {
const routedId = ($('outDevice') && $('outDevice').value) || '';
const routedLabel = (state.cfg && state.cfg.outputDeviceLabel) || '';
if (!routedId && !routedLabel) return ''; // no hardware output in use
const strip = (l) => String(l || '').toLowerCase().replace(/^default\s*-\s*/, '').trim();
let outs = [];
try {
outs = (await navigator.mediaDevices.enumerateDevices()).filter((d) => d.kind === 'audiooutput');
} catch (_) { /* refused below rather than guessed */ }
const card = strip(((outs.find((d) => d.deviceId === routedId) || {}).label) || routedLabel);
const isCard = (d) => (!!routedId && d.deviceId === routedId) || (!!card && strip(d.label) === card);
const def = outs.find((d) => d.deviceId === 'default');
if (def && !isCard(def)) return ''; // the system default is somewhere else
const other = outs.find((d) => d.deviceId !== 'default' && d.deviceId !== 'communications' && !isCard(d));
if (other) return other.deviceId;
throw new Error('the only output here is the one feeding the hardware channels — auditions stay off it');
}
async function auditionVoice(lang, v, btn, msg) {
if (auditionBusy) return;
auditionBusy = true;
@@ -4667,9 +4726,14 @@ async function auditionVoice(lang, v, btn, msg) {
msg.className = 'vmsg';
}
if (instruct) msg.textContent = 'with your direction…';
/* Where it will play is settled BEFORE the voice process builds anything:
an audition that cannot play should cost the service nothing. */
const sink = await auditionSink();
const out = await window.engineHost.ttsAudition({ lang, voice: v, instruct });
if (!auditionAudio) auditionAudio = new Audio();
if (!out.wav) throw new Error('that voice produced no audio');
if (typeof auditionAudio.setSinkId === 'function') await auditionAudio.setSinkId(sink);
else if (sink) throw new Error('this build cannot keep auditions off the hardware output');
auditionAudio.src = 'data:audio/wav;base64,' + out.wav;
/* play() rejects on autoplay policy or a decode failure; either way the
operator needs the reason, not a silent button. */