christian/church-translate
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 11 Wiki Activity

One voice per listener, and the voice process stops stalling itself

Browse Source 
Build 127. Reported from a live test: part of the translation came out in a
different voice on an Android phone, and iPhones were skipping and repeating.
Three investigations against the campus Mac's own log and transcript, one
test harness driving the voice process directly, and the answer is not what
it sounded like.

WHAT THE MACHINE DID. One voice process ran all afternoon. Every language
kept a single resident engine from its first sentence (14:54:26) to STOP
(15:23:18); 2,800 clips, 560 per language, every one filed under its own
language and spoken by its own bundle. No synthesis-side voice change was
possible with that configuration. What the log shows at the time of the
complaint instead:

  - Someone on the English stream heard the ROOM MICROPHONE for 21 s
    (15:09:10-32). English was the first button on every phone's list, forced
    there by the page whatever the operator had chosen (fixed in build 125).
  - A listener tapped Chinese at 15:11:17. The 121 MB zh model was fetched
    and unpacked THREE TIMES AT ONCE on the voice process's only thread.
    Every stream stalled for about a minute, 26 sentences were dropped across
    all five languages, and the catch-up rushed the Spanish voices to their
    +16% cap. Same voice, sounding wrong.

WHAT WAS WRONG, AND IS NOW FIXED.

The live path filed every engine as an "audition". speak() resolved the
language's default voice, which always has a bundle, and passed it as
`explicit`, so the key was 'audition:'+bundle for every ordinary sentence
while the preload warmed copies under the language name that no sentence
ever read. Five languages asked for nine engines against a cap of six; every
service's first sentence built its real engines on the spot (1.3-2 s); the
operator's voice save flushed nothing that mattered. Engines are now keyed
by bundle and shared by every language, variant and audition that uses one,
auditions take at most two slots of their own and can never evict a live
engine, and the cap is 8, sized to the largest campus from measurement
(~165 MB per engine once it has spoken, 1.3 GB for eight).

Voice installs ran on the thread that answers every sentence, and were not
de-duplicated. They now run in a worker thread (voice-install.js), one per
bundle however many ask, staged and renamed into place whole so a half-
written bundle can never pass isInstalled().

Listener rows are { code, count } and engine.js read { lang, n } in five
places, so the set of "languages someone is on" was always empty — and a late
sentence keeps speech only for those. Under load the app dropped EVERY
language's speech instead of only the unchosen ones. Late became silent.

HLS segment numbers restarted at 1 for every new stream object, under a
'max-age=60, immutable' header. A phone back from the lock screen could be
answered from its cache with the previous stream's audio under the new
number: an earlier sentence, replayed. Numbers are seeded from the clock.

In clips mode a clip still downloading when the listener switched language
landed and played in the voice they had left; the frame now carries its
language and voice and the page checks both when the bytes land. In stream
mode the old playlist stayed loaded through a switch and three things could
restart it; it is stopped, not paused. A listener's alternate voice no longer
loses a round trip to the default on the original-text toggle.

One genuine wrong-voice path, dormant today because natural voices are off on
this campus: a variant waiting on another variant's premium synthesis
inherited that variant's Piper FALLBACK, in that variant's bundle — spa_ES
received the byte-identical Mexican clip and its own voice never loaded. A
follower now shares a fallback only when it is its own voice.

An audition (the Voices card's play button) played on the system default
output, which on the campus Mac is the Dante card feeding the hardware
channels. It now plays on another output, or refuses and says why.

VERIFIED: a harness drives the voice process through the baseline, a batch
of four auditions during live sentences, a voices save mid-stream, and nine
live languages against the cache — every clip spoken by its own bundle, and
auditions of a live voice now cost 78-311 ms with nothing rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-15 16:05:15 -04:00
co-authored by Claude Fable 5.1
parent 95d1a7b9ec
commit 81da657094
7 changed files with 426 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/server/live-audio.js
+15 -1
View File
@@ -219,7 +219,21 @@ class LanguageStream {
this.dropTimes = []; // when, for a "recently" view on the operator page
this.building = []; // frames of the segment being filled
this.segments = []; // { seq, body, frames }
this.nextSeq = 1;
/*
* SEGMENT NUMBERS MUST NEVER REPEAT, and starting at 1 made them repeat.
* A stream object is built when a language's first listener arrives and
* dropped when the last one leaves, so within one service the same
* language can get a fresh stream several times — and a phone that came
* back from the lock screen, or a second listener, asked for
* spa_Latn/1.aac again. The segments are served 'max-age=60, immutable',
* which is the right header for a segment that is never reused and the
* wrong one for a number that is: for up to a minute the browser answered
* from its cache with the PREVIOUS stream's audio under the new number —
* a sentence from earlier in the sermon, replayed. Numbering from the
* clock makes every incarnation's numbers new; HLS only asks that they
* increase by one within a playlist, which they still do.
*/
this.nextSeq = Date.now();
this.timer = null;
this.lastTick = 0;
this.carry = 0; // fractional frames owed from the last tick
app/server/public/listen.js
+62 -8
View File
@@ -475,6 +475,7 @@
drift: null,
watchdog: null, // catches engines that stall on HLS without erroring
everPlayed: false, // this engine has PROVEN it can play HLS
clipsForVoice: false, // on clips because of an alternate voice, not because HLS failed
// clip-fallback state
ctx: null,
keepAlive: null,
@@ -594,9 +595,13 @@
/* Give up on streaming and use per-sentence clips. The clip path was already
unlocked inside the tap precisely so this is possible after the gesture has
been spent — otherwise a browser that lies about HLS would end up silent. */
function fallBackToClips() {
function fallBackToClips(forVoice) {
if (srv.mode === 'clips') return;
srv.mode = 'clips';
/* Remember WHY. Only a page that left the stream for an alternate voice
returns to it when the listener goes back to the default — see
applyVoiceMode(). */
srv.clipsForVoice = !!forVoice;
stopStream();
if (srv.keepAlive) srv.keepAlive.play().catch(function () {});
try { console.warn('live stream unsupported here — using per-sentence clips'); } catch (_) {}
@@ -868,14 +873,27 @@
*/
function applyVoiceMode() {
if (srv.voice) {
fallBackToClips();
fallBackToClips(true);
return;
}
/* Back on the default: allow streaming again. fallBackToClips latches
mode to 'clips' and maybeStartStream refuses to act on that, so the
latch has to be released here or the listener stays on clips for the
rest of the service. */
if (srv.mode === 'clips' && hlsSupported() && srv.unlocked && srv.serverHello) {
/*
* Only when the alternate is what put us on clips. Every engine reconnect
* sends every listener a 'voices' frame, and an engine that cannot play
* HLS used to be handed back to the stream by it — eight seconds of
* dropped clips until the watchdog demoted it again.
*
* And silence the alternate's clips that are still scheduled. Left to
* play out they overlapped the shared stream as it came up: the voice
* the listener left and the default together, for up to the six seconds
* the clip queue holds.
*/
if (srv.mode === 'clips' && srv.clipsForVoice && hlsSupported() && srv.unlocked && srv.serverHello) {
stopClips();
srv.clipsForVoice = false;
srv.mode = '';
maybeStartStream();
}
@@ -909,22 +927,52 @@
function switchAudioLanguage(lang) {
stopClips(); // the old language stops NOW, mid-word
if (srv.mode === 'stream' && srv.el) {
/* Pause the old stream; the new one starts when the server's hello
confirms the new language's stream exists. */
try { srv.el.pause(); } catch (_) {}
if (srv.retry) { clearTimeout(srv.retry); srv.retry = null; }
/*
* STOP the old stream, do not merely pause it; the new one starts when
* the server's hello confirms the new language's stream exists.
*
* Paused, the element still pointed at the OLD playlist until that
* hello, and three things restart a paused element without asking: the
* 8 s watchdog's retry (startStream(srv.lang, true), and srv.lang was
* still the old language), coming back from a locked screen
* (visibilitychange plays it), and the lock-screen play button. Each
* put the previous language's voice back in the listener's ear.
* stopStream() is what the back button already does, and the hello's
* maybeStartStream() brings the new language up from it.
*/
stopStream();
}
setMediaSession();
}
// ------------------------------------------------- per-clip fallback path
/*
* Is this clip for the language AND voice this page is on right now?
*
* Asked twice — when the clip is announced and again when its bytes land —
* because the fetch and the decode take a few hundred milliseconds, and
* stopClips() can only silence what is already scheduled, not what is still
* downloading. A listener who switched language in that window heard the
* previous language's sentence AFTER the switch; for Spanish that is one
* sentence of the Mexican voice in a Spaniard's ear, or the reverse. The
* name carries the language (p<seq>-spa_ES.aac); the server also sends lang
* and voice in the frame, and a frame without them is judged on the name.
*/
function clipIsOurs(frame) {
var lang = frame.lang || (String(frame.name).match(/-([a-z]{3}_[A-Za-z]+)\.aac$/) || [])[1];
if (lang !== app.lang) return false;
if ('voice' in frame && String(frame.voice || '') !== String(srv.voice || '')) return false;
return true;
}
/* Fetch, decode and schedule one clip so it follows the previous one. Only
used where native HLS is missing. */
function playClip(frame) {
var name = frame && frame.name;
if (srv.mode === 'stream') return; // the stream already carries it
if (!tts.on || !srv.ctx || !name) return;
if (!clipIsOurs(frame)) return;
fetch('/audio/' + encodeURIComponent(name))
.then(function (r) { return r.ok ? r.arrayBuffer() : Promise.reject(new Error('missing')); })
@@ -936,6 +984,8 @@
})
.then(function (audio) {
if (!tts.on || !srv.ctx) return;
/* Landed after a switch of language, voice or mode: not ours any more. */
if (srv.mode === 'stream' || !clipIsOurs(frame)) return;
var src = srv.ctx.createBufferSource();
src.buffer = audio;
src.connect(srv.ctx.destination);
@@ -1424,7 +1474,11 @@
}
/* The playlist is guaranteed to exist from this moment — the server
creates the stream when this listener registers. */
maybeStartStream();
/* Not for a listener on an alternate: the shared stream is the
operator's default voice, and starting it here played that voice
until the server's 'voices' reply moved the page to clips. */
if (srv.voice) fallBackToClips(true);
else maybeStartStream();
/* The picker was rendered before this socket existed, when we still had
to assume the device's own voice. Now we know better — re-render, or
a listener keeps staring at a voice chooser that controls nothing. */
app/server/server.js
+22 -6
View File
@@ -1763,12 +1763,20 @@ wss.on('connection', (ws, req) => {
if (!m.voice) row.audio[m.lang] = m.name;
delayed((who) => {
const frame = JSON.stringify({
t: 'audio', id: row.id, name: m.name, seconds: m.seconds || 0
});
/* Which voice this clip IS. '' is the operator's default, which is
what a listener who never opened the chooser is on. */
const clipVoice = String(m.voice || '');
/*
* The frame says which language and voice it carries, and the page
* checks both again when the clip's bytes land (clipIsOurs() in
* listen.js). The routing below is decided at SEND time; a listener
* who switches language or voice while a clip is still downloading
* would otherwise hear it after the switch, in the voice they left.
*/
const frame = JSON.stringify({
t: 'audio', id: row.id, name: m.name, seconds: m.seconds || 0,
lang: m.lang, voice: clipVoice
});
for (const [ws, meta] of state.listeners) {
if (ws.readyState !== 1) continue;
if (!who(meta)) continue;
@@ -2054,14 +2062,22 @@ wss.on('connection', (ws, req) => {
if (m.t === 'lang') {
const meta = state.listeners.get(ws);
if (!meta) return;
meta.lang = String(m.lang || config.sourceLanguage);
const next = String(m.lang || config.sourceLanguage);
/*
* A voice belongs to a language. Carrying a Spanish speaker across to
* Portuguese would ask the engine for a voice that language does not
* have, so switching language always returns the listener to the
* default and hands them the new language's options.
*
* Only a CHANGE of language, though. The page re-sends the language it
* is already on to have its captions re-rendered (the original-text
* toggle does exactly that), and resetting the voice there put a
* listener on an alternate back on the default for a round trip —
* until the page re-asserted its pick, they were sent the default
* voice's clips: one voice, then the other, then theirs again.
*/
meta.voice = '';
if (next !== meta.lang) meta.voice = '';
meta.lang = next;
pushLanguagesToEngine();
/*
* builtInVoice MUST be here, not just on the first hello. The page does
@@ -2073,7 +2089,7 @@ wss.on('connection', (ws, req) => {
*/
send(ws, { t: 'hello', live: state.live, lang: meta.lang, serviceName: config.serviceName, build: BUILD,
builtInVoice: Voices.hasVoice(meta.lang),
voiceOptions: voiceOptionsFor(meta.lang), voice: '',
voiceOptions: voiceOptionsFor(meta.lang), voice: meta.voice || '',
scrollback: scrollbackFor(meta.lang, meta.proxied) });
broadcastOperators();
}