christian/church-translate
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 11 Wiki Activity

One voice per listener, and the voice process stops stalling itself

Browse Source 
Build 127. Reported from a live test: part of the translation came out in a
different voice on an Android phone, and iPhones were skipping and repeating.
Three investigations against the campus Mac's own log and transcript, one
test harness driving the voice process directly, and the answer is not what
it sounded like.

WHAT THE MACHINE DID. One voice process ran all afternoon. Every language
kept a single resident engine from its first sentence (14:54:26) to STOP
(15:23:18); 2,800 clips, 560 per language, every one filed under its own
language and spoken by its own bundle. No synthesis-side voice change was
possible with that configuration. What the log shows at the time of the
complaint instead:

  - Someone on the English stream heard the ROOM MICROPHONE for 21 s
    (15:09:10-32). English was the first button on every phone's list, forced
    there by the page whatever the operator had chosen (fixed in build 125).
  - A listener tapped Chinese at 15:11:17. The 121 MB zh model was fetched
    and unpacked THREE TIMES AT ONCE on the voice process's only thread.
    Every stream stalled for about a minute, 26 sentences were dropped across
    all five languages, and the catch-up rushed the Spanish voices to their
    +16% cap. Same voice, sounding wrong.

WHAT WAS WRONG, AND IS NOW FIXED.

The live path filed every engine as an "audition". speak() resolved the
language's default voice, which always has a bundle, and passed it as
`explicit`, so the key was 'audition:'+bundle for every ordinary sentence
while the preload warmed copies under the language name that no sentence
ever read. Five languages asked for nine engines against a cap of six; every
service's first sentence built its real engines on the spot (1.3-2 s); the
operator's voice save flushed nothing that mattered. Engines are now keyed
by bundle and shared by every language, variant and audition that uses one,
auditions take at most two slots of their own and can never evict a live
engine, and the cap is 8, sized to the largest campus from measurement
(~165 MB per engine once it has spoken, 1.3 GB for eight).

Voice installs ran on the thread that answers every sentence, and were not
de-duplicated. They now run in a worker thread (voice-install.js), one per
bundle however many ask, staged and renamed into place whole so a half-
written bundle can never pass isInstalled().

Listener rows are { code, count } and engine.js read { lang, n } in five
places, so the set of "languages someone is on" was always empty — and a late
sentence keeps speech only for those. Under load the app dropped EVERY
language's speech instead of only the unchosen ones. Late became silent.

HLS segment numbers restarted at 1 for every new stream object, under a
'max-age=60, immutable' header. A phone back from the lock screen could be
answered from its cache with the previous stream's audio under the new
number: an earlier sentence, replayed. Numbers are seeded from the clock.

In clips mode a clip still downloading when the listener switched language
landed and played in the voice they had left; the frame now carries its
language and voice and the page checks both when the bytes land. In stream
mode the old playlist stayed loaded through a switch and three things could
restart it; it is stopped, not paused. A listener's alternate voice no longer
loses a round trip to the default on the original-text toggle.

One genuine wrong-voice path, dormant today because natural voices are off on
this campus: a variant waiting on another variant's premium synthesis
inherited that variant's Piper FALLBACK, in that variant's bundle — spa_ES
received the byte-identical Mexican clip and its own voice never loaded. A
follower now shares a fallback only when it is its own voice.

An audition (the Voices card's play button) played on the system default
output, which on the campus Mac is the Dante card feeding the hardware
channels. It now plays on another output, or refuses and says why.

VERIFIED: a harness drives the voice process through the baseline, a batch
of four auditions during live sentences, a voices save mid-stream, and nine
live languages against the cache — every clip spoken by its own bundle, and
auditions of a live voice now cost 78-311 ms with nothing rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
christian
2026-10-15 16:05:15 -04:00
co-authored by Claude Fable 5.1
parent 95d1a7b9ec
commit 81da657094
7 changed files with 426 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/tts-service.js
+173 -84
View File
@@ -42,9 +42,6 @@ let VOICE_DIR = process.platform === 'win32'
: path.join(os.homedir(), 'Library/Application Support/Church Translate Voices');
let CLIP_DIR = path.join(os.tmpdir(), 'church-translate-audio');
/* Loaded engines, one per language. Kept small: each costs ~330-400 MB of ONNX
arena regardless of the 63 MB model, so we do not hold voices nobody is
listening to. */
/* What each voice says when auditioned — a real line of scripture, so the
operator hears the voice on the material it will actually read. */
const SAMPLE_TEXT = {
@@ -65,8 +62,25 @@ const SAMPLE_TEXT = {
hat_Latn: 'Senyè a se gadò mwen, mwen p ap manke anyen.'
};
const loaded = new Map();
const MAX_LOADED = 6;
/*
* Loaded engines, ONE PER BUNDLE, shared by every language, variant and
* audition that speaks with it. Kept bounded, not small: MEASURED on an M2
* Max, an engine costs ~75 MB at construction and ~165 MB once it has spoken
* (8 engines, 1.3 GB after one sentence each), not the 330-400 MB this used
* to assume. The cap is sized to the largest campus — eight languages — so a
* live engine is never evicted to make room for another live one in normal
* service; below the cap an eviction means a reload of ~0.3-0.5 s on this
* process's only thread, in front of the next sentence, every time.
*
* `live` marks an engine a sentence or the preload asked for. Auditions keep
* to MAX_AUDITION slots of their own, always go first, and can never push a
* live engine out: with every slot speaking, an auditioned voice is built,
* played once and dropped. See engineFor() for why this is keyed by bundle.
*/
const loaded = new Map(); // 'bundle/model' -> { tts, usedAt, live }
const installing = new Map(); // bundle -> Promise: one download, however many ask
const MAX_LOADED = 8;
const MAX_AUDITION = 2;
/*
* The operator's per-language voice choices, made by ear in the Voices
@@ -142,81 +156,120 @@ function isInstalled(v) {
* 359 files, same total bytes — and it is now the same code on both platforms.
*/
function installVoice(v) {
/*
* OFF THIS THREAD. The fetch, the bzip2 and the unpack all ran here, on the
* voice process's only thread, from inside engineFor() — so the first
* sentence of a language whose voice was not installed yet stalled every
* OTHER language's sentences for as long as the install took. MEASURED on a
* campus Mac, 2026-10-15 15:11: a listener tapped Chinese, the 121 MB zh
* model was fetched and unpacked three times at once on this thread, every
* stream stalled, 26 sentences were dropped and the catch-up rushed the
* Spanish voices to their +16% cap. The work now happens in a worker thread
* (voice-install.js); this thread only relays its progress and keeps
* speaking. The worker unpacks into a staging folder and renames it into
* place whole, so isInstalled() never sees a half-written bundle.
*/
return new Promise((resolve, reject) => {
fs.mkdirSync(VOICE_DIR, { recursive: true });
const url = voices.bundleUrl(v.bundle);
const fail = (why) => reject(new Error('could not install ' + v.bundle + ': ' + why));
reply({ type: 'voiceProgress', bundle: v.bundle, status: 'downloading' });
(async () => {
const res = await fetch(url, { redirect: 'follow' });
if (!res.ok) throw new Error('HTTP ' + res.status);
const body = Buffer.from(await res.arrayBuffer());
/* A proxy or a captive portal returns an HTML error page with HTTP 200;
the bzip2 magic is the cheapest way to notice before unpacking it. */
if (body.length < 1024 || body.toString('latin1', 0, 3) !== 'BZh') {
throw new Error('that download was not a voice bundle');
}
reply({ type: 'voiceProgress', bundle: v.bundle, status: 'installing' });
const bunzip = require('unbzip2-stream');
const tar = require('tar-stream');
const { Readable } = require('stream');
await new Promise((res2, rej2) => {
const extract = tar.extract();
extract.on('entry', (header, stream, next) => {
/* Never write outside VOICE_DIR, whatever the archive claims. */
const dest = path.resolve(VOICE_DIR, header.name);
if (dest !== VOICE_DIR && !dest.startsWith(VOICE_DIR + path.sep)) {
stream.resume(); return next();
}
if (header.type === 'directory') {
fs.mkdirSync(dest, { recursive: true });
stream.resume(); return next();
}
fs.mkdirSync(path.dirname(dest), { recursive: true });
stream.pipe(fs.createWriteStream(dest)).on('finish', next).on('error', rej2);
});
extract.on('finish', res2);
extract.on('error', rej2);
Readable.from(body).pipe(bunzip()).on('error', rej2).pipe(extract);
let w;
try {
const { Worker } = require('worker_threads');
w = new Worker(path.join(__dirname, 'voice-install.js'), {
workerData: { url: voices.bundleUrl(v.bundle), voiceDir: VOICE_DIR, bundle: v.bundle }
});
reply({ type: 'voiceProgress', bundle: v.bundle, status: 'ready' });
})().then(resolve, (err) => reject(new Error('could not install ' + v.bundle + ': ' + err.message)));
} catch (e) { return fail(e.message); }
let settled = false;
const done = (err) => { if (settled) return; settled = true; err ? fail(err) : resolve(); };
w.on('message', (m) => {
if (!m || !m.status) return;
if (m.status === 'error') { done(m.message || 'the installer failed'); return; }
reply({ type: 'voiceProgress', bundle: v.bundle, status: m.status });
if (m.status === 'ready') done();
});
w.on('error', (e) => done((e && e.message) || String(e)));
w.on('exit', (code) => { if (!settled) done('the installer exited (' + code + ')'); });
});
}
async function engineFor(lang, explicit) {
/* Auditions key the cache by bundle so trying five voices for one language
does not evict the voice that language is actually using. */
const v = explicit || builtinVoiceFor(lang);
const key = explicit ? "audition:" + v.bundle : lang;
if (loaded.has(key)) {
const e = loaded.get(key);
e.usedAt = Date.now();
return e.tts;
/*
* The engine for one voice, built at most once however many ask for it.
*
* KEYED BY BUNDLE — and the old keying is what put batches of four
* "audition:" loads in the production log with nobody at the Voices card.
* Engines were keyed two ways: by language ('spa_Latn') for the preload, and
* 'audition:<bundle>' whenever a voice was passed in, which speak() did for
* EVERY ordinary sentence because the resolved default always has a bundle.
* So the live path was filed as an audition, every bundle was built at least
* twice (five languages asked for nine engines against a cap of six), the
* preload warmed copies no sentence ever read, and the first sentence of
* every service built the real ones on the spot. MEASURED on the Mini,
* 2026-10-15: the 14:26 service's first sentence was said at 14:54:26.291
* and its four engines were ready at 26.995, 27.309, 27.833 and 28.344 —
* ~0.3-0.5 s each on this process's only thread — while what got evicted
* was the startup preload's unused copies.
*
* The engine's config is a function of the bundle alone (voices.ttsConfig;
* speed, gain and sid are applied per sentence), so sharing is exact:
* spa_Latn and spa_MX read one es_MX-ald engine, and auditioning a voice that
* is already speaking reads the live engine instead of building a second.
*/
async function engineFor(v, live, who) {
if (!v || !v.bundle) throw new Error('no built-in voice' + (who ? ' for ' + who : ''));
const key = v.bundle + '/' + (v.model || '');
/*
* One download per bundle. A language joining mid-service is asked for by
* the preload AND by its first sentences at once, and each used to fetch
* and unpack the whole bundle itself — MEASURED on the Mini: pt_BR-jeff
* three times between 11:39:05 and 11:39:50, zh-ll three times between
* 15:11:17 and 15:12:16, two 'voice timed out' each time. The unpack is staged and
* renamed into place whole (voice-install.js), so isInstalled() stays false
* until the bundle is complete — but a second asker must still wait on the
* first install, not start its own.
*/
if (!loaded.has(key) && (installing.has(v.bundle) || !isInstalled(v))) {
if (!installing.has(v.bundle)) {
installing.set(v.bundle, installVoice(v).finally(() => installing.delete(v.bundle)));
}
await installing.get(v.bundle);
}
if (!v) throw new Error('no built-in voice for ' + lang);
if (!isInstalled(v)) await installVoice(v);
if (!sherpa) sherpa = require('sherpa-onnx-node');
const dir = voicePath(v);
const tts = new sherpa.OfflineTts(voices.ttsConfig(dir, v));
/* Evict the least recently used rather than growing without bound. */
if (loaded.size >= MAX_LOADED) {
let oldest = null;
for (const [k, e] of loaded) if (!oldest || e.usedAt < loaded.get(oldest).usedAt) oldest = k;
if (oldest) { loaded.delete(oldest); log('evicted voice ' + oldest); }
/* Looked up again: whoever finished waiting first has built it. Nothing can
interleave from here to the set — construction is synchronous. */
let e = loaded.get(key);
if (!e) {
if (!sherpa) sherpa = require('sherpa-onnx-node');
e = { tts: new sherpa.OfflineTts(voices.ttsConfig(voicePath(v), v)), usedAt: 0, live: !!live };
if (makeRoom(!!live)) loaded.set(key, e);
log('voice ready: ' + v.bundle + (who ? ' for ' + who : '') +
(live ? '' : loaded.get(key) === e ? ' (audition)' : ' (audition, not kept: every slot is speaking)'));
}
e.usedAt = Date.now();
/* An audition never demotes a live engine; a sentence promotes an auditioned one. */
if (live) e.live = true;
return e.tts;
}
loaded.set(key, { tts, usedAt: Date.now() });
log('voice ready: ' + key + ' (' + v.bundle + ')');
return tts;
/*
* Room for one more engine, or false when an audition should not take one.
* Least recently used goes first, as before — auditioned engines before live
* ones, and a live engine only ever gives way to another live one.
*/
function makeRoom(live) {
const stalest = (wantLive) => {
let k = null;
for (const [key, x] of loaded) if (x.live === wantLive && (!k || x.usedAt < loaded.get(k).usedAt)) k = key;
return k;
};
const evict = (k) => { log('evicted voice ' + k + (loaded.get(k).live ? '' : ' (audition)')); loaded.delete(k); };
const auditions = Array.from(loaded.values()).filter((x) => !x.live).length;
if (!live && auditions >= MAX_AUDITION) evict(stalest(false));
if (loaded.size < MAX_LOADED) return true;
const k = stalest(false) || (live ? stalest(true) : null);
if (!k) return false;
evict(k);
return true;
}
// ------------------------------------------------------------- cloud voices
@@ -928,6 +981,9 @@ async function speak(job) {
text, Math.round((Number(job.gainDb) || 0) * 2) / 2, pinRate
].join('\u0001');
/* Set when the leader's result was a Piper fallback in ANOTHER variant's
voice — this job then speaks its own fallback, straight to Piper. */
let piperOnly = false;
if (key) {
const hit = synthCache.get(key);
if (hit) {
@@ -941,7 +997,22 @@ async function speak(job) {
counted as one; the key is deleted below so the NEXT utterance
tries the natural voice again. */
if (premiumPick && done.source !== premiumPick.provider) { cloudStats.fallback++; fallbackAt = Date.now(); }
return cloneClip(done, job.lang, started);
/*
* ...but only a fallback in THIS language's own Piper voice. The key
* names the PREMIUM voice, and spa_Latn, spa_ES and spa_MX share it
* (KOKORO_LANG maps all three to 'e', both Portugueses to 'p'). Their
* Piper fallbacks they do not share: spa_ES falls back to davefx,
* spa_Latn to ald, por_PT to miro, por_Latn to jeff. Cloning the
* leader's fallback put the Mexican voice in the Castilian stream for
* that sentence — measured in a harness against this file: spa_ES's
* clip was the byte-identical ald file and davefx never even loaded.
* So the follower still skips the sidecar that just missed (the
* variants still switch together), but in its own voice.
*/
if (done.source !== 'piper' || done.bundle === (voice.bundle || '')) {
return cloneClip(done, job.lang, started);
}
piperOnly = true;
}
} catch (_) { /* the first caller failed — make our own */ }
}
@@ -956,7 +1027,7 @@ async function speak(job) {
* alternate skips the premium path entirely — otherwise cloudSynth would
* fall back to the operator's own pick and the page would lie.
*/
const cloudOut = (alt && alt.engine === 'builtin')
const cloudOut = (piperOnly || (alt && alt.engine === 'builtin'))
? null
: await cloudSynth(job.lang, text, (alt && alt.engine !== 'builtin') ? alt : null, { speed: kokoroSpeed, deadline });
if (cloudOut) {
@@ -984,13 +1055,16 @@ async function speak(job) {
source: cloudOut.provider
};
}
if (!(alt && alt.engine === 'builtin') && cloudVoiceFor(job.lang)) {
/* A piperOnly job's fallback was already counted at the cache hit. */
if (!piperOnly && !(alt && alt.engine === 'builtin') && cloudVoiceFor(job.lang)) {
cloudStats.fallback++;
fallbackAt = Date.now();
if (cloudStats.fallback === 1 || cloudStats.fallback % 10 === 0) pushStats('fallback');
}
const tts = await engineFor(job.lang, voice.bundle ? voice : null);
/* Live: a sentence is speaking with this voice, so no audition may take
its engine. A listener's built-in alternate counts — someone is on it. */
const tts = await engineFor(voice.bundle ? voice : null, true, job.lang);
if (!sherpa) sherpa = require('sherpa-onnx-node');
/*
@@ -1067,6 +1141,9 @@ async function speak(job) {
const seconds = secs;
return {
source: 'piper',
/* WHICH Piper voice spoke it. A follower may share a fallback only when
it is its own voice — see the cache hit above. */
bundle: voice.bundle || '',
/* What the sentence ended on — the stream gives a full stop a full breath
and a mid-thought fragment only a beat. */
terminal: text.slice(-1),
@@ -1080,8 +1157,12 @@ async function speak(job) {
};
};
/* A piperOnly job is a fallback by construction. It must not stand in the
cache as a leader, or the next identical line would inherit the miss
without trying the natural voice — the chorus case below. */
const cacheKey = piperOnly ? null : key;
const promise = produce();
if (key) synthCache.set(key, { promise, at: Date.now() });
if (cacheKey) synthCache.set(cacheKey, { promise, at: Date.now() });
try {
const out = await promise;
/*
@@ -1091,10 +1172,10 @@ async function speak(job) {
* miss would have re-created the very voice switch this file exists to
* prevent (measured in the harness before shipping, not after).
*/
if (key && premiumPick && out && out.source !== premiumPick.provider) synthCache.delete(key);
if (cacheKey && premiumPick && out && out.source !== premiumPick.provider) synthCache.delete(cacheKey);
return out;
} catch (e) {
if (key) synthCache.delete(key);
if (cacheKey) synthCache.delete(cacheKey);
throw e;
}
}
@@ -1187,11 +1268,15 @@ process.on('message', async (msg) => {
if (msg.type === 'voices') {
/* The operator changed a per-language voice choice. */
voiceChoice = msg.choices || {};
/* Drop cached engines for languages whose choice changed, so the next
sentence uses the new voice rather than the one already resident. */
for (const k of Array.from(loaded.keys())) {
if (!String(k).startsWith("audition:")) loaded.delete(k);
}
/*
* Nothing to drop. Engines are keyed by bundle, so a changed choice
* resolves to a different engine on the next sentence by itself, and
* the one it replaced ages out like any other. The loop that was here
* only ever deleted the preload's per-language copies (the live path was
* filed under 'audition:'); under bundle keys it would delete the live
* engines too, and every "Use it" would make the next sentence rebuild
* every language's voice.
*/
reply({ type: 'voicesOk', count: Object.keys(voiceChoice).length });
return;
}
@@ -1222,7 +1307,9 @@ process.on('message', async (msg) => {
} finally { voiceChoice = saved; }
} else {
if (!sherpa) sherpa = require('sherpa-onnx-node');
const tts = await engineFor(msg.lang, spec.bundle ? spec : null);
/* Not live: shares the engine when this voice is already speaking,
otherwise takes an audition slot — never a live one. */
const tts = await engineFor(spec.bundle ? spec : builtinVoiceFor(msg.lang), false, 'audition');
const audio = tts.generate({ text, sid: spec.sid || 0, speed: spec.speed || 1.0 });
if (spec.gain && spec.gain !== 1) {
for (let i = 0; i < audio.samples.length; i++) {
@@ -1256,7 +1343,9 @@ process.on('message', async (msg) => {
/* Warm the languages people are actually listening to, so the first
sentence of a service is not the one that pays for the download. */
for (const lang of msg.langs || []) {
try { await engineFor(lang); } catch (e) { log('preload failed for ' + lang + ': ' + e.message); }
/* Keyed exactly as speak() keys it, so this warms the engine the first
sentence will read, not a copy beside it. */
try { await engineFor(builtinVoiceFor(lang), true, lang); } catch (e) { log('preload failed for ' + lang + ': ' + e.message); }
}
reply({ type: 'preloaded', langs: msg.langs || [] });
return;