church-translate/docs/PLAN-bible-fidelity.md

Plan — Bible-faithful translation, "bulletproof for live preaching"

The complaint

Operator, watching the Spanish listener page during a live test:

• "why is the voice asking questions to me? it keeps saying '¿Qué es eso?'"
• "if i say 'john' it must be translated properly"
• "make this bulletproof so native speakers in any language can completely understand what is being said in english. this will primarily be used for live preaching of the christian bible."

Measured, not assumed

Ran Xenova/nllb-200-distilled-600M fp32 (the shipped model, shipped dtype, shipped cache) against the exact lines from the screenshot plus stress cases, under four generation configs. Results:

English in	Spanish out	verdict
Donald Trump.	Donald Trump. ¿Qué es eso?	invented question
Amen.	- Sí, estoy bien.	"Yes, I'm fine." — catastrophic
Hello, hello.	- Hola, está bien.	invented content + subtitle dash
His name was John.	Se llamaba John.	name left in English
Turn with me to John chapter 3.	…a Juan capítulo 3.	same name, translated
Open up to the book, Matthew.	Abre al libro, Matthew.	name left in English

The decisive finding: ¿Qué es eso? appears under ALL FOUR configs — no params, max_new_tokens cap, no_repeat_ngram_size:3, and num_beams:2. It is not a decoding-budget artifact. Generation parameters cannot fix it. Post-processing is the only reliable lever. (My first hypothesis was wrong; the probe is why this plan is not built on it.)

Secondary finding: cap + no_repeat_ngram_size:3 cut the 11-case suite from 5897ms to 3924ms. Strictly faster AND safer, so it goes in regardless.

Existing machinery to build on (do not reinvent)

• applyTable() / termMatcher() in engine.js — a single-pass, longest-first find-and-replace that already survived three real bugs: terms ending in punctuation, CJK word boundaries, and case-sensitivity ("job" → "Job").
• glossary.fixes — source-side transcript corrections, operator-editable.
• glossary.overrides[lang] — target-side per-language pins. Currently {}.
• Both MT paths converge on applyOverrides(text, lang): translateSentence() (live, per sentence) and translateAll() (batch).
• translateSentence receives ONE sentence, so a 1-in-1-out clamp is safe there.

The five layers

L1 — source-side Bible vocabulary (before MT)

Built-in fixes for common Whisper mishearings of scripture vocabulary ("have a cook" → "Habakkuk" already exists as the only entry). Ship a real list. Must NOT clobber the operator's own fixes.

L2 — generation parameters (mt-service.js)

max_new_tokens derived from input token count, no_repeat_ngram_size: 3. Measured: faster and safer. Does NOT fix the hallucination — that is L3.

L3 — post-MT hallucination guard ← the fix for the actual complaint

Applied to every language, no per-language data needed:

1. Invented-question rule. Source has no ?/？, output contains ?/？ → the question is hallucinated. Drop the offending clause. This is the direct fix for ¿Qué es eso?. Precision is high: real scripture questions carry a ? in the source.
2. Sentence-count clamp. N sentence terminators in → at most N out (min 1). Kills appended-sentence hallucination generally.
3. Subtitle-dash strip. NLLB emits leading - / – from its subtitle training data.
4. Length-ratio guard. Output wildly longer than input → keep first sentence only. Needs a per-script ratio (CJK is far shorter per character).

L4 — Bible lexicon, target side (per language)

Canonical forms for the 66 book names, ~120 proper nouns (people, places), and key doctrinal terms. Applied through applyTable — the machinery is already hardened. Built-in table, SEPARATE from operator overrides, with the operator's entries winning so an app update never clobbers their work.

Honesty constraint: ship a language only where the canonical forms are verifiable. A wrong Korean Bible name is worse than NLLB's output. Structure the data so unverified languages simply have no entry and fall through to NLLB.

L5 — liturgical phrasebook (skip MT entirely)

Amen. must never reach NLLB. Short liturgical constants — Amen, Hallelujah, Hosanna, Glory to God, Praise the Lord — have stable known forms and are the highest-frequency utterances in preaching. Direct lookup, no inference. This is also the class that produces the worst failures, because NLLB has almost no context to work from on a 1-3 word input.

What must not regress

• Latency: EN 2-4s / ES 4-7s worst case. Every layer here is string work except L2, which is faster. No layer may add an inference pass.
• mtWorkers stays 1 (benchmarked: 2 workers = 0.92x, slower).
• Portuguese variants (PtVariants) run BEFORE applyOverrides; ordering with the new lexicon must be decided explicitly.
• Operator's saved glossary must survive app updates.
• por_PT is a pseudo-code aliased to por_Latn at the MT boundary only.

Open questions for the audit

1. Does the invented-question rule ever fire on legitimate output? Spanish inverts questions (¿…?); other languages may add a question particle where English used intonation with no ?.
2. Ordering: lexicon before or after the hallucination guard? Before or after PtVariants?
3. Can Whisper be biased toward scripture vocabulary in transformers.js (an initial prompt), or is L1 find-and-replace the only available lever?
4. Is a length-ratio guard safe across scripts, or does it need per-script calibration that we cannot verify for 200 languages?
5. Which languages can we honestly ship a lexicon for, and how should the unverified ones behave?