docs/PLAN-bible-fidelity.md

# Plan — Bible-faithful translation, "bulletproof for live preaching"

## The complaint
Operator, watching the Spanish listener page during a live test:
- "why is the voice asking questions to me? it keeps saying '¿Qué es eso?'"
- "if i say 'john' it must be translated properly"
- "make this bulletproof so native speakers in any language can completely
  understand what is being said in english. this will primarily be used for
  live preaching of the christian bible."

## Measured, not assumed
Ran `Xenova/nllb-200-distilled-600M` fp32 (the shipped model, shipped dtype,
shipped cache) against the exact lines from the screenshot plus stress cases,
under four generation configs. Results:

| English in | Spanish out | verdict |
|---|---|---|
| `Donald Trump.` | `Donald Trump. ¿Qué es eso?` | invented question |
| `Amen.` | `- Sí, estoy bien.` | "Yes, I'm fine." — catastrophic |
| `Hello, hello.` | `- Hola, está bien.` | invented content + subtitle dash |
| `His name was John.` | `Se llamaba John.` | name left in English |
| `Turn with me to John chapter 3.` | `…a Juan capítulo 3.` | same name, translated |
| `Open up to the book, Matthew.` | `Abre al libro, Matthew.` | name left in English |

**The decisive finding:** `¿Qué es eso?` appears under ALL FOUR configs —
no params, `max_new_tokens` cap, `no_repeat_ngram_size:3`, and `num_beams:2`.
It is not a decoding-budget artifact. Generation parameters cannot fix it.
Post-processing is the only reliable lever. (My first hypothesis was wrong;
the probe is why this plan is not built on it.)

Secondary finding: cap + `no_repeat_ngram_size:3` cut the 11-case suite from
5897ms to 3924ms. Strictly faster AND safer, so it goes in regardless.

## Existing machinery to build on (do not reinvent)
- `applyTable()` / `termMatcher()` in engine.js — a single-pass, longest-first
  find-and-replace that already survived three real bugs: terms ending in
  punctuation, CJK word boundaries, and case-sensitivity ("job" → "Job").
- `glossary.fixes` — source-side transcript corrections, operator-editable.
- `glossary.overrides[lang]` — target-side per-language pins. Currently `{}`.
- Both MT paths converge on `applyOverrides(text, lang)`:
  `translateSentence()` (live, per sentence) and `translateAll()` (batch).
- `translateSentence` receives ONE sentence, so a 1-in-1-out clamp is safe there.

## The five layers

### L1 — source-side Bible vocabulary (before MT)
Built-in `fixes` for common Whisper mishearings of scripture vocabulary
("have a cook" → "Habakkuk" already exists as the only entry). Ship a real
list. Must NOT clobber the operator's own fixes.

### L2 — generation parameters (mt-service.js)
`max_new_tokens` derived from input token count, `no_repeat_ngram_size: 3`.
Measured: faster and safer. Does NOT fix the hallucination — that is L3.

### L3 — post-MT hallucination guard  ← the fix for the actual complaint
Applied to every language, no per-language data needed:
1. **Invented-question rule.** Source has no `?`/`？`, output contains `?`/`？`
   → the question is hallucinated. Drop the offending clause. This is the
   direct fix for `¿Qué es eso?`. Precision is high: real scripture questions
   carry a `?` in the source.
2. **Sentence-count clamp.** N sentence terminators in → at most N out
   (min 1). Kills appended-sentence hallucination generally.
3. **Subtitle-dash strip.** NLLB emits leading `- ` / `– ` from its subtitle
   training data.
4. **Length-ratio guard.** Output wildly longer than input → keep first
   sentence only. Needs a per-script ratio (CJK is far shorter per character).

### L4 — Bible lexicon, target side (per language)
Canonical forms for the 66 book names, ~120 proper nouns (people, places),
and key doctrinal terms. Applied through `applyTable` — the machinery is
already hardened. Built-in table, SEPARATE from operator overrides, with the
operator's entries winning so an app update never clobbers their work.

**Honesty constraint:** ship a language only where the canonical forms are
verifiable. A wrong Korean Bible name is worse than NLLB's output. Structure
the data so unverified languages simply have no entry and fall through to NLLB.

### L5 — liturgical phrasebook (skip MT entirely)
`Amen.` must never reach NLLB. Short liturgical constants — Amen, Hallelujah,
Hosanna, Glory to God, Praise the Lord — have stable known forms and are the
highest-frequency utterances in preaching. Direct lookup, no inference. This
is also the class that produces the worst failures, because NLLB has almost no
context to work from on a 1-3 word input.

## What must not regress
- Latency: EN 2-4s / ES 4-7s worst case. Every layer here is string work except
  L2, which is faster. No layer may add an inference pass.
- `mtWorkers` stays 1 (benchmarked: 2 workers = 0.92x, slower).
- Portuguese variants (`PtVariants`) run BEFORE `applyOverrides`; ordering with
  the new lexicon must be decided explicitly.
- Operator's saved glossary must survive app updates.
- `por_PT` is a pseudo-code aliased to `por_Latn` at the MT boundary only.

## Open questions for the audit
1. Does the invented-question rule ever fire on legitimate output? Spanish
   inverts questions (`¿…?`); other languages may add a question particle where
   English used intonation with no `?`.
2. Ordering: lexicon before or after the hallucination guard? Before or after
   PtVariants?
3. Can Whisper be biased toward scripture vocabulary in transformers.js
   (an initial prompt), or is L1 find-and-replace the only available lever?
4. Is a length-ratio guard safe across scripts, or does it need per-script
   calibration that we cannot verify for 200 languages?
5. Which languages can we honestly ship a lexicon for, and how should the
   unverified ones behave?