/* * voice-audit.mjs — does each built-in voice actually SAY what it was given? * * The captions can be perfect while the audio is wrong. Text goes through * espeak-ng's phonemizer before it ever reaches the VITS model, and a wrong * espeak voice, a missing dictionary, or a numeral read in the wrong language * all produce fluent-sounding audio that says something else. Nobody on staff * speaks thirty-one languages, so the ear cannot be the check. This is. * * The loop, per language: * * reference text (in-language) * -> sherpa-onnx VITS, the exact config tts-service.js uses * -> WAV (optionally through the same afconvert AAC the listeners get) * -> Whisper large-v3-turbo, language FORCED to the target * -> normalized character similarity against the reference * * Big divergence means the voice mispronounced it, dropped it, or is the wrong * voice entirely. Whisper is an imperfect judge, so the scoring is built to * avoid crying wolf — see "FALSE-POSITIVE CONTROLS" below. A harness that fails * clean languages gets ignored, which is worse than no harness. * * MEMORY. A loaded VITS voice costs ~350 MB of ONNX arena and sherpa-onnx-node * exposes no free() — the native handle goes when the JS finalizer runs, which * is not something to bet a 31-language sweep on. So synthesis for each language * happens in a short-lived CHILD process that exits when the language is done, * and the OS reclaims the arena. Peak is one voice plus one ASR model, never a * pile of voices. * * EXIT CODE. Whisper lives in a child too, and for a different reason: with * onnxruntime-node loaded, this process aborts during teardown — * * libc++abi: terminating due to uncaught exception of type * std::__1::system_error: mutex lock failed: Invalid argument * * — turning a perfectly clean audit into exit 134. A release gate that reports * failure when everything passed is worse than no gate, so the parent stays pure * JavaScript: it loads no native model, orchestrates the children, does the * scoring, and exits with the code it actually means. * * PLAIN NODE ONLY. Same reason as mt-service.js and tts-service.js: Chromium's * PartitionAlloc traps on the allocations a model load makes. Never run this * inside Electron. * * --------------------------------------------------------------------------- * FALSE-POSITIVE CONTROLS * * Every one of these exists because a naive string compare fails clean audio: * * 1. Normalization. NFKC, casefold, punctuation stripped, whitespace collapsed. * Whisper writes "Jesús dijo, yo soy" where the reference says * "Jesús dijo: Yo soy" — identical speech, different glyphs. * 2. Character-level similarity, not exact match. One wrong word in a * twelve-word sentence should cost a little, not everything. * 3. …but characters ALONE are too kind to short sentences, so a word-level * score has to clear its own bar too. "Oremos juntos" came back as "aremos * puntos" — both words wrong, two characters different, 0.846 on characters * and a pass it did not deserve. On words it is 0.0. Scripts without word * breaks (Chinese, Japanese, Thai, Khmer, Lao, Burmese) get characters only, * with spaces stripped from both sides, because Whisper's word breaks there * are arbitrary and carry no information. * 4. Numerals, in BOTH directions. A sentence is scored strictly and again * "numeral-blind" whenever either side contains digits, and the better score * wins with the variant named in the report. It has to work both ways: a * reference reading "3:16" meets a transcript that spells it out, and a * reference reading "veintitrés" meets a transcript that writes "23". All * four Spanish voices scored an identical 0.62 on "Salmo veintitrés" purely * because Whisper wrote digits — four voices failing identically is the * signature of an orthography artifact, not a defect. The leniency is capped * at three dropped tokens per numeral so it cannot become a free pass. * 5. Diacritics. Never folded for scoring — a wrong vowel is a real defect — * but a folded score is reported alongside so a human can tell instantly * whether accents were the entire difference. * 6. ASR failure is not TTS failure. An empty transcript, one that loops a * phrase, one that is a known Whisper hallucination (it answered "Amén." * with "¡Gracias por ver el video!"), or audio too short to hold the text at * all become ASR_EMPTY / ASR_LOOP / ASR_HALLUCINATION / SHORT_AUDIO and are * reported without counting for or against the language. * 7. Every failure is re-heard before it is believed. VITS samples noise on * each call, so the same text synthesized twice is not the same audio — a * single unlucky take can fail a healthy voice. A FAIL is resynthesized * (--retakes, default 2) and judged on the MEDIAN of the takes, never the * best, so retries cannot buy a broken voice a pass. All takes are printed: * three different garbles of the same words is proof the voice is at fault. * 8. Two duration checks that do not involve Whisper at all. Seconds-per- * character far below the language's own median means text was dropped * before it was ever spoken. And an absolute floor besides, because when a * voice is broken for EVERY sentence the language's own median is broken * too and the relative test sees nothing wrong — which is exactly how the * Ukrainian voice hid. * 9. Audio over Whisper's 30 s window is chunked rather than silently * truncated to its first 30 seconds and scored against text it never heard. * 10. --selftest is the falsification test. It synthesizes each language's text * with a DIFFERENT language's voice and asserts the score collapses. If the * wrong voice scores as well as the right one, the harness has no * discriminating power and its PASSes mean nothing — so it says so and * exits non-zero. Only languages whose own voice works can be targets: * asking whether the right voice beats a wrong one is meaningless when the * right voice is itself broken. * * WHAT THIS DOES NOT PROVE. The negative control also measures the method's * ceiling, and reports it: within one script the audit cannot tell a Spanish * voice from a Portuguese one, because sherpa takes the espeak voice from the * model's own metadata and a Portuguese phonemization of Spanish words is still * legible to Whisper. This verifies that a voice says the RIGHT WORDS. It does * not verify that it has the right accent. * * --------------------------------------------------------------------------- * USAGE * * node tools/voice-audit.mjs # every installed voice * node tools/voice-audit.mjs spa_Latn deu_Latn # named FLORES codes * node tools/voice-audit.mjs --all # all 31, downloading as needed * node tools/voice-audit.mjs --aac # audit the AAC listeners hear * node tools/voice-audit.mjs --selftest # prove the harness discriminates * node tools/voice-audit.mjs --keep-audio # leave the WAVs for a human ear * node tools/voice-audit.mjs --json out.json # machine-readable results * * # audition a replacement voice against the identical corpus * node tools/voice-audit.mjs spa_Latn --voice-bundle vits-piper-es_MX-claude-high * * Exits non-zero if any language is SUSPECT, so it can gate a release. */ import fs from 'node:fs'; import path from 'node:path'; import os from 'node:os'; import { fileURLToPath } from 'node:url'; import { execFileSync, fork, spawnSync } from 'node:child_process'; import { createRequire } from 'node:module'; const require = createRequire(import.meta.url); const HERE = path.dirname(fileURLToPath(import.meta.url)); const APP = path.join(HERE, '..'); const voices = require(path.join(APP, 'voices.js')); const VOICE_DIR = path.join(os.homedir(), 'Library/Application Support/Church Translate Voices'); const MODELS_DIR = path.join(os.homedir(), 'Library/Application Support/Church Translate Models'); const CORPUS_FILE = path.join(HERE, 'voice-audit-corpus.json'); /* A sentence must clear BOTH bars — see wordSimilarity() for why characters alone are not enough. A language is SUSPECT below either language bar. */ const SENTENCE_PASS = 0.80; // character similarity const SENTENCE_PASS_WORDS = 0.55; // word similarity, space-delimited scripts only const LANG_MIN_PASS_RATE = 0.80; const LANG_MIN_MEAN = 0.85; /* * Han script gets its own, looser bars — not to be kind to it, but because the * measurement genuinely means something different there. * * Chinese ASR picks characters, and Mandarin is full of homophones, so a * PERFECTLY pronounced sentence still comes back with different glyphs: * 耶稣基督 (Yēsū Jīdū) transcribed as 夜宿季度 (yèsù jìdù), 把 as 拜, 洁净 as * 结尽. Every one of those is the right sound and the wrong character, and no * amount of normalization can tell them apart without a pronunciation table * this tool does not carry. Scoring them as defects would fail a healthy voice * three sentences out of twelve — the definition of crying wolf. * * The bars below sit between what a working Han voice scores and what a broken * one does, and the separation is not subtle: the real Chinese voice measured * 0.878-0.904 across runs, while the negative control (Chinese text spoken by * the Spanish voice) measured 0.000. A voice as broken as Ukrainian's would be * caught here just as loudly. * * The honest fix is to compare pinyin rather than characters. That needs a * romanization table nobody has vendored yet, so until then the report says * plainly that a Han-script PASS means "said something that sounds right", and * the per-sentence evidence is printed for a reader who can adjudicate. */ const HAN_MIN_PASS_RATE = 0.60; const HAN_MIN_MEAN = 0.78; const HAN_SCRIPTS = new Set(['Hans', 'Hant', 'Jpan']); /* FLORES codes are _