bible-app/tools/interlinear/README.md

Interlinear re-conversion (STEPBible TAHOT → interlinear/*.json)

These scripts exist because the ORIGINAL converter did not. It was a one-off, was never kept, and its reference regex silently dropped 2,083 OT verses — 96 whole chapters, including 62 of the 150 Psalms — leaving the word-popup card blank on all of them for months. Keep these files so the data can be regenerated and audited instead of re-derived from scratch.

tools/ is excluded from build.files, so nothing here ships in the installer.

Rebuilding the data

# 1. Fetch the four TAHOT OT files (~67 MB) into ./tahot/
#    https://github.com/STEPBible/STEPBible-Data  ->  "Translators Amalgamated OT+NT"
#    TAHOT Gen-Deu / Jos-Est / Job-Sng / Isa-Mal   (CC BY 4.0, Tyndale House Cambridge)

# 2. Sanity-check the converter against the data already shipping.
#    It must reproduce existing verses almost exactly before you trust it on missing ones.
node tools/interlinear/tahot-convert.cjs tools/interlinear/tahot/TAHOT_Gen-Deu.txt . Genesis

# 3. Dry-run, then merge (additive: never changes a verse that already exists).
node tools/interlinear/merge.cjs . "" --dry
node tools/interlinear/merge.cjs . ../bible-app-interlinear-backup-<date>

# 4. Rebuild the usage stats from the merged corpus.
node tools/interlinear/gen-stats.cjs interlinear compare   # measure against the current file
node tools/interlinear/gen-stats.cjs interlinear write

The traps, in the order they bite

1. The reference parenthetical — this was the bug. TAHOT writes Gen.32.1(32.2): the English reference (NRSV numbering) with the Hebrew reference in brackets, wherever the two traditions number verses differently. A regex anchored on Book.ch.v#word rejects it. Every verse in such a chapter carries the bracket, so whole chapters disappear — and because scattered single verses carry it too (Gen.31.55(32.1)), a whole-chapter audit alone will not find them all. Key on the ENGLISH reference; the app's chapter|verse keys are English.

2. Punctuation. / separates prefixes/suffixes, \ separates punctuation. Drop a trailing \׃ (sof pasuq) and nothing else: the maqqef ־ and paseq ׀ stay, and a sof pasuq followed by a paragraph marker also stays (אֶחָֽד\׃\ \פ → אֶחָֽד׃ פ, 90 such words in Genesis alone).

3. Word numbers are two digits, plus two more for a sub-position. #0501 is "word 5, sub 1" — the LXX-restored words that sit between words 5 and 6 in Gen 4:8. Parsing it as the number 501 sorts them to the end of the verse.

4. Strong's numbers lose their disambiguation letter. H7225G → H7225; the app's dictionaries are keyed on the plain 4-digit form.

5. Do not "fix" the 36 verses that disagree. A fresh conversion differs from the shipped data on 36 verses: Qere/Ketiv variant picks, empty-w placeholders (14 rows across all of TAHOT, all Q(K) with no Strong's), and the verse-boundary cases TAHOT's own header lists (Num.26.1, 1Sa.21.1, 1Ki.18.33, 20.3, 22.22, 22.43, 1Ch.12.4, Isa.64.1). merge.cjs refuses to write if any existing verse would change — leave that guard in.

6. Psalm titles arrive as verse 0 (116 keys). Harmless: the app never requests verse 0.

Field mapping

app	TAHOT column	transform
w	2 Hebrew	drop trailing \׃, then remove / and \
tr	3 transliteration	remove /
g	4 English	/ → space, collapse
s	5 dStrongs	the {...} root, disambiguation letter dropped, padded to 4 digits
m	6 grammar	verbatim
l, lg	12 expanded tags	lemma + gloss from {...}; strip leading : , » → », _ → space

kjv is NT-only (from TAGNT) — no OT word carries it.

Verifying

• node src/coverage.js in ../bible-lexicon-export — expects zero missing chapters.
• Against the app's own bible-structure.js, 31,069 of 31,078 verses carry tagged words (99.97%). The 9 short ones are all NT versification differences (John 7:53, Acts 19:41, Rom 16:25-27, 2Co 13:13-14, Php 1:29-30) — TAGNT vs the NLT-derived structure, not gaps.