# Interlinear re-conversion (STEPBible TAHOT → `interlinear/*.json`)

These scripts exist because the ORIGINAL converter did not. It was a one-off, was never kept, and
its reference regex silently dropped 2,083 OT verses — 96 whole chapters, including 62 of the 150
Psalms — leaving the word-popup card blank on all of them for months. Keep these files so the data
can be regenerated and audited instead of re-derived from scratch.

`tools/` is excluded from `build.files`, so nothing here ships in the installer.

## Rebuilding the data

```bash
# 1. Fetch the four TAHOT OT files (~67 MB) into ./tahot/
#    https://github.com/STEPBible/STEPBible-Data  ->  "Translators Amalgamated OT+NT"
#    TAHOT Gen-Deu / Jos-Est / Job-Sng / Isa-Mal   (CC BY 4.0, Tyndale House Cambridge)

# 2. Sanity-check the converter against the data already shipping.
#    It must reproduce existing verses almost exactly before you trust it on missing ones.
node tools/interlinear/tahot-convert.cjs tools/interlinear/tahot/TAHOT_Gen-Deu.txt . Genesis

# 3. Dry-run, then merge (additive: never changes a verse that already exists).
node tools/interlinear/merge.cjs . "" --dry
node tools/interlinear/merge.cjs . ../bible-app-interlinear-backup-<date>

# 4. Rebuild the usage stats from the merged corpus.
node tools/interlinear/gen-stats.cjs interlinear compare   # measure against the current file
node tools/interlinear/gen-stats.cjs interlinear write
```

## The traps, in the order they bite

1. **The reference parenthetical — this was the bug.** TAHOT writes `Gen.32.1(32.2)`: the English
   reference (NRSV numbering) with the Hebrew reference in brackets, wherever the two traditions
   number verses differently. A regex anchored on `Book.ch.v#word` rejects it. Every verse in such
   a chapter carries the bracket, so whole chapters disappear — and because scattered single verses
   carry it too (`Gen.31.55(32.1)`), a whole-chapter audit alone will not find them all. Key on the
   ENGLISH reference; the app's `chapter|verse` keys are English.

2. **Punctuation.** `/` separates prefixes/suffixes, `\` separates punctuation. Drop a **trailing**
   `\׃` (sof pasuq) and nothing else: the maqqef `־` and paseq `׀` stay, and a sof pasuq *followed
   by* a paragraph marker also stays (`אֶחָֽד\׃\ \פ` → `אֶחָֽד׃ פ`, 90 such words in Genesis alone).

3. **Word numbers are two digits, plus two more for a sub-position.** `#0501` is "word 5, sub 1" —
   the LXX-restored words that sit *between* words 5 and 6 in Gen 4:8. Parsing it as the number 501
   sorts them to the end of the verse.

4. **Strong's numbers lose their disambiguation letter.** `H7225G` → `H7225`; the app's dictionaries
   are keyed on the plain 4-digit form.

5. **Do not "fix" the 36 verses that disagree.** A fresh conversion differs from the shipped data on
   36 verses: Qere/Ketiv variant picks, empty-`w` placeholders (14 rows across all of TAHOT, all
   `Q(K)` with no Strong's), and the verse-boundary cases TAHOT's own header lists (Num.26.1,
   1Sa.21.1, 1Ki.18.33, 20.3, 22.22, 22.43, 1Ch.12.4, Isa.64.1). `merge.cjs` refuses to write if any
   existing verse would change — leave that guard in.

6. **Psalm titles arrive as verse 0** (116 keys). Harmless: the app never requests verse 0.

## Field mapping

| app | TAHOT column | transform |
|---|---|---|
| `w`  | 2 Hebrew | drop trailing `\׃`, then remove `/` and `\` |
| `tr` | 3 transliteration | remove `/` |
| `g`  | 4 English | `/` → space, collapse |
| `s`  | 5 dStrongs | the `{...}` root, disambiguation letter dropped, padded to 4 digits |
| `m`  | 6 grammar | verbatim |
| `l`, `lg` | 12 expanded tags | lemma + gloss from `{...}`; strip leading `: `, `»` → ` » `, `_` → space |

`kjv` is **NT-only** (from TAGNT) — no OT word carries it.

## Verifying

- `node src/coverage.js` in `../bible-lexicon-export` — expects zero missing chapters.
- Against the app's own `bible-structure.js`, 31,069 of 31,078 verses carry tagged words (99.97%).
  The 9 short ones are all NT versification differences (John 7:53, Acts 19:41, Rom 16:25-27,
  2Co 13:13-14, Php 1:29-30) — TAGNT vs the NLT-derived structure, not gaps.