fix(whatsapp): resolve msgstore.db columns by name, and refuse a schema the data contradicts

[h4x0r]

Summary

A real 2023-era Android msgstore.db produced a structurally complete but
entirely wrong report: every message dated 1970-01-01, senders misattributed,
message text wrong, 62 calls instead of 166, and no forensic warning raised.
The pipeline was fine; the WhatsApp plugin read records by hardcoded column
ordinals that matched only our own fixtures.

The fixtures could never have caught it. modern_schema.sql was authored to the
same layout the ordinals assumed, so code and fixture agreed while both were
wrong about real data.

What changed

Columns are resolved by name from the database's own DDL (src/columns.rs).
A depth- and quote-aware parser reads each CREATE TABLE out of sqlite_master
and builds a name to position map: commas inside NUMERIC(10, 2) and
DEFAULT 'x,y' do not split a column, table-level constraints occupy no
position, quoted identifiers are unwrapped, malformed DDL yields an empty map
rather than a panic. A column the schema does not declare resolves to None, so
no field is ever read off a neighbouring column.

This applies to every table the extractor reads, not just the four in the
diagnosis. The modern schema has no call_row_id, so calls are no longer merged
on duration — that alone was collapsing unrelated calls of equal length into
single group-call records.

A wrong map is now an error, not a report (src/schema_gate.rs). Before
interpreting anything, up to 512 rows are sampled and 90% must carry a plausible
epoch-millisecond timestamp. The error names the table, column, resolved
ordinal, observed pass rate, expected bounds, and up to three offending values
with their _id. Not 100%: the real database has one message whose year does
not render, and rejecting good evidence over it would be the worse failure.

Refusing beats reporting empty. A database with no modern message table
aborts and names the legacy messages table if it finds one, or lists the
tables that are there. Zero messages must not look like a clean device.

A distribution-level detector. detect_timestamp_anomalies compares
neighbouring messages, and 245,981 messages all dated 1970 are in perfect order,
so it had nothing to report. TimestampDistributionAnomaly looks at the set and
fires above a 5% share of pre-2009 timestamps. Its standing job is genuine
tampering, which has the same signature.

detect_schema_version classifies on table presence. user_version is
app-defined and reads 1 on the real device, so the >= 100 arm was noise. The
plugin now passes the real table list instead of an empty slice.

Fixtures that can falsify a positional reader

Fixture	Provenance
real_modern_schema.sql	Real device DDL, synthetic rows. Confirmed positions read off the case device; intervening columns from published documentation, position-preserving and never read
shuffled_schema.sql	Same names and data, every column order permuted, _id declared mid-table
legacy_schema.sql	Pre-2018 generation, from published documentation — not the case device, and labelled so
modern_schema.sql	Kept, but now states it is simplified and why

The real and shuffled pair makes position-independence a property under test:
two databases differing only in declaration order must extract identically. A
characterisation test pins the invariant underneath — SQLite writes an
INTEGER PRIMARY KEY as a NULL at its declared position, so values[i] is
the column at DDL position i even when _id is not declared first.

Validation

tests/real_msgstore_validation.rs reconciles against sqlite3 reading the
same evidence: counts, the per-year histogram bucket for bucket, no message
dated 1970, senders carrying an @, hourly spread, and no anomaly warning
raised against the extraction's own dates. Expected values are derived from the
oracle at run time, not transcribed. A count mismatch fails with a breakdown by
evidence source and a distinct-rowid tally.

This Tier-1 run has not been executed. It needs CHAT4N6_REAL_MSGSTORE and
CHAT4N6_REAL_INPUT_DIR, which are not set in this environment; the harness was
exercised against a synthetic stand-in to prove the oracle plumbing works.
Everything else runs in CI. docs/validation.md records the method and tiers
each check by who confirms it.

Deliberately not done

Per the handoff's deferred list: message_media support (so media metadata is
absent, not wrong, on a modern schema), and auditing the iOS/Signal/Telegram
plugins for the same pattern. Auxiliary-table column names remain unverified
against a device — where a real schema spells one differently, the field
resolves to None. All recorded in docs/validation.md.

Note on CI

cargo test --workspace is green: 1183 passed. cargo clippy --workspace -D warnings and cargo fmt --all --check were already failing on main before
this branch — 19 clippy findings across chat4n6-report and
chat4n6-sqlite-forensics, plus rustfmt drift in cli/ and chat4n6-fs/.
Measured for the chat4n6-whatsapp package tree: 32 clippy findings on main,
15 here, none introduced. Clearing the rest is a separate sweep rather than
churn folded into this fix.

Tests were written first throughout: four RED commits (24, 8, 4 and 1 failing
tests) each followed by the GREEN commit that satisfies them.

🤖 Generated with Claude Code