feat(housenumber+address-points): suffix discipline, AU unit shorthand, FLAT_NUMBER storage - #37
Open
pdeaudney wants to merge 3 commits into
Open
feat(housenumber+address-points): suffix discipline, AU unit shorthand, FLAT_NUMBER storage#37pdeaudney wants to merge 3 commits into
pdeaudney wants to merge 3 commits into
Conversation
Phase 1 of the address-coverage work — query-side only, no rebuild needed. Phase 2 (G-NAF storage of FLAT_NUMBER + on-disk unit_id field) ships separately. # Changes ## 1. Suffix discipline in `housenumber_matches` PR #36 was correct on the no-suffix-query → suffixed-stored axis ("42" matches "42A") but too permissive when the query carries an explicit suffix: - `1327A` query vs `1327B` stored → was matching, NOW does not. These are typically distinct cadastral parcels in G-NAF (separate duplex units). - `1327A` vs `1327` (bare) → was matching, NOW does not. Stored `1327` is a different parcel from the suffixed subdivision. - `1327A` vs `1320-1340` (true range containing 1327) → still matches. Range form denotes a span of building numbers; suffix quirks within that span aren't decisive. - `1327` vs `1327A` → still matches (the documented permissive behaviour from PR #36 — bare query, suffix is sub-address detail the user didn't specify). Implementation: `parse_range` becomes `parse_hn` returning `ParsedHn { first, last, suffix }`. Suffix discipline applies before the range-overlap check. ## 2. `parse_au_unit_address` helper Detects the AU shorthand `<unit>/<housenumber>`: `3/827a` → Some(("3", "827a")) `12 / 45` → Some(("12", "45")) `12-14` → None (range, not unit/hn) `Apt 3/45` → None (let OSM word-prefix path handle it) `827a` → None ## 3. Pre-tokenisation pass in `parse_freeform_query` Tokenise splits on `/`, so by the time the per-token loop runs the "/" signal is gone and we can't tell `["3","827a"]` apart from `3 827a`. Pre-pass scans the input head for the AU pattern via a hand-written scanner (`consume_au_unit_prefix`) that tolerates whitespace around the slash. On match, sets `unit` + `house_number` and strips the prefix before tokenising the rest. ## 4. Leading digit-led tokens accepted as house_number Previous rule required ALL-DIGIT leading token to become house_number. `1327A Old Northern Road` would tokenise to `["1327a","old",...]` and fall into the rest bag — losing the housenumber. New rule: any leading digit-LED token becomes house_number when one isn't already set. ## 5. `unit` field on ParsedQuery + SearchParams ParsedQuery gains `unit: Option<String>`, populated from the AU shorthand. SearchParams gains `unit: Option<String>` so callers can also pass it explicitly. Echoed back to the response under `address.unit` via enrich_hit. Currently informational only — G-NAF's on-disk address-point format doesn't carry FLAT_NUMBER (Phase 2 wires that in), so the unit doesn't yet narrow the address-point lookup. # Verified end-to-end against AU G-NAF + OSM build q=3%2F827a+Old+Northern+Road+Dural → response includes "unit": "3" (from AU shorthand) + Old Northern Road street hit q=1327A+Old+Northern+Road+Middle+Dural → response includes house_number now extracted as "1327a" # Tests - 13 housenumber tests (added `suffix_discipline` covering the new tighter rule, `parse_au_unit_address_basic` and `parse_au_unit_address_rejects_non_pattern`) - 5 new parse_freeform_query tests covering AU shorthand, whitespace around slash, leading-digit-with-suffix capture, plain leading digit still works, and embedded slash NOT firing - All 208 lib tests pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ddresses
Adds a 4-byte unit_id slot to the AddressPoint format (24 → 28 bytes), so
G-NAF FLAT_NUMBER and OpenAddresses UNIT round-trip end-to-end:
* Builder side: build_gnaf_index.rs reads FLAT_NUMBER + prefix +
suffix from ADDRESS_DETAIL columns 9/10/11; build_openaddresses_index
reads the UNIT column. Both intern the value into strings.bin and
store its id on the AddressPoint, with the 0 sentinel preserved
for the common unit-less case (~95 % of rows) so we don't burn
16M non-zero ids on empty strings.
* Runtime side: find_by_housenumber takes an optional unit_hint and
applies two-tier matching — prefer a stored record whose unit_id
matches case-insensitively, fall back to a unit-less stored record
(the building entrance) when no unit-tagged record fits. Without a
unit hint, the matcher returns the geographically-nearest record
regardless of its unit, preserving prior behaviour.
* Same discipline mirrored in the OSM addr_points fallback in lib.rs
(the OSM AddrPoint already carries unit_id from the existing
addr:unit ingestion in PR #19), so all three sources speak the
same unit-matching contract.
* /search and /validate now accept and surface unit. enrich_hit
threads the freeform AU shorthand parse (Phase 1) through to
find_addr_point_in_country and prefers the stored unit on the
matched AddressPoint over the echoed input — clients see the
authoritative G-NAF unit when a match exists, fall back to their
typed unit otherwise.
* C++ normalise_housenumber gains a digit/digit-led split for
the "3/827a" AU/NZ unit shorthand (the existing word-prefix path
handled "Apt 3/45" but missed numeric-only LHS). Restricted to
`/` separators so it can't collide with a "12-14" range.
The format break is pinned at compile time by struct_layout.rs's new
AddressPoint size assertion (28 bytes), so a future field add surfaces
as a build-time failure rather than a corrupt 13h G-NAF rebuild.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
…5-10 feat(address-points): store + match unit (FLAT_NUMBER) on G-NAF & OpenAddresses
pdeaudney
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two-phase address-coverage work, surfaced from the
1327A Old Northern Roadand3/827a Old Northern Roadtest cases. Originally split as Phase 1 (#37, query-side only) and Phase 2 (#38, on-disk format change). #38 has been merged into this branch, so this PR is now the single umbrella that ships both.Phase 1 — query-side, no rebuild needed
Suffix discipline in
housenumber_matchesPR #36 was correct on the no-suffix-query → suffixed-stored axis (
"42"→"42A") but too permissive when the query carries an explicit suffix:1327A1327A1327A1327B1327A13271327is the original parcel;1327Ais its subdivision1327A1320-134013271327AAU
<unit>/<housenumber>shorthand3/827a Old Northern Roadused to lose the unit-vs-housenumber distinction at tokenisation (["3", "827a"]is identical to3 827a). New pre-pass scans the input head with a hand-written scanner that tolerates whitespace around/, extracts unit + housenumber, and strips the prefix before tokenising.Leading digit-LED tokens accepted as house_number
Previous rule required ALL-DIGIT leading token.
1327A Old Northern Roadwould lose the housenumber. New rule: any leading digit-LED token (digit + optional alpha suffix) becomes house_number when one isn't already set.unitfield on responseParsedQuery,SearchParams, andValidateParamsgainunit: Option<String>. Echoed in the response underaddress.unit. Phase 2 (below) wires the actual storage match.Phase 2 — on-disk format break, rebuild required
AddressPoint24 → 28 bytesNew
unit_id: u32slot, pinned at compile/test time bystruct_layout::address_point_sizeso a future field add surfaces as a build break, not corrupt data.Builders
FLAT_NUMBER_PREFIX | FLAT_NUMBER | FLAT_NUMBER_SUFFIX(PSV cols 9/10/11), interns the concatenated form intounit_id. 0 sentinel preserved for the unit-less common case (~95 % of rows).UNITcolumn with the same 0-sentinel rule.Two-tier matching in
find_by_housenumberWhen a unit hint is supplied: prefer a stored record whose unit matches (case-insensitive), fall back to a unit-less stored record (the building entrance) when no unit-tagged record fits. Without a unit hint, behaviour matches prior runs (geographic-nearest). Same discipline mirrored in the OSM addr_points fallback in
lib.rs(the OSMAddrPointalready carriesunit_idfrom PR #19).enrich_hitthreads the unit throughThe freeform AU shorthand parse is plumbed into
find_addr_point_in_country. Response surfaces the stored G-NAF unit on the matched AddressPoint, falling back to the echoed input when no AddressPoint match was found.C++
normalise_housenumberdigit/digit-led splitFor OSM-tagged addresses using AU shorthand (
addr:housenumber=3/827a), the existing word-prefix path missed numeric-only LHS. New rule: when the separator is/(not,/;, which would conflict with ranges) and both sides are digit-led with a digit/optional-alpha LHS, split as unit/housenumber.Verified end-to-end
Local AU build (
australia-latest.osm.pbf+ G-NAF, 16,418,248 records, 459 MB points.bin = 28 B × 16M ✓):/search?q=4+Burbank+Place+Norwest(no unit input) →unit: "1"from the matched G-NAF AddressPoint. Confirms storage→retrieval round-trip./search?q=3%2F4+Burbank+Place+Norwest(AU shorthand) → matches FLAT=3 record, surfacesunit: "3"./search?q=99%2F4+Burbank+Place+Norwest(fake unit) → no AddressPoint match, falls back to street centroid, echoes input "99". Confirms no-match path.Test plan
cargo build --releaseclean (server + builders + C++ build-index)🤖 Generated with Claude Code