fix(build-db): make deploys resilient to malformed tenant/job data - #179
Open
datascry wants to merge 3 commits into
Open
fix(build-db): make deploys resilient to malformed tenant/job data#179datascry wants to merge 3 commits into
datascry wants to merge 3 commits into
Conversation
A single tenant or job row that fails schema validation used to throw at build-db and abort the whole deploy: the SQLite build never finished, the Pages deploy was skipped, and the live manifest froze at the last good build while scrapes kept succeeding — a silent outage. One underscore or dotted domain in a slug was enough to stall the site indefinitely. Parse each row with safeParse: valid rows proceed unchanged, invalid rows are dropped and recorded, and one summary line per kind names the offenders on stderr. Jobs are now validated per-row rather than per-file, so one bad posting no longer discards a tenant's entire corpus. Orphan jobs whose tenant row was skipped are left in place — the slim index reads jobs directly with no tenant join, matching prior behavior.
build-db runs only post-merge, so a malformed tenant row passes pull-request CI and first surfaces at deploy — the exact blind spot that froze the live site. Validate every committed data/tenants/*.json entry against TenantSchema in the test suite (which pr.yml runs), failing with the file, row index, slug, and violated rule so the source gets fixed before merge rather than silently dropped on every refresh.
The refresh chain can fail after the scrape succeeds — build-db aborting on a bad row, a Pages hiccup — leaving the live manifest frozen while every check stays green. Nothing surfaces the freeze. Add a workflow that runs every 6h (and on dispatch), fetches the live manifest, and fails when built_at is older than 36h. A red scheduled run plus GitHub's failed-workflow notification makes a stalled deploy visible within a day; the failure message reports the actual staleness. It needs no new deps — plain curl, jq, and date on the runner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three layered defenses so a single malformed row can never silently take the live site offline again — the failure mode behind today's outage (a lone bad tenant slug aborted build-db, skipped deploy, and froze the manifest while scrape kept succeeding, so there was no signal).
1. build-db: skip-and-warn instead of abort
cli.tsbuild-db no longer does eagerz.array(TenantSchema).parse(...)(throws on the first bad row). New pure helpers indb/resilient-parse.ts:partitionTenants— per-rowsafeParse; valid rows pass, invalid rows are dropped + recorded.partitionScrapeOutput— envelope validated strictly, but jobs per-row, so one malformed posting no longer discards a tenant's whole corpus (upgrade from the old per-file behavior). A corrupt envelope still skips the file.formatSkipSummary— one stderr line naming up to 20 offenders asats/slug (field)+ the true total.The build always completes from valid rows. Orphan jobs (whose tenant row was skipped) are left in place — verified safe: the slim-index reads
companydirectlyFROM jobswith no tenant join, so they render correctly.2. PR-time guard
scraper/tests/tenant-data.test.tsvalidates everydata/tenants/*.jsonentry againstTenantSchema, so bad tenant data failsbun run test(→pr.yml) at PR time — not post-merge at deploy. Message names the file, row, slug, and violated rule.3. Deploy-freshness watchdog
.github/workflows/deploy-freshness.yml(cron every 6h + dispatch) curls the live manifest and fails ifbuilt_atis >36h old — a frozen deploy now surfaces as a red check instead of silence. Fails-safe on curl error / missing field / unparseable date. No new deps.Gates
typecheck + lint clean;
bun run test1754 pass / 0 fail;resilient-parse.ts100% line+func,cli.ts95.66%/98.41%. Passed the fresh-Opus phase-audit gate (verdict SHIP; orphan-job safety confirmed via a throwaway build-db integration test).Follow-up (optional, noted in audit): tighten
tenant_resultsenvelope validation to per-row too, matching the per-row spirit.