[P2] Add a bounded JSON parser and writer usable from Semaprax programs

[wsdt]

Audit: 2026-09-05, commit 2b3b1ee24da2.
Classification: Documented standard-library gap; proposed data-processing enhancement. Priority: P2.

Finding

std.data.json is explicitly missing in docs/STANDARD-LIBRARY-V1.md. The Rust compiler's use of serde_json does not expose JSON processing to a Semaprax program. The existing byte/text/encoding primitives and project library packaging are a starting point, not a complete application JSON API.

For an agent-native language this is a high-value everyday capability: model responses, configuration, structured tool results, and API payloads are JSON.

First implementation boundary

Create a versioned pure parser over an explicit borrowed byte input with fixed input, nesting, token/node, decoded-string, and output limits. Start with a bounded token/arena document whose views cannot outlive the source, or use an owned representation only after its collection/cleanup contract exists. Do not depend on an unimplemented general map.

Support the complete JSON lexical forms in the admitted slice. Specify duplicate-key policy, exact integer handling, fractional/exponent representation, UTF-8 and surrogate rules, and source-offset diagnostics. A writer should produce valid deterministic JSON and never silently round an i64/usize through an f64.

Acceptance criteria

• Parse/serialize round-trips across objects, arrays, strings/escapes, booleans, null, signed limits, and fractional/exponent examples.
• Invalid UTF-8, lone surrogates, invalid number grammar, excessive depth, capacity exhaustion, and duplicate keys follow the documented deterministic policy.
• No input/output truncation is accepted as a complete document.
• Borrowed values cannot escape their source; owned values clean up exactly once on every failure path.
• Interpreter, native, and Wasm conformance use the same corpus, including exact numeric boundaries and Unicode.
• Add an offline project that validates an agent-style response and writes a structured result.
• Register the package in std/packages.json, regenerate both catalogs, and extend tests/project/standard_library.rs.

Dependencies: a token-view parser can be independent; an arbitrary owned JSON tree depends on an admitted collection and ownership slice. Keep parser data separate from executable tool authority.

Agent handoff

Recheck the working revision and existing issues first. Read AGENTS.md, the relevant completion-matrix row, and the owning versioned specifications. This proposed slice must preserve canonical source, independent replay, precise target admission, and separately granted authority. Place regressions in the existing owning harness. Execute the applicable quality gates and record what actually ran; unrun native, hosted, or device evidence must stay unpromoted.

Audit revision scope

Runtime observations in this ticket belong to commit 2b3b1ee. A final source-tree comparison against e1c5b56 was used to check for obvious superseding changes. That later snapshot was not rebuilt or exhaustively retested. Reproduce on the intended fixing revision before implementation.

Audit ID: SPX-AUDIT-20260905-15

Suggested scope: L. Dependencies: None required.

[wsdt]

Partially implemented in a59a68d3 (on main). Leaving open — a genuinely-complete slice landed, but it is much smaller than this issue asks for, and the reason is a measured compiler bound rather than a design choice.

What landed

A new bundled package std.data.json (std/data-json/) — an allocation-free scanner over borrow Slice<u8> — plus docs/BOUNDED-JSON-SCANNER-V1.md. Twelve functions: at_is, at_in, failure, is_failure, failure_offset, skip_whitespace, hex_at, code_unit, escape_kind, escape_end, string_end, is_string.

Design points that satisfy this issue's stated requirements:

• Source-offset diagnostics with no allocation and no second scan. One usize carries both answers: r <= byte_len(input) is an exclusive end offset, r > byte_len(input) is a rejection whose first offending byte is at r - byte_len(input) - 1. failure_offset is total (it takes a fallback), so it can never be called out of contract.
• Views cannot outlive the source, in the strongest available sense: the scanner returns only scalars, never a view.
• Strict surrogate rules. A high surrogate must be immediately followed by \u plus a low surrogate; a lone high surrogate and any unpaired low surrogate are rejected at the offset of the backslash that opened them. escape_kind exposes the classification.
• No truncated prefix is accepted as complete. Control bytes 0x00–0x1F are rejected inside strings, and an unterminated string is rejected at end-of-input.

Verified across all three backends, which is this issue's conformance criterion for the implemented subset: cargo test --locked -p semaprax --test project -- standard_library → 12 passed, 0 failed, 1 ignored (421s), including examples_and_conformance_return_zero_on_interpreter_native_and_wasm (interpreter, clang C11 at -O0 and -O2, Core Wasm under Node) and every_library_module_is_self_contained_when_vendored. Both catalogs regenerated; tests/project/standard_library.rs extended with package_manifest_links_bundled_std_data_json, as the issue required. Coordinator-verified independently: semaprax check → verified, semaprax test → project tests passed, semaprax fmt --check → canonical.

Not implemented, and documented as absent in the spec, the matrix and both catalogs

• number tokens, exact i64 decoding, fractional/exponent representation
• literal tokens (true / false / null)
• structural document validation — and therefore no nesting-depth limit, no trailing-byte rejection, no duplicate-key policy, no round-trip
• UTF-8 validation and lone-surrogate rejection in raw bytes (escape-level surrogate rules are implemented)
• a writer, decoded strings, an owned document tree, the offline agent-response project

The cause is a hard compiler bound, and it was measured rather than guessed

All of the above was working — 13.8 KB including an iterative depth-bounded document scanner with a base-2 container stack, strict UTF-8, and exact i64 decoding — and had to be cut. Two limits bite:

1. SPX-G171 workspace-graph pre-bound. The conformance module must import every library function, and each import re-charges that function's tree, so a library is charged roughly twice. Measured on this exact code: a 4,679-byte library module is admitted; a 5,575-byte one fails. Splitting into sibling packages does not help — a consumer linking a 4.3 KB module plus a 4.7 KB dependency also fails SPX-G171, so two packages could never be used together.
2. SPX-H006 cleanup-replay path budget (MAX_REPLAY_PATHS = 65_536, src/cleanup_plan/replay.rs:47) — roughly 16 lazy-boolean decision points per function. A first escape_end with ~24 was rejected and had to be restructured into escape_kind + escape_end with match tables.

This is the real blocker for std.* generally, not JSON specifically, and it is filed separately as #78. Completing this issue is gated on that.

[wsdt]

Status update: #78 landed, and it does NOT unblock this issue. Recording that explicitly so nobody assumes otherwise.

#78 (dd7640a8) fixed a real double-charge — an imported function was charged as a full second copy of its provider tree (142,043 bytes against the library's own 142,332, then expanded ×24) because the cost function walked the whole AST while the projection only retains a stub. Conformance import_raw dropped 142,043 → 12,407, and the package pre-bound fell about 20–25%.

The authoring ceiling for a single std/ library module moved:

	admitted	first SPX-G171
before #78	4,769 B	5,277 B
after #78	6,801 B	7,309 B

The 13.8 KB implementation discarded during this issue's work is still inadmissible at 6,801 B. +43% is real, but the remaining cost is the module's own source at roughly 1,200–1,400 estimated bytes per source byte, driven by the documented HIR_STRUCTURE_EXPANSION_FACTOR = 24, HIR_STRING_COPY_FACTOR = 64 and the identity-slot terms. Those are deliberate upper bounds on retained resolver memory, backed by compile-time assertions — tightening them means re-deriving a bound, not fixing an accounting error, and #78 deliberately left them alone.

So the deferred work here — number and literal tokens, structural document validation with a nesting limit, trailing-byte rejection, duplicate-key policy, UTF-8 validation, decoded strings, a writer, and the offline agent-response project — remains gated on either:

1. a separate, evidence-backed re-derivation of those expansion factors (a real piece of work, not a constant bump); or
2. splitting the parser across modules in a way that keeps each under the per-module ceiling — noting that [P1] SPX-G171 pre-bound caps std.* packages at ~4.7 KiB and double-charges imported trees #78 confirmed sibling packages now compose (package_manifest_links_two_large_sibling_packages links std.bytes 4,705 B + std.data.json 4,769 B and executes), which was not true before [P1] SPX-G171 pre-bound caps std.* packages at ~4.7 KiB and double-charges imported trees #78 and is the one structural avenue this fix did open.

Option 2 is now viable where it previously was not, and is probably the cheaper next step for this issue.

[wsdt]

Substantial progress landed in 4f81ec27 (on main). This issue stays open — structural document validation, decoded strings, an output buffer and the offline agent-response project are still missing.

First, a correction to my earlier comment on this issue

I wrote that the constraint was a 6,801-byte per-module ceiling and that the route was "sibling packages compose". Both framings were wrong in a way that matters:

1. The pre-bound is charged per package, not per module. Splitting into modules inside one package does nothing — three modules in one package hit SPX-G171 at 13.4 KB regardless.
2. A [dependencies] edge spends the whole dependency's source against the consumer's budget. All three new packages initially depended on std.data.json and all three failed. Dropping the dependency and restating a three-line at_is/at_in helper locally — about 180 B, against 4,769 B of vendored dependency — made all three pass.

So sibling packages do compose, but not because they share a budget gracefully: it is not depending on each other that makes them fit. That is close to the opposite of the intuition, and it is the most useful finding here. It is now recorded in docs/STANDARD-LIBRARY-V1.md rather than left in a comment thread.

The real ceiling is also higher and more variable than the single number I quoted: 13.3–15.9 KB of package source, varying with declaration and expression structure rather than byte count alone.

What landed — four new sibling packages

Package	Tier / profile	Scope
std.data.json.token	portable / useful-data.v1	RFC 8259 number grammar (integer_end, fraction_end, exponent_end, number_end, is_number), true/false/null (literal_kind, literal_end, is_literal), exact i64 decoding (i64_or)
std.data.json.utf8	portable / useful-data.v1	Raw-byte UTF-8 validation: sequence_kind, scalar_at, is_shortest, is_surrogate, sequence_end, utf8_end, is_utf8
std.data.json.write	portable / useful-data.v1	Pull-based string encoder: quoted_len/quoted_byte, escape_len/escape_byte, width_at/encoded_at
std.data.json.digits	core / scalar	Exact i64 rendering (i64_len/i64_byte) and literal bytes (literal_len/literal_word/literal_byte)

write had to split again on a profile boundary: under scalar, usize is not an admitted signature type (SPX-G174); under useful-data.v1, every web export must take only borrow Slice<u8> (SPX-W121). The numeric encoder has no slice parameter, so it cannot live in a useful-data.v1 package, and the string encoder cannot live in a scalar one.

Measured headroom

Probed by appending padding to each library module and its conformance imports, bisecting until SPX-G171 fires:

Package	source	admitted extra	first SPX-G171
data-json-digits	5,190 B	+10,411 B	+10,661 B
data-json-token	11,087 B	+3,601 B	+3,847 B
data-json-utf8	9,746 B	+3,541 B	+3,783 B
data-json-write	9,486 B	+4,339 B	+4,585 B

Correctness points

• i64_or accumulates negatively (value * 10 - digit), so -9223372036854775808 decodes exactly with no intermediate overflow. It returns the caller's fallback — never a rounded value — on overflow or when the token carries ./e/E. i64_byte renders toward zero the same way, so i64::MIN renders as its exact 20 bytes. Nothing anywhere touches f64, which is this issue's explicit requirement.
• The writer is pull-based (len + byte(index)), which is how it stays allocation-free inside the profile.
• Two SPX-H006 replay-budget hits were real and fixed by flattening, not by weakening: literal_byte's four parallel match tables plus nested ifs became one packed big-endian word plus a shift loop.

Coordinator-verified

cargo test --locked -p semaprax --test project -- standard_library → 15 passed, 0 failed, 1 ignored (149.59 s), re-run by me on the rebased tree after confirming the rebase completed. That covers interpreter + clang C11 -O0/-O2 + Core Wasm under Node for all 21 packages, plus every_library_module_is_self_contained_when_vendored and committed_catalogs_match_the_sources. cargo fmt --all --check clean. Package registration confirmed: 21 packages, 9 core / 11 portable / 1 test, with all five std.data.json* modules listed.

Two new gate cases: package_manifest_links_json_scanner_and_token_siblings (9 imports) and package_manifest_links_json_writer_siblings (8 imports), both checking, executing entry and test, and asserting both dependency paths.

Still unimplemented — why this issue stays open

• Structural document validation — no object/array grammar, nesting-depth limit, trailing-byte rejection, or duplicate-key policy. The largest remaining piece; it needs its own sibling package, and no half-wired version was attempted.
• Decoded strings — escapes are validated and measured, never expanded; needs an output buffer with explicit capacity.
• An output buffer or owned document tree — the writer reports bytes, it does not store them.
• Floating-point numbers — i64_or refuses a fraction/exponent rather than converting.
• The offline agent-response project, and round-trip over a whole document.

Unrelated staleness noticed, not touched

docs/AGENT-QUICK-REFERENCE.md cites a "22,076-byte, 6,662-unit full catalog"; the committed catalog was already 23,828 B before this change and is 30,003 B now. The cli_help_surface_v1 guards are relative ratios and still pass, so this is stale prose rather than a broken gate. That page is printed verbatim by semaprax help language, so editing it is a separate change.

[wsdt]

The structural document layer landed in 639ed5e0 (on main). This issue stays open, but the remaining work is now gated on one measured fact, recorded below so no future attempt wastes effort rediscovering it.

What landed — std.data.json.doc, the sixth sibling

Twelve library functions, no [dependencies]:

• Object/array grammar — a six-mode state machine over a base-2 container stack packed into one i64 with a sentinel bit (object push stack*2+1, array push stack*2, pop stack/2, stack == 1 = top level). No recursion, which the language has no budget for.
• Depth limit — document_end(input, start, depth_limit), explicit and clamped to 32; a container opened past it is rejected at the offset of its {/[.
• Trailing-byte rejection — whole_end(input, depth_limit) requires only whitespace after the value, rejecting at the first trailing non-whitespace byte. is_document is the whole-input form.
• Number, literal and string-framing tokens validated in the same pass; raw 0x00–0x1F and unterminated strings rejected. The family result encoding is preserved exactly.

Policies chosen, and why

• Duplicate keys: accepted, nothing discarded, stated explicitly in the spec. Rejecting them needs either a retained key list — and no growable collection exists ([P2] Implement a first owned bounded vector slice for everyday data processing #62) — or a second pass, which does not fit the budget.
• Escape/surrogate and raw UTF-8 validation: delegated, with the composition stated rather than left to inference. string_end treats \ + next byte as two bytes, which finds the correct end of every string including \uXXXX, so structural correctness is exact; escape-form and surrogate validity remain in std.data.json, raw UTF-8 in std.data.json.utf8, and the spec states the composition as is_document && is_utf8, plus is_string per span. Restating them costs ~1.2 KB the bound does not admit.

The measurement that governs everything remaining

12,216 B admitted, 12,292 B first SPX-G171. The committed package is exactly 12,216 B. Zero headroom.

Four configurations were measured, three rejected:

Configuration	Own source	Result
Self-contained, full escape + surrogate validation	15,748 B	SPX-G171
Depends on std.data.json + std.data.json.token	8,502 B (+9,333 vendored)	SPX-G171
Depends on std.data.json only, restating tokens	10,852 B (+4,769)	SPX-G171
Self-contained, escapes delegated	12,216 B	verified

Note the second and third rows: a [dependencies] edge is fatal even when the consumer's own source is smaller than the standalone version.

Also: the bound is charged partly against maximum_identity_bytes (identity length × 64), not byte count alone. Shortening the module segment document → doc bought ~960 B of admitted source; std.j would have bought ~1,800 B. Module naming is a budget decision in this codebase.

Consequence for this issue: no further split will help. Duplicate-key rules, escape/UTF-8 re-checking inside the document layer, decoded strings, an output buffer, an owned document tree, floating-point numbers, whole-document round-trip and the offline agent-response project now all require the SPX-G171 expansion factors (HIR_STRUCTURE_EXPANSION_FACTOR = 24, HIR_STRING_COPY_FACTOR = 64, and the identity terms) to be re-derived with evidence — a real analysis, not more packaging. That is the single next step, and it belongs in its own issue rather than here.

Two undocumented language rules found

• SPX-T252: match is not admitted in a while body. Every table lookup inside a loop must be a helper call. This forced the advance / step_action / next_state split.
• SPX-H006 path counts multiply across sequential statements (sequence_path_counts) and only sum across match arms. Packing four tables into one function tripped the replay budget; splitting one back out fixed it — by flattening, not by weakening.

Coordinator-verified

cargo test --locked -p semaprax --test project -- standard_library → 16 passed, 0 failed, 1 ignored (147.02 s), re-run by me after rebasing onto current main and confirming the rebase completed — which mattered here, because with zero headroom a rebase could plausibly have pushed the package over its bound. It did not. Coverage includes interpreter + clang C11 -O0/-O2 + Core Wasm under Node for all 22 packages, every_library_module_is_self_contained_when_vendored, committed_catalogs_match_the_sources, and the new package_manifest_links_the_json_document_layer. cargo fmt --all --check clean.

The lane confirmed the 6 --lib failures are pre-existing by reverting its own change, forcing a full recompile, and reproducing them identically.