fix(llm): downgrade unverifiable code-typed semantic nodes to UNVERIFIED#1949
Closed
TPAteeq wants to merge 1 commit into
Closed
fix(llm): downgrade unverifiable code-typed semantic nodes to UNVERIFIED#1949TPAteeq wants to merge 1 commit into
TPAteeq wants to merge 1 commit into
Conversation
The semantic (LLM) extraction runs on documents/papers/images; code files are handled by the deterministic AST engine and never reach the model. A node the model tags file_type="code" is therefore a symbol it surfaced from within a document (a name in a fenced code block, an API referenced in a paper), and it enters graph.json today with no check that the symbol actually appears in the source the model read. `_out_of_scope` (Graphify-Labs#1895) only rejects nodes attributed to a file that was NOT dispatched, so a fabricated symbol on a dispatched file passes. `extract_files_direct` now verifies every file_type=="code" node whose source_file was dispatched in the call: if no identifier from its label occurs (case-insensitive substring) in that file's source bytes, the node's confidence is downgraded to "UNVERIFIED" (never dropped) and the count is reported to stderr. Concept/document nodes, nodes without a source_file, nodes attributed to undispatched files, and labels with no checkable identifier are left untouched. Best-effort; never aborts extraction; one unreadable file is skipped, not fatal.
safishamsi
added a commit
that referenced
this pull request
Jul 16, 2026
…follow-up to #1949) The PR flagged unverifiable code nodes as confidence="UNVERIFIED", but node confidence is read by nothing and UNVERIFIED isn't in the validated (edge-only) confidence vocabulary {EXTRACTED,INFERRED,AMBIGUOUS} — a dead, colliding field. - Move the flag to a dedicated verification="unverified" node field, off the confidence key. A node the model already hedged (INFERRED/AMBIGUOUS) is left untouched, as before. - Also check the node id's identifiers (not just the label): ids carry the verbatim symbol, cutting false flags on prettified labels. - Skip the (potentially PDF-re-extracting) source read entirely when a result has no code-typed node with a source_file. - Wire a consumer: diagnose_extraction now counts and reports unverified nodes, so the persisted flag is actually surfaced. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
safishamsi
added a commit
that referenced
this pull request
Jul 16, 2026
…angelog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
|
Landed on v8 (ships in 0.9.18), cherry-picked to preserve your authorship. Thanks @TPAteeq — code-typed semantic nodes with no evidence in the source are now flagged rather than presented as read facts. Follow-up added: moved the flag off the confidence key (whose validated vocabulary is edge-only) onto a dedicated verification="unverified" node field, made it check the node id as well as the label (fewer false flags on prettified labels), skipped the source re-read when a result has no code node, and wired a consumer so it is not a dead field (graphify diagnose reports the count). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The semantic (LLM) extraction pass only ever runs on documents, papers, and images — code files are handled by the deterministic AST engine and are never sent to the model (
cli.pybuilds the semantic worklist assemantic_files = doc_files + paper_files + image_files). So when the model returns a node taggedfile_type="code", that symbol was surfaced from inside a document — a name in a fenced code block, an API referenced in a paper, a symbol mentioned in prose. Today such a node entersgraph.jsonwith no check that the symbol actually occurs in the source the model read. The existing_out_of_scopefilter (#1895) only rejects a node whosesource_fileresolves to a real file that was not dispatched in the batch; a fabricated symbol attributed to a file that was dispatched sails straight through it. That directly undercuts the "grounded in the source" guarantee: a plausible-looking but invented symbol becomes a permanent, first-class graph node.Fix
extract_files_directnow performs an evidence check on its own output before returning. For everyfile_type=="code"node whosesource_fileresolves to one of the files dispatched in this call, it verifies that at least one identifier token from the node's label (≥3 chars, case-insensitive) appears as a substring of that file's source bytes — the exact bytes the model saw, read through the same_read_filespath and capped at_FILE_CHAR_CAP. If none of the label's identifiers occur, the node'sconfidenceis downgraded to"UNVERIFIED"rather than dropped, and the number downgraded is reported to stderr so the signal is visible during extraction (and left on the node ingraph.jsonfor programmatic inspection).The check is precision-first and deliberately conservative: concept/document nodes, nodes with no
source_file, nodes attributed to a file not dispatched in this call (left to_out_of_scope), and labels with no checkable identifier are all untouched. The whole pass is best-effort — a single unreadable file (e.g. a malformed PDF) is skipped rather than aborting the chunk, and any unexpected error disables the check without failing extraction.Tests
tests/test_evidence_binding.py: fabricated symbol downgraded; real and qualified/prettified labels (PaymentProcessor.charge_card()) not false-flagged; concept, document, and no-source nodes untouched; symbols attributed to an undispatched file left to_out_of_scope; short/uncheckable labels skipped; existing lower confidence never clobbered;FileSliceinput; absolutesource_file; stderr surfacing; and that a downgraded node still passes validation. Full test suite green (3295 passed).