#1915 follow-up: semantic-doc gate only accepts file_type=="document", so docs whose LLM layer is concept/rationale are still AST-quick-scanned

[jw3b-dev]

#1915 follow-up: the semantic-doc gate only accepts file_type == "document", so docs whose LLM layer is concept/rationale still get AST-quick-scanned

Version: 0.9.17
Affects: graphify update / watch / graphify hook install on any corpus whose semantic layer was produced by the skill's extraction subagents.

Summary

#1915 fixed doc double-representation by skipping the AST quick-scan for docs that already carry semantic nodes. The gate that decides "does this doc have semantic nodes" requires a node with file_type == "document":

# watch.py:915-927
# Semantic doc nodes lack the AST origin marker. Gate on
# file_type=="document" as well so a pre-#1865 graph whose
# AST nodes lack the ``_origin`` marker isn't misread as
# semantic-backed via some other marker-less node kind.
semantic_doc_identities: set[str] = set()
for node in prior.get("nodes", []):
    if node.get("_origin") == "ast":
        continue
    if node.get("file_type") != "document":   # <-- too narrow
        continue
    identity = prior_paths.identity(node.get("source_file"))
    if identity:
        semantic_doc_identities.add(identity)

But references/extraction-spec.md instructs subagents to emit concept and rationale for exactly this material:

For rationale (WHY decisions were made, trade-offs, design intent): store as a rationale attribute on the relevant concept node — do NOT create a separate rationale node or fragment node. Only create a node for something that is itself a named entity or concept. Use file_type:"rationale" for concept-like nodes (ideas, principles, mechanisms, design patterns). file_type MUST be one of exactly these six values: code, document, paper, image, rationale, concept.

So a doc whose extraction produced only concept/rationale nodes — which the spec actively encourages, and which is the desirable outcome for a well-extracted doc — is invisible to the gate. It gets AST-quick-scanned, and #1915's symptom returns for that file: heading nodes are minted alongside the semantic nodes and the doc is represented twice.

Docs that happen to have at least one document-typed node (e.g. a file-level title node) are correctly protected. The result is a silent, per-file split: some docs are safe, some are not, with no diagnostic.

Impact (measured on a real 1,070-file corpus)

Semantic layer built via the /graphify skill (17 extraction subagents, ~2.4M tokens), then a single graphify update:

	before update	after update
nodes	8,304	8,717
document	339	1,004

• 249 .md files had a document-typed node → protected, unchanged.
• 62 .md files had only concept/rationale nodes → each grew AST heading nodes.
• Predicted-vs-actual: predicted 62, observed 63 fragmented files, of which 62 exactly matched the "no document-typed node" set. (The 63rd was a genuinely new doc with no semantic layer — correct fallback behavior.)

Worst cases:

48 heading nodes  research/audit-platform/2026-05-29-architecture-research.md
30 heading nodes  .agents/sops/requirements-architect.md
29 heading nodes  .agents/sops/qa.md
21 heading nodes  docs/KTHULHU_ADVISOR_WHITEPAPER.md
19 heading nodes  .agents/rules/business-analyst.md

Example of the double representation after one graphify update — .agents/rules/business-analyst.md, 26 nodes:

[document ] origin=ast   business-analyst.md
[document ] origin=ast   IT Business Analyst — Operational Rules
[document ] origin=ast   1. Core Identity
[document ] origin=ast   2. Analytical Principles (Non-Negotiable)
[document ] origin=ast   3. Requirement Standards
[document ] origin=ast   3.1 Business Objective Format
...19 total AST heading nodes...
[concept   ] origin=None  <3 real LLM concept nodes>
[rationale ] origin=None  <4 real LLM rationale nodes>

The 7 semantic nodes are the ones worth keeping; the 19 heading nodes are the fragments #1915 set out to eliminate.

Reproduction

1. Build a graph with the skill so the doc layer is LLM-extracted (/graphify .).
2. Ensure at least one doc's extraction yields only concept/rationale nodes and no document node — the spec's preferred shape for a doc full of named concepts.
3. Run graphify update ..
4. That doc now carries _origin=ast heading nodes in addition to its semantic nodes; a doc that happens to own a document-typed node does not.

Suggested fix

Treat any non-AST node as evidence of a semantic layer, rather than allowlisting one file_type. The stated reason for the narrow gate is to avoid misreading a pre-#1865 graph whose AST nodes lack _origin; that can be handled by gating on the graph rather than on each node's type — e.g. only apply the _origin-based rule when the graph contains at least one _origin=="ast" node (the same graph_has_legacy_ids style check already used by _semantic_id_remap), and otherwise fall back to the current conservative behavior.

graph_has_ast_markers = any(n.get("_origin") == "ast" for n in prior.get("nodes", []))
SEMANTIC_TYPES = {"document", "concept", "rationale", "paper"}
for node in prior.get("nodes", []):
    if node.get("_origin") == "ast":
        continue
    if not graph_has_ast_markers and node.get("file_type") not in SEMANTIC_TYPES:
        continue          # legacy graph: keep the conservative type check
    if graph_has_ast_markers and node.get("file_type") not in SEMANTIC_TYPES:
        continue          # still exclude code/image nodes
    ...

Minimally: widen the check from != "document" to not in {"document", "concept", "rationale", "paper"}. That alone fixes all 62 files in our corpus and stays consistent with the six-value file_type enum the extraction spec defines.

Workaround

A scoped post-commit hook that filters the changed set through _CODE_EXTENSIONS (.md is correctly absent from it) and calls _rebuild_code(changed_paths=code_only), so docs are never AST-scanned at all. Doc changes are recorded in the needs_update flag for a deliberate /graphify --update instead.

Aside

.claude/skills/graphify/references/extraction-spec.md and watch.py disagree about what a semantic doc node looks like. Whatever the fix, it may be worth asserting the six-value file_type enum in one place both sides import, so the extractor's vocabulary and the incremental gate's vocabulary can't drift apart again.

[safishamsi]

Fixed on v8 (ships in 0.9.18) via #1957 (5163a62), thanks @alphanury and @jw3b-dev. The #1915 semantic-doc gate now recognizes the full doc-shaped file_type subset (document/concept/rationale/paper), so a doc represented purely as concept/rationale nodes is no longer AST-quick-scanned and double-represented on rebuilds; bloated graphs self-heal on the next rebuild.