fix(detect): clear stale semantic_hash for dispatched-but-omitted files (#1948)

CHANGELOG.md

@@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu
 
 ## 0.9.18 (unreleased)
 
+- Fix: `save_manifest` no longer seeds a stale `semantic_hash` for a file that was dispatched this run but produced no stamped output, masking an LLM-omitted doc on the next run (#1948, thanks @rsolanilla). A file omitted from the semantic result (a `--force` re-run where the model drops its chunk) is dropped from the `files` dict `cli.py`'s `_stamped_manifest_files()` passes to `save_manifest`, per the #933/#1890 never-stamp-a-failed-chunk contract — but the seed loop that carries forward untouched rows for subset saves (#917) then copied that file's row from the on-disk manifest verbatim, including its `semantic_hash` from an earlier successful run. `detect_incremental(kind="semantic")` compared current content against that inherited hash, found a match, and silently reported the file unchanged — defeating the #1890 retry promise the exact way the issue's manual "blank the hash by hand" workaround worked around. `save_manifest` now accepts a `clear_semantic` set; the seed loop forces `semantic_hash` to `""` for any file in it instead of inheriting the stale value. `cli.py` derives the set as `semantic_files` — what was actually sent to the backend this run (narrowed by the incremental gate and `--code-only`, widened by deep mode) — minus `_stamped_manifest_files()`'s result: dispatched-but-not-stamped, regardless of why. Untouched live files that were never dispatched are deliberately not in the set, so a partial incremental run cannot blank the rest of the corpus's stamps. The set is passed at all three `_save_manifest` call sites. `clear_semantic` defaults to `None` (no-op), so every existing caller and the #917/#1908 seed/pruning behavior for untouched and excluded-but-alive rows is unchanged.
 - Fix: the semantic cache no longer replays extractions from an older prompt after an upgrade (#1939, thanks @HunterMcGrew and @SinghAman21). Entries were keyed on `sha256(file content + path)` alone, with no component for the extraction prompt that produced them, so a release that changed the prompt left every unchanged file a cache hit: the run exited 0, `cost.json` looked cheap, and the graph silently carried two prompt generations side by side. Semantic entries are now namespaced by a fingerprint of the extraction prompt (`cache/semantic/p{fingerprint}/`, mirroring the AST cache's `v{version}/` layout), keeping both properties #1252 wanted — entries survive releases that don't touch the prompt, and invalidate only when it actually changed. The fingerprint normalizes line endings so a CRLF checkout doesn't look like a prompt change. Both extraction paths pass their prompt: the Python/CLI path (`llm.py`'s `_EXTRACTION_SYSTEM`, all backends) automatically, and the skill path via a new `prompt_file` argument in Step B0/B3 pointing at the `references/extraction-spec.md` the subagents were handed. Pre-existing entries predate fingerprinting and have unknowable vintage: they are still served rather than re-billing a whole corpus, but `check_semantic_cache` now warns with the count, so the "no signal at all" the report describes becomes a visible one; `--force` (or `GRAPHIFY_FORCE=1`) re-extracts them. Old-fingerprint entries are pruned by liveness only, never swept wholesale the way stale AST versions are — two hosts with different prompts can share one `graphify-out/`, and a wholesale sweep would have each run delete the other's entries. (The two monolith skills, aider and devin, inline their prompt instead of shipping a spec sidecar and stay on the unfingerprinted path for now.)
 - Fix: PostgreSQL foreign-key `references` edges are no longer dropped when a routine in the same schema is unparseable (#1854, thanks @sekmur). `pg_introspect` builds one synthetic DDL document and parsed it with the function stubs emitted before the FK `ALTER TABLE`s, so a C-language (or otherwise unparseable) routine's stub parsed as a tree-sitter ERROR node that swallowed the trailing FK statements into the error region, losing every FK edge after it. The FK DDL is now emitted before the function stubs, so table-to-table `references` edges are produced first and can't be eaten by a later unparseable routine.
 

graphify/cli.py

@@ -2728,6 +2728,22 @@ def _progress(idx: int, total: int, _result: dict) -> None:
         # absolute file lists.
         _manifest_files = _stamped_manifest_files(files_by_type, sem_result, target)
 
+        # Files dispatched this run but dropped by _stamped_manifest_files
+        # above (failed chunk, LLM omission, or any future exclusion) still
+        # carry a stale semantic_hash from a prior successful run in the
+        # on-disk manifest; save_manifest's seed loop would otherwise copy it
+        # verbatim and mask the omission (#1948). Derived from semantic_files
+        # — what was actually SENT to the backend this run (narrowed by the
+        # incremental gate and --code-only, widened by deep mode) — NOT from
+        # files_by_type: the full live corpus includes untouched files that
+        # were never dispatched, and clearing those would blank the whole
+        # manifest on every partial incremental run, forcing a full-corpus
+        # re-extraction on the next one.
+        _stamped_semantic = {
+            f for _flist in _manifest_files.values() for f in _flist
+        }
+        _cleared_semantic = {str(p) for p in semantic_files} - _stamped_semantic
+
         # Full-scan manifest saves prune rows for in-root files that left the
         # scan corpus but still exist on disk (#1908). The corpus must be the
         # RAW detect output (files_by_type), NOT the #933-stamp-filtered
@@ -2776,7 +2792,7 @@ def _progress(idx: int, total: int, _result: dict) -> None:
                     "(--no-cluster); outputs left untouched."
                 )
                 try:
-                    _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus)
+                    _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
                 except Exception as exc:
                     print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
                 stages.total()
@@ -2813,7 +2829,7 @@ def _progress(idx: int, total: int, _result: dict) -> None:
                     f"est. cost: ${cost:.4f}"
                 )
             try:
-                _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus)
+                _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
             except Exception as exc:
                 print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
             if global_merge:
@@ -2915,7 +2931,7 @@ def _progress(idx: int, total: int, _result: dict) -> None:
         }
         analysis_path.write_text(json.dumps(analysis, indent=2), encoding="utf-8")
         try:
-            _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus)
+            _save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
         except Exception as exc:
             print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
 

graphify/detect.py

@@ -1478,6 +1478,7 @@ def save_manifest(
     kind: str = "both",
     root: Path | None = None,
     scan_corpus: set[str] | list[str] | None = None,
+    clear_semantic: set[str] | list[str] | None = None,
 ) -> None:
     """Save current file mtimes + content hashes for change detection.
 
@@ -1504,10 +1505,19 @@ def save_manifest(
     --code-only doc rows). Out-of-root entries are never pruned. Callers
     saving a SUBSET of files (changed_paths hooks, skill runbooks, #917)
     must leave this None so their untouched rows are preserved.
+
+    ``clear_semantic`` (#1948): files that were dispatched this run but
+    produced no stamped output (e.g. the LLM omitted their chunk on a
+    --force re-run) are absent from ``files``, so the seed loop below would
+    otherwise copy their prior semantic_hash verbatim — masking the omission
+    and making detect_incremental(kind="semantic") report them unchanged.
+    Pass the set of such files (any path form ``scan_corpus`` accepts) to
+    force their seeded semantic_hash to "" instead of inheriting it.
     """
     existing = load_manifest(manifest_path, root=root)
 
     scan_set: set[str] | None = set(scan_corpus) if scan_corpus is not None else None
+    clear_set: set[str] | None = set(clear_semantic) if clear_semantic is not None else None
     try:
         root_res: Path | None = Path(root).resolve() if root is not None else None
     except (OSError, RuntimeError):
@@ -1521,6 +1531,14 @@ def _in_scan(path_str: str) -> bool:
         except (OSError, RuntimeError):
             return False
 
+    def _in_clear(path_str: str) -> bool:
+        if path_str in clear_set:
+            return True
+        try:
+            return str(Path(path_str).resolve()) in clear_set
+        except (OSError, RuntimeError):
+            return False
+
     def _in_root(path_str: str) -> bool:
         # Without a root we cannot tell in-root from out-of-root; fail open
         # (keep the row) so out-of-root corpora are never pruned by accident.
@@ -1566,6 +1584,10 @@ def _normalise_entry(entry):
             continue
         if scan_set is not None and not _in_scan(f) and _in_root(f):
             continue  # excluded-but-alive: drop the stale row (#1908)
+        if clear_set is not None and _in_clear(f):
+            # Dispatched-but-omitted this run: don't inherit the stale
+            # semantic_hash, or detect_incremental would call it unchanged (#1948).
+            normalised = {**normalised, "semantic_hash": ""}
         manifest[f] = normalised
 
     all_files = [f for file_list in files.values() for f in file_list]

tests/test_detect.py

@@ -1135,6 +1135,42 @@ def test_save_manifest_skips_semantic_hash_for_files_without_cache(tmp_path):
     assert str(doc2) not in manifest, "failed-chunk file must be absent from manifest"
 
 
+def test_save_manifest_clear_semantic_erases_stale_hash_for_omitted_file(tmp_path):
+    """#1948: a file stamped in an earlier run, then omitted from ``files`` on
+    a later run (LLM dropped its chunk / #1890 retry), must not keep surviving
+    with its stale semantic_hash from the prior run — the seed loop copies
+    the on-disk row verbatim otherwise, and detect_incremental(kind='semantic')
+    reports it unchanged, silently defeating the #1890 retry promise."""
+    import json
+
+    doc = tmp_path / "docs" / "doc.md"
+    doc.parent.mkdir()
+    doc.write_text("# Doc\n\ncontent")
+    manifest_path = str(tmp_path / "graphify-out" / "manifest.json")
+
+    # Run 1: doc.md is dispatched and stamped.
+    corpus = {str(doc)}
+    save_manifest({"document": [str(doc)]}, manifest_path, root=tmp_path, scan_corpus=corpus)
+    manifest = json.loads(Path(manifest_path).read_text(encoding="utf-8"))
+    assert manifest["docs/doc.md"]["semantic_hash"] != ""
+
+    # Run 2 (--force re-run): the model omits doc.md this time, so cli.py's
+    # _stamped_manifest_files() drops it from the files dict passed here —
+    # but it was still dispatched, so the caller passes it via clear_semantic.
+    save_manifest(
+        {"document": []}, manifest_path, root=tmp_path,
+        scan_corpus=corpus, clear_semantic={str(doc)},
+    )
+    manifest = json.loads(Path(manifest_path).read_text(encoding="utf-8"))
+    assert manifest["docs/doc.md"]["semantic_hash"] == "", (
+        "omitted file must have its stale semantic_hash cleared, not inherited"
+    )
+
+    inc = detect_incremental(tmp_path, manifest_path, kind="semantic")
+    assert [Path(f).name for f in inc["new_files"]["document"]] == ["doc.md"], (
+        "cleared file must be re-queued for semantic extraction"
+    )
+
 
 def test_save_manifest_without_filter_unchanged_for_code(tmp_path):
     """Code files must be stamped in the manifest regardless of semantic cache."""

tests/test_extract_cli.py

@@ -137,6 +137,81 @@ def _capture_semantic_cache(*args, **kwargs):
     } == {str(corpus / "README.md")}
 
 
+def test_incremental_partial_run_preserves_untouched_semantic_hash(
+    monkeypatch, tmp_path
+):
+    """#1948 caller-side guard: an incremental run that only re-dispatches the
+    CHANGED subset must not blank semantic_hash for live-but-untouched files.
+
+    clear_semantic must be derived from what was actually SENT to the backend
+    this run (semantic_files), not from the full live corpus (files_by_type):
+    with the latter, every unchanged doc lands in the clear set on every
+    incremental run, so the very next run re-extracts the whole corpus,
+    forever."""
+    import json
+
+    corpus = _make_corpus(tmp_path)  # main.go + README.md
+    (corpus / "OTHER.md").write_text("# Other\nAn independent second doc.\n")
+    out_dir = tmp_path / "out"
+    monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test-fake-key")
+
+    dispatched: list[list[str]] = []
+
+    def _stamp_everything_sent(paths, **kwargs):
+        sent = sorted(os.path.relpath(str(p), str(corpus)) for p in paths)
+        dispatched.append(sent)
+        on_chunk = kwargs.get("on_chunk_done")
+        if on_chunk:
+            on_chunk(0, 1, {"nodes": [], "edges": [], "hyperedges": []})
+        return {
+            "nodes": [{"id": f"n-{rel}", "source_file": rel,
+                       "file_type": "document"} for rel in sent],
+            "edges": [],
+            "hyperedges": [],
+            "input_tokens": 10,
+            "output_tokens": 5,
+        }
+
+    monkeypatch.setattr(
+        "graphify.llm.extract_corpus_parallel", _stamp_everything_sent
+    )
+    monkeypatch.setattr(mainmod, "_check_skill_version", lambda _: None)
+
+    def _run_extract():
+        monkeypatch.setattr(
+            mainmod.sys, "argv",
+            ["graphify", "extract", str(corpus), "--backend", "claude",
+             "--no-cluster", "--out", str(out_dir)],
+        )
+        try:
+            mainmod.main()
+        except SystemExit as exc:
+            assert exc.code in (None, 0), f"unexpected exit code {exc.code}"
+
+    # Run 1: full scan — both docs dispatched and stamped.
+    _run_extract()
+    manifest_path = out_dir / "graphify-out" / "manifest.json"
+    m1 = json.loads(manifest_path.read_text())
+    assert m1["README.md"].get("semantic_hash")
+    assert m1["OTHER.md"].get("semantic_hash")
+
+    # Run 2: only README.md changes → the incremental gate dispatches it alone.
+    (corpus / "README.md").write_text("# Notes\nChanged content, new hash.\n")
+    _run_extract()
+    assert dispatched[-1] == ["README.md"], (
+        f"run 2 should dispatch only the changed doc, got {dispatched[-1]}"
+    )
+    m2 = json.loads(manifest_path.read_text())
+    assert m2["README.md"].get("semantic_hash")
+    # The heart of the guard: an untouched, never-dispatched live doc keeps
+    # its stamp across a partial incremental run.
+    assert m2["OTHER.md"].get("semantic_hash"), (
+        "untouched doc's semantic_hash was blanked by a partial incremental "
+        "run — clear_semantic was derived from the full live corpus instead "
+        "of the dispatched subset (#1948)"
+    )
+
+
 def test_manifest_stamps_freshly_extracted_semantic_docs(monkeypatch, tmp_path):
     """#1897: fresh extraction returns nodes with ROOT-RELATIVE source_file,
     while the #933 manifest filter compared them against detect()'s ABSOLUTE