refactor(filesystem): simplify PIFS core surface#47
Conversation
Align the CLI, public Python API, SQLite schemas, and summary projection with the locally approved PR VectifyAI#302 simplification spec. Add fail-first behavior coverage and real migration compatibility fixtures.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Pageindex/pageindex/filesystem/core.py
Lines 582 to 586 in 8a9f590
Preserve browse hits for duplicate display names
When browsing a physical folder, this branch builds a plain /folder/display_name locator and then drops the candidate if that locator is ambiguous. If two files in the same folder have different titles but the same file_folders.metadata_json.display_name, tree already exposes /folder/name~1 and /folder/name~2, but browse /folder "..." catches the RuntimeError from _stable_file_locator and returns no documents for those hits. Use the scoped locator path here, as metadata scopes do, so summary results remain selectable.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fail closed on pseudo-v2 catalog and projection databases, validate projection topology before any workspace mutation, and keep migration subprocess coverage outside the PageIndex test suite.
Require an exact catalog whenever projection databases exist, validate canonical vec0 and persisted embedding identities, and align the external migration's staged and idempotent checks with the runtime schema contract.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 121bef95e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fail closed before mutating an existing catalog and reject projection rows that do not reference active catalog files. Validate the canonical root and cross-database references at workspace open. Refs #47
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 120b1ea5f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Treat existing symlink entries as projection paths even when their targets are missing. Share the same presence rule between core preflight and topology validation so damaged projections fail closed without workspace mutation. Refs #47
Derive setmeta output from the updated document instead of echoing its input target. Reuse the tree scope collision mapping for physical browse and stable locators so duplicate display names remain ranked and resolvable. Refs #47
|
Also addressed the earlier review-summary feedback about duplicate physical |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81ee9baebf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Open or create the Summary Projection for Python registrations, require ready status before returning, and roll back fresh batches when projection completion fails. Clean partial fresh projection databases when schema creation aborts.
Track only folders, metadata fields, and embedding cache keys introduced by a failed registration batch, then remove them without disturbing preexisting workspace state. Generalize Summary Projection lifecycle readiness messages for add and registration.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eea969c1bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Capture PageIndex document IDs before registration preparation and remove only IDs introduced by a failed batch. Prepare records inside the rollback boundary so PageIndex after-write failures also restore the logical workspace baseline.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 867fd8f778
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Validate copied caller metadata before PageIndex and artifact preparation. Capture exact registration-owned artifact baselines so prepare-stage failures restore existing bytes and remove only newly created paths.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89c62087f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Validate every batch metadata payload before PageIndex work begins. Reuse one canonical raw-artifact policy for both success-path writes and rollback baseline capture.
Validate active catalog and Summary Projection references in both directions. Existing catalogs with active files now fail closed when the projection pair is absent, while fresh and deleted-only catalog listing remains lazy.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a784f7bd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Restore the canonical metadata-axis paths, load embedding credentials from the configured precedence, and validate projection rows on open. Make add and registration failures restore catalog, projection, cache, PageIndex, and owned artifact state exactly, including updates to existing records. Refs #47
Percent-encode dot-only metadata values so tree locators survive path normalization and remain actionable through tree, browse, and stat. Refs #47
1 participant
Summary
Implements the locally approved PR VectifyAI#302 PIFS simplification contract. PIFS now has one public Python entry point, five agent commands, one summary projection, exact schema-v2 validation, and fail-closed workspace opening and migration behavior.
Goal
Keep the essential PIFS document workflow while removing pre-release compatibility surface and making the catalog, Summary Projection, and embedding cache safe to reopen and migrate as one workspace.
What Changed
pageindex.PageIndexFileSystemas the supported Python entry point.tree,browse,stat,cat, andgrepin the shell executor.@field/valuesegments, including three-layer tree navigation, trailing-slash normalization, fixed 50-value pages, encoded slash/dot-only values, and actionable multi-axis locators; the pre-release@field=valueform is rejected.setmetaderive its returned locator from the updated document rather than echoing the input target. File-ref targets and stale metadata-scope targets therefore return a fresh actionable physical path.display_name.path.exists() or path.is_symlink(); broken summary/cache symlinks therefore fail closed instead of being treated as an absent projection.folder_root / /in every existing catalog.deleted_at IS NULL) and projected document refs; missing and orphan refs both fail closed.filesystem.sqliteas fresh. Existing zero-byte, non-SQLite, directory, legacy, pseudo-v2, or inconsistent catalog paths fail closed without mutation.register_file()/register_files()synchronously create or open the Summary Projection and persistready. Failed registration restores all new-batch state and exact pre-call state for updated existing records across catalog rows/memberships/metadata, summary/vector rows, cache keys, owned text/raw artifacts, and PageIndex doc/meta artifacts; preparation itself is inside the same rollback boundary.SummaryProjectionlifecycle.PIFS_EMBEDDING_API_KEY> user PIFS configembedding_api_key>OPENAI_API_KEY;pifs set workspacepreserves the config key while command output and SQLite remain secret-free.pifs-data/scripts.already_migrated.Command Surface
pifs add <source> <folder>: atomically import PDF or Markdown.pifs tree <scope> [-L depth] [--page N]: inspect workspace structure.pifs browse <scope> "<query>" [--page N] [--where JSON] [-R]: rank files by summary inside a scope.pifs stat <file>: inspect one document.pifs cat <file> --structure: inspect PageIndex structure.pifs cat <file> --page N[-M]: read a bounded page range.pifs grep <query> <file>: search evidence in one resolved document.pifs ask/pifs chat: run the agent against the configured workspace.pifs setmeta [--clear] <file> [JSON]: replace custom metadata and return the updated actionable locator.pifs set ...: persist workspace and embedding connection settings; an optional config API key is preserved but never emitted or written into workspace databases.Key Files
pageindex/filesystem/cli.py: add/setmeta/config/agent orchestration and command-facing output.pageindex/filesystem/core.py: workspace lifecycle, registration artifact cleanup, post-update locators, scoped collision mapping, and summary-backed browse.pageindex/filesystem/_projection_topology.py: shared path-presence semantics and lightweight summary/cache pair detection.pageindex/filesystem/_workspace_consistency.py: read-only canonical-root and active projection-reference validation.pageindex/filesystem/store.py: exact catalog v2 validation and absent-only fresh initialization.pageindex/filesystem/semantic_projection.py: unified projection lifecycle, cache validation, topology ownership, and precise cache-key rollback.pageindex/filesystem/semantic_index.py: exact summary/config/vec0 validation and retrieval.pageindex/filesystem/_embedding_identity.py: canonical persisted identity rules.pageindex/filesystem/_sqlite_schema.py: shared exact SQLite signatures.pageindex/filesystem/commands.py: five-command contract and boundary identity mapping.tests/test_pifs_simplification_acceptance.py: real temporary-workspace acceptance, actionable-locator, collision, logical-baseline rollback, and no-mutation coverage.examples/pifs_demo.py: minimal supported walkthrough.Verification
setmetaproduced1 failedbecause the successful command returnedpath=null.same.md~1/~2files while physical browse returned zero documents (1 failed).8 passed.2 failed, 1 passedto4 passed, including the truly absent catalog-only control.pending_index, and a second-record embedding failure returned success —3 failed.summary.sqlitewithout its cache —1 failed.6 passed.yearfield, and one cache row; the preexisting-state control retained new batch-only catalog state —2 failed.2 passed.3 failedacross the three fail-first slices.8 passed.2 failed, 75 deselected.1 failed.stat,cat,grep, and PageIndex structure remain available; combined rollback controls —11 passed, 67 deselected.100/100 PASS, including forced partial-write restoration.1 failed, 1 passed.3 passed, 77 deselected; combined registration/add rollback controls —13 passed, 67 deselected.100/100 PASS, including a canonical-equivalent alias-path restoration probe.2 failed, 4 passed, 80 deselected.7 passed, 80 deselected.2 failed, 23 deselected.3 passed, 22 deselected.100/100 PASS; focused PageIndex10 passed, migration full25 passed.pifs set workspace, and precedence was wrong —3 failed, 3 passed; green precedence/config-preservation/secret-absence gate —7 passed./documents/@year=2024instead of/documents/@year/2024; three-layer CLI/trailing-slash/50-value pagination and actionable locator gate —8 passed.4 passed, with migration parity —4 passed.2 passed.category=financewas silently dropped and an invalid legacy field migrated successfully —2 failed; green preservation/fail-closed/already-migrated controls are included in the 32-test migration suite./documents/@tag/.and/documents/@tag/..; green%2E/%2E%2Etree/browse/stat round trip —1 passed; independent re-review100/100 PASS.env PYTHONPATH=. .../pytest -q— 172 passed.python -m compileall -q pageindex tests.fb3f5edb9f0342d470e65924ede305e7aac330a534a1bdd01097aadb9abea577(1341 lines), tests1e02132b6f4aa5c3637a4a40fc7c47eeeb048d448c40df1b1adc7687cb2e114f(921 lines).git diff --checkand staged diff check.PageIndexFileSystem, then succeeds throughtreeand cachedbrowse.