diff --git a/.planning/README.md b/.planning/README.md index a7c9b9bc..becf84a8 100644 --- a/.planning/README.md +++ b/.planning/README.md @@ -128,6 +128,7 @@ The plan aligns to: 71. [Phase 77 - Prompt Context Memory Boundary And Namespace Foundation](https://github.com/mikehostetler/jido_code/blob/main/.planning/phase-77-prompt-context-memory-boundary-and-namespace-foundation.md): introduce `jido_memory` as a bounded prompt-context layer with a product-owned adapter, explicit namespace policy, and provider-safe rollout defaults so conversation runtime can gain recall without turning prompt memory into product truth. 72. [Phase 78 - Conversation Runtime Prompt Recall And Bounded Capture Adoption](https://github.com/mikehostetler/jido_code/blob/main/.planning/phase-78-conversation-runtime-prompt-recall-and-bounded-capture-adoption.md): wire prompt-context retrieval and explicit bounded capture into the real conversation runtime so each turn can reuse the right short-term memory without replaying raw transcript history. 73. [Phase 79 - Prompt Memory Lifecycle Hardening And Contributor Convergence](https://github.com/mikehostetler/jido_code/blob/main/.planning/phase-79-prompt-memory-lifecycle-hardening-and-contributor-convergence.md): harden provider behavior, retention and cleanup policy, verification defaults, and contributor guidance so prompt memory remains bounded, explainable, and clearly separate from provenance and durable repository memory. +74. [Phase 80 - Source Code Graph Save-Triggered Refresh Adoption](https://github.com/mikehostetler/jido_code/blob/main/.planning/phase-80-source-code-graph-save-triggered-refresh-adoption.md): add repository-scoped source-change observation and debounced refresh scheduling so the `source_code` graph updates after code saves from either a human editor or product-managed LLM write path. Chronology note: Phase 55 now owns the previously landed `55.6.*` memory ontology and governed-reference verification so the planning sequence once @@ -158,6 +159,11 @@ Phase 79 hardens lifecycle, provider behavior, and contributor guidance while preserving the provenance-first and explicit-adoption split for long-term memory. +Source-code graph live-refresh note: Phase 80 adds save-triggered refresh on +top of the existing explicit analyze, load, refresh, status, query, and recovery +lifecycle. Repo-scoped monitoring owns source-change observation while +`AgentWorkspace` and `SourceCodeGraphPod` continue to own graph mutation. + ## Shared Conventions - Numbering: - Phases: `N` diff --git a/.planning/phase-80-source-code-graph-save-triggered-refresh-adoption.md b/.planning/phase-80-source-code-graph-save-triggered-refresh-adoption.md new file mode 100644 index 00000000..e1eafc2e --- /dev/null +++ b/.planning/phase-80-source-code-graph-save-triggered-refresh-adoption.md @@ -0,0 +1,247 @@ +# Phase 80 - Source Code Graph Save-Triggered Refresh Adoption + + + + + + +Back to index: [README](https://github.com/mikehostetler/jido_code/blob/main/.planning/README.md) + +## Relevant Shared APIs / Interfaces +- `.planning/phase-20-source-code-graph-pod-foundation.md` +- `.planning/phase-23-source-code-graph-hardening-and-operational-convergence.md` +- `.planning/phase-53-source-code-graph-enablement-and-hardening.md` +- `docs/developer/03-agent-workspace-and-runtime-topology.md` +- `docs/developer/07-source-code-graph-and-semantic-services.md` +- `docs/developer/13-source-code-graph-operations.md` +- `lib/jido_code/agents/repo_monitor.ex` +- `lib/jido_code/pods/repo_pod.ex` +- `lib/jido_code/pods/source_code_graph_pod.ex` +- `lib/jido_code/agent_workspace.ex` +- `lib/jido_code/source_code_graph.ex` +- `lib/jido_code/source_code_graph/store.ex` +- `lib/jido_code/actions/load_source_code_graph.ex` +- `lib/jido_code/actions/refresh_source_code_graph.ex` +- `lib/jido_code/actions/get_source_code_graph_status.ex` +- `lib/jido_code/actions/source_code_graph_support.ex` +- `lib/jido_code/conversations/runtime.ex` +- `lib/jido_code/forge/runners/workflow.ex` +- `lib/jido_code/forge/runners/claude_code.ex` +- `mix.exs` +- `config/config.exs` +- `config/runtime.exs` +- `test/jido_code/source_code_graph_workspace_test.exs` +- `test/jido_code/agent_os/phase_twenty_three_integration_test.exs` +- `test/jido_code/source_code_graph_workflow_service_test.exs` + +## Relevant Assumptions / Defaults +- The source-code graph remains repository-local runtime state and not product + truth; save-triggered refresh should keep it useful without making it a hidden + dependency for ordinary source reads or edits. +- `RepoPod` and `RepoMonitor` are the right repository-scoped observation seam + for human editor saves and runtime-managed code writes, while + `SourceCodeGraphPod` and `AgentWorkspace` remain the only boundaries that + analyze, load, refresh, query, or recover the graph. +- File-save detection must catch both human editor changes in a local workspace + and product-controlled LLM writes. Remote or virtual write paths should emit + the same normalized workspace-source-change signal after successful writes. +- Refresh work is potentially expensive, so saves should enqueue a debounced, + coalesced refresh rather than running analysis synchronously in the save path. +- Query behavior remains bounded: stale queries still require `allow_stale?: true` + unless a refresh has already completed. +- The source graph's refresh input set must be shared with revision detection so + watcher filters, stale detection, and analysis do not drift. + +[x] 80 Phase 80 - Source Code Graph Save-Triggered Refresh Adoption + Add repository-scoped source-change observation and debounced refresh scheduling + so the `source_code` graph updates after code saves from either a human editor + or product-managed LLM write path, while preserving explicit graph lifecycle, + stale-state visibility, and fallback behavior. + + [x] 80.1 Section - Repository-Scoped Source Change Observation Boundary + Establish one product-owned way to detect and normalize source file changes + without teaching LiveViews, tool handlers, or graph query code how filesystem + watching works. + + [x] 80.1.1 Task - Expose the canonical source graph file scope + Make the graph input set reusable so watcher filtering and revision + detection agree about what counts as source-code graph input. + + [x] 80.1.1.1 Subtask - Promote the current private source globs and + exclusion rules in `JidoCode.SourceCodeGraph` into a public helper such + as `source_file?/2`, `source_files/1`, or `source_patterns/1`. + [x] 80.1.1.2 Subtask - Keep the canonical input set aligned with current + revision detection: `mix.exs`, `lib/**/*.ex`, `lib/**/*.exs`, + `test/**/*.ex`, `test/**/*.exs`, and `config/**/*.exs`, excluding + `deps`, `_build`, `node_modules`, and repository-local graph storage. + [x] 80.1.1.3 Subtask - Add coverage proving the watcher filter and + revision identity use the same source-scope helper instead of duplicating + path rules. + + [x] 80.1.2 Task - Add repo-monitor source change events + Turn workspace source saves into repository-scoped observations that can + be consumed by graph refresh scheduling and operator status surfaces. + + [x] 80.1.2.1 Subtask - Extend `RepoMonitor` or a RepoPod-owned helper with + a normalized event shape such as `:workspace_source_changed`, carrying + managed repo id, workspace path, changed paths, detected revision, event + source, and observed timestamp. + [x] 80.1.2.2 Subtask - Add a local filesystem watcher for repo-scoped + local workspaces, using a direct runtime dependency if needed rather than + relying on transitive development-only watcher packages. + [x] 80.1.2.3 Subtask - Ignore irrelevant and self-generated paths such as + `.jido_code/source_code_graph`, `.jido_code/memory_graph`, `_build`, + `deps`, and `node_modules` so refresh does not recursively trigger from + graph writes. + [x] 80.1.2.4 Subtask - Ensure watcher lifecycle follows managed repository + workspace binding changes, repo runtime startup, and repo runtime + shutdown without leaving orphaned OS watcher processes. + + [x] 80.2 Section - Save-Origin Coverage For Human And LLM Writes + Make both external editor saves and product-managed code writes converge on + the same source-change event instead of creating actor-specific refresh + behavior. + + [x] 80.2.1 Task - Cover human editor saves through the workspace watcher + Treat ordinary editor saves as external repository changes that mark the + graph stale and enqueue refresh without requiring a product UI action. + + [x] 80.2.1.1 Subtask - Detect create, modify, rename, and delete events + for source-scope files under the managed repository workspace path. + [x] 80.2.1.2 Subtask - Debounce editor write bursts and atomic-save rename + patterns into one normalized source-change observation per quiet window. + [x] 80.2.1.3 Subtask - Preserve current stale status behavior while a + refresh is queued or running so product surfaces remain honest about + graph freshness. + + [x] 80.2.2 Task - Add explicit product write notifications for LLM save paths + Ensure runtime-managed writes emit the same source-change signal even when + the storage layer is remote, virtualized, or otherwise not visible to a + local OS watcher. + + [x] 80.2.2.1 Subtask - Add a product-owned helper such as + `AgentWorkspace.notify_workspace_source_changed/3` or a RepoMonitor + action that write-capable runtime boundaries can call after successful + code writes. + [x] 80.2.2.2 Subtask - Wire product-controlled write paths, including + Forge/Sprite write helpers and LLM runner save boundaries, to emit the + normalized change event only after the write succeeds. + [x] 80.2.2.3 Subtask - Include event source metadata such as + `:human_watcher`, `:llm_write`, `:tool_write`, or `:runtime_write` + without changing graph refresh semantics by actor type. + [x] 80.2.2.4 Subtask - Keep external LLM or CLI edits covered by the + filesystem watcher whenever they touch a local workspace on disk. + + [x] 80.3 Section - Debounced Source Graph Refresh Scheduling + Convert source-change observations into bounded graph refresh work that is + reliable under save bursts and does not block editing, tool execution, or + conversation progress. + + [x] 80.3.1 Task - Introduce a per-repository refresh coordinator + Add one repo-scoped scheduler that owns debounce, coalescing, in-flight + protection, and retry visibility for source graph refresh requests. + + [x] 80.3.1.1 Subtask - Add a RepoPod-owned or AgentWorkspace-owned + coordinator that queues refresh requests per managed repo and workspace + path with configurable debounce and maximum coalescing windows. + [x] 80.3.1.2 Subtask - Prevent overlapping source graph refreshes for the + same managed repo; if another save arrives while refresh is in flight, + mark a follow-up refresh as pending and run it after the current refresh + completes. + [x] 80.3.1.3 Subtask - Keep refresh work asynchronous from the save path, + returning event acceptance promptly while recording queued, running, + succeeded, skipped, and failed scheduler states. + [x] 80.3.1.4 Subtask - Add bounded retries or handoff to + `recover_source_code_graph/3` for refresh failures without creating an + infinite analysis loop. + + [x] 80.3.2 Task - Reuse explicit graph lifecycle entrypoints + Preserve the current source graph lifecycle by routing scheduler work + through `AgentWorkspace` rather than writing to TripleStore directly. + + [x] 80.3.2.1 Subtask - For ready stale graphs, call + `AgentWorkspace.refresh_source_code_graph/3` so the existing staged + replacement behavior remains canonical. + [x] 80.3.2.2 Subtask - For missing graphs, make policy explicit: either + leave the graph not-ready until semantic use requests `load_if_missing`, + or enqueue `load_source_code_graph/3` behind an opt-in configuration. + [x] 80.3.2.3 Subtask - For disabled source graph configuration, drop or + record the event as skipped without starting watcher or refresh work. + [x] 80.3.2.4 Subtask - Persist refresh diagnostics on the existing source + graph pod metadata so product surfaces and recovery actions can explain + queued, stale, failed, and refreshed states. + + [x] 80.4 Section - Operator Visibility And Runtime Degradation + Surface save-triggered refresh as a bounded runtime capability without + making the graph feel magically current when refresh is queued, disabled, + or degraded. + + [x] 80.4.1 Task - Extend source graph status projections with refresh activity + Make background refresh state visible to product helpers and operator + surfaces through existing bounded projection layers. + + [x] 80.4.1.1 Subtask - Extend source graph status or health projections + with fields such as `auto_refresh`, `last_source_change_at`, + `last_refresh_started_at`, `last_refresh_completed_at`, + `refresh_queued?`, and `refresh_in_flight?`. + [x] 80.4.1.2 Subtask - Keep existing ready, stale, degraded, and recovery + labels authoritative while adding refresh activity as supporting context. + [x] 80.4.1.3 Subtask - Update managed-repo semantic inspection and + dashboard monitoring hints to explain queued or failed background + refresh without exposing watcher internals. + + [x] 80.4.2 Task - Add configuration and contributor guidance + Make save-triggered refresh safe to enable by environment, workspace type, + and repository size. + + [x] 80.4.2.1 Subtask - Add configuration for auto-refresh enablement, + debounce interval, maximum coalescing delay, watcher path limits, and + missing-graph load policy. + [x] 80.4.2.2 Subtask - Default auto-refresh conservatively for production + while keeping local development behavior useful when the source graph is + enabled. + [x] 80.4.2.3 Subtask - Update source graph developer and operations docs + to explain human editor saves, LLM write notifications, debounce, + skipped refreshes, and manual recovery. + [x] 80.4.2.4 Subtask - Align contributor guidance so future write-capable + tools emit the normalized source-change notification rather than calling + graph refresh directly. + + [x] 80.5 Section - Phase 80 Integration Tests + Prove save-triggered refresh updates the source graph for human and LLM + saves while preserving stale-query semantics, explicit lifecycle boundaries, + and operational safety. + + [x] 80.5.1 Task - Add source-change observation and scheduler coverage + Verify file saves become bounded repo-scoped refresh requests without + duplicated path rules or overlapping graph writes. + + [x] 80.5.1.1 Subtask - Add unit coverage for source file scope filtering, + ignored directories, deleted files, atomic-save renames, and graph-store + self-write suppression. + [x] 80.5.1.2 Subtask - Add GenServer or action-level coverage proving + repeated save events coalesce into one refresh during the debounce + window. + [x] 80.5.1.3 Subtask - Add coverage proving in-flight refresh coalescing + schedules one follow-up refresh when more saves arrive during analysis + or load. + [x] 80.5.1.4 Subtask - Add coverage proving disabled configuration skips + watchers and refresh scheduling without breaking manual load, refresh, + status, or query entrypoints. + + [x] 80.5.2 Task - Add end-to-end human and LLM save refresh scenarios + Exercise the full repository-scoped path from save event to refreshed graph + status through product-owned boundaries. + + [x] 80.5.2.1 Subtask - Add an integration test that loads a graph, changes + a local source file through a simulated watcher event, waits for the + debounced refresh, and verifies the graph imports the new revision. + [x] 80.5.2.2 Subtask - Add an integration test that simulates a + product-controlled LLM write notification and verifies it follows the + same refresh path as a watcher event. + [x] 80.5.2.3 Subtask - Add coverage proving stale queries still fail + without `allow_stale?: true` while refresh is queued or failed, and + return current results after refresh completes. + [x] 80.5.2.4 Subtask - Run and document `mix source_graph.verify`, plus + any targeted RepoPod, conversation-runtime, or Forge write-path tests + touched by the implementation. diff --git a/config/config.exs b/config/config.exs index 090fb3d2..defc6ef5 100644 --- a/config/config.exs +++ b/config/config.exs @@ -104,7 +104,17 @@ config :jido_code, system_config_saver: &JidoCode.Setup.SystemConfigPersistence.save/1, ash_authentication: [return_error_on_invalid_magic_link_token?: true], mailer: [from_name: "Jido Code"], - runtime_mode: config_env() + runtime_mode: config_env(), + # Source-code graph save-triggered refresh defaults are conservative outside dev. + source_code_graph_file_watcher_enabled: false, + source_code_graph_file_watcher_debounce_ms: 500, + source_code_graph_file_watcher_max_pending_paths: 500, + source_code_graph_auto_refresh_enabled: false, + source_code_graph_refresh_debounce_ms: 250, + source_code_graph_refresh_max_coalesce_ms: 2_500, + source_code_graph_refresh_max_pending_paths: 500, + source_code_graph_auto_refresh_missing_graph_policy: :skip, + source_code_graph_auto_refresh_max_attempts: 1 config :jido_code, :code_server, data_dir: ".jido", diff --git a/config/dev.exs b/config/dev.exs index 601d5745..a387dc07 100644 --- a/config/dev.exs +++ b/config/dev.exs @@ -117,9 +117,21 @@ config :jido_code, ], # Source code graph configuration (enabled in dev with conservative defaults) source_code_graph_enabled: true, - source_code_graph_analysis_timeout_ms: 300_000, # 5 minutes - source_code_graph_load_timeout_ms: 120_000, # 2 minutes - source_code_graph_query_timeout_ms: 30_000, # 30 seconds + # 5 minutes + source_code_graph_analysis_timeout_ms: 300_000, + # 2 minutes + source_code_graph_load_timeout_ms: 120_000, + # 30 seconds + source_code_graph_query_timeout_ms: 30_000, + source_code_graph_file_watcher_enabled: true, + source_code_graph_file_watcher_debounce_ms: 500, + source_code_graph_file_watcher_max_pending_paths: 500, + source_code_graph_auto_refresh_enabled: true, + source_code_graph_refresh_debounce_ms: 250, + source_code_graph_refresh_max_coalesce_ms: 2_500, + source_code_graph_refresh_max_pending_paths: 500, + source_code_graph_auto_refresh_missing_graph_policy: :skip, + source_code_graph_auto_refresh_max_attempts: 1, source_code_graph_max_retries: 3, source_code_graph_retry_backoff_ms: 1000, source_code_graph_max_file_count: 10_000, @@ -127,13 +139,23 @@ config :jido_code, source_code_graph_allow_partial_results: false, # Memory graph configuration (enabled in dev with conservative defaults) memory_graph_enabled: true, - memory_graph_store_timeout_ms: 30_000, # 30 seconds - TripleStore operations - memory_graph_query_timeout_ms: 60_000, # 1 minute - SPARQL queries - memory_graph_validation_timeout_ms: 120_000, # 2 minutes - ontology validation - memory_graph_recovery_timeout_ms: 300_000, # 5 minutes - recovery operations - memory_graph_max_retries: 3, # maximum retry attempts for transient failures - memory_graph_max_write_retries: 2, # maximum retry attempts for write operations - memory_graph_retry_backoff_ms: 1000, # base backoff time in milliseconds - memory_graph_max_graph_size_mb: 10_000, # maximum graph size in MB (10 GB) - memory_graph_max_query_results: 10_000, # maximum query result count - memory_graph_max_concurrent_operations: 50 # maximum concurrent write operations + # 30 seconds - TripleStore operations + memory_graph_store_timeout_ms: 30_000, + # 1 minute - SPARQL queries + memory_graph_query_timeout_ms: 60_000, + # 2 minutes - ontology validation + memory_graph_validation_timeout_ms: 120_000, + # 5 minutes - recovery operations + memory_graph_recovery_timeout_ms: 300_000, + # maximum retry attempts for transient failures + memory_graph_max_retries: 3, + # maximum retry attempts for write operations + memory_graph_max_write_retries: 2, + # base backoff time in milliseconds + memory_graph_retry_backoff_ms: 1000, + # maximum graph size in MB (10 GB) + memory_graph_max_graph_size_mb: 10_000, + # maximum query result count + memory_graph_max_query_results: 10_000, + # maximum concurrent write operations + memory_graph_max_concurrent_operations: 50 diff --git a/config/runtime.exs b/config/runtime.exs index c7a0b1c8..7d11416a 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -82,6 +82,46 @@ if query_timeout = System.get_env("SOURCE_CODE_GRAPH_QUERY_TIMEOUT_MS") do config :jido_code, source_code_graph_query_timeout_ms: String.to_integer(query_timeout) end +if watcher_enabled = System.get_env("SOURCE_CODE_GRAPH_FILE_WATCHER_ENABLED") do + config :jido_code, source_code_graph_file_watcher_enabled: watcher_enabled == "true" +end + +if watcher_debounce = System.get_env("SOURCE_CODE_GRAPH_FILE_WATCHER_DEBOUNCE_MS") do + config :jido_code, source_code_graph_file_watcher_debounce_ms: String.to_integer(watcher_debounce) +end + +if watcher_max_paths = System.get_env("SOURCE_CODE_GRAPH_FILE_WATCHER_MAX_PENDING_PATHS") do + config :jido_code, source_code_graph_file_watcher_max_pending_paths: String.to_integer(watcher_max_paths) +end + +if auto_refresh_enabled = System.get_env("SOURCE_CODE_GRAPH_AUTO_REFRESH_ENABLED") do + config :jido_code, source_code_graph_auto_refresh_enabled: auto_refresh_enabled == "true" +end + +if refresh_debounce = System.get_env("SOURCE_CODE_GRAPH_REFRESH_DEBOUNCE_MS") do + config :jido_code, source_code_graph_refresh_debounce_ms: String.to_integer(refresh_debounce) +end + +if refresh_max_coalesce = System.get_env("SOURCE_CODE_GRAPH_REFRESH_MAX_COALESCE_MS") do + config :jido_code, source_code_graph_refresh_max_coalesce_ms: String.to_integer(refresh_max_coalesce) +end + +if refresh_max_paths = System.get_env("SOURCE_CODE_GRAPH_REFRESH_MAX_PENDING_PATHS") do + config :jido_code, source_code_graph_refresh_max_pending_paths: String.to_integer(refresh_max_paths) +end + +if missing_graph_policy = System.get_env("SOURCE_CODE_GRAPH_AUTO_REFRESH_MISSING_GRAPH_POLICY") do + case missing_graph_policy do + "skip" -> config :jido_code, source_code_graph_auto_refresh_missing_graph_policy: :skip + "load" -> config :jido_code, source_code_graph_auto_refresh_missing_graph_policy: :load + _other -> raise "SOURCE_CODE_GRAPH_AUTO_REFRESH_MISSING_GRAPH_POLICY must be skip or load" + end +end + +if auto_refresh_attempts = System.get_env("SOURCE_CODE_GRAPH_AUTO_REFRESH_MAX_ATTEMPTS") do + config :jido_code, source_code_graph_auto_refresh_max_attempts: String.to_integer(auto_refresh_attempts) +end + # Memory graph configuration from environment # These can be set to override defaults for production deployment # Default: memory_graph_enabled is false in production diff --git a/config/test.exs b/config/test.exs index 35fd212d..48995f73 100644 --- a/config/test.exs +++ b/config/test.exs @@ -50,6 +50,17 @@ config :jido_code, system_config_loader: &JidoCode.Setup.SystemConfig.default_loader/0, system_config_saver: &JidoCode.Setup.SystemConfig.default_saver/1 +config :jido_code, + source_code_graph_file_watcher_enabled: false, + source_code_graph_file_watcher_debounce_ms: 5, + source_code_graph_file_watcher_max_pending_paths: 100, + source_code_graph_auto_refresh_enabled: false, + source_code_graph_refresh_debounce_ms: 5, + source_code_graph_refresh_max_coalesce_ms: 50, + source_code_graph_refresh_max_pending_paths: 100, + source_code_graph_auto_refresh_missing_graph_policy: :skip, + source_code_graph_auto_refresh_max_attempts: 1 + config :jido_code, :llm_selection, %{default: %{provider: "deterministic", model: "deterministic"}} # Ontology configuration (optional - requires elixir_ontologies package) diff --git a/docs/developer/07-source-code-graph-and-semantic-services.md b/docs/developer/07-source-code-graph-and-semantic-services.md index 5ed06d7b..5afb7a71 100644 --- a/docs/developer/07-source-code-graph-and-semantic-services.md +++ b/docs/developer/07-source-code-graph-and-semantic-services.md @@ -67,6 +67,11 @@ The expected lifecycle is: This explicit lifecycle matters because freshness, stale state, degraded query behavior, and recovery are part of the product contract. +Save-triggered refresh keeps that lifecycle explicit. Repository-scoped source +change events can enqueue debounced background refresh work, but they still route +through `AgentWorkspace` graph actions. A queued or running refresh is supporting +status context, not proof that the graph is current. + ## Product Adoption Pattern The source-code graph is intended to enrich canonical product surfaces rather @@ -101,6 +106,7 @@ Product-facing semantic surfaces should expose: - ready vs not ready - stale vs current - latest failure +- queued or running background refresh activity - explicit recovery affordances The system should not pretend the graph is current when it is degraded. @@ -126,6 +132,8 @@ Prefer ordinary file reads and code inspection when you need: - it is product-owned through bounded services - it should enrich managed-repo and governed surfaces, not replace them - its findings matter only after they rejoin governed records +- write-capable tools should emit the normalized source-change notification + after successful saves instead of calling refresh directly ## Read Next diff --git a/docs/developer/13-source-code-graph-operations.md b/docs/developer/13-source-code-graph-operations.md index 0fb8c0ce..7357607e 100644 --- a/docs/developer/13-source-code-graph-operations.md +++ b/docs/developer/13-source-code-graph-operations.md @@ -43,6 +43,30 @@ export SOURCE_CODE_GRAPH_ENABLED=true | `source_code_graph_retry_backoff_ms` | 1000 | Base backoff for retry (exponential) | | `source_code_graph_allow_partial_results` | false | Whether to allow partial results on failure | +### Save-Triggered Refresh + +The graph can observe source saves and enqueue a debounced background refresh. +Human editor saves are detected by the repository-scoped file watcher when it is +enabled. Product-managed LLM or tool writes should call +`AgentWorkspace.notify_workspace_source_changed/4` after the write succeeds so +local, remote, and virtual write paths converge on the same normalized event. + +| Config | Default | Description | +|--------|---------|-------------| +| `source_code_graph_file_watcher_enabled` | false (`true` in dev) | Enables local filesystem observation for managed repository workspaces | +| `source_code_graph_file_watcher_debounce_ms` | 500 | Quiet window for editor write bursts and atomic-save rename patterns | +| `source_code_graph_file_watcher_max_pending_paths` | 500 | Maximum changed paths carried in one watcher observation | +| `source_code_graph_auto_refresh_enabled` | false (`true` in dev) | Enqueues background refresh work from normalized source-change events | +| `source_code_graph_refresh_debounce_ms` | 250 | Quiet window before the refresh scheduler starts graph work | +| `source_code_graph_refresh_max_coalesce_ms` | 2500 | Upper bound before queued save bursts are flushed into refresh work | +| `source_code_graph_refresh_max_pending_paths` | 500 | Maximum changed paths carried in one scheduler request | +| `source_code_graph_auto_refresh_missing_graph_policy` | `:skip` | `:skip` leaves missing graphs not-ready; `:load` loads missing graphs from save events | +| `source_code_graph_auto_refresh_max_attempts` | 1 | Bounded attempts for one scheduled refresh request | + +Background refresh always routes through `AgentWorkspace.refresh_source_code_graph/3` +or, when explicitly configured, `AgentWorkspace.load_source_code_graph/3`. It does +not write directly to TripleStore. + ## Operational Requirements ### Dependencies @@ -89,6 +113,12 @@ Analysis requires approximately: # - graph_size_mb: Integer | nil ``` +Source graph status projections also include `graph.refresh` with +`auto_refresh_enabled?`, `last_source_change_at`, `last_refresh_started_at`, +`last_refresh_completed_at`, `refresh_queued?`, `refresh_in_flight?`, and +`last_failure`. These fields explain background activity while the existing +ready, stale, degraded, failed, and disabled graph states remain authoritative. + ### Health States | State | Meaning | Action | @@ -142,6 +172,23 @@ Analysis requires approximately: 2. Reduce result set size with `limit` 3. Refresh graph if stale +### Background Refresh Skipped Or Failed + +**Symptom:** status shows `graph.refresh.last_failure` or a skipped refresh +result after a save. + +**Causes:** +- Auto-refresh is disabled in this environment +- The graph has not been loaded and missing-graph policy is `:skip` +- Analysis or TripleStore load failed +- More saves arrived while a previous refresh was already running + +**Solutions:** +1. Confirm `source_code_graph_auto_refresh_enabled` and watcher settings +2. Load or recover the graph manually when status is not ready or failed +3. Keep write-capable tools emitting `AgentWorkspace.notify_workspace_source_changed/4` after successful saves +4. Increase debounce or coalescing settings for repositories with noisy generated files + ### Graceful Degradation The runtime automatically falls back to non-semantic mode on: diff --git a/lib/jido_code/actions/get_source_code_graph_status.ex b/lib/jido_code/actions/get_source_code_graph_status.ex index 08a90b48..9f2972dc 100644 --- a/lib/jido_code/actions/get_source_code_graph_status.ex +++ b/lib/jido_code/actions/get_source_code_graph_status.ex @@ -40,6 +40,7 @@ defmodule JidoCode.Actions.GetSourceCodeGraphStatus do imported_revision: Map.get(latest_import_status, :imported_revision), latest_import_status: latest_import_status, latest_analysis_status: latest_analysis_status, + source_graph_refresh: graph_context.source_graph_refresh, latest_failure: graph_context.latest_failure, dataset: graph_context.dataset_metadata }} diff --git a/lib/jido_code/actions/source_code_graph_support.ex b/lib/jido_code/actions/source_code_graph_support.ex index 63c229d0..e606c028 100644 --- a/lib/jido_code/actions/source_code_graph_support.ex +++ b/lib/jido_code/actions/source_code_graph_support.ex @@ -39,6 +39,13 @@ defmodule JidoCode.Actions.SourceCodeGraphSupport do :latest_failure, latest_failure(context, graph_context.latest_failure) ) + end) + |> then(fn graph_context -> + Map.put( + graph_context, + :source_graph_refresh, + source_graph_refresh(context, graph_context.source_graph_refresh, managed_repo_id) + ) end)} end end @@ -64,6 +71,16 @@ defmodule JidoCode.Actions.SourceCodeGraphSupport do default_failure end + @spec source_graph_refresh(map(), map() | nil, String.t() | nil) :: map() + def source_graph_refresh(context, default_refresh, managed_repo_id \\ nil) do + refresh = + context[:source_graph_refresh] || + get_in(context, [:graph, :source_graph_refresh]) || + default_refresh + + SourceCodeGraph.merge_refresh_status(refresh, managed_repo_id) + end + @spec ready?(map()) :: boolean() def ready?(status) when is_map(status), do: Map.get(status, :ready?, false) diff --git a/lib/jido_code/agent_workspace.ex b/lib/jido_code/agent_workspace.ex index fa360c54..72664fa2 100644 --- a/lib/jido_code/agent_workspace.ex +++ b/lib/jido_code/agent_workspace.ex @@ -52,7 +52,7 @@ defmodule JidoCode.AgentWorkspace do """ alias JidoCode.AgentOS.Manager - alias JidoCode.Agents.{Coder, Explainer, Planner, Reviewer} + alias JidoCode.Agents.{Coder, Explainer, Planner, RepoMonitor, Reviewer} alias JidoCode.Control.Actor alias JidoCode.Conversations alias JidoCode.Conversations.Driver, as: ConversationDriver @@ -707,7 +707,8 @@ defmodule JidoCode.AgentWorkspace do @spec ensure_source_code_graph_pod(managed_repo_id(), String.t(), keyword()) :: {:ok, source_code_graph_summary()} | {:error, term()} def ensure_source_code_graph_pod(managed_repo_id, workspace_path, opts \\ []) do - with :ok <- ensure_source_code_graph_enabled(opts) do + with :ok <- ensure_source_code_graph_enabled(opts), + {:ok, _repo_pod} <- ensure_repo_pod_entry(managed_repo_id) do case Manager.pod_status(managed_repo_id, SourceCodeGraph.pod_id()) do nil -> with {:ok, pod_metadata} <- SourceCodeGraph.pod_metadata(managed_repo_id, workspace_path, opts), @@ -717,12 +718,15 @@ defmodule JidoCode.AgentWorkspace do SourceCodeGraph.pod_id(), SourceCodeGraphPod, pod_metadata - ) do + ), + :ok <- maybe_ensure_source_watcher(managed_repo_id, workspace_path, opts) do {:ok, source_code_graph_summary(managed_repo_id, pod_entry)} end pod_entry -> - {:ok, source_code_graph_summary(managed_repo_id, pod_entry)} + with :ok <- maybe_ensure_source_watcher(managed_repo_id, workspace_path, opts) do + {:ok, source_code_graph_summary(managed_repo_id, pod_entry)} + end end end end @@ -838,6 +842,39 @@ defmodule JidoCode.AgentWorkspace do end end + @doc """ + Notifies the repo-scoped source watcher that product-managed code writes have + changed source graph inputs. + + This keeps LLM/tool writes on the same normalized source-change path as human + editor saves observed by the filesystem watcher. + """ + @spec notify_workspace_source_changed(managed_repo_id(), String.t(), String.t() | [String.t()], keyword()) :: + :ok | {:error, term()} + def notify_workspace_source_changed(managed_repo_id, workspace_path, changed_paths, opts \\ []) do + with {:ok, _repo_pod} <- ensure_repo_pod_entry(managed_repo_id), + {:ok, _watcher_pid} <- + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + Keyword.take(opts, [:start_file_system?, :debounce_ms]) + ) do + changed_paths + |> List.wrap() + |> Enum.reduce_while(:ok, fn changed_path, :ok -> + case RepoMonitor.notify_source_changed( + managed_repo_id, + normalize_changed_source_path(workspace_path, changed_path), + Keyword.get(opts, :file_events, [:modified]), + Keyword.get(opts, :event_source, :runtime_write) + ) do + :ok -> {:cont, :ok} + {:error, reason} -> {:halt, {:error, reason}} + end + end) + end + end + @doc """ Recovers repository-scoped source graph state after analysis, load, refresh, or query failures. """ @@ -1209,6 +1246,19 @@ defmodule JidoCode.AgentWorkspace do defp coding_pod_id(work_item_id), do: "coding-pod-#{work_item_id}" + defp ensure_repo_pod_entry(managed_repo_id) do + Manager.ensure_pod( + managed_repo_id, + @repo_pod_id, + RepoPod, + %{ + scope: :repository, + managed_repo_id: managed_repo_id, + runtime_status: :logical + } + ) + end + defp ensure_repo_pod_runtime(managed_repo_id) do ensure_runtime_pod( managed_repo_id, @@ -2323,6 +2373,22 @@ defmodule JidoCode.AgentWorkspace do end end + defp maybe_ensure_source_watcher(managed_repo_id, workspace_path, opts) do + watcher_opts = Keyword.take(opts, [:start_file_system?, :debounce_ms]) + + case RepoMonitor.ensure_source_watcher(managed_repo_id, workspace_path, watcher_opts) do + {:ok, _pid} -> :ok + {:error, _reason} -> :ok + end + end + + defp normalize_changed_source_path(workspace_path, changed_path) when is_binary(changed_path) do + case Path.type(changed_path) do + :absolute -> changed_path + _relative -> Path.join(workspace_path, changed_path) + end + end + defp ensure_memory_graph_enabled(opts) do if MemoryGraph.capability_enabled?(opts) do :ok @@ -2340,6 +2406,7 @@ defmodule JidoCode.AgentWorkspace do latest_import_status: get_in(pod_entry, [:metadata, :latest_import_status]), latest_analysis_status: get_in(pod_entry, [:metadata, :latest_analysis_status]), latest_failure: get_in(pod_entry, [:metadata, :latest_failure]), + source_graph_refresh: get_in(pod_entry, [:metadata, :source_graph_refresh]), graph: %{revision: Keyword.get(opts, :revision)} } diff --git a/lib/jido_code/agents/repo_monitor.ex b/lib/jido_code/agents/repo_monitor.ex index a135fb43..5244f835 100644 --- a/lib/jido_code/agents/repo_monitor.ex +++ b/lib/jido_code/agents/repo_monitor.ex @@ -1,6 +1,7 @@ defmodule JidoCode.Agents.RepoMonitor do # covers: architecture.agent_os_integration.kernel_per_managed_repo # covers: architecture.agent_os_integration.pod_hierarchy + # covers: architecture.source_code_graph_pod.graph_revision_state_is_explicit_and_explainable @moduledoc """ Eager agent that monitors repository state. @@ -43,4 +44,47 @@ defmodule JidoCode.Agents.RepoMonitor do doc: "The ID of the managed repository being monitored" ] ] + + alias JidoCode.Agents.RepoMonitor.SourceWatcher + + @source_change_topic_prefix "repo_monitor:source_changes" + + @doc """ + Starts or reconfigures the repo-scoped local source watcher. + + The watcher emits normalized `:workspace_source_changed` events for source + files that participate in source-code graph revision detection. + """ + @spec ensure_source_watcher(String.t(), String.t(), keyword()) :: {:ok, pid()} | {:error, term()} + def ensure_source_watcher(managed_repo_id, workspace_path, opts \\ []) do + SourceWatcher.ensure_started(managed_repo_id, workspace_path, opts) + end + + @doc """ + Stops the repo-scoped local source watcher when a repository runtime shuts down + or changes workspace binding. + """ + @spec stop_source_watcher(String.t()) :: :ok + def stop_source_watcher(managed_repo_id) do + SourceWatcher.stop(managed_repo_id) + end + + @doc """ + Emits a normalized source-change event through the repo-scoped watcher. + + This is primarily useful for tests and product-owned write boundaries that + already know a source path changed successfully. + """ + @spec notify_source_changed(String.t(), String.t(), [atom() | String.t()], atom()) :: :ok | {:error, term()} + def notify_source_changed(managed_repo_id, changed_path, file_events \\ [:modified], event_source \\ :runtime_write) do + SourceWatcher.notify_change(managed_repo_id, changed_path, file_events, event_source) + end + + @doc """ + PubSub topic used for normalized workspace source-change observations. + """ + @spec source_change_topic(String.t()) :: String.t() + def source_change_topic(managed_repo_id) when is_binary(managed_repo_id) do + @source_change_topic_prefix <> ":" <> managed_repo_id + end end diff --git a/lib/jido_code/agents/repo_monitor/source_watcher.ex b/lib/jido_code/agents/repo_monitor/source_watcher.ex new file mode 100644 index 00000000..38db34e1 --- /dev/null +++ b/lib/jido_code/agents/repo_monitor/source_watcher.ex @@ -0,0 +1,419 @@ +defmodule JidoCode.Agents.RepoMonitor.SourceWatcher do + # covers: architecture.agent_os_integration.repo_pod_singleton_per_kernel + # covers: architecture.source_code_graph_pod.graph_revision_state_is_explicit_and_explainable + @moduledoc false + + use GenServer + + require Logger + + alias JidoCode.Agents.RepoMonitor + alias JidoCode.AgentOS.Manager + alias JidoCode.SourceCodeGraph + alias JidoCode.SourceCodeGraph.RefreshScheduler + + @registry JidoCode.RepoMonitor.SourceWatcherRegistry + @supervisor JidoCode.RepoMonitor.SourceWatcherSupervisor + @repo_pod_id "repo-pod" + + @type source_change_event :: %{ + kind: :workspace_source_changed, + managed_repo_id: String.t(), + workspace_path: String.t(), + changed_paths: [String.t()], + changed_path: String.t(), + file_events: [atom() | String.t()], + event_source: atom(), + event_sources: [atom()], + current_revision: String.t() | nil, + source_commit: String.t() | nil, + workspace_snapshot_identity: String.t() | nil, + observed_at: DateTime.t() + } + + @spec ensure_started(String.t(), String.t(), keyword()) :: {:ok, pid()} | {:error, term()} + def ensure_started(managed_repo_id, workspace_path, opts \\ []) + when is_binary(managed_repo_id) and is_binary(workspace_path) and is_list(opts) do + child_opts = + opts + |> Keyword.put(:managed_repo_id, managed_repo_id) + |> Keyword.put(:workspace_path, workspace_path) + + case DynamicSupervisor.start_child(@supervisor, {__MODULE__, child_opts}) do + {:ok, pid} -> + {:ok, pid} + + {:error, {:already_started, pid}} -> + GenServer.call(pid, {:configure, workspace_path, opts}) + + {:error, {:shutdown, reason}} -> + {:error, reason} + + {:error, reason} -> + {:error, reason} + end + end + + @spec stop(String.t()) :: :ok + def stop(managed_repo_id) when is_binary(managed_repo_id) do + case Registry.lookup(@registry, managed_repo_id) do + [{pid, _value}] -> + DynamicSupervisor.terminate_child(@supervisor, pid) + :ok + + [] -> + :ok + end + end + + @spec notify_change(String.t(), String.t(), [atom() | String.t()], atom()) :: :ok | {:error, term()} + def notify_change(managed_repo_id, changed_path, file_events, event_source) + when is_binary(managed_repo_id) and is_binary(changed_path) and is_list(file_events) and + is_atom(event_source) do + case Registry.lookup(@registry, managed_repo_id) do + [{pid, _value}] -> + GenServer.cast(pid, {:notify_change, changed_path, file_events, event_source}) + + [] -> + {:error, :source_watcher_not_started} + end + end + + @spec via(String.t()) :: {:via, Registry, {module(), String.t()}} + def via(managed_repo_id) when is_binary(managed_repo_id) do + {:via, Registry, {@registry, managed_repo_id}} + end + + def child_spec(opts) do + managed_repo_id = Keyword.fetch!(opts, :managed_repo_id) + + %{ + id: {__MODULE__, managed_repo_id}, + start: {__MODULE__, :start_link, [opts]}, + restart: :transient, + shutdown: 5_000, + type: :worker + } + end + + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) when is_list(opts) do + managed_repo_id = Keyword.fetch!(opts, :managed_repo_id) + GenServer.start_link(__MODULE__, opts, name: via(managed_repo_id)) + end + + @impl true + def init(opts) do + Process.flag(:trap_exit, true) + + with {:ok, state} <- build_state(opts), + {:ok, state} <- maybe_start_file_watcher(state, opts) do + {:ok, persist_watcher_state(state)} + else + {:error, reason} -> {:stop, reason} + end + end + + @impl true + def handle_call({:configure, workspace_path, opts}, _from, state) do + with {:ok, normalized_workspace_path} <- normalize_workspace_path(workspace_path) do + if normalized_workspace_path == state.workspace_path do + {:reply, {:ok, self()}, state} + else + next_state = + state + |> stop_file_watcher() + |> Map.put(:workspace_path, normalized_workspace_path) + |> Map.put(:debounce_ms, debounce_ms(opts)) + |> Map.put(:max_pending_paths, max_pending_paths(opts)) + |> Map.put(:pending_change, nil) + + case maybe_start_file_watcher(next_state, opts) do + {:ok, configured_state} -> + {:reply, {:ok, self()}, persist_watcher_state(configured_state)} + + {:error, reason} -> + {:reply, {:error, reason}, persist_watcher_state(%{next_state | watcher_status: :failed})} + end + end + else + {:error, reason} -> {:reply, {:error, reason}, state} + end + end + + @impl true + def handle_cast({:notify_change, path, events, event_source}, state) do + {:noreply, queue_source_change(state, path, List.wrap(events), event_source)} + end + + @impl true + def handle_info({:file_event, watcher_pid, {path, events}}, %{file_watcher_pid: watcher_pid} = state) do + {:noreply, queue_source_change(state, path, List.wrap(events), :human_watcher)} + end + + def handle_info({:file_event, watcher_pid, :stop}, %{file_watcher_pid: watcher_pid} = state) do + {:noreply, persist_watcher_state(%{state | file_watcher_pid: nil, watcher_status: :stopped})} + end + + def handle_info({:file_event, _watcher_pid, _event}, state), do: {:noreply, state} + + def handle_info(:flush_source_changes, state) do + case source_change_event(state) do + {:ok, event} -> + publish_source_change(event) + {:noreply, persist_source_change(%{state | debounce_timer: nil, pending_change: nil}, event)} + + :ignore -> + {:noreply, %{state | debounce_timer: nil, pending_change: nil}} + end + end + + def handle_info({:EXIT, pid, reason}, %{file_watcher_pid: pid} = state) do + Logger.warning( + "source_file_watcher_exited managed_repo_id=#{inspect(state.managed_repo_id)} reason=#{inspect(reason)}" + ) + + {:noreply, persist_watcher_state(%{state | file_watcher_pid: nil, file_watcher_ref: nil, watcher_status: :stopped})} + end + + def handle_info({:DOWN, ref, :process, pid, reason}, %{file_watcher_ref: ref, file_watcher_pid: pid} = state) do + Logger.warning( + "source_file_watcher_stopped managed_repo_id=#{inspect(state.managed_repo_id)} reason=#{inspect(reason)}" + ) + + {:noreply, persist_watcher_state(%{state | file_watcher_pid: nil, file_watcher_ref: nil, watcher_status: :stopped})} + end + + def handle_info(_message, state), do: {:noreply, state} + + defp build_state(opts) do + managed_repo_id = Keyword.fetch!(opts, :managed_repo_id) + + with {:ok, workspace_path} <- normalize_workspace_path(Keyword.get(opts, :workspace_path)) do + {:ok, + %{ + managed_repo_id: managed_repo_id, + workspace_path: workspace_path, + file_watcher_pid: nil, + file_watcher_ref: nil, + watcher_status: :starting, + debounce_ms: debounce_ms(opts), + max_pending_paths: max_pending_paths(opts), + debounce_timer: nil, + pending_change: nil, + latest_source_change: nil + }} + end + end + + defp normalize_workspace_path(path) when is_binary(path) do + case SourceCodeGraph.normalize_workspace_path(path) do + {:ok, workspace_path} -> + if File.dir?(workspace_path) do + {:ok, workspace_path} + else + {:error, :missing_workspace_path} + end + + {:error, reason} -> + {:error, reason} + end + end + + defp normalize_workspace_path(_path), do: {:error, :missing_workspace_path} + + defp maybe_start_file_watcher(state, opts) do + start_file_system? = + Keyword.get( + opts, + :start_file_system?, + Application.get_env(:jido_code, :source_code_graph_file_watcher_enabled, false) + ) + + if start_file_system? do + start_file_watcher(state) + else + {:ok, %{state | watcher_status: :disabled}} + end + end + + defp debounce_ms(opts) do + Keyword.get(opts, :debounce_ms, Application.get_env(:jido_code, :source_code_graph_file_watcher_debounce_ms, 500)) + end + + defp max_pending_paths(opts) do + Keyword.get( + opts, + :max_pending_paths, + Application.get_env(:jido_code, :source_code_graph_file_watcher_max_pending_paths, 500) + ) + end + + defp start_file_watcher(state) do + with {:module, FileSystem} <- Code.ensure_loaded(FileSystem), + {:ok, watcher_pid} <- FileSystem.start_link(dirs: [state.workspace_path]), + :ok <- FileSystem.subscribe(watcher_pid) do + {:ok, + %{ + state + | file_watcher_pid: watcher_pid, + file_watcher_ref: Process.monitor(watcher_pid), + watcher_status: :watching + }} + else + {:error, reason} -> + {:error, {:file_watcher_start_failed, reason}} + + other -> + {:error, {:file_watcher_unavailable, other}} + end + end + + defp stop_file_watcher(%{file_watcher_pid: nil} = state), do: state + + defp stop_file_watcher(%{file_watcher_pid: pid, file_watcher_ref: ref} = state) do + if is_reference(ref), do: Process.demonitor(ref, [:flush]) + if Process.alive?(pid), do: GenServer.stop(pid) + + %{state | file_watcher_pid: nil, file_watcher_ref: nil, watcher_status: :stopped} + end + + defp queue_source_change(state, path, file_events, event_source) do + if SourceCodeGraph.source_file?(state.workspace_path, path) do + changed_path = Path.relative_to(Path.expand(path), state.workspace_path) + + pending_change = + merge_pending_change(state.pending_change, changed_path, file_events, event_source, state.max_pending_paths) + + state + |> Map.put(:pending_change, pending_change) + |> schedule_debounce() + else + state + end + end + + defp merge_pending_change(nil, changed_path, file_events, event_source, _max_pending_paths) do + %{ + changed_paths: MapSet.new([changed_path]), + file_events: MapSet.new(normalize_file_events(file_events)), + event_sources: MapSet.new([event_source]), + first_observed_at: DateTime.utc_now() + } + end + + defp merge_pending_change(pending_change, changed_path, file_events, event_source, max_pending_paths) do + pending_change + |> update_in([:changed_paths], &bounded_path_put(&1, changed_path, max_pending_paths)) + |> update_in([:file_events], fn existing -> + Enum.reduce(normalize_file_events(file_events), existing, fn event, acc -> MapSet.put(acc, event) end) + end) + |> update_in([:event_sources], &MapSet.put(&1, event_source)) + end + + defp bounded_path_put(paths, changed_path, max_pending_paths) do + cond do + MapSet.member?(paths, changed_path) -> paths + MapSet.size(paths) < max_pending_paths -> MapSet.put(paths, changed_path) + true -> paths + end + end + + defp schedule_debounce(%{debounce_timer: timer} = state) when is_reference(timer) do + Process.cancel_timer(timer) + schedule_debounce(%{state | debounce_timer: nil}) + end + + defp schedule_debounce(state) do + %{state | debounce_timer: Process.send_after(self(), :flush_source_changes, state.debounce_ms)} + end + + defp source_change_event(%{pending_change: nil}), do: :ignore + + defp source_change_event(state) do + revision_metadata = + case SourceCodeGraph.current_revision_metadata(state.workspace_path) do + {:ok, metadata} -> metadata + {:error, _reason} -> %{} + end + + changed_paths = state.pending_change.changed_paths |> MapSet.to_list() |> Enum.sort() + event_sources = state.pending_change.event_sources |> MapSet.to_list() |> Enum.sort() + + {:ok, + %{ + kind: :workspace_source_changed, + managed_repo_id: state.managed_repo_id, + workspace_path: state.workspace_path, + changed_paths: changed_paths, + changed_path: List.first(changed_paths), + file_events: state.pending_change.file_events |> MapSet.to_list() |> Enum.sort(), + event_source: event_source(event_sources), + event_sources: event_sources, + current_revision: Map.get(revision_metadata, :current_revision), + source_commit: Map.get(revision_metadata, :source_commit), + workspace_snapshot_identity: Map.get(revision_metadata, :workspace_snapshot_identity), + observed_at: DateTime.utc_now() + }} + end + + defp event_source([event_source]), do: event_source + defp event_source(_event_sources), do: :mixed + + defp normalize_file_events(events) do + events + |> Enum.map(fn + event when is_atom(event) -> event + event when is_binary(event) -> event + event -> event + end) + end + + defp publish_source_change(event) do + Phoenix.PubSub.broadcast( + JidoCode.PubSub, + RepoMonitor.source_change_topic(event.managed_repo_id), + {:workspace_source_changed, event} + ) + + RefreshScheduler.enqueue(event) + end + + defp persist_source_change(state, event) do + next_state = %{state | latest_source_change: event} + + persist_watcher_state(next_state, %{ + latest_source_change: Map.drop(event, [:kind]), + latest_source_change_at: event.observed_at + }) + end + + defp persist_watcher_state(state, extra_updates \\ %{}) do + updates = + Map.merge( + %{ + source_watcher: %{ + workspace_path: state.workspace_path, + status: state.watcher_status, + latest_source_change_at: state.latest_source_change && state.latest_source_change.observed_at + } + }, + extra_updates + ) + + safe_update_pod_metadata(state.managed_repo_id, updates) + + state + end + + defp safe_update_pod_metadata(managed_repo_id, updates) do + case Manager.update_pod_metadata(managed_repo_id, @repo_pod_id, updates) do + {:ok, _pod_entry} -> :ok + {:error, _reason} -> :ok + end + rescue + _error -> :ok + catch + :exit, _reason -> :ok + end +end diff --git a/lib/jido_code/application.ex b/lib/jido_code/application.ex index a8ac7794..a75e5134 100644 --- a/lib/jido_code/application.ex +++ b/lib/jido_code/application.ex @@ -25,6 +25,12 @@ defmodule JidoCode.Application do {DynamicSupervisor, name: JidoCode.Forge.SpriteSupervisor, strategy: :one_for_one}, {DynamicSupervisor, name: JidoCode.Forge.ExecSessionSupervisor, strategy: :one_for_one}, JidoCode.Forge.Manager, + # Repository source-change monitoring + {Registry, keys: :unique, name: JidoCode.RepoMonitor.SourceWatcherRegistry}, + {DynamicSupervisor, name: JidoCode.RepoMonitor.SourceWatcherSupervisor, strategy: :one_for_one}, + {Registry, keys: :unique, name: JidoCode.SourceCodeGraph.RefreshSchedulerRegistry}, + {DynamicSupervisor, name: JidoCode.SourceCodeGraph.RefreshSchedulerSupervisor, strategy: :one_for_one}, + {Task.Supervisor, name: JidoCode.SourceCodeGraph.RefreshTaskSupervisor}, # AgentOS supervision tree {JidoCode.AgentOS.Manager.Server, []}, {JidoCode.AgentOS.Manager.Supervisor, []} diff --git a/lib/jido_code/forge/sprite_client.ex b/lib/jido_code/forge/sprite_client.ex index bd125fb8..bab7f88a 100644 --- a/lib/jido_code/forge/sprite_client.ex +++ b/lib/jido_code/forge/sprite_client.ex @@ -15,6 +15,7 @@ defmodule JidoCode.Forge.SpriteClient do @behaviour JidoCode.Forge.SpriteClient.Behaviour alias JidoCode.Forge.SpriteClient.Fake + alias JidoCode.AgentWorkspace defp impl do Application.get_env(:jido_code, :forge_sprite_client, Fake) @@ -55,6 +56,25 @@ defmodule JidoCode.Forge.SpriteClient do impl_for(client).write_file(client, path, content) end + @doc """ + Writes a file and optionally notifies the repo-scoped source watcher. + + Pass `:managed_repo_id` and `:workspace_path` when the write is known to touch + a managed repository workspace. Notification only happens after the write + succeeds. + """ + @spec write_file(term(), String.t(), binary(), keyword()) :: :ok | {:error, term()} + def write_file(client, path, content, opts) when is_list(opts) do + case write_file(client, path, content) do + :ok -> + maybe_notify_source_write(path, opts) + :ok + + {:error, _reason} = error -> + error + end + end + @impl true def read_file(client, path) do impl_for(client).read_file(client, path) @@ -69,4 +89,25 @@ defmodule JidoCode.Forge.SpriteClient do def destroy(client, sprite_id) do impl_for(client).destroy(client, sprite_id) end + + defp maybe_notify_source_write(path, opts) do + managed_repo_id = Keyword.get(opts, :managed_repo_id) + workspace_path = Keyword.get(opts, :workspace_path) + + if is_binary(managed_repo_id) and is_binary(workspace_path) do + changed_path = Keyword.get(opts, :source_path, path) + + AgentWorkspace.notify_workspace_source_changed( + managed_repo_id, + workspace_path, + changed_path, + event_source: Keyword.get(opts, :event_source, :tool_write), + file_events: Keyword.get(opts, :file_events, [:modified]), + start_file_system?: Keyword.get(opts, :start_file_system?, false), + debounce_ms: Keyword.get(opts, :debounce_ms, 500) + ) + end + + :ok + end end diff --git a/lib/jido_code/source_code_graph.ex b/lib/jido_code/source_code_graph.ex index 84df4778..913cf596 100644 --- a/lib/jido_code/source_code_graph.ex +++ b/lib/jido_code/source_code_graph.ex @@ -66,6 +66,32 @@ defmodule JidoCode.SourceCodeGraph do |> then(&Path.join([&1, ".jido_code", "source_code_graph", "triple_store"])) end + @spec source_file_patterns(workspace_path()) :: [String.t()] + def source_file_patterns(workspace_path) when is_binary(workspace_path), do: source_globs(Path.expand(workspace_path)) + + @spec source_files(workspace_path()) :: [String.t()] + def source_files(workspace_path) when is_binary(workspace_path) do + workspace_path + |> Path.expand() + |> source_globs() + |> Enum.flat_map(&Path.wildcard/1) + |> Enum.filter(&source_file?(workspace_path, &1)) + |> Enum.uniq() + |> Enum.sort() + end + + @spec source_file?(workspace_path(), String.t()) :: boolean() + def source_file?(workspace_path, path) when is_binary(workspace_path) and is_binary(path) do + workspace_path = Path.expand(workspace_path) + path = Path.expand(path) + + inside_workspace?(workspace_path, path) and + source_relative_file?(Path.relative_to(path, workspace_path)) and + not excluded_source_file?(path) + end + + def source_file?(_workspace_path, _path), do: false + @spec base_iri(managed_repo_id()) :: String.t() def base_iri(managed_repo_id) when is_binary(managed_repo_id) do "https://jido.run/managed_repos/#{managed_repo_id}/source_code#" @@ -155,6 +181,7 @@ defmodule JidoCode.SourceCodeGraph do revision_source: nil, failure: nil }, + source_graph_refresh: default_refresh_status(managed_repo_id), latest_failure: nil }} end @@ -172,12 +199,46 @@ defmodule JidoCode.SourceCodeGraph do graph_store_path: context.graph_store_path, latest_analysis_status: context.latest_analysis_status, latest_import_status: context.latest_import_status, + source_graph_refresh: context.source_graph_refresh, latest_failure: context.latest_failure, dataset_metadata: context.dataset_metadata }} end end + @spec default_refresh_status(managed_repo_id() | nil) :: map() + def default_refresh_status(managed_repo_id \\ nil) do + %{ + managed_repo_id: managed_repo_id, + state: :idle, + auto_refresh_enabled?: Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled, false), + file_watcher_enabled?: Application.get_env(:jido_code, :source_code_graph_file_watcher_enabled, false), + file_watcher_debounce_ms: Application.get_env(:jido_code, :source_code_graph_file_watcher_debounce_ms, 500), + file_watcher_max_pending_paths: + Application.get_env(:jido_code, :source_code_graph_file_watcher_max_pending_paths, 500), + refresh_debounce_ms: Application.get_env(:jido_code, :source_code_graph_refresh_debounce_ms, 250), + refresh_max_coalesce_ms: Application.get_env(:jido_code, :source_code_graph_refresh_max_coalesce_ms, 2_500), + refresh_max_pending_paths: Application.get_env(:jido_code, :source_code_graph_refresh_max_pending_paths, 500), + missing_graph_policy: + Application.get_env(:jido_code, :source_code_graph_auto_refresh_missing_graph_policy, :skip), + max_refresh_attempts: Application.get_env(:jido_code, :source_code_graph_auto_refresh_max_attempts, 1), + refresh_queued?: false, + refresh_in_flight?: false, + pending_changed_paths: [], + last_source_change_at: nil, + last_refresh_started_at: nil, + last_refresh_completed_at: nil, + last_result: nil, + last_failure: nil + } + end + + @spec merge_refresh_status(map() | nil, managed_repo_id() | nil) :: map() + def merge_refresh_status(status, managed_repo_id \\ nil) do + default_refresh_status(managed_repo_id) + |> Map.merge(if(is_map(status), do: status, else: %{})) + end + defp default_analysis_status(revision_metadata) do %{ state: :not_analyzed, @@ -251,15 +312,6 @@ defmodule JidoCode.SourceCodeGraph do end end - defp source_files(workspace_path) do - workspace_path - |> source_globs() - |> Enum.flat_map(&Path.wildcard/1) - |> Enum.reject(&excluded_source_file?/1) - |> Enum.uniq() - |> Enum.sort() - end - defp source_globs(workspace_path) do [ Path.join(workspace_path, "mix.exs"), @@ -271,9 +323,28 @@ defmodule JidoCode.SourceCodeGraph do ] end + defp inside_workspace?(workspace_path, path) do + path == workspace_path or String.starts_with?(path, workspace_path <> "/") + end + + defp source_relative_file?("mix.exs"), do: true + + defp source_relative_file?(relative_path) do + segments = Path.split(relative_path) + extension = Path.extname(relative_path) + + case segments do + ["lib" | _] when extension in [".ex", ".exs"] -> true + ["test" | _] when extension in [".ex", ".exs"] -> true + ["config" | _] when extension == ".exs" -> true + _ -> false + end + end + defp excluded_source_file?(path) do String.contains?(path, "/deps/") or String.contains?(path, "/_build/") or - String.contains?(path, "/node_modules/") + String.contains?(path, "/node_modules/") or + String.contains?(path, "/.jido_code/") end end diff --git a/lib/jido_code/source_code_graph/analysis.ex b/lib/jido_code/source_code_graph/analysis.ex index efb8f49a..8693be03 100644 --- a/lib/jido_code/source_code_graph/analysis.ex +++ b/lib/jido_code/source_code_graph/analysis.ex @@ -32,21 +32,23 @@ defmodule JidoCode.SourceCodeGraph.Analysis do analyzed_at: DateTime.utc_now(), source_commit: revision_metadata.source_commit, workspace_snapshot_identity: revision_metadata.workspace_snapshot_identity, - failure: reason, + failure: normalize_failure(reason), details: details }} end end defp do_analyze(graph_context, analysis_options, revision_metadata, started_at, timeout, opts) do - analysis_task = Task.async(fn -> retry_opts = Keyword.take(opts, [:max_retries, :retry_backoff_ms, :on_retry]) - RetryPolicy.retry(fn -> - ElixirOntologies.analyze_project(graph_context.workspace_path, analysis_options) - end, retry_opts) + RetryPolicy.retry( + fn -> + ElixirOntologies.analyze_project(graph_context.workspace_path, analysis_options) + end, + retry_opts + ) end) case Task.yield(analysis_task, timeout) || Task.shutdown(analysis_task, :brutal_kill) do diff --git a/lib/jido_code/source_code_graph/product_feedback.ex b/lib/jido_code/source_code_graph/product_feedback.ex index d1960cb9..2461c59b 100644 --- a/lib/jido_code/source_code_graph/product_feedback.ex +++ b/lib/jido_code/source_code_graph/product_feedback.ex @@ -12,6 +12,7 @@ defmodule JidoCode.SourceCodeGraph.ProductFeedback do imported_revision: nil, current_revision: nil, latest_failure: nil, + refresh: %{}, recovery_action: :none } @@ -35,6 +36,7 @@ defmodule JidoCode.SourceCodeGraph.ProductFeedback do imported_revision: Map.get(graph, :imported_revision), current_revision: Map.get(graph, :current_revision), latest_failure: normalize_failure(Map.get(graph, :latest_failure)), + refresh: Map.get(graph, :refresh, %{}), recovery_action: Map.get(graph, :recovery_action, :none) } end diff --git a/lib/jido_code/source_code_graph/refresh_scheduler.ex b/lib/jido_code/source_code_graph/refresh_scheduler.ex new file mode 100644 index 00000000..2c8858ac --- /dev/null +++ b/lib/jido_code/source_code_graph/refresh_scheduler.ex @@ -0,0 +1,403 @@ +defmodule JidoCode.SourceCodeGraph.RefreshScheduler do + # covers: architecture.source_code_graph_pod.explicit_actions_drive_analyze_load_refresh_and_query + # covers: architecture.source_code_graph_pod.graph_refresh_replaces_named_graph_coherently + @moduledoc false + + use GenServer + + alias JidoCode.AgentOS.Manager + alias JidoCode.AgentWorkspace + alias JidoCode.SourceCodeGraph + + @registry JidoCode.SourceCodeGraph.RefreshSchedulerRegistry + @supervisor JidoCode.SourceCodeGraph.RefreshSchedulerSupervisor + @task_supervisor JidoCode.SourceCodeGraph.RefreshTaskSupervisor + + @type source_change_event :: map() + + @spec enqueue(source_change_event()) :: :ok | {:error, term()} + def enqueue(%{managed_repo_id: managed_repo_id} = event) when is_binary(managed_repo_id) do + if Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled, false) do + with {:ok, _pid} <- ensure_started(managed_repo_id) do + GenServer.cast(via(managed_repo_id), {:enqueue, event}) + end + else + :ok + end + end + + @spec ensure_started(String.t(), keyword()) :: {:ok, pid()} | {:error, term()} + def ensure_started(managed_repo_id, opts \\ []) when is_binary(managed_repo_id) and is_list(opts) do + child_opts = Keyword.put(opts, :managed_repo_id, managed_repo_id) + + case DynamicSupervisor.start_child(@supervisor, {__MODULE__, child_opts}) do + {:ok, pid} -> {:ok, pid} + {:error, {:already_started, pid}} -> {:ok, pid} + {:error, reason} -> {:error, reason} + end + end + + @spec status(String.t()) :: {:ok, map()} | {:error, :not_started} + def status(managed_repo_id) when is_binary(managed_repo_id) do + case Registry.lookup(@registry, managed_repo_id) do + [{pid, _value}] -> GenServer.call(pid, :status) + [] -> {:error, :not_started} + end + end + + @spec stop(String.t()) :: :ok + def stop(managed_repo_id) when is_binary(managed_repo_id) do + case Registry.lookup(@registry, managed_repo_id) do + [{pid, _value}] -> + DynamicSupervisor.terminate_child(@supervisor, pid) + :ok + + [] -> + :ok + end + end + + @spec via(String.t()) :: {:via, Registry, {module(), String.t()}} + def via(managed_repo_id) when is_binary(managed_repo_id) do + {:via, Registry, {@registry, managed_repo_id}} + end + + def child_spec(opts) do + managed_repo_id = Keyword.fetch!(opts, :managed_repo_id) + + %{ + id: {__MODULE__, managed_repo_id}, + start: {__MODULE__, :start_link, [opts]}, + restart: :transient, + shutdown: 5_000, + type: :worker + } + end + + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) when is_list(opts) do + managed_repo_id = Keyword.fetch!(opts, :managed_repo_id) + GenServer.start_link(__MODULE__, opts, name: via(managed_repo_id)) + end + + @impl true + def init(opts) do + {:ok, + %{ + managed_repo_id: Keyword.fetch!(opts, :managed_repo_id), + pending_event: nil, + refresh_timer: nil, + refresh_task: nil, + refresh_debounce_ms: Keyword.get(opts, :refresh_debounce_ms, default_refresh_debounce_ms()), + refresh_max_coalesce_ms: Keyword.get(opts, :refresh_max_coalesce_ms, default_refresh_max_coalesce_ms()), + max_pending_paths: Keyword.get(opts, :max_pending_paths, default_max_pending_paths()), + missing_graph_policy: Keyword.get(opts, :missing_graph_policy, default_missing_graph_policy()), + refresh_fun: Keyword.get(opts, :refresh_fun, &run_agent_workspace_refresh/4), + max_refresh_attempts: Keyword.get(opts, :max_refresh_attempts, default_max_refresh_attempts()), + state: :idle, + first_queued_at: nil, + last_source_change_at: nil, + last_refresh_started_at: nil, + last_refresh_completed_at: nil, + last_result: nil, + last_failure: nil, + pending_after_refresh?: false + }} + end + + @impl true + def handle_call(:status, _from, state), do: {:reply, {:ok, status_projection(state)}, state} + + @impl true + def handle_cast({:enqueue, event}, state) do + next_state = + state + |> Map.put(:last_source_change_at, Map.get(event, :observed_at) || DateTime.utc_now()) + |> enqueue_event(event) + + {:noreply, persist_scheduler_state(next_state)} + end + + @impl true + def handle_info(:run_refresh, state) do + state = %{state | refresh_timer: nil} + + if state.refresh_task do + {:noreply, persist_scheduler_state(%{state | pending_after_refresh?: true, state: :queued})} + else + {:noreply, start_refresh_task(state)} + end + end + + def handle_info({ref, result}, %{refresh_task: %{ref: ref}} = state) do + Process.demonitor(ref, [:flush]) + + next_state = + state + |> Map.put(:refresh_task, nil) + |> Map.put(:last_refresh_completed_at, DateTime.utc_now()) + |> apply_refresh_result(result) + + if next_state.pending_after_refresh? do + rescheduled = + next_state + |> Map.put(:pending_after_refresh?, false) + |> schedule_refresh() + + {:noreply, persist_scheduler_state(rescheduled)} + else + {:noreply, persist_scheduler_state(%{next_state | pending_event: nil, first_queued_at: nil})} + end + end + + def handle_info({:DOWN, ref, :process, _pid, reason}, %{refresh_task: %{ref: ref}} = state) do + next_state = + state + |> Map.put(:refresh_task, nil) + |> Map.put(:last_refresh_completed_at, DateTime.utc_now()) + |> Map.put(:last_failure, %{reason: inspect(reason), recorded_at: DateTime.utc_now()}) + |> Map.put(:state, :failed) + + {:noreply, persist_scheduler_state(next_state)} + end + + def handle_info(_message, state), do: {:noreply, state} + + defp enqueue_event(%{refresh_task: task} = state, event) when not is_nil(task) do + %{ + state + | pending_event: merge_events(state.pending_event, event, state.max_pending_paths), + first_queued_at: state.first_queued_at || DateTime.utc_now(), + pending_after_refresh?: true, + state: :queued + } + end + + defp enqueue_event(state, event) do + state + |> Map.put(:pending_event, merge_events(state.pending_event, event, state.max_pending_paths)) + |> Map.put(:first_queued_at, state.first_queued_at || DateTime.utc_now()) + |> schedule_refresh() + end + + defp schedule_refresh(%{refresh_timer: timer} = state) when is_reference(timer) do + Process.cancel_timer(timer) + schedule_refresh(%{state | refresh_timer: nil}) + end + + defp schedule_refresh(state) do + %{state | refresh_timer: Process.send_after(self(), :run_refresh, refresh_delay_ms(state)), state: :queued} + end + + defp start_refresh_task(%{pending_event: nil} = state), do: %{state | state: :idle} + + defp start_refresh_task(state) do + event = state.pending_event + refresh_fun = state.refresh_fun + + task = + Task.Supervisor.async_nolink(@task_supervisor, fn -> + run_refresh_with_retry( + refresh_fun, + state.managed_repo_id, + event.workspace_path, + event, + [missing_graph_policy: state.missing_graph_policy], + state.max_refresh_attempts + ) + end) + + %{ + state + | refresh_task: task, + pending_event: nil, + first_queued_at: nil, + state: :running, + last_refresh_started_at: DateTime.utc_now(), + last_failure: nil + } + |> persist_scheduler_state() + end + + defp apply_refresh_result(state, {:ok, result}) do + state + |> Map.put(:last_result, result) + |> Map.put(:last_failure, nil) + |> Map.put(:state, refresh_result_state(result)) + end + + defp apply_refresh_result(state, {:error, reason, detail}) do + state + |> Map.put(:last_failure, %{reason: reason, detail: detail, recorded_at: DateTime.utc_now()}) + |> Map.put(:state, :failed) + end + + defp apply_refresh_result(state, {:error, reason}) do + state + |> Map.put(:last_failure, %{reason: reason, recorded_at: DateTime.utc_now()}) + |> Map.put(:state, :failed) + end + + defp apply_refresh_result(state, other) do + state + |> Map.put(:last_failure, %{reason: {:unexpected_result, other}, recorded_at: DateTime.utc_now()}) + |> Map.put(:state, :failed) + end + + defp refresh_result_state(%{status: status}) when status in [:refresh_skipped_current, :refresh_skipped_not_ready], + do: :skipped + + defp refresh_result_state(%{status: :refresh_skipped_disabled}), do: :skipped + defp refresh_result_state(_result), do: :succeeded + + defp merge_events(nil, event, max_pending_paths), do: normalize_event(event, max_pending_paths) + + defp merge_events(existing, event, max_pending_paths) do + event = normalize_event(event, max_pending_paths) + + existing + |> Map.put(:observed_at, event.observed_at) + |> Map.put(:changed_paths, merge_lists(existing.changed_paths, event.changed_paths, max_pending_paths)) + |> Map.put(:file_events, merge_lists(existing.file_events, event.file_events)) + |> Map.put(:event_sources, merge_lists(existing.event_sources, event.event_sources)) + |> then(fn merged -> Map.put(merged, :event_source, event_source(merged.event_sources)) end) + end + + defp normalize_event(event, max_pending_paths) do + changed_paths = Map.get(event, :changed_paths) || List.wrap(Map.get(event, :changed_path)) + event_sources = Map.get(event, :event_sources) || [Map.get(event, :event_source, :unknown)] + + %{ + kind: :workspace_source_changed, + managed_repo_id: event.managed_repo_id, + workspace_path: event.workspace_path, + changed_paths: + changed_paths |> Enum.reject(&is_nil/1) |> Enum.uniq() |> Enum.sort() |> Enum.take(max_pending_paths), + file_events: (Map.get(event, :file_events) || []) |> Enum.uniq() |> Enum.sort(), + event_source: event_source(event_sources), + event_sources: event_sources |> Enum.reject(&is_nil/1) |> Enum.uniq() |> Enum.sort(), + observed_at: Map.get(event, :observed_at) || DateTime.utc_now() + } + end + + defp merge_lists(left, right), do: (left ++ right) |> Enum.uniq() |> Enum.sort() + defp merge_lists(left, right, max), do: left |> merge_lists(right) |> Enum.take(max) + + defp refresh_delay_ms(%{first_queued_at: nil} = state), do: state.refresh_debounce_ms + + defp refresh_delay_ms(state) do + elapsed_ms = DateTime.diff(DateTime.utc_now(), state.first_queued_at, :millisecond) + remaining_ms = max(state.refresh_max_coalesce_ms - elapsed_ms, 0) + + min(state.refresh_debounce_ms, remaining_ms) + end + + defp event_source([event_source]), do: event_source + defp event_source(_event_sources), do: :mixed + + defp run_agent_workspace_refresh(managed_repo_id, workspace_path, _event, opts) do + case AgentWorkspace.source_code_graph_status(managed_repo_id, workspace_path) do + {:ok, %{ready?: true, stale?: true}} -> + AgentWorkspace.refresh_source_code_graph(managed_repo_id, workspace_path) + + {:ok, %{ready?: true, stale?: false} = status} -> + {:ok, %{status: :refresh_skipped_current, graph_status: status}} + + {:ok, %{ready?: false} = status} -> + case Keyword.get(opts, :missing_graph_policy, :skip) do + :load -> AgentWorkspace.load_source_code_graph(managed_repo_id, workspace_path) + _skip -> {:ok, %{status: :refresh_skipped_not_ready, graph_status: status}} + end + + {:error, :source_code_graph_disabled} -> + {:ok, %{status: :refresh_skipped_disabled}} + + {:error, reason} -> + {:error, reason} + + {:error, reason, detail} -> + {:error, reason, detail} + end + end + + defp run_refresh_with_retry(refresh_fun, managed_repo_id, workspace_path, event, opts, attempts_left) + when attempts_left <= 1 do + refresh_fun.(managed_repo_id, workspace_path, event, opts) + end + + defp run_refresh_with_retry(refresh_fun, managed_repo_id, workspace_path, event, opts, attempts_left) do + case refresh_fun.(managed_repo_id, workspace_path, event, opts) do + {:error, _reason} -> + run_refresh_with_retry(refresh_fun, managed_repo_id, workspace_path, event, opts, attempts_left - 1) + + {:error, _reason, _detail} -> + run_refresh_with_retry(refresh_fun, managed_repo_id, workspace_path, event, opts, attempts_left - 1) + + result -> + result + end + end + + defp status_projection(state) do + %{ + managed_repo_id: state.managed_repo_id, + state: state.state, + auto_refresh_enabled?: Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled, false), + file_watcher_enabled?: Application.get_env(:jido_code, :source_code_graph_file_watcher_enabled, false), + file_watcher_debounce_ms: Application.get_env(:jido_code, :source_code_graph_file_watcher_debounce_ms, 500), + file_watcher_max_pending_paths: + Application.get_env(:jido_code, :source_code_graph_file_watcher_max_pending_paths, 500), + refresh_debounce_ms: state.refresh_debounce_ms, + refresh_max_coalesce_ms: state.refresh_max_coalesce_ms, + refresh_max_pending_paths: state.max_pending_paths, + missing_graph_policy: state.missing_graph_policy, + max_refresh_attempts: state.max_refresh_attempts, + refresh_queued?: not is_nil(state.refresh_timer) or state.pending_after_refresh?, + refresh_in_flight?: not is_nil(state.refresh_task), + pending_changed_paths: pending_changed_paths(state.pending_event), + last_source_change_at: state.last_source_change_at, + last_refresh_started_at: state.last_refresh_started_at, + last_refresh_completed_at: state.last_refresh_completed_at, + last_result: state.last_result, + last_failure: state.last_failure + } + end + + defp pending_changed_paths(nil), do: [] + defp pending_changed_paths(event), do: Map.get(event, :changed_paths, []) + + defp persist_scheduler_state(state) do + updates = %{source_graph_refresh: status_projection(state)} + + case Manager.update_pod_metadata(state.managed_repo_id, SourceCodeGraph.pod_id(), updates) do + {:ok, _pod_entry} -> :ok + {:error, _reason} -> :ok + end + + state + rescue + _error -> state + catch + :exit, _reason -> state + end + + defp default_refresh_debounce_ms do + Application.get_env(:jido_code, :source_code_graph_refresh_debounce_ms, 250) + end + + defp default_refresh_max_coalesce_ms do + Application.get_env(:jido_code, :source_code_graph_refresh_max_coalesce_ms, 2_500) + end + + defp default_max_pending_paths do + Application.get_env(:jido_code, :source_code_graph_refresh_max_pending_paths, 500) + end + + defp default_missing_graph_policy do + Application.get_env(:jido_code, :source_code_graph_auto_refresh_missing_graph_policy, :skip) + end + + defp default_max_refresh_attempts do + Application.get_env(:jido_code, :source_code_graph_auto_refresh_max_attempts, 1) + end +end diff --git a/lib/jido_code/source_code_graph/view_model.ex b/lib/jido_code/source_code_graph/view_model.ex index 11078c49..f5fe1b39 100644 --- a/lib/jido_code/source_code_graph/view_model.ex +++ b/lib/jido_code/source_code_graph/view_model.ex @@ -174,6 +174,7 @@ defmodule JidoCode.SourceCodeGraph.ViewModel do imported_revision: nil, current_revision: nil, latest_failure: nil, + refresh: refresh_state(nil), recovery_action: graph_recovery_action(nil, reason) } end @@ -205,6 +206,7 @@ defmodule JidoCode.SourceCodeGraph.ViewModel do imported_revision: Map.get(status_result, :imported_revision), current_revision: Map.get(status_result, :current_revision), latest_failure: normalize_failure(latest_failure), + refresh: refresh_state(Map.get(status_result, :source_graph_refresh)), recovery_action: graph_recovery_action(status_result, reason) } end @@ -327,4 +329,40 @@ defmodule JidoCode.SourceCodeGraph.ViewModel do message: Map.get(failure, :message) || Map.get(failure, "message") } end + + defp refresh_state(nil) do + refresh_state(%{}) + end + + defp refresh_state(refresh) when is_map(refresh) do + %{ + state: map_get(refresh, :state, "state", :idle), + auto_refresh_enabled?: map_get(refresh, :auto_refresh_enabled?, "auto_refresh_enabled?", false), + file_watcher_enabled?: map_get(refresh, :file_watcher_enabled?, "file_watcher_enabled?", false), + file_watcher_debounce_ms: map_get(refresh, :file_watcher_debounce_ms, "file_watcher_debounce_ms"), + file_watcher_max_pending_paths: + map_get(refresh, :file_watcher_max_pending_paths, "file_watcher_max_pending_paths"), + refresh_debounce_ms: map_get(refresh, :refresh_debounce_ms, "refresh_debounce_ms"), + refresh_max_coalesce_ms: map_get(refresh, :refresh_max_coalesce_ms, "refresh_max_coalesce_ms"), + refresh_max_pending_paths: map_get(refresh, :refresh_max_pending_paths, "refresh_max_pending_paths"), + missing_graph_policy: map_get(refresh, :missing_graph_policy, "missing_graph_policy", :skip), + max_refresh_attempts: map_get(refresh, :max_refresh_attempts, "max_refresh_attempts"), + refresh_queued?: map_get(refresh, :refresh_queued?, "refresh_queued?", false), + refresh_in_flight?: map_get(refresh, :refresh_in_flight?, "refresh_in_flight?", false), + pending_changed_paths: map_get(refresh, :pending_changed_paths, "pending_changed_paths", []), + last_source_change_at: map_get(refresh, :last_source_change_at, "last_source_change_at"), + last_refresh_started_at: map_get(refresh, :last_refresh_started_at, "last_refresh_started_at"), + last_refresh_completed_at: map_get(refresh, :last_refresh_completed_at, "last_refresh_completed_at"), + last_result: map_get(refresh, :last_result, "last_result"), + last_failure: map_get(refresh, :last_failure, "last_failure") + } + end + + defp map_get(map, atom_key, string_key, default \\ nil) when is_map(map) do + cond do + Map.has_key?(map, atom_key) -> Map.get(map, atom_key) + Map.has_key?(map, string_key) -> Map.get(map, string_key) + true -> default + end + end end diff --git a/lib/jido_code/workbench/project_semantic_inspection.ex b/lib/jido_code/workbench/project_semantic_inspection.ex index 4822b85e..579e0732 100644 --- a/lib/jido_code/workbench/project_semantic_inspection.ex +++ b/lib/jido_code/workbench/project_semantic_inspection.ex @@ -25,6 +25,16 @@ defmodule JidoCode.Workbench.ProjectSemanticInspection do imported_revision: nil, current_revision: nil, latest_failure: nil, + refresh: %{ + state: :idle, + auto_refresh_enabled?: false, + refresh_queued?: false, + refresh_in_flight?: false, + last_source_change_at: nil, + last_refresh_started_at: nil, + last_refresh_completed_at: nil, + last_failure: nil + }, recovery_action: :none } @@ -307,10 +317,39 @@ defmodule JidoCode.Workbench.ProjectSemanticInspection do defp semantic_notice_kind(graph), do: ProductFeedback.notice_kind(graph) - defp hint_detail(%{graph: graph, error: error}), do: ProductFeedback.for_graph(graph, error).detail + defp hint_detail(%{graph: graph, error: error}) do + refresh_hint_detail(graph) || ProductFeedback.for_graph(graph, error).detail + end + defp hint_detail(_status), do: "Semantic repository state is unavailable." - defp hint_remediation(graph), do: ProductFeedback.for_graph(graph).remediation + defp hint_remediation(graph), do: refresh_hint_remediation(graph) || ProductFeedback.for_graph(graph).remediation + + defp refresh_hint_detail(%{refresh: %{last_failure: last_failure}}) when is_map(last_failure) do + "Background source-code graph refresh failed after a source save." + end + + defp refresh_hint_detail(%{refresh: %{refresh_in_flight?: true}}) do + "Background source-code graph refresh is running after a source save." + end + + defp refresh_hint_detail(%{refresh: %{refresh_queued?: true}}) do + "Background source-code graph refresh is queued after a source save." + end + + defp refresh_hint_detail(_graph), do: nil + + defp refresh_hint_remediation(%{refresh: %{last_failure: last_failure}}) when is_map(last_failure) do + "Review semantic graph status and run recovery if refresh keeps failing." + end + + defp refresh_hint_remediation(%{refresh: %{refresh_in_flight?: true}}), + do: "Refresh status will update when analysis completes." + + defp refresh_hint_remediation(%{refresh: %{refresh_queued?: true}}), + do: "Refresh status will update after the debounce window." + + defp refresh_hint_remediation(_graph), do: nil defp recovery_feedback(%{status: :source_code_graph_recovery_not_needed}) do %{ diff --git a/mix.exs b/mix.exs index fe09df15..4112baff 100644 --- a/mix.exs +++ b/mix.exs @@ -173,6 +173,7 @@ defmodule JidoCode.MixProject do {:swoosh, "~> 1.16"}, {:heroicons, github: "tailwindlabs/heroicons", tag: "v2.2.0", sparse: "optimized", app: false, compile: false, depth: 1}, + {:file_system, "~> 1.1"}, {:phoenix_live_reload, "~> 1.2", only: :dev}, # Observability & monitoring diff --git a/test/jido_code/phase_eighty_integration_test.exs b/test/jido_code/phase_eighty_integration_test.exs new file mode 100644 index 00000000..55c26ba1 --- /dev/null +++ b/test/jido_code/phase_eighty_integration_test.exs @@ -0,0 +1,231 @@ +defmodule JidoCode.PhaseEightyIntegrationTest do + # covers: architecture.source_code_graph_pod.explicit_actions_drive_analyze_load_refresh_and_query + # covers: architecture.source_code_graph_pod.graph_refresh_replaces_named_graph_coherently + # covers: architecture.source_code_graph_pod.stale_queries_and_failures_remain_bounded + # covers: architecture.agent_os_integration.repo_pod_singleton_when_enabled + # covers: package.jido_code.version_controlled_quality_surfaces + use JidoCode.DataCase, async: false + + alias JidoCode.Agents.RepoMonitor + alias JidoCode.AgentWorkspace + alias JidoCode.SourceCodeGraph.RefreshScheduler + + setup do + previous_source_graph = Application.get_env(:jido_code, :source_code_graph_enabled) + previous_auto_refresh = Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled) + previous_refresh_debounce = Application.get_env(:jido_code, :source_code_graph_refresh_debounce_ms) + previous_refresh_coalesce = Application.get_env(:jido_code, :source_code_graph_refresh_max_coalesce_ms) + + Application.put_env(:jido_code, :source_code_graph_enabled, true) + Application.put_env(:jido_code, :source_code_graph_auto_refresh_enabled, true) + Application.put_env(:jido_code, :source_code_graph_refresh_debounce_ms, 1) + Application.put_env(:jido_code, :source_code_graph_refresh_max_coalesce_ms, 1) + + on_exit(fn -> + restore_env(:source_code_graph_enabled, previous_source_graph) + restore_env(:source_code_graph_auto_refresh_enabled, previous_auto_refresh) + restore_env(:source_code_graph_refresh_debounce_ms, previous_refresh_debounce) + restore_env(:source_code_graph_refresh_max_coalesce_ms, previous_refresh_coalesce) + end) + + :ok + end + + describe "80.5 save-triggered source graph refresh" do + test "80.5.2.1 simulated watcher save refreshes the loaded graph and preserves stale-query semantics while running" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!("PhaseEighty.HumanBefore") + + on_exit(fn -> + RepoMonitor.stop_source_watcher(managed_repo_id) + RefreshScheduler.stop(managed_repo_id) + end) + + assert {:ok, _load_result} = AgentWorkspace.load_source_code_graph(managed_repo_id, workspace_path) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: blocking_refresh_fun(self()), + refresh_debounce_ms: 1, + refresh_max_coalesce_ms: 1 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + rewrite_workspace_module!(workspace_path, "PhaseEighty.HumanAfter") + + assert {:ok, _watcher_pid} = + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + start_file_system?: false, + debounce_ms: 1 + ) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, "lib/example_workspace.ex"), + [:modified], + :human_watcher + ) + + assert_receive {:workspace_source_changed, %{event_source: :human_watcher}}, 200 + assert_receive {:refresh_started, refresh_task, %{event_source: :human_watcher}}, 500 + + assert {:error, :source_code_graph_stale, _message} = + AgentWorkspace.query_source_code_graph( + managed_repo_id, + workspace_path, + "SELECT * WHERE { ?s ?p ?o }" + ) + + assert {:ok, stale_query} = + AgentWorkspace.query_source_code_graph( + managed_repo_id, + workspace_path, + "SELECT * WHERE { ?s ?p ?o } LIMIT 5", + allow_stale?: true + ) + + assert stale_query.degraded? == true + assert stale_query.stale_graph? == true + + send(refresh_task, :release_refresh) + assert_receive {:refresh_completed, {:ok, refresh_result}}, 5_000 + assert refresh_result.latest_import_status.ready? == true + + assert {:ok, refreshed_status} = AgentWorkspace.source_code_graph_status(managed_repo_id, workspace_path) + assert refreshed_status.ready? == true + assert refreshed_status.stale? == false + assert refreshed_status.imported_revision == refreshed_status.current_revision + + assert {:ok, modules_after} = + AgentWorkspace.find_source_code_graph_modules( + managed_repo_id, + workspace_path, + module_name_contains: "PhaseEighty" + ) + + assert module_present?(modules_after, "PhaseEighty.HumanAfter") + refute module_present?(modules_after, "PhaseEighty.HumanBefore") + end + + test "80.5.2.2 product LLM write notification follows the same refresh path" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!("PhaseEighty.LlmBefore") + + on_exit(fn -> + RepoMonitor.stop_source_watcher(managed_repo_id) + RefreshScheduler.stop(managed_repo_id) + end) + + assert {:ok, _load_result} = AgentWorkspace.load_source_code_graph(managed_repo_id, workspace_path) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: blocking_refresh_fun(self()), + refresh_debounce_ms: 1, + refresh_max_coalesce_ms: 1 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + rewrite_workspace_module!(workspace_path, "PhaseEighty.LlmAfter") + + assert :ok = + AgentWorkspace.notify_workspace_source_changed( + managed_repo_id, + workspace_path, + "lib/example_workspace.ex", + event_source: :llm_write, + start_file_system?: false, + debounce_ms: 1 + ) + + assert_receive {:workspace_source_changed, %{event_source: :llm_write}}, 200 + assert_receive {:refresh_started, refresh_task, %{event_source: :llm_write}}, 500 + + send(refresh_task, :release_refresh) + assert_receive {:refresh_completed, {:ok, _refresh_result}}, 5_000 + + assert {:ok, refreshed_status} = AgentWorkspace.source_code_graph_status(managed_repo_id, workspace_path) + assert refreshed_status.ready? == true + assert refreshed_status.stale? == false + + assert {:ok, modules_after} = + AgentWorkspace.find_source_code_graph_modules( + managed_repo_id, + workspace_path, + module_name_contains: "PhaseEighty" + ) + + assert module_present?(modules_after, "PhaseEighty.LlmAfter") + refute module_present?(modules_after, "PhaseEighty.LlmBefore") + end + end + + defp blocking_refresh_fun(parent) do + fn managed_repo_id, workspace_path, event, _opts -> + send(parent, {:refresh_started, self(), event}) + + receive do + :release_refresh -> :ok + after + 5_000 -> :ok + end + + result = AgentWorkspace.refresh_source_code_graph(managed_repo_id, workspace_path) + send(parent, {:refresh_completed, result}) + result + end + end + + defp create_workspace_path!(module_name) do + workspace_path = + Path.join( + System.tmp_dir!(), + "jido_code_phase_eighty_#{System.unique_integer([:positive])}" + ) + + File.mkdir_p!(Path.join(workspace_path, "lib")) + + File.write!( + Path.join(workspace_path, "mix.exs"), + """ + defmodule PhaseEighty.MixProject do + use Mix.Project + + def project do + [app: :phase_eighty_example, version: "0.1.0", elixir: "~> 1.18", deps: []] + end + end + """ + ) + + rewrite_workspace_module!(workspace_path, module_name) + + on_exit(fn -> File.rm_rf!(workspace_path) end) + workspace_path + end + + defp rewrite_workspace_module!(workspace_path, module_name) do + File.write!( + Path.join(workspace_path, "lib/example_workspace.ex"), + """ + defmodule #{module_name} do + def greet(name) when is_binary(name), do: "hello " <> name + end + """ + ) + end + + defp module_present?(result, module_name) do + Enum.any?(Map.get(result, :bindings, []), fn row -> + get_in(row, ["module_name", :value]) == module_name + end) + end + + defp restore_env(key, nil), do: Application.delete_env(:jido_code, key) + defp restore_env(key, value), do: Application.put_env(:jido_code, key, value) +end diff --git a/test/jido_code/repo_monitor_source_watcher_test.exs b/test/jido_code/repo_monitor_source_watcher_test.exs new file mode 100644 index 00000000..1da1d09e --- /dev/null +++ b/test/jido_code/repo_monitor_source_watcher_test.exs @@ -0,0 +1,261 @@ +defmodule JidoCode.RepoMonitorSourceWatcherTest do + # covers: architecture.agent_os_integration.repo_pod_singleton_per_kernel + # covers: architecture.source_code_graph_pod.graph_revision_state_is_explicit_and_explainable + use JidoCode.DataCase, async: false + + alias JidoCode.Agents.RepoMonitor + alias JidoCode.AgentWorkspace + alias JidoCode.Forge.SpriteClient + alias JidoCode.SourceCodeGraph + + describe "source graph source scope" do + test "keeps watcher filtering aligned with graph revision inputs" do + workspace_path = create_workspace_path!() + + assert SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "mix.exs")) + assert SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "lib/example.ex")) + assert SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "lib/example_script.exs")) + assert SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "test/example_test.exs")) + assert SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "config/runtime.exs")) + + refute SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "lib/example.txt")) + refute SourceCodeGraph.source_file?(workspace_path, Path.join(workspace_path, "deps/package/lib/example.ex")) + + refute SourceCodeGraph.source_file?( + workspace_path, + Path.join(workspace_path, ".jido_code/source_code_graph/triple_store/generated.ex") + ) + + refute SourceCodeGraph.source_file?(workspace_path, Path.join(System.tmp_dir!(), "outside.ex")) + + assert Enum.all?(SourceCodeGraph.source_files(workspace_path), &SourceCodeGraph.source_file?(workspace_path, &1)) + end + end + + describe "repo monitor source watcher" do + test "publishes normalized source change events for source files" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + assert {:ok, _pid} = + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + start_file_system?: false, + debounce_ms: 1 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + changed_path = Path.join(workspace_path, "lib/example.ex") + assert :ok = RepoMonitor.notify_source_changed(managed_repo_id, changed_path, [:modified], :human_watcher) + + assert_receive {:workspace_source_changed, event}, 100 + + assert event.kind == :workspace_source_changed + assert event.managed_repo_id == managed_repo_id + assert event.workspace_path == Path.expand(workspace_path) + assert event.changed_paths == ["lib/example.ex"] + assert event.changed_path == "lib/example.ex" + assert event.file_events == [:modified] + assert event.event_source == :human_watcher + assert event.event_sources == [:human_watcher] + assert is_binary(event.current_revision) + assert %DateTime{} = event.observed_at + end + + test "coalesces source write bursts into one normalized event" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + assert {:ok, _pid} = + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + start_file_system?: false, + debounce_ms: 10 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, "lib/example.ex"), + [:modified], + :human_watcher + ) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, "config/runtime.exs"), + [:created], + :runtime_write + ) + + assert_receive {:workspace_source_changed, event}, 100 + + assert event.changed_paths == ["config/runtime.exs", "lib/example.ex"] + assert event.file_events == [:created, :modified] + assert event.event_source == :mixed + assert event.event_sources == [:human_watcher, :runtime_write] + refute_receive {:workspace_source_changed, _event}, 50 + end + + test "coalesces delete and rename-style source events while capping pending paths" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + assert {:ok, _pid} = + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + start_file_system?: false, + debounce_ms: 10, + max_pending_paths: 1 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, "lib/alpha.ex"), + [:deleted], + :human_watcher + ) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, "lib/beta.ex"), + [:renamed], + :human_watcher + ) + + assert_receive {:workspace_source_changed, event}, 100 + + assert event.changed_paths == ["lib/alpha.ex"] + assert event.file_events == [:deleted, :renamed] + assert event.event_source == :human_watcher + refute_receive {:workspace_source_changed, _event}, 50 + end + + test "ignores non-source files and graph store self writes" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + assert {:ok, _pid} = + RepoMonitor.ensure_source_watcher( + managed_repo_id, + workspace_path, + start_file_system?: false, + debounce_ms: 1 + ) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + assert :ok = + RepoMonitor.notify_source_changed( + managed_repo_id, + Path.join(workspace_path, ".jido_code/source_code_graph/triple_store/generated.ex"), + [:modified], + :human_watcher + ) + + refute_receive {:workspace_source_changed, _event}, 50 + end + + test "AgentWorkspace exposes product-owned source change notification" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + assert :ok = + AgentWorkspace.notify_workspace_source_changed( + managed_repo_id, + workspace_path, + "lib/example.ex", + event_source: :llm_write, + start_file_system?: false, + debounce_ms: 1 + ) + + assert_receive {:workspace_source_changed, event}, 100 + + assert event.changed_paths == ["lib/example.ex"] + assert event.event_source == :llm_write + assert event.event_sources == [:llm_write] + end + + test "SpriteClient write_file can notify source graph changes after successful writes" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = create_workspace_path!() + + on_exit(fn -> RepoMonitor.stop_source_watcher(managed_repo_id) end) + + Phoenix.PubSub.subscribe(JidoCode.PubSub, RepoMonitor.source_change_topic(managed_repo_id)) + + assert {:ok, client, sprite_id} = SpriteClient.create(%{base_dir: System.tmp_dir!()}) + + on_exit(fn -> + if Process.alive?(client.agent_pid), do: SpriteClient.destroy(client, sprite_id) + end) + + changed_path = Path.join(workspace_path, "lib/example.ex") + + assert :ok = + SpriteClient.write_file(client, changed_path, "defmodule Example do\n def changed, do: :ok\nend\n", + managed_repo_id: managed_repo_id, + workspace_path: workspace_path, + event_source: :tool_write, + start_file_system?: false, + debounce_ms: 1 + ) + + assert_receive {:workspace_source_changed, event}, 100 + + assert event.changed_paths == ["lib/example.ex"] + assert event.event_source == :tool_write + end + end + + defp create_workspace_path! do + workspace_path = + Path.join( + System.tmp_dir!(), + "jido_code_repo_monitor_source_watcher_#{System.unique_integer([:positive])}" + ) + + File.rm_rf!(workspace_path) + File.mkdir_p!(Path.join(workspace_path, "lib")) + File.mkdir_p!(Path.join(workspace_path, "test")) + File.mkdir_p!(Path.join(workspace_path, "config")) + File.mkdir_p!(Path.join(workspace_path, "deps/package/lib")) + File.mkdir_p!(Path.join(workspace_path, ".jido_code/source_code_graph/triple_store")) + + File.write!(Path.join(workspace_path, "mix.exs"), "defmodule Example.MixProject do\nend\n") + File.write!(Path.join(workspace_path, "lib/example.ex"), "defmodule Example do\nend\n") + File.write!(Path.join(workspace_path, "lib/example_script.exs"), "defmodule ExampleScript do\nend\n") + File.write!(Path.join(workspace_path, "test/example_test.exs"), "defmodule ExampleTest do\nend\n") + File.write!(Path.join(workspace_path, "config/runtime.exs"), "import Config\n") + File.write!(Path.join(workspace_path, "lib/example.txt"), "not source\n") + File.write!(Path.join(workspace_path, "deps/package/lib/example.ex"), "defmodule Ignored do\nend\n") + + on_exit(fn -> File.rm_rf!(workspace_path) end) + + workspace_path + end +end diff --git a/test/jido_code/source_code_graph_product_service_test.exs b/test/jido_code/source_code_graph_product_service_test.exs index 88708de7..36ec7b6c 100644 --- a/test/jido_code/source_code_graph_product_service_test.exs +++ b/test/jido_code/source_code_graph_product_service_test.exs @@ -6,6 +6,8 @@ defmodule JidoCode.SourceCodeGraphProductServiceTest do alias JidoCode.AgentWorkspace alias JidoCode.SourceCodeGraph.ProductService + alias JidoCode.SourceCodeGraph.RefreshScheduler + alias JidoCode.Workbench.ProjectSemanticInspection setup do previous = Application.get_env(:jido_code, :source_code_graph_enabled, false) @@ -57,6 +59,95 @@ defmodule JidoCode.SourceCodeGraphProductServiceTest do assert summary.groups.modules.status == :unavailable assert summary.groups.runtime_patterns.status == :unavailable end + + test "semantic status hints explain queued background refresh activity" do + previous_auto_refresh = Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled) + Application.put_env(:jido_code, :source_code_graph_auto_refresh_enabled, true) + + managed_repo_id = "repo-#{System.unique_integer()}" + workspace_path = create_workspace_path!() + + on_exit(fn -> + restore_env(:source_code_graph_auto_refresh_enabled, previous_auto_refresh) + RefreshScheduler.stop(managed_repo_id) + end) + + assert {:ok, _pod} = + AgentWorkspace.ensure_source_code_graph_pod( + managed_repo_id, + workspace_path, + start_file_system?: false + ) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: fn _managed_repo_id, _workspace_path, _event, _opts -> + {:ok, %{status: :graph_refreshed}} + end, + refresh_debounce_ms: 1_000 + ) + + assert :ok = RefreshScheduler.enqueue(source_change_event(managed_repo_id, workspace_path, ["lib/example.ex"])) + assert {:ok, _scheduler_status} = RefreshScheduler.status(managed_repo_id) + + project_like = %{ + managed_repo_id: managed_repo_id, + settings: %{ + "workspace" => %{ + "workspace_environment" => "local", + "workspace_path" => workspace_path + } + } + } + + assert %{} = hint = ProjectSemanticInspection.status_hint(project_like) + assert hint.detail == "Background source-code graph refresh is queued after a source save." + assert hint.remediation == "Refresh status will update after the debounce window." + end + end + + describe "status/3" do + test "projects background refresh activity without changing graph state" do + previous_auto_refresh = Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled) + Application.put_env(:jido_code, :source_code_graph_auto_refresh_enabled, true) + + managed_repo_id = "repo-#{System.unique_integer()}" + workspace_path = create_workspace_path!() + + on_exit(fn -> + restore_env(:source_code_graph_auto_refresh_enabled, previous_auto_refresh) + RefreshScheduler.stop(managed_repo_id) + end) + + assert {:ok, _pod} = + AgentWorkspace.ensure_source_code_graph_pod( + managed_repo_id, + workspace_path, + start_file_system?: false + ) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: fn _managed_repo_id, _workspace_path, _event, _opts -> + {:ok, %{status: :graph_refreshed}} + end, + refresh_debounce_ms: 1_000 + ) + + assert :ok = RefreshScheduler.enqueue(source_change_event(managed_repo_id, workspace_path, ["lib/example.ex"])) + assert {:ok, scheduler_status} = RefreshScheduler.status(managed_repo_id) + assert scheduler_status.refresh_queued? == true + + assert {:ok, status} = ProductService.status(managed_repo_id, workspace_path) + + assert status.graph.state == :not_ready + assert status.graph.refresh.auto_refresh_enabled? == true + assert status.graph.refresh.refresh_queued? == true + assert status.graph.refresh.refresh_in_flight? == false + assert status.graph.refresh.pending_changed_paths == ["lib/example.ex"] + end end describe "bounded lookup projections" do @@ -206,4 +297,21 @@ defmodule JidoCode.SourceCodeGraphProductServiceTest do """ ) end + + defp source_change_event(managed_repo_id, workspace_path, changed_paths) do + %{ + kind: :workspace_source_changed, + managed_repo_id: managed_repo_id, + workspace_path: workspace_path, + changed_paths: changed_paths, + changed_path: List.first(changed_paths), + file_events: [:modified], + event_source: :human_watcher, + event_sources: [:human_watcher], + observed_at: DateTime.utc_now() + } + end + + defp restore_env(key, nil), do: Application.delete_env(:jido_code, key) + defp restore_env(key, value), do: Application.put_env(:jido_code, key, value) end diff --git a/test/jido_code/source_code_graph_refresh_scheduler_test.exs b/test/jido_code/source_code_graph_refresh_scheduler_test.exs new file mode 100644 index 00000000..a5c9b8ac --- /dev/null +++ b/test/jido_code/source_code_graph_refresh_scheduler_test.exs @@ -0,0 +1,193 @@ +defmodule JidoCode.SourceCodeGraphRefreshSchedulerTest do + # covers: architecture.source_code_graph_pod.explicit_actions_drive_analyze_load_refresh_and_query + # covers: architecture.source_code_graph_pod.graph_refresh_replaces_named_graph_coherently + use ExUnit.Case, async: false + + alias JidoCode.SourceCodeGraph.RefreshScheduler + + setup do + previous_auto_refresh = Application.get_env(:jido_code, :source_code_graph_auto_refresh_enabled) + Application.put_env(:jido_code, :source_code_graph_auto_refresh_enabled, true) + + on_exit(fn -> + restore_env(:source_code_graph_auto_refresh_enabled, previous_auto_refresh) + end) + + :ok + end + + test "coalesces queued source changes into one refresh request" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = System.tmp_dir!() + parent = self() + + refresh_fun = fn _managed_repo_id, _workspace_path, event, _opts -> + send(parent, {:refresh, event}) + {:ok, %{status: :graph_refreshed}} + end + + on_exit(fn -> RefreshScheduler.stop(managed_repo_id) end) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: refresh_fun, + refresh_debounce_ms: 10 + ) + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/alpha.ex"])) + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["config/runtime.exs"])) + + assert_receive {:refresh, refresh_event}, 200 + + assert refresh_event.changed_paths == ["config/runtime.exs", "lib/alpha.ex"] + assert refresh_event.event_sources == [:human_watcher] + refute_receive {:refresh, _event}, 50 + + assert {:ok, status} = RefreshScheduler.status(managed_repo_id) + assert status.state in [:running, :succeeded] + end + + test "uses maximum coalescing window as an upper debounce bound" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = System.tmp_dir!() + parent = self() + + refresh_fun = fn _managed_repo_id, _workspace_path, event, _opts -> + send(parent, {:refresh, event}) + {:ok, %{status: :graph_refreshed}} + end + + on_exit(fn -> RefreshScheduler.stop(managed_repo_id) end) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: refresh_fun, + refresh_debounce_ms: 1_000, + refresh_max_coalesce_ms: 0 + ) + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/alpha.ex"])) + + assert_receive {:refresh, refresh_event}, 100 + assert refresh_event.changed_paths == ["lib/alpha.ex"] + end + + test "caps pending changed paths in scheduler status" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = System.tmp_dir!() + parent = self() + + refresh_fun = fn _managed_repo_id, _workspace_path, event, _opts -> + send(parent, {:refresh, event}) + {:ok, %{status: :graph_refreshed}} + end + + on_exit(fn -> RefreshScheduler.stop(managed_repo_id) end) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: refresh_fun, + refresh_debounce_ms: 1, + max_pending_paths: 1 + ) + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/alpha.ex", "lib/beta.ex"])) + + assert_receive {:refresh, refresh_event}, 100 + assert refresh_event.changed_paths == ["lib/alpha.ex"] + end + + test "runs one follow-up refresh when source changes arrive during an in-flight refresh" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = System.tmp_dir!() + parent = self() + + refresh_fun = fn _managed_repo_id, _workspace_path, event, _opts -> + send(parent, {:refresh_started, self(), event}) + + receive do + :release_refresh -> :ok + after + 1_000 -> :ok + end + + {:ok, %{status: :graph_refreshed}} + end + + on_exit(fn -> RefreshScheduler.stop(managed_repo_id) end) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: refresh_fun, + refresh_debounce_ms: 1 + ) + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/alpha.ex"])) + assert_receive {:refresh_started, first_task, first_event}, 200 + assert first_event.changed_paths == ["lib/alpha.ex"] + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/beta.ex"])) + send(first_task, :release_refresh) + + assert_receive {:refresh_started, second_task, second_event}, 300 + assert second_event.changed_paths == ["lib/beta.ex"] + send(second_task, :release_refresh) + end + + test "disabled auto refresh accepts source changes without starting scheduler work" do + Application.put_env(:jido_code, :source_code_graph_auto_refresh_enabled, false) + + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, System.tmp_dir!(), ["lib/alpha.ex"])) + assert {:error, :not_started} = RefreshScheduler.status(managed_repo_id) + end + + test "bounds retry attempts for failed refresh work" do + managed_repo_id = "repo-#{System.unique_integer([:positive])}" + workspace_path = System.tmp_dir!() + parent = self() + + refresh_fun = fn _managed_repo_id, _workspace_path, _event, _opts -> + send(parent, :refresh_attempt) + {:error, :source_code_graph_store_failed} + end + + on_exit(fn -> RefreshScheduler.stop(managed_repo_id) end) + + assert {:ok, _pid} = + RefreshScheduler.ensure_started( + managed_repo_id, + refresh_fun: refresh_fun, + refresh_debounce_ms: 1, + max_refresh_attempts: 2 + ) + + assert :ok = RefreshScheduler.enqueue(event(managed_repo_id, workspace_path, ["lib/alpha.ex"])) + + assert_receive :refresh_attempt, 100 + assert_receive :refresh_attempt, 100 + refute_receive :refresh_attempt, 50 + end + + defp event(managed_repo_id, workspace_path, changed_paths) do + %{ + kind: :workspace_source_changed, + managed_repo_id: managed_repo_id, + workspace_path: workspace_path, + changed_paths: changed_paths, + changed_path: List.first(changed_paths), + file_events: [:modified], + event_source: :human_watcher, + event_sources: [:human_watcher], + observed_at: DateTime.utc_now() + } + end + + defp restore_env(key, nil), do: Application.delete_env(:jido_code, key) + defp restore_env(key, value), do: Application.put_env(:jido_code, key, value) +end