Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
145 lines (116 loc) · 6.83 KB

ARCHITECTURE.md

File metadata and controls

145 lines (116 loc) · 6.83 KB
Raw

Architecture

A short map of the implementation. Module-level rustdoc inside each file is the source of truth; this document points to the right files for each concern. Only shipped phases (0–3) are documented in any detail.

Crates

synapse-core    →  graph engine. Phases 1–3 implemented (store, indexes, tx).
synapse-agent   →  typed Goal/Plan/Step/Outcome API. Skeleton; Phase 4.
synapse-cypher  →  Cypher subset parser. Skeleton; Phase 5.
synapse-proto   →  gRPC + HTTP wire types. Skeleton; Phase 8.
synapsed        →  out-of-process daemon binary. Skeleton; Phase 8.
synapse-mcp     →  MCP server adapter. Skeleton; Phase 8.
synapse-cli     →  `synapse` CLI. Skeleton; Phase 8/9.

synapse-core is the only crate with code worth reading right now. The others are intentional empty shells with module-level rustdoc that pins their scope so later phases land in a known shape.

synapse-core — kernel (Phase 1, M1)

crates/synapse-core/src/store/

  • arena.rs — slot-based arena. u32 internal index, opaque NodeId / RelId handles that pair a process-unique StoreId with a packed (generation, index) raw word. Two safety properties matter:
    • Cross-store rejection. Ids minted by store A cannot operate on store B even though both reuse low slot indices. Write paths reject with StoreError::WrongStore; read paths resolve to None / empty iter.
    • Stale-slot rejection. Generation is bumped with checked_add on every free; overflow retires the slot rather than reusing it.
  • tokens.rs — string interner shared by labels, rel types, and prop keys (FxHashMap<SmolStr, Id> + Vec<SmolStr>).
  • record.rs / mod.rs — NodeRecord, RelRecord, the intrusive per-node rel list, and the Store trait + InMemoryStore impl.
  • value.rs — heterogeneous Value (incl. FiniteF64, Vector, Bytes) and the IndexValue projection used by the property index, with a stable kind_tag-pinned ordering across enum variants.

The kernel is string-keyed end to end on its public surface — token ids are crate-private and have no public constructor or Deserialize impl, so a token from store A cannot be smuggled into store B.

The Phase 1 InMemoryStore is not Send + Sync; that bound lands with the tx layer in Phase 3, which owns its own concurrency model.

synapse-core::index — indexes (Phase 2, M2)

crates/synapse-core/src/index/

  • label.rs — always-on label index, FxHashMap<LabelId, FixedBitSet> over arena slot indexes (not full NodeIds — the bitset stays dense). scan_label reconstructs current-generation ids through the arena, so stale slots never leak.
  • btree.rs — opt-in property B-tree index keyed by (LabelId, PropKeyId, IndexValue). Buckets are SmallVec<[NodeId; 1]> so the unique-value-per-key case stays inline. Bucket order is unspecified — read paths sort before equality-comparing (the "sort-before-compare convention" referenced in the tx differential tests).

Index maintenance is synchronous and inline with every mutating store call. Mutations are prevalidate-then-mutate: if any index would reject the op (non-indexable existing value, NaN on a float index, …) the call errors and no state changes. That keeps per-record atomicity inside the kernel without paying for a writeahead log.

synapse-core::tx — MVCC (Phase 3, M3 — internal preview)

crates/synapse-core/src/tx/

The whole module is documented at the top of mod.rs; the summary:

  • Isolation. Snapshot isolation. A transaction sees its begin-snapshot view plus its own writes. Commits are rejected with TxError::StaleSnapshot if any commit landed between begin() and commit(); callers retry.
  • Mechanism. Single-writer / multi-reader. One writer thread owns a mutable Inner. On commit it clones Inner, applies the batch, wraps in a new Arc, and publishes through an ArcSwap<PublishedSnapshot>. Acquisition is a lock-free relaxed pointer load; publish is a single relaxed pointer store. Readers don't block writers and vice versa.
  • Atomicity layers. Per-record atomicity comes from the Phase 2 prevalidate-then-mutate contract. Per-batch atomicity comes from writer::try_apply holding the staged Inner on the stack and only publishing the finished Arc<Inner> via one ArcSwap::store — any panic during apply unwinds through staged's Drop before publish, so readers can never observe a partially-mutated commit.
  • Writer-thread death surfaces as WriterDisconnected, not silent progress.

Release-gate status

The commit_clone_cost bench measures ~468 µs against a 15 µs gate at the 10k-node row. M3 ships as internal preview; the record_step P99 = 50 µs headline budget is suspended until M3.5 (versioned tombstones with snapshot-filtered indexes) lands. The escalation order is locked in tx/mod.rs:

  1. ArcSwap<PublishedSnapshot> — already adopted, not the gate-mover.
  2. Per-field Arc<T> inside Inner + Arc::make_mut — useful intermediate, projected ~200–300 µs at the gate row, still over budget.
  3. M3.5 — versioned tombstones is the only design that reaches the gate at 10k+ nodes. Multi-week rewrite touching every storage type.

Multi-op transactions clone Inner twice at present: once on the tx's first write (for read-your-own-writes), once again in the writer's staged clone. Until M3.5, that doubling is a tracked follow-up; add a bench_tx_commit_cost to make it visible.

Commit log

CommitLog is a bounded in-memory ring (COMMIT_LOG_CAPACITY = 1024). M6 will frame the same records to a WAL; until then there is no consumer, so retaining every commit forever is a permanent leak — the ring drops the oldest entries silently. When M5/M6 wires a consumer, this becomes a bounded handoff queue with the same shape.

Test layout

crates/synapse-core/tests/:

  • index_proptest.rs — invariants on the label + B-tree index layers.
  • tx_proptest.rs — randomized commit / read interleavings.
  • tx_differential.rs — multi-tx histories vs. a model store, with the sort-before-compare convention.
  • tx_stress.rs — long-running write/read mixes.
  • tx_concurrent_snapshot.rs — reader / writer race surfaces.
  • tx_bug_fixes.rs — regression suite for issues found during M3 review.

Fuzz harness under crates/synapse-core/fuzz/ targets the arena's generation / cross-store id surface.

What's next

Phase 4 (M4) — the typed Goal / Plan / Step / Tool / Artifact / Outcome agent API on top of synapse-core, exposed by synapse-agent. See the module-level rustdoc in crates/synapse-agent/src/lib.rs for the intended shape. Subsequent phases (Cypher, persistence, vector recall, daemon + MCP, polish) are sequenced in the README; the design lives in internal planning notes that are not part of the public repo.