diff --git a/.gitignore b/.gitignore
index 6f5b75ea..298fbaeb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,3 +6,5 @@ logs/
pkg
.DS_Store
.claude
+bench/
+
diff --git a/CLAUDE.md b/CLAUDE.md
index eb56bbc0..f49d5e09 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -4,135 +4,176 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
-Sourced is an Event Sourcing / CQRS library for Ruby built around the "Decide, Evolve, React" pattern. It provides eventual consistency by default with an actor-like execution model for building event-sourced applications.
+Sourced is a Ruby library for **aggregateless, stream-less event sourcing**. Messages go into a flat, globally-ordered log (SQLite via Sequel). Consistency context is assembled dynamically by querying relevant facts via key-value pairs extracted from event payloads, rather than being pre-assigned to fixed streams. Reactors declare partition keys which the store uses to build query conditions and claim work.
## Core Architecture
-### Key Components
-- **Actors**: Classes that hold state, handle commands, produce events, and react to events (lib/sourced/actor.rb)
-- **Commands**: Intents to effect change in the system
-- **Events**: Facts describing state changes that have occurred
-- **Projectors**: React to events to build views, caches, or other representations (lib/sourced/projector.rb)
-- **Backends**: Storage adapters (ActiveRecord, Sequel, test backend) in lib/sourced/backends/
-- **Router**: Routes commands and events to appropriate handlers (lib/sourced/router.rb)
-- **Supervisor**: Manages background worker processes (lib/sourced/supervisor.rb)
+### Key abstractions
-### Message Flow
-Commands → Actors (Decide) → Events → Storage → Reactors (React) → New Commands
+- **Message** (`lib/sourced/message.rb`) — base class for all commands/events. No `stream_id` or `seq`; gets a global `position` when stored. Provides `causation_id` / `correlation_id`, `#correlate`, `#extracted_keys`, and a `Registry`. Subclasses: `Sourced::Command`, `Sourced::Event`.
+- **Store** (`lib/sourced/store.rb`) — SQLite-backed append-only log with key-pair indexing, consumer groups, scheduled messages, and stale-claim reaping. Returns `ReadResult`, `ClaimResult`, `ConsistencyGuard`, `PositionedMessage`, `Stats`, `OffsetsResult`, `ReadAllResult`.
+- **Reactor base classes** — all `extend Sourced::Consumer` and declare `partition_by :key` (+ other keys) to define their consistency boundary:
+ - `Decider` (`lib/sourced/decider.rb`) — handles commands, produces events via `event` helper.
+ - `Projector` (`lib/sourced/projector.rb`) — builds read models. Two flavors: `Projector::StateStored` and `Projector::EventSourced`.
+ - `DurableWorkflow` (`lib/sourced/durable_workflow.rb`) — long-running workflows with step memoisation via `durable`/`wait`/`context`/`execute` and `catch(:halt)`.
+ - Plain `Consumer` reactors (extend `Sourced::Consumer` directly) for side-effect-only handlers.
+- **Mixins**: `Sourced::Evolve` (state evolution from history), `Sourced::React` (event → command/event reactions), `Sourced::Sync` (post-append side effects).
+- **Router** (`lib/sourced/router.rb`) — registers reactors, dispatches claimed batches, manages consumer-group lifecycle hooks.
+- **Dispatcher / Worker / WorkQueue** (`lib/sourced/{dispatcher,worker,work_queue}.rb`) — claim-and-drain processing; signal-driven via `InlineNotifier` + `CatchUpPoller`.
+- **StaleClaimReaper** (`lib/sourced/stale_claim_reaper.rb`) — releases abandoned partition claims from dead workers via heartbeats.
+- **ScheduledMessagePoller** (`lib/sourced/scheduled_message_poller.rb`) — promotes due scheduled messages into the main log.
+- **Supervisor** (`lib/sourced/supervisor.rb`) — top-level process entry point wiring Dispatcher, executor, and reactors.
+- **CommandContext** (`lib/sourced/command_context.rb`) — builds commands from raw attributes; supports per-message and `any` hooks.
+- **Topology** (`lib/sourced/topology.rb`) — graph of reactors / message flows.
+- **Installer + migrations** (`lib/sourced/installer.rb`, `lib/sourced/migrations/`) — Sequel migration template for installing store tables.
+- **Falcon integration** (`lib/sourced/falcon/`) — `Environment` + `Service` for deferred post-fork setup.
-### Concurrency Model
-Sourced processes events by acquiring locks on `[reactor_group_id][stream_id]` combinations, ensuring sequential processing within streams while allowing concurrent processing across different streams.
+### Message flow
+
+`Command → Decider.decide → Events → Store.append → Router claims → Reactor.handle_claim → (Projections / Sync actions)`
+
+Reactions are **deferred**: a Decider's `react` blocks don't run inline with the command that produced the triggering event. When the Decider appends events, its own subscription (`handled_messages_for_react`) picks them up on the next claim cycle and runs the reaction in a separate `handle_batch`. Consequence: the originating command's `after_sync` commits as soon as its events commit, not after reactions finish. Trade-off: command and reactions are no longer in the same transaction — a failing reaction does not roll back the command.
+
+All reactors implement `.handle_claim(claim, history:)` and/or `.handle_batch(partition_values, new_messages, history:, replaying:)` with a uniform signature so GWT helpers and partial-ack logic work across types.
+
+### Partition-based consistency
+
+Reactors declare `partition_by :key1, :key2`. The store indexes every payload attribute into `sourced_key_pairs` at append time, and reads use AND-filtered conditions over these keys. `ConsistencyGuard` (returned by `read` / `claim_next`) detects conflicting appends via `messages_since(conditions, position)`.
## Development Commands
### Testing
-```bash
-# Run all tests (default rake task)
-rake
-# Run specific test file
-bundle exec rspec spec/actor_spec.rb
-
-# Run backend tests
-bundle exec rspec spec/backends/
+```bash
+# Full suite
+bundle exec rake
-# Run with specific database (PostgreSQL required for some tests)
-DATABASE_URL=postgres://localhost/sourced_test bundle exec rspec
+# Specific file
+bundle exec rspec spec/store_spec.rb
+bundle exec rspec spec/decider_spec.rb
```
-### Database Setup for Tests
-The gem supports multiple backends:
-- PostgreSQL (via Sequel or ActiveRecord)
-- SQLite (via Sequel or ActiveRecord)
-- In-memory test backend
+Tests use in-memory SQLite by default. `spec/store_spec.rb` is the central integration suite; `spec/testing/rspec_spec.rb` covers the GWT helpers in `lib/sourced/testing/rspec.rb`.
-Test databases are automatically created/cleared by the test suite.
+### Console
-### Console/IRB
```bash
-# Interactive console for experimentation
bin/console
```
-## Configuration Patterns
+## Configuration
-### Backend Configuration
```ruby
-# PostgreSQL via Sequel (default production setup)
Sourced.configure do |config|
- config.backend = Sequel.connect(ENV.fetch('DATABASE_URL'))
+ config.store = Sequel.sqlite('my_app.db') # auto-wraps in Sourced::Store
+ # or: config.store = Sourced::Store.new(db)
+ # or: any object matching Configuration::StoreInterface
+ config.worker_count = 2
+ config.batch_size = 50
+ config.catchup_interval = 5
+ config.claim_ttl_seconds = 120
end
-# Test backend (default, in-memory)
-Sourced.configure do |config|
- config.backend = Sourced::Backends::TestBackend.new
-end
-```
-
-### Registering Components
-```ruby
-# Register actors and projectors for background processing
-Sourced.register(SomeActor)
+Sourced.register(SomeDecider)
Sourced.register(SomeProjector)
```
-## Key DSL Patterns
+- `Sourced.configure` stores the block and calls `setup!`; re-runnable post-fork to re-establish DB connections (used by Falcon integration).
+- `Sourced.store`, `Sourced.router`, `Sourced.topology`, `Sourced.reset!` — module-level accessors.
+- `Sourced.handle!(ReactorClass, command)` — synchronous command dispatch (for web controllers): validates, loads history via partition read, decides, appends with guard, advances registered offsets. Returns `HandleResult(command, reactor, events)`.
+- `Sourced.load(ReactorClass, **partition_values)` — loads a reactor instance by evolving over AND-filtered partition history. Returns `[instance, read_result]`.
+
+## DSL Patterns
+
+### Decider
-### Actor Definition
```ruby
-class SomeActor < Sourced::Actor
- # Initial state factory
- state do |id|
- { id: id, status: 'new' }
- end
-
- # Command handler
- command :create_something, name: String do |state, cmd|
- event :something_created, cmd.payload
+class Courses < Sourced::Decider
+ partition_by :course_id
+
+ command CreateCourse do |_state, cmd|
+ event CourseCreated, course_id: cmd.payload.course_id, course_name: cmd.payload.course_name
end
-
- # Event handler (state evolution)
- event :something_created, name: String do |state, event|
- state[:name] = event.payload.name
+
+ event CourseCreated do |state, evt|
+ state[:course_id] = evt.payload.course_id
+ state[:name] = evt.payload.course_name
end
-
- # Reaction (workflow orchestration)
- reaction :something_created do |event|
- stream_for(event).command :next_step
+
+ reaction CourseCreated do |evt|
+ dispatch SendWelcomeEmail, course_id: evt.payload.course_id
end
end
```
-### Message Definitions
+### Message definition
+
```ruby
-# Expanded syntax for complex validation/coercion
-CreateLead = Sourced::Command.define('leads.create') do
- attribute :name, Types::String.present
- attribute :email, Types::Email.present
+CreateCourse = Sourced::Command.define('courses.create') do
+ attribute :course_id, Types::String.present
+ attribute :course_name, Types::String.present
end
-LeadCreated = Sourced::Event.define('leads.created') do
- attribute :name, String
- attribute :email, String
+CourseCreated = Sourced::Event.define('courses.created') do
+ attribute :course_id, String
+ attribute :course_name, String
end
```
-## Backend Implementation Notes
+`Sourced::Command` and `Sourced::Event` each have their own `Registry` (both reachable from `Sourced::Message.registry` via recursive lookup).
+
+### Projector flavors
+
+- `Projector::StateStored` — evolves only the claimed batch on top of the stored state snapshot.
+- `Projector::EventSourced` — evolves from full history every claim (via `context_for`).
+
+### Scheduled / delayed messages
+
+```ruby
+cmd = SendReminder.new(payload: { course_id: 'c1' }).at(Time.now + 3600)
+store.schedule_messages([cmd])
+store.update_schedule! # manual promotion (normally done by ScheduledMessagePoller)
+```
+
+In reactions: `dispatch(Cmd, ...).at(time)`.
+
+## Store API highlights
-- All backends must implement the BackendInterface defined in lib/sourced/configuration.rb
-- SequelBackend is the main production backend (lib/sourced/backends/sequel_backend.rb)
-- ActiveRecordBackend provides Rails integration (lib/sourced/backends/active_record_backend.rb)
-- TestBackend provides in-memory storage for testing (lib/sourced/backends/test_backend.rb)
+- `append(messages, guard: nil)` — writes + auto-indexes payload keys; raises `ConcurrentAppendError` if guard is violated.
+- `read(conditions, after_position:, limit:)` → `ReadResult(messages, guard)`.
+- `read_partition(partition_attrs, handled_types:)` — AND-filtered read for loading reactor state.
+- `read_all(after_position:, limit:, order: :asc, conditions: nil)` → `ReadAllResult` (lazy pagination via `to_enum`).
+- `claim_next(reactor, worker_id:)` → `ClaimResult` with partition batch + guard. Supports compound partitions and replaying flag.
+- `ack(claim, last_position:)` / `release(claim)` / `advance_offset(group_id, partition:, position:)`.
+- `register_consumer_group`, `start_consumer_group`, `stop_consumer_group`, `reset_consumer_group`.
+- `read_offsets(group_id:, limit:, from_id:)` → `OffsetsResult` (cursor-paginated, `to_enum`).
+- `stats` → `Stats(max_position, groups)` including `error_context`.
+- `worker_heartbeat` / `release_stale_claims` — claim liveness.
-## Testing Considerations
+## Testing
-- Use shared examples from spec/shared_examples/backend_examples.rb when testing backends
-- Time manipulation available via Timecop gem
-- Database isolation handled automatically per test
-- Concurrent testing patterns available for testing race conditions
+- `lib/sourced/testing/rspec.rb` provides GWT helpers (`given`/`when_`/`then_`) usable across all reactor types since `#handle_batch` has a uniform signature.
+- Shared store behaviour concentrated in `spec/store_spec.rb` (2400+ lines).
+- Durable workflow specs demonstrate the step-memoisation pattern.
## Error Handling
-- Default error strategy logs exceptions and stops consumer groups
-- Configurable retry/backoff strategies available
-- Consumer groups can be stopped/started programmatically via backend API
\ No newline at end of file
+- `error_strategy` on `Configuration` — configurable retry / backoff / fail. See `lib/sourced/error_strategy.rb`.
+- Consumer groups have `running` / `stopped` / `failed` states. `on_fail` fires on terminal failures.
+- `PartialBatchError` carries successfully-processed `action_pairs` plus the failing message so batches can be partially acked.
+
+## Key Files
+
+- Entrypoint: `lib/sourced.rb` (top-level API, `handle!`, `load`)
+- Store: `lib/sourced/store.rb` + `lib/sourced/installer.rb` + `lib/sourced/migrations/`
+- Reactors: `lib/sourced/{decider,projector,durable_workflow,consumer}.rb`
+- Mixins: `lib/sourced/{evolve,react,sync}.rb`
+- Dispatch: `lib/sourced/{dispatcher,worker,work_queue,stale_claim_reaper,scheduled_message_poller,inline_notifier}.rb`
+- Router/topology: `lib/sourced/{router,topology}.rb`
+- Messages: `lib/sourced/message.rb` (includes `QueryCondition`, `ConsistencyGuard`)
+- Falcon: `lib/sourced/falcon/{environment,service}.rb`
+- Testing: `lib/sourced/testing/rspec.rb`
+
+## Local scratch (untracked)
+
+`examples/app/` and `bench/*` are intentionally untracked (see commits `Untrack examples/app` / `Untrack bench files`). Files may exist locally for experimentation but must not be re-added to the repo without explicit approval.
diff --git a/Gemfile b/Gemfile
index fb62b510..4a9f830d 100644
--- a/Gemfile
+++ b/Gemfile
@@ -14,8 +14,6 @@ group :development do
end
group :test do
- gem 'dotenv'
- gem 'pg'
gem 'rspec', '~> 3.0'
gem 'sequel'
gem 'sqlite3'
diff --git a/Gemfile.lock b/Gemfile.lock
index bb1843f8..b7a50169 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -33,7 +33,6 @@ GEM
irb (~> 1.10)
reline (>= 0.3.8)
diff-lcs (1.5.1)
- dotenv (3.1.4)
fiber-annotation (0.2.0)
fiber-local (1.1.0)
fiber-storage
@@ -57,7 +56,6 @@ GEM
parser (3.3.7.1)
ast (~> 2.4.1)
racc
- pg (1.5.8)
plumb (0.0.17)
bigdecimal
concurrent-ruby
@@ -119,9 +117,7 @@ PLATFORMS
DEPENDENCIES
debug
docco!
- dotenv
logger
- pg
rake (~> 13.0)
rspec (~> 3.0)
rubocop
diff --git a/README.md b/README.md
index e15b9c86..901dfca0 100644
--- a/README.md
+++ b/README.md
@@ -1,1619 +1,1141 @@
-# sourced
+# Sourced — Stream-less Event Sourcing
-**WORK IN PROGRESS**
+Sourced is a Ruby library for aggregateless, stream-less event sourcing. Events go into a flat, globally-ordered log. Consistency context is assembled dynamically by querying relevant facts via key-value pairs extracted from event payloads, rather than being pre-assigned to fixed streams.
-Event Sourcing / CQRS library for Ruby.
-There's many ES gems available already. The objectives here are:
-* Cohesive and toy-like DX.
-* Eventual consistency by default. Actor-like execution model.
-* Low-level APIs for durable messaging.
-* Supports the [Decide, Evolve, React pattern](https://ismaelcelis.com/posts/decide-evolve-react-pattern-in-ruby/)
-* Control concurrency by modeling.
-* Simple to operate: it should be as simple to run as most Ruby queuing systems.
-* Explore ES as a programming model for Ruby apps.
+## Core Concepts
-A small demo app [here](https://github.com/ismasan/sourced_todo).
+- **No streams or aggregates** — all messages share a single append-only log with auto-increment positions.
+- **Partitioning by attributes** — reactors declare which payload attributes define their consistency boundary (e.g. `partition_by :course_id`). The store uses these to build query conditions and claim work.
+- **Decide → Evolve → React** — reactors handle commands, evolve state from history, and react to events.
+- **Optimistic concurrency** — reads return a `ConsistencyGuard` that detects conflicting writes at append time.
-## The programming model
+## Messages
-If you're unfamiliar with Event Sourcing, you can read this first: [Event Sourcing from the ground up, with Ruby examples](https://ismaelcelis.com/posts/event-sourcing-ruby-examples)
-For a high-level overview of the mental model, [read this](https://ismaelcelis.com/posts/2025-04-give-it-time/). Or the video version, [here](https://www.youtube.com/watch?v=EgUwnzUJHMA).
+Messages have no `stream_id` or `seq` — they get a global `position` when stored.
-The entire behaviour of an event-sourced app is described via **commands**, **events** and **reactions**.
+```ruby
+# Define base classes for your domain (optional but recommended)
+class MyEvent < Sourced::Event; end
+class MyCommand < Sourced::Command; end
-
+# Define typed messages with payload attributes
+CreateCourse = MyCommand.define('courses.create') do
+ attribute :course_id, String
+ attribute :course_name, String
+end
+CourseCreated = MyEvent.define('courses.created') do
+ attribute :course_id, String
+ attribute :course_name, String
+end
+```
-* **Commands** are _intents_ to effect some change in the state of the system. Ex. `Add cart item`, `Place order`, `Update email`, etc.
-* **Events** are produced after handling a command and they describe _facts_ or state changes in the system. Ex. `Item added to cart`, `order placed`, `email updated`. Events are stored and you can use them to build views ("projections"), caches and reports to support UIs, or other artifacts.
-* **Reactions** are blocks of code that run _after_ an event has been processed and can dispatch new commands in a workflow or automation.
-* **State** is whatever object you need to hold the current state of a part of the system. It's usually derived from past events, and it's just enough to interrogate the state of the system and make the next decision.
+### Message features
-## Actors
+- **Auto-generated UUIDs** for `id`, `causation_id`, and `correlation_id`
+- **Causal tracing** via `#correlate(other_message)` — sets `causation_id` and `correlation_id`
+- **Auto-indexed keys** — `#extracted_keys` returns `[["course_id", "c1"], ["course_name", "Algebra"]]` from payload attributes, used by the store to index messages for querying
+- **Registry** — `Sourced::Message.registry` indexes defined types. Use `.from(type: "courses.created", payload: {...})` to instantiate from a type string.
+- **Typed payloads** — Plumb/Types DSL for attribute coercion and validation
-### Overview
+## Store
-Actors are classes that encapsulate the full life-cycle of a concept in your domain, backed by an event stream. This includes loading state from past events and handling commands for a part of your system. They can also define reactions to their own events, or events emitted by other actors. This is a simple shopping cart actor.
+`Sourced::Store` is an SQLite-backed store (via Sequel) providing the flat message log, key-pair indexing, and consumer group management.
```ruby
-class Cart < Sourced::Actor
- # Define what cart state looks like.
- # This is the initial state which will be updated by applying events.
- # The state holds whatever data is relevant to decide how to handle a command.
- # It can be any object you need. A custom class instance, a Hash, an Array, etc.
- CartState = Struct.new(:id, :status, :items) do
- def total = items.sum { |it| it.price * it.quantity }
- end
-
- CartItem = Struct.new(:product_id, :price, :quantity)
-
- # This factory is called to initialise a blank cart.
- state do |id|
- CartState.new(id:, status: 'open', items: [])
- end
-
- # Define a command and its handling logic.
- # The command handler will be passed the current state of the cart,
- # and the command instance itself.
- # Its main job is to validate business rules and decide whether new events
- # can be emitted to update the state
- command :add_item, product_id: String, price: Integer, quantity: Integer do |cart, cmd|
- # Validate that this command can run
- raise "cart is not open!" unless cart.status == 'open'
- # Produce a new event with the same attributes as the command
- event :item_added, cmd.payload
- end
-
- # Define an event handler that will "evolve" the state of the cart by adding an item to it.
- # These handlers are also used to "hydrate" the initial state from Sourced's storage backend
- # when first handling a command
- event :item_added, product_id: String, price: Integer, quantity: Integer do |cart, event|
- cart.items << CartItem.new(**event.payload.to_h)
- end
-
- # Optionally, define how this actor reacts to the event above.
- # .reaction blocks can dispatch new commands that will be routed to their handlers.
- # This allows you to build workflows.
- reaction :item_added do |event|
- # Evaluate whether we should dispatch the next command.
- # Here we could fetch some external data or query that might be needed
- # to populate the new commands.
- # Here we dispatch a command to the same stream_id present in the event
- dispatch(:send_admin_email, product_id: event.payload.product_id)
- end
-
- # Handle the :send_admin_email dispatched by the reaction above
- command :send_admin_email, product_id: String do |cart, cmd|
- # maybe produce new events
- end
-end
-```
-
-Using the `Cart` actor in an IRB console. This will use Sourced's in-memory backend by default.
+require 'sequel'
-```ruby
-cart = Cart.new(id: 'test-cart')
-cart.state.total # => 0
-# Instantiate a command and handle it
-cmd = Cart::AddItem.build('test-cart', product_id: 'p123', price: 1000, quantity: 2)
-events = cart.decide(cmd)
-# => [Cart::ItemAdded.new(...)]
-cmd.valid? # true
-# Inspect state
-cart.state.total # 2000
-cart.items.items.size # 1
-# Inspect that events were stored
-cart.seq # 2 the sequence number or "version" in storage. Ie. how many commands / events exist for this cart
-# Append new messages to the backend
-Sourced.config.backend.append_to_stream('test-cart', events)
-# Load events for cart
-events = Sourced.history_for(cart)
-# => an array with instances of [Cart::AddItem, Cart::ItemAdded]
-events.map(&:type) # ['cart.add_item', 'cart.item_added']
+db = Sequel.sqlite('my_app.db')
+store = Sourced::Store.new(db)
+store.install! # creates tables (idempotent)
```
-Try loading a new cart instance from recorded events
+### Appending messages
```ruby
-cart2, events = Sourced.load(Cart, 'test-cart')
-cart2.seq # 2
-cart2.state.total # 2000
-cart2.state.items.size # 1
+cmd = CreateCourse.new(payload: { course_id: 'c1', course_name: 'Algebra' })
+position = store.append(cmd) # returns the assigned position
```
-### Registering actors
-
-Invoking commands directly on an actor instance works in an IRB console or a synchronous-only web handler, but for actors to be available to background workers, and to react to other actor's events, you need to register them.
+### Reading with query conditions
```ruby
-Sourced.register(Cart)
-```
+# Build conditions for a specific course
+conditions = CourseCreated.to_conditions(course_id: 'c1')
+# => [QueryCondition('courses.created', attrs: { course_id: 'c1' })]
-This achieves two things:
+# Read matching messages
+result = store.read(conditions)
+result.messages # => [PositionedMessage, ...]
+result.guard # => ConsistencyGuard (for optimistic concurrency)
+```
-1. Messages can be routed to this actor by background processes, using `Sourced.dispatch(message)`.
-2. The actor can _react_ to other events in the system (more on event choreography later), via its low-level `.handle(event)` [Reactor Interface](#the-reactor-interface).
+### Optimistic concurrency
-These two properties are what enables asynchronous, eventually-consistent systems in Sourced.
+```ruby
+result = store.read(conditions)
-### Expanded message syntax
+# ... later, append with conflict detection
+store.append(new_events, guard: result.guard)
+# raises Sourced::ConcurrentAppendError if conflicting messages
+# were appended after the read
+```
-Commands and event structs can also be defined separately as `Sourced::Command` and `Sourced::Event` sub-classes.
+### Delayed messages
-These definitions include a message _type_ (for storage) and payload attributes schema, if any.
+Any message can be stamped with a future time via `#at(time)`. Scheduled messages live in a separate `sourced_scheduled_messages` table and are promoted into the main log when their `available_at` passes.
```ruby
-module Carts
- # A command to add an item to the cart
- # Commands may come from HTML forms, so we use Types::Lax to coerce attributes
- AddItem = Sourced::Command.define('carts.add_item') do
- attribute :product_id, Types::Lax::Integer
- attribute :quantity, Types::Lax::Integer.default(1)
- attribute :price, Types::Lax::Integer.default(0)
- end
-
- # An event to track items added to the cart
- # Events are only produced by valid commands, so we don't
- # need validations or coercions
- ItemAdded = Sourced::Event.define('carts.item_added') do
- attribute :product_id, Integer
- attribute :quantity, Integer
- attribute :price, Integer
- end
-
- ## Now define command and event handlers in a Actor
- class Cart < Sourced::Actor
- # Initial state, etc...
-
- command AddItem do |cart, cmd|
- # logic here
- event ItemAdded, cmd.payload
- end
-
- event ItemAdded do |cart, event|
- cart.items << CartItem.new(**event.payload.to_h)
- end
- end
-end
-```
+cmd = SendReminder.new(payload: { course_id: 'c1' }).at(Time.now + 3600)
+store.schedule_messages([cmd])
-### `.command` block
+# Normally the ScheduledMessagePoller (started by the Dispatcher) promotes
+# due messages automatically. In tests or scripts, do it manually:
+store.update_schedule! # => number of messages promoted
+```
-The class-level `.command` block defines a _command handler_. Its job is to take a command (from a user, an automation, etc), validate it, and apply state changes by publishing new events.
+In a reaction handler, `dispatch(Cmd, ...).at(time)` uses this pipeline under the hood — see [Reactions](#reactions).
-
+### Partition reads
+`read_partition` uses AND semantics — a message is included only when every partition attribute it declares matches the given value.
```ruby
-command AddItem do |cart, cmd|
- # logic here...
- # apply and publish one or more new events
- # using instance-level #event(event_type, **payload)
- event ItemAdded, product_id: cmd.payload.product_id
-end
+result = store.read_partition(
+ { course_id: 'c1' },
+ handled_types: ['courses.created', 'courses.enrolled']
+)
```
+### Browsing the global log
+`read_all` paginates the entire message log, without requiring query conditions or partition attributes. It returns a `ReadAllResult` with `messages` and `last_position` (the current max position in the store), so clients know whether more pages exist.
-### `.event` block
+```ruby
+# First page (default limit: 50, ascending order)
+result = store.read_all(limit: 20)
+result.messages # => [PositionedMessage, ...]
+result.last_position # => 100 (max position in the store)
-The class-level `.event` block registers an _event handler_ used to _evolve_ the actor's internal state.
+# Next page — pass the last message's position as cursor
+result = store.read_all(from_position: result.messages.last.position, limit: 20)
-These blocks are used both to load the initial state when handling a command, and to apply new events to the state in command handlers.
+# Check if there are more pages
+has_more = result.messages.any? && result.messages.last.position < result.last_position
-
+# Destructuring is also supported
+messages, last_position = store.read_all(limit: 20)
+```
+Use `order: :desc` for reverse-chronological browsing (newest first). Pagination works the same way — `from_position` fetches messages *before* the given position.
```ruby
-event ItemAdded do |cart, event|
- cart.items << CartItem.new(**event.payload.to_h)
-end
-```
+result = store.read_all(order: :desc, limit: 20)
-These handlers are pure: given the same state and event, they should always update the state in the same exact way. They should never reach out to the outside (API calls, current time, etc), and they should never run validations. They work on events already committed to history, which by definition are assumed to be valid.
+# Next page of older messages
+result = store.read_all(from_position: result.messages.last.position, order: :desc, limit: 20)
+```
-### `.before_evolve` block
+#### Iterating all messages with `to_enum`
-The class-level `.before_evolve` block registers a callback that runs **before** each registered event handler during state evolution. This is useful for common logic that should run before all event handlers, such as updating timestamps or recording metadata.
+`ReadAllResult#to_enum` returns a lazy `Enumerator` that transparently fetches subsequent pages as you iterate, using the same `order` and `limit` from the original query.
```ruby
-class CartListings < Sourced::Projector::StateStored
- state do |id|
- { id:, items: [], updated_at: nil, seq: 0 }
- end
-
- # This block runs before any .event handler
- before_evolve do |state, event|
- state[:updated_at] = event.created_at
- state[:seq] = event.seq
- end
+# Iterate all messages in pages of 50
+store.read_all(limit: 50).to_enum.each do |msg|
+ puts "#{msg.position}: #{msg.type}"
+end
- event Cart::ItemAdded do |state, event|
- state[:items] << event.payload.to_h
- end
+# Works with Enumerable methods
+store.read_all(order: :desc, limit: 100).to_enum.map(&:type)
- event Cart::Placed do |state, event|
- state[:status] = :placed
- end
-end
+# Supports lazy enumeration — stops fetching pages once satisfied
+store.read_all(limit: 20).to_enum.lazy.select { |m|
+ m.type == 'courses.created'
+}.first(5)
```
-The `before_evolve` callback only runs for events that have a registered handler via the `.event` macro. If an event is not handled by this class, the callback is skipped for that event.
+## Reactors
-### `.reaction` block
+Sourced provides three reactor base classes that share the same lifecycle (claim → evolve → handle → append → ack) and the same consumer-group machinery:
-The class-level `.reaction` block registers an event handler that _reacts_ to events already published by this or other Actors.
+- [Deciders](#deciders) — handle commands, enforce invariants, produce events
+- [Projectors](#projectors) — build read models by consuming events
+- [Durable workflows](#durable-workflows) — imperative orchestrations whose steps are memoised across re-entries
-`.reaction` blocks can dispatch the next command in a workflow with the instance-level `#dispatch` helper.
+Any reactor can produce follow-up messages via [Reactions](#reactions), and run side effects via [Sync and after-sync blocks](#sync-and-after-sync-blocks).
-
+### Deciders
+Deciders handle commands, enforce invariants, and produce events. They rebuild state from event history before each decision.
```ruby
-reaction ItemAdded do |cart, event|
- # dispatch the next command to the event's stream_id
- dispatch(
- CheckInventory,
- product_id: event.payload.product_id,
- quantity: event.payload.quantity
- )
-end
-```
+class CourseDecider < Sourced::Decider
+ # Defines the consistency boundary
+ partition_by :course_name
-You can also dispatch commanda to _other_ streams. For example for starting concurrent workflows.
+ # Initial state factory (receives partition values hash)
+ state do |_partition_values|
+ { name_taken: false }
+ end
-```ruby
-# dispatch a command to a new custom-made stream_id
-dispatch(CheckInventory, event.payload).to("cart-#{Time.now.to_i}")
+ # Evolve state from events (rebuilds history)
+ evolve CourseCreated do |state, _event|
+ state[:name_taken] = true
+ end
-# Or use Sourced.new_stream_id
-dispatch(CheckInventory, event.payload).to(Sourced.new_stream_id)
+ # Command handler — enforce invariants, then produce events
+ command CreateCourse do |state, cmd|
+ raise "Course '#{cmd.payload.course_name}' already exists" if state[:name_taken]
-# Or start a new stream and dispatch commands to another actor
-dispatch(:notify, message: 'hello!').to(NotifierActor)
+ event CourseCreated,
+ course_id: cmd.payload.course_id,
+ course_name: cmd.payload.course_name
+ end
+end
```
-#### `.reaction` block with actor state
+#### Synchronous command handling
- `.reaction` blocks receive the actor state, which is derived by applying past events to it (same as when handling commands).
+`Sourced.handle!` loads history, runs the decider, appends the command + events, and advances consumer group offsets — all in one call. Designed for web controllers.
```ruby
-# Define an event handler to evolve state
-event ItemAdded do |state, event|
- state[:item_count] += 1
-end
+cmd = CreateCourse.new(payload: { course_id: 'c1', course_name: 'Algebra' })
+cmd, decider, events = Sourced.handle!(CourseDecider, cmd)
-# Now react to it and check state
-reaction ItemAdded do |state, event|
- if state[:item_count] > 30
- dispatch NotifyBigCart
- end
+if cmd.valid?
+ # Success — events were appended
+else
+ # Validation failure — cmd.errors has details
end
```
-#### `.reaction` with state for all events
+Raises `Sourced::ConcurrentAppendError` on conflicts, or `RuntimeError` on domain invariant violations (e.g. "Course already exists").
-If the event name or class is omitted, the `.reaction` macro registers reaction handlers for all events already registered for the actor with the `.event` macro, minus events that have specific reaction handlers defined.
-
-```ruby
-# wildcard reaction for all evolved events
-reaction do |state, event|
- if state[:item_count] > 30
- dispatch NotifyBigCart
- end
-end
-```
+#### CommandContext
-#### `.reaction` for multiple events
+`Sourced::CommandContext` is a factory for building Sourced commands from raw Hash attributes (e.g. HTTP params), injecting defaults like `metadata`. It mirrors `Sourced::CommandContext` but without `stream_id`, since Sourced messages are stream-less.
```ruby
-reaction ItemAdded, InventoryChecked do |state, event|
- # etc
-end
-```
+# In a web controller, build a context with shared metadata
+ctx = Sourced::CommandContext.new(
+ metadata: { user_id: session[:user_id] }
+)
-It also works with symbols, for messages that have been defined as symbols (ex `event :item_added`)
+# Build from a type string + payload hash (e.g. from JSON params)
+cmd = ctx.build(type: 'courses.create', payload: { course_id: 'c1', course_name: 'Algebra' })
+cmd.metadata[:user_id] # => session[:user_id]
-```ruby
-reaction :item_added, InventoryChecked do |state, event|
- # etc
-end
+# Or pass an explicit command class
+cmd = ctx.build(CreateCourse, payload: { course_id: 'c1', course_name: 'Algebra' })
```
-## Causation and correlation
-
-When a command produces events, or when an event makes a reactor dispatch a new command, the cause-and-effect relationship between these messages is tracked by Sourced in the form of `correlation_id` and `causation_id` properties in each message's metadata.
+String keys are automatically symbolized, so `ctx.build('type' => '...', 'payload' => { ... })` works too.
-
+##### Callback hooks (`on` and `any`)
+Subclass `CommandContext` and register class-level hooks to enrich or transform commands at build time — e.g. injecting session data or adding metadata from the request scope.
-This helps the system keep a full audit trail of the cause-and-effect behaviour of the entire system.
+- **`on(MessageClass, ...)`** — runs for one or more command types. Multiple `on` calls for the same class accumulate (all blocks run in registration order).
+- **`any`** — runs for all commands (multiple blocks allowed, executed in order)
-
+Both receive the `app` scope and the command, and must return the (possibly modified) command. `on` blocks run before `any` blocks. Blocks are evaluated in the context of the `CommandContext` instance, so they can call private helper methods defined on the subclass.
-## Background vs. foreground execution
+```ruby
+class AppCommandContext < Sourced::CommandContext
+ # Enrich a specific command with data from the app scope
+ on CreateCourse do |app, cmd|
+ cmd.with_payload(created_by: app.current_user.id)
+ end
-By default Sourced processes commands and events **asynchronously** through background workers. Each reactor picks up messages at its own pace — the system is eventually consistent.
+ # Same block for multiple command types
+ on EnrolStudent, DropStudent do |app, cmd|
+ cmd.with_metadata(campus: app.current_campus)
+ end
-Sometimes you need **synchronous, all-or-nothing** execution: a web request handler that must know the full outcome before responding, or a test that needs deterministic behaviour. `Sourced::Unit` provides this.
+ # Additional block for EnrolStudent — both blocks run in order
+ on EnrolStudent do |app, cmd|
+ cmd.with_metadata(enrolment_source: 'web')
+ end
-### `Sourced::Unit`
+ # Add metadata to every command
+ any do |app, cmd|
+ cmd.with_metadata(
+ request_id: app.request_id,
+ session_id: app.session_id
+ )
+ end
+end
+```
-A Unit wires one or more reactors together and runs the full command → event → reaction → command chain inside a **single backend transaction**, using breadth-first traversal of the message graph. If any step raises, the entire transaction rolls back.
+Pass the request-scoped `app` object at construction time:
```ruby
-unit = Sourced::Unit.new(
- OrderActor,
- PaymentActor,
- InventoryProjector,
- backend: Sourced.config.backend
+# In a web controller
+ctx = AppCommandContext.new(
+ metadata: { user_id: session[:user_id] },
+ app: self # e.g. Sinatra app instance, Rack env wrapper, etc.
)
-cmd = PlaceOrder.new(stream_id: 'order-1', payload: { amount: 100 })
-results = unit.handle(cmd)
+cmd = ctx.build(type: 'courses.create', payload: { course_id: 'c1', course_name: 'Algebra' })
+cmd.metadata[:request_id] # => set by the `any` hook
```
-Messages produced by one reactor are immediately routed to any other reactor in the unit that handles them — no background workers needed.
+`app` defaults to `nil`, so existing callers without hooks are unaffected. Hooks are inherited by subclasses.
-#### Extracting results
-
-`Unit#handle` returns a `Results` object you can query per reactor class.
+Since blocks run in instance context, you can extract shared logic into private methods:
```ruby
-results = unit.handle(cmd)
+class AppCommandContext < Sourced::CommandContext
+ on CreateCourse do |app, cmd|
+ cmd.with_metadata(user_id: build_user_id(app))
+ end
-# Hash of { instance => [events] } for a given reactor
-results[OrderActor].each do |instance, events|
- puts "#{instance.id}: #{events.map(&:type)}"
-end
+ private
-# Flat list of events
-results.events_for(OrderActor)
-# => [OrderPlaced, ...]
+ def build_user_id(app)
+ "user-#{app.session_id}"
+ end
+end
```
-#### Skipping command persistence
+##### Scoping to a command subset
-By default every message (commands and events) is written to the store. Pass `persist_commands: false` to write only events.
+By default, `CommandContext` looks up types in `Sourced::Command.registry`. Pass a `scope:` to restrict lookups to a specific command subclass — attempts to build commands outside the scope raise `Sourced::UnknownMessageError`.
```ruby
-unit = Sourced::Unit.new(OrderActor, backend: backend, persist_commands: false)
-unit.handle(cmd) # only events are persisted; commands still flow through the chain
-```
+class PublicCommand < Sourced::Command; end
-This is useful when commands are transient intents that don't need an audit trail.
+CreateCourse = PublicCommand.define('courses.create') do
+ attribute :course_id, String
+ attribute :course_name, String
+end
-#### Loop detection
+# Only PublicCommand subclasses are allowed
+ctx = Sourced::CommandContext.new(scope: PublicCommand)
+ctx.build(type: 'courses.create', payload: { ... }) # OK
+ctx.build(type: 'admin.delete_all', payload: {}) # raises UnknownMessageError
+```
-The BFS traversal is capped at 100 iterations by default. If a reaction dispatches a command whose event triggers the same reaction, the unit raises `Sourced::Unit::InfiniteLoopError` before running away.
+#### Loading a decider's state
```ruby
-unit = Sourced::Unit.new(LoopyActor, backend: backend, max_iterations: 10)
-unit.handle(cmd)
-# => raises Sourced::Unit::InfiniteLoopError after 10 steps
+decider, read_result = Sourced.load(CourseDecider, course_name: 'Algebra')
+decider.state # => { name_taken: true }
```
-#### ACK tracking
+### Projectors
-After the BFS completes, the unit ACKs every handled message for each reactor's consumer group. This means background workers won't re-process messages that the unit already handled.
+Projectors consume events to build read models. Two flavours:
-#### When to use Unit vs. background workers
+#### EventSourced projector
-| | `Sourced::Unit` | Background workers |
-|---|---|---|
-| Consistency | Immediate (single transaction) | Eventual |
-| Failure mode | All-or-nothing rollback | Per-message retry / stop |
-| Concurrency | Single-threaded BFS | Concurrent across streams |
-| Use case | Web request handlers, tests, scripts | Long-running workflows, side-effects |
-
-You can combine both: use a Unit for the synchronous core of a request, while other reactors pick up the same events asynchronously in the background.
-
-### Actions
-
-Actions are the return values of reactor `.handle` methods. They tell the runtime (Unit or background worker) what side-effects to perform. Each persistable action class implements an `#execute(backend, source_message)` method that correlates messages and persists them via the backend.
-
-| Action | Description | `#execute` behaviour |
-|---|---|---|
-| `Actions::AppendAfter` | Append to a stream with optimistic locking (sequence check) | Correlate → `backend.append_to_stream` |
-| `Actions::AppendNext` | Append to stream(s), auto-incrementing sequence | Correlate → `backend.append_next_to_stream` per stream |
-| `Actions::Schedule` | Schedule messages for future delivery | Correlate → `backend.schedule_messages` |
-| `Actions::Sync` | Run a synchronous side-effect (cache write, API call) | Call the work block, return nil |
-| `Actions::OK` | Acknowledge the message (no persistence) | — |
-| `Actions::RETRY` | Tell the runtime to retry this message later | — |
-| `Actions::Ack` | ACK an arbitrary message by ID | — |
-
-`OK`, `RETRY`, and `Ack` are caller-specific signals — they don't implement `#execute`.
+Rebuilds state from full history on every batch (like deciders).
```ruby
-# Inside a reactor's .handle method:
-def self.handle(message)
- started = Order::Started.build(message.stream_id)
- [Sourced::Actions::AppendNext.new([started])]
-end
-```
-
-## Projectors
-
-Projectors react to events published by actors and update views, search indices, caches, or other representations of current state useful to the app. They can both react to events as they happen in the system, and also "catch up" to past events. Sourced keeps track of where in the global event stream each projector is.
-
-From the outside-in, projectors are classes that implement the _Reactor interface_.
+class CourseCatalogProjector < Sourced::Projector::EventSourced
+ partition_by :course_id
-Sourced ships with two ready-to-use projectors, but you can also build your own.
-
-### State-stored projector
+ state do |_partition_values|
+ { course_id: nil, course_name: nil, students: [] }
+ end
-A state-stored projector fetches initial state from storage somewhere (DB, files, API), and then after reacting to events and updating state, it can save it back to the same or different storage.
+ evolve CourseCreated do |state, event|
+ state[:course_id] = event.payload.course_id
+ state[:course_name] = event.payload.course_name
+ end
-```ruby
-class CartListings < Sourced::Projector::StateStored
- # Fetch listing record from DB, or new one.
- state do |id|
- CartListing.find_or_initialize(id)
+ evolve StudentEnrolled do |state, event|
+ state[:students] << event.payload.student_id
end
- # Evolve listing record from events
- event Carts::ItemAdded do |listing, event|
- listing.total += event.payload.price
+ # Sync block runs within the store transaction after evolving
+ sync do |state:, messages:, **|
+ next unless state[:course_id]
+ # Write projection to disk, database, cache, etc.
+ File.write("projections/#{state[:course_id]}.json", state.to_json)
end
- # Sync listing record back to DB
- sync do |state:, events:, replaying:|
- state.save!
+ # After-sync block runs after the transaction commits.
+ # Use for side effects that should only happen on successful commit
+ # (e.g. sending emails, HTTP calls, pushing to external queues).
+ after_sync do |state:, messages:, **|
+ NotificationService.notify("Course #{state[:course_name]} updated")
end
end
```
-### Event-sourced projector
+#### StateStored projector
-An event-sourced projector fetches initial state from past events in the event store, and then after reacting to events and updating state, it can save it to a DB table, a file, etc.
+Loads persisted state via the `state` block, evolves only new (unprocessed) messages.
```ruby
-class CartListings < Sourced::Projector::EventSourced
- # Initial in-memory state
- state do |id|
- { id:, total: 0 }
+class MyProjector < Sourced::Projector::StateStored
+ partition_by :course_id
+
+ state do |partition_values|
+ # Load existing state from your storage
+ existing = MyDB.find(partition_values[:course_id])
+ existing || { course_id: nil, students: [] }
end
- # Evolve listing record from events
- event Carts::ItemAdded do |listing, event|
- listing[:total] += event.payload.price
+ evolve StudentEnrolled do |state, event|
+ state[:students] << event.payload.student_id
end
- # Sync listing record to a file
- sync do |state:, events:, replaying:|
- File.write("/listings/#{state[:id]}.json", JSON.dump(state))
+ sync do |state:, messages:, **|
+ MyDB.upsert(state)
end
end
```
-### Registering projectors
-
-Like any other _reactor_, projectors need to be registered for background workers to route events to them.
-
-```ruby
-# In your app's configuration
-Sourced.register(CartListings)
-```
-
-### Reacting to events and scheduling the next command from projectors
-
-Sourced projectors can define `.reaction` handlers that will be called after evolving state via their `.event` handlers, in the same transaction.
+### Durable workflows
-This can be useful to implement TODO List patterns where a projector persists projected data, and then reacts to the data update using the data to schedule the next command in a workflow.
-
-
+`Sourced::DurableWorkflow` models long-running, side-effect-heavy workflows as an imperative `execute` method whose steps are memoised across re-entries. A workflow instance is identified by a `workflow_id` (auto-assigned) which doubles as its partition key.
+Subclassing `DurableWorkflow` auto-registers a full lifecycle event set (`WorkflowStarted`, `StepStarted`, `StepComplete`, `StepFailed`, `ContextUpdated`, `WaitStarted`, `WaitEnded`, `WorkflowComplete`, `WorkflowFailed`) under the class name.
```ruby
-class ReadyOrders < Sourced::Projector::StateStored
- # Fetch listing record from DB, or new one.
- state do |id|
- OrderListing.find_or_initialize(id)
+class GreetingTask < Sourced::DurableWorkflow
+ def execute(name)
+ ip = resolve_ip
+ location = geolocate(ip)
+ "Hello #{name}, your IP is #{ip} and its location is #{location}"
end
- event Orders::ItemAdded do |listing, event|
- listing.line_items << event.payload
- end
-
- # Evolve listing record from events
- event Orders::PaymentConfirmed do |listing, event|
- listing.payment_confirmed = true
+ durable def resolve_ip
+ IPResolver.resolve
end
- event Orders::BuildConfirmed do |listing, event|
- listing.build_confirmed = true
- end
-
- # Sync listing record back to DB
- sync do |state:, events:, replaying:|
- state.save!
- end
-
- # If a listing has both the build and payment confirmed,
- # automate dispatching the next command in the workflow
- reaction do |listing, event|
- if listing.payment_confirmed? && listing.build_confirmed?
- dispatch Orders::Release, **listing.attributes
- end
+ def geolocate(ip)
+ Geolocator.locate(ip)
end
+ # retry the step up to 3 times before failing the workflow
+ durable :geolocate, retries: 3
end
```
-Projectors can also define `.reaction event_class do |state, event|` to react to specific events, or `reaction event1, event2` to react to more than one event with the same block.
-
-### Skipping projector reactions when replaying events
+`durable` wraps a method so its result is memoised (via `StepComplete` events) and its failures are recorded (`StepFailed`). Re-entering `execute` replays past steps from memory instead of re-running their side effects. The `durable def foo; ...; end` form works because `def` returns the method's symbol.
-When a projector's offsets are reset (so that it starts re-processing events and re- building projections), Sourced skips invoking a projector's `.reaction` handlers. This is because building projections should be deterministic, and rebuilding them should not trigger side-effects such as automations (we don't want to call 3rd party APIs, send emails, or just dispatch the same commands over and over when rebuilding projections).
+#### Starting and observing a workflow
-To do this, Sourced keeps track of each consumer groups' highest acknowledged event sequence. When a consumer group is reset and starts re-processing past events, this sequence number is compared with each event's sequence, which tells us whether the event has been processed before.
-
-## Concurrency model
-
-Concurrency in Sourced is achieved by explicitely _modeling it in_.
-
-Sourced workers process messages by acquiring locks on `[reactor group ID][stream ID]`. For example `"CartActor:cart-123"`
+```ruby
+# Register the workflow so the Dispatcher drives it forward
+Sourced.register(GreetingTask)
-This means that all events for a given reactor/stream are processed in order, but events for different streams can be processed concurrently. You can define workflows where some work is done concurrently by modeling them as a collaboration of streams.
+# Start a new run — appends WorkflowStarted and returns a Waiter
+waiter = GreetingTask.execute('Ada', store: Sourced.store)
+waiter.workflow_id # => "workflow-"
-### Single-stream sequential execution
+# Block until the workflow reaches a terminal state (:complete or :failed)
+instance = waiter.wait(timeout: 30)
+instance.status # => :complete
+instance.output # => "Hello Ada, your IP is ... and its location is ..."
+```
-In the following (simplified!) example, a Holiday Booking workflow is modelled as a single stream ("Actor"). The infrastructure makes sure these steps are run sequentially.
+#### Rehydrating from the store
-
+```ruby
+instance = GreetingTask.load('workflow-abc-123')
+instance.status # => :started | :complete | :failed
+instance.context # => hash set via the `context` DSL
+```
-The Actor glues its steps together by reacting to events emitted by the previous step, and dispatching the next command.
+#### Initial context
```ruby
-class HolidayBooking < Sourced::Actor
- # State and details omitted...
-
- command :start_booking do |state, cmd|
- event :booking_started
- end
-
- reaction :booking_started do |event|
- dispatch :book_flight
+class IndexPages < Sourced::DurableWorkflow
+ context do
+ { visited: [] }
end
-
- command :book_flight do |state, cmd|
- event :flght_booked
- end
-
- reaction :flight_booked do |event|
- dispatch :book_hotel
+
+ def execute(urls)
+ urls.each { |u| fetch(u) }
end
-
- command :book_hotel do |state, cmd|
- event :hotel_booked
+
+ durable def fetch(url)
+ # ... returns parsed page
+ context[:visited] << url
end
-
- # Define event handlers if you haven't...
- event :booking_started, # ..etc
- event :flight_booked, # ..etc
end
```
-### Multi-stream concurrent execution
+The `context` block runs once per replay and seeds `#context`, which is persisted as `ContextUpdated` events whenever it changes.
-In this other example, the same workflow is split into separate streams/actors, so that Flight and Hotel bookings can run concurrently from each other. When completed, they each notify the parent Holiday actor, so the whole process coalesces into a sequential operation again.
+#### Waiting inside a workflow
-
+Inside `execute`, `wait(seconds)` suspends the run by appending `WaitStarted` with an `at:` time. The `ScheduledMessagePoller` resumes the workflow by promoting the corresponding `WaitEnded` when the time elapses.
```ruby
-# An actor dispatches a message to different stream
-# messages for different streams are processed concurrently
-reaction BookingStarted do |state, event|
- dispatch(BookHotel).to("#{event.stream_id}-hotel")
+def execute
+ notify_started
+ wait(300) # sleep for 5 minutes
+ notify_finished
end
```
-### Units of work
+### Reactions
-
+Both deciders and projectors can react to events to produce follow-up messages, enabling workflow orchestration. Reaction blocks queue messages via the `dispatch` helper rather than returning them — `dispatch` correlates each message with the triggering event, tags it with the reactor's `group_id` as `metadata[:producer]`, and returns a chainable `Dispatcher`.
-The diagram shows the units of work in an example Sourced workflow. The operations within each of the red boxes are protected by a combination of transactions and locking strategies on the consumer group + stream ID, so they are isolated from other concurrent processing. They can be said to be **immediately consistent**.
-The data-flow _between_ these boxes is propagated asynchronously by Sourced's infrastructure so, relative to each other, the entire system is **eventually consistent**.
-
-These transactional boundaries are guarded by the same locks that enforce the concurrency model, so that for example the same message can't be processed twice by the same Reactor (workflow, projector, etc).
+```ruby
+class EnrolmentDecider < Sourced::Decider
+ partition_by :course_id
-## Durable workflows
+ # ... evolve and command handlers ...
-There's a `Sourced::DurableWorkflow` class that can be subclassed to define Reactors with a synchronous-looking API. This is *work in progress*.
+ # Dispatch a follow-up message by class
+ reaction StudentEnrolled do |state, event|
+ dispatch(NotifyStudent, student_id: event.payload.student_id)
+ end
-```ruby
-class BookHoliday < Sourced::DurableWorkflow
- # This method can be called like a regular method
- # The methods inside also have blocking semantics
- # but they're in fact event-sourced, and will be
- # retried on failure until the booking completes.
- # Methods that were succesful will be idempotent on retry
- def execute(flight_info, hotel_info)
- flight = book_flight(flight_info)
- hotel = book_hotel(hotel_info)
- confirm_booking(flight, hotel)
+ # Dispatch multiple messages from one block
+ reaction OrderPlaced do |_state, event|
+ dispatch(ReserveInventory, order_id: event.payload.order_id)
+ dispatch(ChargePayment, order_id: event.payload.order_id)
end
-
- # The .durable macro turns a regular method
- # into an event-sourced workflow
- durable def book_flight(info)
- FlightsAPI.book(info)
+
+ # Chain .with_metadata and .at for metadata/delay
+ reaction StudentEnrolled do |_state, event|
+ dispatch(SendReminder, student_id: event.payload.student_id)
+ .with_metadata(channel: 'email')
+ .at(Time.now + 300)
end
-
- durable def book_hotel(info)
- HotelsAPI.book(info)
+
+ # Dispatch by symbol (resolved via the reactor's message registry)
+ reaction :student_enrolled do |_state, event|
+ dispatch(:notify_student, student_id: event.payload.student_id)
end
-
- durable def confirm_booking(flight, hotel)
- # etc,
+
+ # Wildcard: react to every evolve type without an explicit handler.
+ # Useful for side-channel pipelines (audit logs, outbox, etc.).
+ reaction do |_state, event|
+ dispatch(ForwardToOutbox, event_id: event.id)
end
end
```
-These executions will be handed off to the runtime to be run by one or more workers, while preserving ordering. You can optionally wait for a result.
+`dispatch` accepts either a message class or a symbol. Symbols are looked up via `self.class[symbol]` against the reactor's command/event registry and raise if unresolved. The returned `Dispatcher` supports:
-```ruby
-result = BookHoliday.execute(flight_info, hotel_info).wait.output
-# Confirmed booking, or whatever error result your code returns
-```
+- `.with_metadata(hash)` — merges into the message metadata (the `producer` key is already set).
+- `.at(time)` — stamps the message for delayed delivery (promoted by the `ScheduledMessagePoller`).
-Events for the full execution are recorded to the backend.
-
+Reactions are skipped during replay (when `replaying: true`), so side effects don't re-fire.
-Durable workflows must be registered with the runtime, like any other Reactor.
+### Sync and After-Sync Blocks
-```ruby
-Sourced.register BookHoliday
-```
+Both deciders and projectors support `sync` and `after_sync` blocks for running side effects during message processing.
-## Handler DSL
+- **`sync`** blocks run **inside** the store transaction, alongside event persistence and offset acknowledgement. Use them for writes that must be atomic with the event append (e.g. updating a database projection).
+- **`after_sync`** blocks run **after** the transaction commits. Use them for side effects that should only happen if the commit succeeds (e.g. sending emails, HTTP calls, pushing to external queues).
-The `Sourced::Handler` mixin provides a lighter-weight DSL for simple reactors.
+Both receive the same keyword arguments as the reactor's action-building step:
-```ruby
-class OrderTelemetry
- include Sourced::Handler
-
- # Handle these Order events
- # and log them
- on Order::Started do |event|
- Logger.info ['order started', event.stream_id]
- []
- end
-
- on Order::Placed do |event|
- Logger.info ['order placed', event.stream_id]
- []
- end
-end
+| Reactor type | Keyword arguments |
+|--------------|---------------------------------------|
+| Decider | `state:`, `messages:`, `events:` |
+| Projector | `state:`, `messages:`, `replaying:` |
-# Register it
-Sourced.register OrderTelemetry
-```
+```ruby
+class OrderDecider < Sourced::Decider
+ partition_by :order_id
-Handlers can optionally define the `:history` argument. The runtime will provide the full message history for the stream ID being handled.
+ # ... evolve / command handlers ...
-```ruby
-on Order::Placed do |event, history:|
- total = history
- .filter { |e| Order::ProductAdded === e }
- .reduce(0) { |n, e| n + e.payload.price }
-
- if total > 10000
- return [Order::AddDiscount.build(event.stream_id, amount: 100)]
+ sync do |state:, messages:, events:|
+ # Runs inside the transaction
+ OrderCache.update(state[:order_id], state)
end
-
- []
-end
-```
-
-It also supports multiple event types, for generic handling.
-```ruby
-on Order::Placed, Order::Complete do |event|
- Logger.info "received event #{event.inspect}"
- []
+ after_sync do |state:, messages:, events:|
+ # Runs after successful commit
+ Mailer.send_confirmation(state[:order_id]) if events.any? { |e| e.is_a?(OrderPlaced) }
+ end
end
```
-## Command methods for Actors
-
-The optional `Sourced::CommandMethods` mixin allows invoking an Actor's commands as regular methods.
-
-`CommandMethods` automatically generates instance methods from command definitions,
-allowing you to invoke commands in two ways:
+Multiple `sync` and `after_sync` blocks can be registered; they execute in registration order. Blocks are inherited by subclasses.
-1. **In-memory version** (e.g., `actor.start(name: 'Joe')`)
- - Validates the command and executes the decision handler
- - Returns a tuple of [cmd, new_events]
- - Does NOT persist events to backend
-2. **Durable version** (e.g., `actor.start!(name: 'Joe')`)
- - Same as in-memory, but also appends events to backend
- - Raises `FailedToAppendMessagesError` if backend fails
-
-Include the module in an Actor and define commands normally:
+## Registering reactors
```ruby
-class MyActor < Sourced::Actor
- include Sourced::CommandMethods
-
- command :create_item, name: String do |state, cmd|
- event :item_created, cmd.payload
- end
-end
-
-actor = MyActor.new(id: 'actor-123')
-cmd, events = actor.create_item(name: 'Widget') # In-memory
-cmd, events = actor.create_item!(name: 'Widget') # Persists to backend
+Sourced.register(CourseDecider)
+Sourced.register(EnrolmentDecider)
+Sourced.register(CourseCatalogProjector)
```
+This registers the reactor's consumer group with the store and adds it to the global router.
+## Background processing
-## Orchestration and choreography
+### Falcon (recommended)
-### Orchestration
-
-Orchestration is when the flow control of a multi-collaborator workflow is centralised into a single entity. This can be achieved by having one Actor coordinate the communication by reacting to events and sending commands to other actors.
+`Sourced::Falcon` provides a ready-made Falcon service that runs both the web server and Sourced background workers as sibling fibers. No separate worker process needed.
```ruby
-class HolidayBooking < Sourced::Actor
- state do |id|
- BookingState.new(id)
- end
-
- command StartBooking do |booking, cmd|
- # validations, etc
- event BookingStarted, cmd.payload
- end
-
- event BookingStarted
-
- # React to BookingStarted and start sub-workflows
- reaction BookingStarted do |booking, event|
- dispatch(HotelBooking::Start)
- end
-
- # React to events emitted by sub-workflows
- reaction HotelBooking::Started do |booking, event|
- dispatch(ConfirmHotelBooking, event.payload)
- end
-
- command ConfirmHotelBooking do |booking, cmd|
- unless booking.hotel.booked?
- event HotelBookingConfirmed, cmd.payload
- end
- end
-
- event HotelBookingConfirmed do |booking, event|
- # update booking state
- booking.confirm_hotel(event.payload)
- end
-end
-```
-
-This is a verbose step-by-step choreography, but it can be made more succint by ommiting the mirroring of commands/events, if needed (or by using the [Reactor Interface](#the-reactor-interface) directly).
-
-*TODO*: a way for Actors to initialise their internal state with event attributes other than the `stream_id`. For example, events may carry a `booking_id` for the overall workflow.
-
-### Choreography
+# falcon.rb
+#!/usr/bin/env falcon-host
+require_relative 'domain'
+require_relative 'app'
+require 'sourced/falcon'
-Choreography is when each component reacts to other components' events without centralised control. The overall workflow "emerges" from this collaboration.
+service "my-app" do
+ include Sourced::Falcon::Environment
+ include Falcon::Environment::Rackup
-```ruby
-class HotelBooking < Sourced::Actor
- # The HotelBooking defines its own
- # reactions to booking events
- reaction HolidayBooking::StartBooking do |state, event|
- # dispatch a command to itself to start its own life-cycle
- dispatch Start, event.payload
- end
-
- command Start do |state, cmd|
- # validations, etc
- # other Actors in the choreography
- # can choose to react to events emitted here
- event Started, cmd.payload
- end
-
- event Started do |state, event|
- # update state, etc
- end
+ url "http://localhost:9292"
+ count 1
end
```
-## Appending and reading messages
+Start with:
-### Appending messages without optimistic locking
+```bash
+bundle exec falcon host
+```
-Use `Backend#append_next_to_stream` to append messages to a stream, with no questions asked.
+The service automatically calls `Sourced.setup!` in each forked process, which replays the `Sourced.configure` block to create fresh database connections. This is necessary because SQLite connections are not fork-safe.
-```ruby
-message = ProductAdded.build('order-123', product_id: 123, price: 100)
-Sourced.config.backend.append_next_to_stream('order-123', [message])
+#### How it works
-# Shortcut:
-Sourced.dispatch(message)
-```
+- `Sourced::Falcon::Environment` — mixin that sets the `service_class` to `Sourced::Falcon::Service`. Include it in your Falcon service definition alongside `Falcon::Environment::Rackup`.
+- `Sourced::Falcon::Service` — extends `Falcon::Service::Server`. On `run`, it calls `Sourced.setup!`, starts the web server, and spawns a `Sourced::Dispatcher` with all settings from `Sourced.config`. On `stop`, it shuts down the dispatcher before the server.
+- No separate HouseKeeper fibers are needed — the `StaleClaimReaper` is embedded in the Sourced Dispatcher.
-### Appending messages with optimistic locking
+### Supervisor (standalone)
-Using `Backend#append_to_stream`, the backend expects the new messages `seq` property (sequence number) to be greater than the last message in storage for the same stream. This is to catch concurrent writes where a different client or thread may append to the stream while your code was preparing for it.
+For running workers without a web server, the supervisor starts workers that claim partitions, process messages, and ack offsets.
```ruby
-# Your code must make sure to increment sequence numbers
-past_events = Sourced.config.backend.read_stream('order-123')
-last_known_seq = past_events.last&.seq # ex. 10
-# Instantiate new messages and make sure to increment their sequences
-message = ProductAdded.new(
- stream_id: 'order-123',
- seq: last_known_seq + 1, # <== incremented sequence
- payload: { product_id: 123, price: 100 }
-)
+# Start blocking (handles INT/TERM signals for graceful shutdown)
+Sourced::Supervisor.start
-# This will raise an exception if there's already a message
-# for this stream with this sequence number in storage.
-Sourced.backend.append_to_stream('order-123', [message])
+# Or create and start manually
+supervisor = Sourced::Supervisor.new(
+ router: Sourced.router,
+ count: 4
+)
+supervisor.start
```
-`Sourced::Actor` classes do this incrementing automatically when they produce new messages.
+### How it works
-### Scheduling messages in the future
+1. **Store** appends messages and notifies listeners of new message types
+2. **Dispatcher** routes notifications to a `WorkQueue`, mapping message types to interested reactors
+3. **Workers** pop reactors from the queue, claim a partition via `Router#handle_next_for`, process messages, and ack
+4. **CatchUpPoller** periodically pushes all reactors as a safety net (handles missed notifications)
+5. **Store#schedule_messages** persists delayed messages in a separate `sourced_scheduled_messages` table keyed by `available_at`
+6. **ScheduledMessagePoller** runs on an interval and promotes any messages whose `available_at` is in the past into the main log
+7. **StaleClaimReaper** releases claims held by dead workers
-You can append messages to a separate log, with a schedule time. Sourced workers will periodically poll this log and move these messages into the main log at the right time.
-
-```ruby
-message = ProductAdded.build('order-123', product_id: 123, price: 100)
-Sourced.config.backend.schedule_messages([message], at: Time.now + 20)
-```
+### Router (direct usage)
-Actor reactions can use the `#dispatch` and `#at` helpers to schedule commands to run at a future time.
+The router can also be used directly for testing or scripting:
```ruby
-reaction ProductAdded do |order, event|
- dispatch(NotifyNewProduct).at(Time.now + 20)
-end
-```
+router = Sourced.router
-## Replaying messages
+# Process one batch for a specific reactor
+router.handle_next_for(CourseDecider)
-You can use the backend API to reset offsets for a specific consumer group, which will cause workers to start replaying messages for that group.
-
-```ruby
-Sourced.config.backend.reset_consumer_group(ReadyOrder)
+# Drain all pending work across all reactors
+router.drain
```
-See [below](#stopping-and-starting-consumer-groups) for other consumer lifecycle methods.
+## Failure handling and retries
-## The Reactor Interface
+Sourced already supports consumer-group retries on failure.
-All built-in Reactors (Actors, Projections) build on the low-level Reactor Interface.
+- On reactor errors, `Router#handle_next_for` calls the reactor's `on_exception` hook.
+- If a batch fails mid-way, it is raised as `Sourced::PartialBatchError`. The error carries the already-processed `action_pairs` (which are still ack'd) and the `failed_message` the hook receives — so partial progress is not lost.
+- By default, the hook uses `Sourced.config.error_strategy`.
+- The default `Sourced::ErrorStrategy` marks the consumer group as failed immediately.
+- If you configure a retrying error strategy, Sourced stores the next retry time in the consumer group's `retry_at` column and skips claiming work for that group until that time has passed.
-The runtime invokes `.handle_batch` on each reactor, passing a batch of `[message, replaying]` pairs from the same stream. The `Sourced::Consumer` module provides a default `handle_batch` that delegates to a per-message `.handle` method, so simple reactors only need to implement `.handle`.
+So retries are built in already, but they are opt-in via the error strategy configuration.
-```ruby
-class MyReactor
- extend Sourced::Consumer
-
- # The runtime will poll and hand over messages of this type
- def self.handled_messages = [Order::Started, Order::Placed]
-
- # The default handle_batch (from Consumer) calls this per message.
- # Return an Array of one or more Actions.
- def self.handle(new_message)
- actions = []
-
- # Just acknowledge new_message
- actions << Sourced::Actions::OK
-
- # Append these new messages to the event store
- # Sourced will automatically increment the stream's sequence number
- # (ie. no optimistic locking)
- started = Order::Started.build(new_message.stream_id)
- actions << Sourced::Actions::AppendNext.new([started])
-
- # Append these new messages to the event store.
- # The messages are expected to have a :seq incremented after new_message.seq
- # Messages will fail to append if other messages have been appended
- # with overlapping sequence numbers (optimistic locking)
- started = Order::Started.new(stream_id: new_message.stream_id, seq: new_message.seq + 1)
- actions << Sourced::Actions::AppendAfter.new(new_message.stream_id, [started])
-
- # Tell the runtime to retry this message
- actions << Sourced::Actions::RETRY
-
- actions
- end
-end
-```
-
-You can implement your own low-level reactors following the interface above. Then register them as normal.
+### Example: exponential backoff retries
```ruby
-Sourced.register MyReactor
-```
+require 'sourced'
-### Batch processing
+Sourced.configure do |c|
+ c.store = Sequel.sqlite('my_app.db')
-Workers fetch up to `worker_batch_size` messages per lock cycle (default: 50). Built-in reactors optimize batch processing automatically:
-
-- **Projector::StateStored**: loads state once, evolves all batch messages, syncs once (instead of N state loads + N syncs).
-- **Projector::EventSourced**: evolves history once, syncs once (instead of N x O(H) evolves + N syncs).
-- **Actor**: replaying messages return OK immediately; live messages are handled individually.
-
-For custom reactors, you can override `handle_batch` directly for full control:
-
-```ruby
-class MyBatchReactor
- extend Sourced::Consumer
+ c.error_strategy = Sourced::ErrorStrategy.new do |s|
+ s.retry(
+ times: 5,
+ after: 2,
+ backoff: ->(retry_after, retry_count) { retry_after * (2**(retry_count - 1)) }
+ )
- def self.handled_messages = [Order::Started, Order::Placed]
+ s.on_retry do |retry_count, exception, message, later|
+ LOGGER.warn(
+ "Sourced retry ##{retry_count} for #{message.type} (#{message.id}) " \
+ "at #{later}: #{exception.class}: #{exception.message}"
+ )
+ end
- # Override handle_batch for custom batch processing.
- # Must return Array of [actions, source_message] pairs.
- def self.handle_batch(batch)
- batch.map do |message, replaying|
- actions = process(message)
- [actions, message]
+ s.on_fail do |exception, message|
+ LOGGER.error(
+ "Sourced failing consumer group after retries for #{message.type} (#{message.id}): " \
+ "#{exception.class}: #{exception.message}"
+ )
end
end
end
```
-Individual reactors can override the global `worker_batch_size` via the `consumer` DSL:
-
-```ruby
-class OrderProjector < Sourced::Projector::StateStored
- consumer do |c|
- c.batch_size = 100
- end
-end
-```
+With the configuration above, failures retry after:
-When set, the reactor's `batch_size` takes precedence over the global `worker_batch_size`. When not set (default), the global value is used.
+- retry 1: 2 seconds
+- retry 2: 4 seconds
+- retry 3: 8 seconds
+- retry 4: 16 seconds
+- retry 5: 32 seconds
-#### Partial ACK on failure
+After the configured retries are exhausted, the consumer group is marked as failed.
-When a message raises mid-batch, Sourced ACKs up to the last successfully processed message instead of retrying the entire batch. The failed message and any subsequent messages are retried in the next batch. This is handled automatically by `each_with_partial_ack` in the Consumer module, which all built-in reactor types use.
+## Consumer groups
-**Actors and plain Consumer reactors** process each message independently (a new instance per message), so partial ACK is straightforward and safe with any batch size.
+Each reactor class is a consumer group. The store tracks per-partition offsets so multiple reactors process the same events independently.
-**Projectors** use an evolve-all-sync-once optimization: all batch messages are evolved into a single instance's state, then synced once. Reactions are processed one by one — if a reaction fails mid-batch, all previously successful messages (including their reactions and a correct partial sync) are ACKed, and only the failed message onward is retried. On partial failure, the projector automatically rebuilds a fresh instance with only the successfully processed messages to produce a correct sync (via the `on_partial_sync` block in `sync_and_react`).
+The lifecycle methods (`stop_consumer_group`, `start_consumer_group`, `reset_consumer_group`, `consumer_group_active?`) accept either a String group ID or any object responding to `#group_id` (e.g. a reactor class).
```ruby
-class PaymentProcessor < Sourced::Projector::StateStored
- consumer do |c|
- c.batch_size = 10
- end
+store = Sourced.store
- reaction PaymentStarted do |state, evt|
- # If this HTTP call succeeds for messages 1-3 but fails on message 4,
- # messages 1-3 are fully ACKed (reactions + sync). Message 4 onward is retried.
- response = PaymentGateway.post(:process_payment, state[:payment_info])
- if response.ok?
- dispatch ConfirmPayment, payment_id: response.body[:payment_id]
- else
- dispatch RejectPayment, errors: response.body[:errors]
- end
- end
-end
+# Pass reactor classes directly
+store.stop_consumer_group(CourseDecider)
+store.start_consumer_group(CourseDecider)
+store.reset_consumer_group(CourseDecider) # reprocess from beginning
+store.consumer_group_active?(CourseDecider) # => true/false
+
+# Or use plain strings
+store.stop_consumer_group('CourseApp::CourseDecider')
```
-### Reactors that require message history
+When retries are configured via `Sourced.config.error_strategy`, failed consumer groups remain active but paused until their `retry_at` time. Once that time passes, they become claimable again automatically.
-Reactors that declare the `:history` keyword in their `.handle_batch` (or `.handle`) signature will be provided the full message history for the stream being handled.
+### Lifecycle hooks via Router
-This is how event-sourced Actors are implemented.
+The Router provides lifecycle methods that wrap the Store operations and invoke optional callbacks on the reactor class. This lets reactors run cleanup or setup logic when their consumer group is stopped, reset, or started.
```ruby
-def self.handle(new_message, history:)
- # evolve state from history,
- # handle command, return new events, etc
- []
-end
-```
-
-### `:replaying` flag
+# Accept a reactor class or a string group_id
+Sourced.stop_consumer_group(CourseDecider, 'maintenance window')
+Sourced.reset_consumer_group(CourseDecider)
+Sourced.start_consumer_group(CourseDecider)
-Your `.handle` method can also declare a `:replaying` boolean, which tells the reactor whether the stream is replaying events, or handling new messages. Reactors use this to run or omit side-effects (for example, replaying Projectors don't run `reaction` blocks).
-
-```ruby
-def self.handle(new_message, history:, replaying:)
- if replaying
- # Omit side-effects
- else
- # Trigger side-effects
- end
-end
+# String group_id works too — the router resolves it to the registered class
+Sourced.stop_consumer_group('CourseApp::CourseDecider')
```
-## Testing
-
-There's a couple of experimental RSpec helpers that allow testing Sourced reactors in GIVEN, WHEN, THEN style.
+These delegate to `Router#stop_consumer_group`, `Router#reset_consumer_group`, and `Router#start_consumer_group`, which:
-*GIVEN* existing events A, B, C
-WHEN new command D is sent
-THEN I expect new events E and F
+1. Resolve the argument to a registered reactor class (raising `ArgumentError` if the string doesn't match any registered reactor)
+2. Call the corresponding `Store` method
+3. Invoke the reactor's callback (`on_stop`, `on_reset`, `on_start`)
-### Single reactor
+#### Defining callbacks
-Use `with_reactor` to unit-test the life-cycle of a single reactor.
+Override the no-op class methods on your reactor to hook into lifecycle events:
```ruby
-require 'sourced/testing/rspec'
+class CourseDecider < Sourced::Decider
+ partition_by :course_name
-RSpec.describe Order do
- include Sourced::Testing::RSpec
-
- it 'adds product to order' do
- with_reactor(Order, 'order-123')
- .when(Order::AddProduct, product_id: 1, price: 100)
- .then(Order::ProductAdded.build('order-123', product_id: 1, price: 100))
+ # Called when the consumer group is stopped.
+ # `message` is the optional reason string passed to stop_consumer_group.
+ def self.on_stop(message = nil)
+ Rails.logger.info "CourseDecider stopped: #{message}"
end
- it 'is a noop if product already in order' do
- with_reactor(Order, 'order-123')
- .given(Order::ProductAdded, product_id: 1, price: 100)
- .when(Order::AddProduct, product_id: 1, price: 100)
- .then([])
+ # Called when the consumer group is reset (offsets cleared).
+ def self.on_reset
+ Rails.cache.delete_matched('course_projections/*')
end
-end
-```
-
-`#then` can also take a block, which will be given the low level `Sourced::Actions` objects returned by your `.handle()` interface.
-You can use this block to test reactors that trigger side effects.
-
-```ruby
-with_reactor(Webhooks, 'webhook-1')
- .when(Webooks::Dispatch, name: 'Joe')
- .then do |actions|
- expect(api_request).to have_been_requested
+ # Called when the consumer group is started.
+ def self.on_start
+ Rails.logger.info 'CourseDecider started'
end
+end
```
-You can mix argument and block assertions with `.then()`
+Reactors without custom callbacks work fine — the defaults are no-ops.
-```ruby
-with_reactor(Webhooks, 'webhook-1')
- .when(Webooks::Dispatch, name: 'Joe')
- .then do |_|
- expect(api_request).to have_been_requested
- end
- .then(Webhooks::Dispatched, reference: 'webhook-abc')
-```
+## Monitoring
-For reactors that have `sync` blocks for side-effects (ex. Projectors), use `#then!` to trigger those side-effects and assert their results.
+`Store#stats` returns system-wide diagnostics for monitoring and debugging Sourced deployments.
```ruby
-with_reactor(PlacedOrders, 'order-123')
- .given(Order::Started)
- .given(Order::ProductAdded, product_id: 1, price: 100, units: 2)
- .given(Order::Placed)
- .then! do |_|
- expect(OrderRecord.find('order-123').total).to eq(200)
- end
+stats = store.stats
+stats.max_position # => 42 (latest position in the message log)
+stats.groups # => array of per-consumer-group hashes
```
-### Testing exceptions
+Each group hash contains:
-`#then` also accepts an exception class or instance, to assert that a command handler raises a specific error.
+| Key | Description |
+|----------------------|----------------------------------------------------------------|
+| `group_id` | Consumer group identifier (e.g. `"CourseDecider"`) |
+| `status` | `"active"`, `"stopped"`, or `"failed"` |
+| `retry_at` | `Time` of next retry, or `nil` |
+| `error_context` | Hash with error details (`{}` when healthy, see below) |
+| `oldest_processed` | `MIN(last_position)` across partitions where processing started |
+| `newest_processed` | `MAX(last_position)` across partitions |
+| `partition_count` | Number of offset rows (partitions) for this group |
-```ruby
-# Match on exception class
-with_reactor(Order, 'order-123')
- .given(Order::Placed)
- .when(Order::Place)
- .then(Order::AlreadyPlacedError)
-
-# Match on exception class and message
-with_reactor(Order, 'order-123')
- .given(Order::Placed)
- .when(Order::Place)
- .then(Order::AlreadyPlacedError.new('order already placed'))
-```
+### `error_context`
-When given an exception class, the test passes if any error of that type is raised. When given an exception instance, it also checks that the error message matches.
+The `error_context` hash is empty (`{}`) for healthy groups. When a group is stopped or has failed, it may contain:
-### Multiple reactors (A.K.A "Sagas")
+| Key | Present when | Description |
+|----------------------|--------------|--------------------------------------|
+| `:message` | Stopped | Operator-supplied reason for stopping |
+| `:exception_class` | Failed | Exception class name (e.g. `"RuntimeError"`) |
+| `:exception_message` | Failed | Exception message string |
-Use `with_reactors` to test the collaboration of multiple reactors sending and picking up eachother's messages.
+When retries are configured, `error_context` also accumulates retry state set by `GroupUpdater#retry_later`.
```ruby
-it 'tests collaboration of reactors' do
- order_stream = 'actor-1'
- payment_stream = 'actor-1-payment'
- telemetry_stream = Testing::Telemetry::STREAM_ID
-
- # With these reactors
- with_reactors(Order, Payment, Telemetry)
- # GIVEN that these events exist in history
- .given(Order::Started.build(order_stream, name: 'foo'))
- # WHEN I dispatch this new command
- .when(Order::StartPayment.build(order_stream))
- # Then I expect
- .then do |stage|
- # The different reactors collaborated and
- # left this message trail behind
- # Backend#messages is only available in the TestBackend
- expect(stage.backend.messages).to match_sourced_messages([
- Order::Started.build(order_stream, name: 'foo'),
- Order::StartPayment.build(order_stream),
- Order::PaymentStarted.build(order_stream),
- Telemetry::Logged.build(telemetry_stream, source_stream: order_stream),
- Payment::Process.build(payment_stream),
- Payment::Processed.build(payment_stream),
- Telemetry::Logged.build(telemetry_stream, source_stream: payment_stream),
- ])
- end
+stats = store.stats
+stats.groups.each do |g|
+ puts "#{g[:group_id]}: #{g[:status]} (#{g[:partition_count]} partitions, up to position #{g[:newest_processed]})"
+ if g[:status] == 'failed'
+ puts " error: #{g[:error_context][:exception_class]}: #{g[:error_context][:exception_message]}"
+ end
end
```
-`with_reactors` sets up its own in-memory backend, so you can test multi-reactor workflows in terms of what messages they produce without database or network requests, and there's no need for database setup or tear-down. Just test the behaviour!
+### `Store#read_offsets` — inspecting partition offsets
-The `.then` block can take an optional second argument, which will be passed as only the _new_ messages produced by the reactors, appended after any messages setup with `given`.
+`read_offsets` lists individual consumer group offsets with optional filtering and cursor-based pagination. Useful for inspecting the progress and claim status of each partition.
```ruby
-.then do |stage, new_messages|
- expect(new_messages).to match_sourced_messages([...])
-end
+result = store.read_offsets
+result.offsets # => array of offset hashes
+result.total_count # => total number of matching offsets (ignoring pagination)
```
+#### Parameters
+| Parameter | Type | Default | Description |
+|-------------|----------------|---------|----------------------------------------------------------|
+| `group_id:` | `String`, `nil` | `nil` | Filter by consumer group. `nil` returns all groups. |
+| `limit:` | `Integer` | `50` | Max offsets per page. |
+| `from_id:` | `Integer`, `nil`| `nil` | Cursor — return offsets with `id >= from_id` (inclusive). |
-## Setup
+#### Offset hash fields
-Sourced uses the Sequel gem for database access. It supports **PostgreSQL** (recommended for production) and **SQLite** (useful for development, scripts, and single-process apps).
+Each offset in the result is a Hash with:
-### PostgreSQL
+| Key | Type | Description |
+|------------------|---------------|------------------------------------------------------|
+| `:id` | `Integer` | Offset primary key (used as pagination cursor) |
+| `:group_name` | `String` | Consumer group identifier |
+| `:group_status` | `String` | `"active"`, `"stopped"`, or `"failed"` |
+| `:partition_key` | `String` | Partition identifier (e.g. `"device_id:dev-1"`) |
+| `:last_position` | `Integer` | Highest acked position for this partition |
+| `:claimed` | `Boolean` | Whether a worker currently holds this partition |
+| `:claimed_at` | `String`, `nil`| ISO8601 timestamp of the claim |
+| `:claimed_by` | `String`, `nil`| Worker ID holding the claim |
-You'll need the `pg` and `sequel` gems.
+#### Filtering by group
```ruby
-gem 'sourced', github: 'ismasan/sourced'
-gem 'pg'
-gem 'sequel'
+result = store.read_offsets(group_id: 'CourseDecider')
+result.offsets.each do |o|
+ puts "#{o[:partition_key]}: position #{o[:last_position]}, claimed=#{o[:claimed]}"
+end
```
-Create a Postgres database and configure the backend.
+#### Pagination
```ruby
-Sourced.configure do |config|
- config.backend = Sequel.connect(ENV.fetch('DATABASE_URL'))
-
- # Worker and housekeeping options (shown with defaults)
- config.worker_count = 2 # Number of worker fibers
- config.worker_batch_size = 50 # Messages fetched per lock cycle (batch processing)
- config.housekeeping_count = 1 # Number of housekeeper fibers
- config.housekeeping_interval = 3 # Seconds between scheduling cycles
- config.housekeeping_heartbeat_interval = 5 # Seconds between worker heartbeats
- config.housekeeping_claim_ttl_seconds = 120 # Seconds before stale claims are reaped
-end
+# First page
+page1 = store.read_offsets(limit: 20)
-Sourced.config.backend.install unless Sourced.config.backend.installed?
+# Next page using cursor
+page2 = store.read_offsets(limit: 20, from_id: page1.offsets.last[:id] + 1)
```
-Passing a `Sequel::Postgres::Database` connection auto-selects `PGBackend`, which supports PG-specific features: `LISTEN/NOTIFY` for real-time worker dispatch, advisory locks, and `FOR UPDATE SKIP LOCKED` for concurrent stream processing.
+#### Auto-pagination with `to_enum`
-These options are used by both `Sourced::Supervisor` and the Falcon integration. When running workers alongside a web server (Falcon, or any other Async-compatible server), these control how many worker and housekeeper fibers are spawned per OS process. The `worker_batch_size` controls how many messages from the same stream are fetched and processed in a single lock cycle (see [Batch processing](#batch-processing)).
-
-### SQLite
-
-You'll need the `sqlite3` and `sequel` gems.
+`OffsetsResult#to_enum` returns a lazy `Enumerator` that fetches subsequent pages automatically.
```ruby
-gem 'sourced', github: 'ismasan/sourced'
-gem 'sqlite3'
-gem 'sequel'
-```
-
-Configure with a Sequel SQLite connection.
-
-```ruby
-Sourced.configure do |config|
- # File-based database
- config.backend = Sequel.sqlite('myapp.db')
-
- # Or in-memory (useful for scripts and tests)
- # config.backend = Sequel.sqlite
+# Iterate all offsets in pages of 20
+store.read_offsets(limit: 20).to_enum.each do |offset|
+ puts "#{offset[:group_name]} / #{offset[:partition_key]}: #{offset[:last_position]}"
end
-Sourced.config.backend.install unless Sourced.config.backend.installed?
+# Works with Enumerable methods
+behind = store.read_offsets(limit: 50).to_enum.lazy.select { |o|
+ o[:last_position] < store.latest_position - 100
+}.to_a
```
-Passing a `Sequel::SQLite::Database` connection auto-selects `SQLiteBackend`. The SQLite backend sets up WAL mode and busy timeouts automatically.
-
-**Differences from PostgreSQL:**
-
-- **In-process pub/sub**: Uses an in-memory, thread/fiber-safe pub/sub (`PubSub::Test`) instead of PG `LISTEN/NOTIFY`. Worker dispatch is synchronous (the `InlineNotifier` pushes work to the `WorkQueue` inline when messages are appended), with the `CatchUpPoller` as a safety net.
-- No advisory locks or `SKIP LOCKED` — concurrency is handled via SQLite's transaction-level write locks.
-- Best suited for single-process deployments, development, scripts, and tests.
-
-See `examples/lite_cart.rb` for a complete working example.
-
-### Generating Sequel migrations
-
-If your app already uses Sequel's migrator, you can copy Sourced's migration into your migrations directory instead of using `backend.install`.
+#### Array destructuring
```ruby
-backend = Sourced.config.backend
-backend.copy_migration_to("db/migrations")
-# => writes db/migrations/001_create_sourced_tables.rb
+offsets, total_count = store.read_offsets(group_id: 'CourseDecider')
```
-Or use a block to control the file name (e.g. timestamped migrations):
+## Topology introspection
-```ruby
-backend.copy_migration_to do
- "db/migrations/#{Time.now.strftime('%Y%m%d%H%M%S')}_create_sourced_tables.rb"
-end
-```
-
-The generated file is a standard `Sequel.migration { change { ... } }` that works with `Sequel::Migrator`. It respects the `prefix` and `schema` options passed when configuring the backend:
+`Sourced.topology` analyses all registered reactors (deciders and projectors) and returns a flat array of node structs describing the message-flow graph. Nodes are `CommandNode`, `EventNode`, `AutomationNode` (for reactions), and `ReadModelNode` (for projectors). Each node has at least `type`, `id`, `name`, `group_id`, and depending on kind, `consumes` / `produces` / `schema`.
```ruby
-Sourced.configure do |config|
- db = Sequel.connect(ENV.fetch('DATABASE_URL'))
- config.backend = Sourced::Backends::SequelBackend.new(db, prefix: 'myapp', schema: 'events')
-end
+Sourced.register(CourseDecider)
+Sourced.register(EnrolmentDecider)
+Sourced.register(CourseCatalogProjector)
-# Migration will create tables like events.myapp_messages, events.myapp_streams, etc.
-Sourced.config.backend.copy_migration_to("db/migrations")
-```
-
-Register your Actors and Reactors.
-
-```ruby
-Sourced.register(Leads::Actor)
-Sourced.register(Leads::Listings)
-Sourced.register(Webooks::Dispatcher)
+Sourced.topology.each do |node|
+ puts "#{node.type}: #{node.name}"
+end
+# command: CourseApp::CreateCourse
+# event: CourseApp::CourseCreated
+# command: CourseApp::EnrolStudent
+# automation: reaction(CourseCreated)
+# readmodel: CourseApp::CourseCatalogProjector
+# ...
```
-### Running workers as a separate process
-
-When using a web server that doesn't share Sourced's Async event loop (e.g. Puma), or in non-web applications, run workers as a standalone process using `Sourced::Supervisor`:
+The graph is cached; `Sourced.register` invalidates the cache automatically, and you can force a rebuild with `Sourced.reset_topology`. Typical uses are generating [Event Modeling](https://eventmodeling.org/) diagrams, debugging "which reactor produces this event", or driving visual service-dependency tooling.
```ruby
-# worker.rb
-require_relative 'config/environment'
-# start workers with 10 worker fibers or threads per OS process
-# depending on Sourced.config.executor (:async, :thread, or custom)
-Sourced::Supervisor.start(count: 10)
+# Which commands produce the `courses.created` event?
+Sourced.topology
+ .select { |n| n.type == 'command' && n.produces.include?('courses.created') }
+ .map(&:name)
```
-This requires managing two processes in deployment: one for your web server, one for workers.
+`produces` / `consumes` are resolved statically by parsing reactor source with Prism, so they are only populated when the `prism` gem is available.
-### Running workers with Falcon
-
-If you use [Falcon](https://github.com/socketry/falcon) as your web server, you can run Sourced workers in the same process. Both Falcon and Sourced use the [Async](https://github.com/socketry/async) gem, so workers run as lightweight fibers alongside web requests — no separate worker process needed.
-
-This requires `Sourced.config.executor = :async` (the default). Do not change it to `:thread` when using Falcon, as workers must run as fibers to share Falcon's event loop.
+## Testing
-Add a `./falcon.rb` file to the root of your app, which requieres `sourced/falcon` (no hard dependency on Falcon in sourced.gemspec):
+Sourced ships with RSpec helpers for Given-When-Then testing of deciders and projectors. The helpers call `handle_batch` directly — no store, router, or consumer group setup needed.
```ruby
-# falcon.rb
-#!/usr/bin/env falcon-host
-require 'bundler/setup'
-require 'sourced/falcon'
-require_relative 'config/environment' # <= YOUR app setup, Sourced.configure, register reactors, etc.
+require 'sourced/testing/rspec'
-service "my-app" do
- include Sourced::Falcon::Environment
- include Falcon::Environment::Rackup # loads config.ru
-
- # -- Falcon / Async options --
- url "http://[::]:9292" # Server bind URL (default: "http://[::]:9292")
- count 2 # Number of OS processes to fork (default: Etc.nprocessors)
- timeout 30 # Connection timeout in seconds (default: nil)
- verbose false # Enable verbose logging (default: false)
- cache true # Enable HTTP response caching (default: false)
-
- # Sourced worker options default to Sourced.config values.
- # Override per-service if needed:
- # sourced_worker_count 4
- # sourced_worker_batch_size 50
- # sourced_housekeeping_count 1
- # sourced_housekeeping_interval 3
- # sourced_housekeeping_heartbeat_interval 5
- # sourced_housekeeping_claim_ttl_seconds 120
+RSpec.configure do |config|
+ config.include Sourced::Testing::RSpec
end
```
-Run with:
+### Testing deciders
-```
-falcon host
-```
+`with_reactor` takes a decider class and partition attributes, then chains `.given` (history), `.when` (command), and `.then` (expected outcomes).
-Total Sourced workers = `count * sourced_worker_count`. For example, `count 2` and `sourced_worker_count 4` gives 8 worker fibers across 2 OS processes, all competing for events via database locks (same as running multiple Supervisors).
-
-Set `config.worker_count = 0` to run Falcon as a web-only process with no Sourced workers. This is useful if you want to run workers separately via `Sourced::Supervisor` while still using Falcon for HTTP, or if you explicitely don't want workers adding unnecessary pressure on the database.
-
-On shutdown (`Ctrl-C` / `SIGTERM`), Falcon signals workers to stop. Their poll loops exit gracefully with no stale claims.
-
-### How worker dispatch works
-
-Instead of each worker polling the database independently, Sourced uses a signal-driven dispatch model. Workers block on a shared `WorkQueue` waiting for signals, and two sources feed that queue:
+```ruby
+RSpec.describe CourseDecider do
+ include Sourced::Testing::RSpec
-1. **Backend notifier** (real-time): The backend exposes a generic pub/sub notifier (`PGNotifier` for PostgreSQL, `InlineNotifier` for others). When messages are appended, the backend publishes a `messages_appended` event with the message types. When a stopped reactor is resumed, it publishes a `reactor_resumed` event with the consumer group ID. For PostgreSQL, these are delivered over PG `LISTEN/NOTIFY`; for other backends, they fire synchronously.
+ it 'creates a course' do
+ with_reactor(CourseDecider, course_name: 'Algebra')
+ .when(CreateCourse, course_id: 'c1', course_name: 'Algebra')
+ .then(CourseCreated, course_id: 'c1', course_name: 'Algebra')
+ end
-2. **CatchUpPoller** (safety net): A single fiber pushes all registered reactors into the WorkQueue every few seconds (default 5). This covers startup catch-up, missed notifications, offset resets, and PG reconnections.
+ it 'rejects duplicate course names' do
+ with_reactor(CourseDecider, course_name: 'Algebra')
+ .given(CourseCreated, course_id: 'c1', course_name: 'Algebra')
+ .when(CreateCourse, course_id: 'c2', course_name: 'Algebra')
+ .then(RuntimeError, "Course 'Algebra' already exists")
+ end
-The `Dispatcher` subscribes to the backend notifier and maps these events to reactor classes, pushing them onto the `WorkQueue`. For `messages_appended`, it resolves message types to interested reactors via an eager lookup table. For `reactor_resumed`, it resolves the group ID directly to the reactor class.
+ it 'produces no events for a no-op command' do
+ with_reactor(CourseDecider, course_name: 'Algebra')
+ .when(SomeNoopCommand, course_name: 'Algebra')
+ .then([])
+ end
+end
+```
-When a worker pops a reactor from the queue, it enters a **bounded drain loop**: it processes up to `max_drain_rounds` batches (default 10) for that reactor, then re-enqueues the reactor if it hit the cap. This ensures no single reactor monopolizes workers, and multiple workers can drain the same reactor concurrently on different streams (via `SKIP LOCKED`).
+#### Multiple expected messages
-The `WorkQueue` caps pending entries per reactor (equal to the worker count), so notification bursts are coalesced without queue bloat.
+When a decider produces events and reactions, pass all expected messages as instances:
-```
-Backend notifier ────┐
- (PG LISTEN or ├──▶ WorkQueue (capped/reactor) ──▶ Worker fibers
- inline pub/sub) │ │ │
-CatchUpPoller (5s) ──┘ │◀── re-enqueue ────────────┘
- (if max_drain_rounds hit)
+```ruby
+it 'produces event and reaction' do
+ with_reactor(EnrolmentDecider, course_id: 'c1')
+ .given(CourseCreated, course_id: 'c1', course_name: 'Algebra')
+ .when(EnrolStudent, course_id: 'c1', student_id: 's1')
+ .then(
+ StudentEnrolled.new(payload: { course_id: 'c1', student_id: 's1' }),
+ NotifyStudent.new(payload: { student_id: 's1' })
+ )
+end
```
-This design preserves natural back-pressure (workers only fetch when ready), eliminates polling-interval lag for new messages, and handles both real-time and catch-up work in a single operating mode.
+#### Block form
-## Custom attribute types and coercions.
-
-Define a module to hold your attribute types using [Plumb](https://github.com/ismasan/plumb)
+Pass a block to `.then` to receive the raw action pairs for custom assertions:
```ruby
-module Types
- include Plumb::Types
-
- # Your own types here.
- CorporateEmail = Email[/@apple\.com^/]
+it 'inspects action pairs' do
+ with_reactor(CourseDecider, course_name: 'Algebra')
+ .when(CreateCourse, course_id: 'c1', course_name: 'Algebra')
+ .then { |pairs|
+ actions, source_msg = pairs.first
+ append = Array(actions).find { |a| a.is_a?(Sourced::Actions::Append) }
+ expect(append.messages.first).to be_a(CourseCreated)
+ }
end
```
-Then you can use any [built-in Plumb types](https://github.com/ismasan/plumb?tab=readme-ov-file#built-in-types), as well as your own, when defining command or event structs (or any other data structures for your app).
+#### `.then!` — run sync and after_sync actions
+
+Use `.then!` instead of `.then` to execute both `sync` and `after_sync` actions before assertions:
```ruby
-UpdateEmail = Sourced::Command.define('accounts.update_email') do
- attribute :email, Types::CorporateEmail
+it 'runs sync block' do
+ with_reactor(CourseDecider, course_name: 'Algebra')
+ .when(CreateCourse, course_id: 'c1', course_name: 'Algebra')
+ .then! { |pairs| ... }
end
```
-## Error handling
-
-Sourced workflows are eventually-consistent by default. This means that commands and events are handled in background processes, and any exceptions raised can't be immediatly surfaced back to the user (and, there might not be a user anyway!).
+### Testing projectors
-Most "domain errors" in command handlers should be handled by the developer and recorded as domain events, so that the domain can react and/or compensate for them.
+Projectors use `.given` (events to evolve) and `.then` with a block that receives the projected state. `.when` is not supported — projectors don't handle commands.
-To handle true _exceptions_ (code or data bugs, network or IO exceptions) Sourced provides a default error strategy that will "stop" the affected consumer group (the Postgres backend will log the exception and offending message in the `consumer_groups` table).
-
-You can configure the error strategy with retries and exponential backoff, as well as `on_retry` and `on_stop` callbacks.
+#### StateStored
```ruby
-Sourced.configure do |config|
- # config.backend = Sequel.connect(ENV.fetch('DATABASE_URL'))
- config.error_strategy do |s|
- s.retry(
- # Retry up to 3 times
- times: 3,
- # Wait 5 seconds before retrying
- after: 5,
- # Custom backoff: given after=5, retries in 5, 10 and 15 seconds before stopping
- backoff: ->(retry_after, retry_count) { retry_after * retry_count }
- )
-
- # Trigger this callback on each retry
- s.on_retry do |n, exception, message, later|
- LOGGER.info("Retrying #{n} times")
- end
+RSpec.describe ItemProjector do
+ include Sourced::Testing::RSpec
- # Finally, trigger this callback
- # after all retries have failed and the consumer group is stopped.
- s.on_stop do |exception, message|
- Sentry.capture_exception(exception)
- end
+ it 'builds state from events' do
+ with_reactor(ItemProjector, list_id: 'L1')
+ .given(ItemAdded, list_id: 'L1', name: 'Apple')
+ .given(ItemAdded, list_id: 'L1', name: 'Banana')
+ .then { |state| expect(state[:items]).to eq(['Apple', 'Banana']) }
end
-end
-```
-### Custom error strategy
-
-You can also configure your own error strategy. It must respond to `#call(exception, message, group)`
-
-```ruby
-CUSTOM_STRATEGY = proc do |exception, message, group|
- case exception
- when Faraday::Error
- group.retry(Time.now + 10)
- else
- group.stop(exception)
+ it 'handles removal' do
+ with_reactor(ItemProjector, list_id: 'L1')
+ .given(ItemAdded, list_id: 'L1', name: 'Apple')
+ .and(ItemArchived, list_id: 'L1', name: 'Apple')
+ .then { |state| expect(state[:items]).to eq([]) }
end
-end
-Sourced.configure do |config|
- # Configure backend, etc
- config.error_strategy = CUSTOM_STRATEGY
+ it 'runs sync actions with then!' do
+ with_reactor(ItemProjector, list_id: 'L1')
+ .given(ItemAdded, list_id: 'L1', name: 'Apple')
+ .then! { |state| expect(state[:synced]).to be true }
+ end
end
```
-## Stopping and starting consumer groups.
+#### EventSourced
-`Sourced.config.backend` provides an API for stopping and starting consumer groups. For example to resume groups that were stopped by raised exceptions, after the error has been corrected.
+Same API — the helper creates an instance, evolves from all given messages, and yields state:
```ruby
-Sourced.config.backend.stop_consumer_group('Carts::Listings')
-Sourced.config.backend.start_consumer_group('Carts::Listings')
-```
-
-## Topology
-
-`Sourced.topology` returns a flat array of node structs describing the message flow graph of all registered reactors. This is useful for building visualizations, documentation, or tooling that needs to understand how commands, events, automations and read models connect.
-
-```ruby
-Sourced.register(Cart)
-Sourced.register(CartListings)
+RSpec.describe CatalogProjector do
+ include Sourced::Testing::RSpec
-nodes = Sourced.topology
-# => [CommandNode, EventNode, AutomationNode, ReadModelNode, ...]
+ it 'rebuilds state from full history' do
+ with_reactor(CatalogProjector, course_id: 'c1')
+ .given(CourseCreated, course_id: 'c1', course_name: 'Algebra')
+ .given(StudentEnrolled, course_id: 'c1', student_id: 's1')
+ .then { |state| expect(state[:students]).to eq(['s1']) }
+ end
+end
```
-The result is memoized. Call `Sourced.reset_topology` to clear the cache after registering new reactors.
-
-### Node types
+### Message matching
-#### CommandNode
+`.then` compares messages by **class** and **payload** only. Fields like `id`, `created_at`, `causation_id`, `correlation_id`, and `metadata` are ignored, so tests don't need to match auto-generated values.
-Represents a command handled by an actor. `produces` lists the event type strings that the command handler can emit (extracted via static analysis).
+## Full example
-```ruby
-# Fields: type, id, name, group_id, produces, schema
-{ type: "command", id: "carts.add_item", name: "Carts::AddItem",
- group_id: "Carts::Cart", produces: ["carts.item_added"],
- schema: { "type" => "object", "properties" => { ... } } }
-```
+See `examples/app/` for a complete Sinatra application with:
+- Two deciders (course creation with name uniqueness, student enrolment with capacity limits)
+- An event-sourced projector writing JSON files
+- Synchronous command handling via `Sourced.handle!` in HTTP endpoints
+- Background worker processing via Falcon
-#### EventNode
+## Setup & configuration
-Represents an event type. Deduplicated across reactors — the first reactor to reference an event owns its `group_id`.
+### Configuration
```ruby
-# Fields: type, id, name, group_id, produces, schema
-{ type: "event", id: "carts.item_added", name: "Carts::ItemAdded",
- group_id: "Carts::Cart", produces: [],
- schema: { "type" => "object", "properties" => { ... } } }
-```
+require 'sourced'
-#### AutomationNode
+Sourced.configure do |c|
+ # Pass a Sequel SQLite connection or a Sourced::Store instance
+ c.store = Sequel.sqlite('my_app.db')
-Represents a `.reaction` block. `consumes` lists what triggers the reaction (event types for actors, readmodel IDs for projectors). `produces` lists the command type strings dispatched by the reaction (extracted via static analysis).
-
-```ruby
-# Fields: type, id, name, group_id, consumes, produces
-# Actor reaction — consumes an event directly:
-{ type: "automation", id: "carts.item_added-Carts::Cart-aut",
- name: "reaction(Carts::ItemAdded)", group_id: "Carts::Cart",
- consumes: ["carts.item_added"], produces: ["carts.check_inventory"] }
-
-# Projector reaction — consumes the readmodel:
-{ type: "automation", id: "carts.item_added-Carts::CartListings-aut",
- name: "reaction(Carts::ItemAdded)", group_id: "Carts::CartListings",
- consumes: ["carts.cart_listings-rm"], produces: ["carts.notify_admin"] }
+ # Optional settings
+ c.worker_count = 4 # background worker fibers (default: 2)
+ c.batch_size = 50 # messages per claim (default: 50)
+ c.catchup_interval = 5 # seconds between catch-up polls (default: 5)
+ c.max_drain_rounds = 10 # max drain iterations per pickup (default: 10)
+ c.claim_ttl_seconds = 120 # stale claim threshold (default: 120)
+ c.housekeeping_interval = 30 # heartbeat/reap cycle (default: 30)
+end
```
-#### ReadModelNode
-
-Represents a projector as a consumer of events. `consumes` lists the event types the projector evolves. `produces` lists the IDs of any automation nodes derived from the projector's reactions.
-
-```ruby
-# Fields: type, id, name, group_id, consumes, produces, schema
-{ type: "readmodel", id: "carts.cart_listings-rm",
- name: "Carts::CartListings", group_id: "Carts::CartListings",
- consumes: ["carts.item_added", "carts.placed"],
- produces: ["carts.item_added-Carts::CartListings-aut"],
- schema: {} }
-```
+### Database setup
-### How nodes link together
+`Store#install!` creates all required tables directly (useful for scripts, tests, and quick prototyping). For production apps using Sequel migrations, the store can export a migration file instead.
-The `produces` and `consumes` fields reference other node IDs, forming a directed graph:
+#### Quick setup (e.g. scripts, tests)
-```
-CommandNode ──produces──▶ EventNode
-EventNode ──consumes──▶ AutomationNode (actor reactions)
-EventNode ──consumes──▶ ReadModelNode ──produces──▶ AutomationNode (projector reactions)
-AutomationNode ──produces──▶ CommandNode
+```ruby
+db = Sequel.sqlite('my_app.db')
+store = Sourced::Store.new(db)
+store.install!
```
-### Catch-all reactions
+#### Exporting a Sequel migration
-When a reactor uses a catch-all `reaction do ... end` (no event argument), the topology collapses all covered events into a single automation node named after the reactor, instead of one automation per event.
+Use `Store#copy_migration_to` to generate a migration file compatible with `Sequel::Migrator`:
```ruby
-class ReadyOrders < Sourced::Projector::StateStored
- event Orders::PaymentConfirmed do |state, event|
- # ...
- end
+db = Sequel.sqlite('my_app.db')
+store = Sourced::Store.new(db)
- event Orders::BuildConfirmed do |state, event|
- # ...
- end
+# Option 1: pass a directory (uses a default filename)
+store.copy_migration_to('db/migrations')
- # Catch-all: reacts to all evolved events
- reaction do |state, event|
- if state[:ready]
- dispatch Orders::Release
- end
- end
+# Option 2: pass a block for full control over the path
+store.copy_migration_to do
+ "db/migrations/#{Time.now.strftime('%Y%m%d%H%M%S')}_create_sourced_tables.rb"
end
```
-This produces a single automation node:
+Then run your migrations as usual:
-```ruby
-{ type: "automation", id: "ready_orders-aut",
- name: "reaction(ReadyOrders)", group_id: "ReadyOrders",
- consumes: ["ready_orders-rm"], produces: ["orders.release"] }
+```bash
+sequel -m db/migrations sqlite://my_app.db
```
-Rather than separate automation nodes for `PaymentConfirmed` and `BuildConfirmed`.
-
-## Rails integration
+#### Custom table prefix
-Soon.
-
-## Sourced vs. ActiveJob
-
-ActiveJob is a great way to handle background jobs in Rails. It's simple and easy to use. However, it's not designed for event sourcing.
-ActiveJob backends (and other job queues) are optimised for parallel processing of jobs, this means that multiple jobs for the same business entity may be processed in parallel without any ordering guarantees.
-
-
-
-Sourced's concurrency model is designed to process events for the same entity in order, while allowing for parallel processing of events for different entities.
-
-
-
-## Gotchas
-
-By default `Sourced` processes commands and events asynchronously through
-background workers. This can be confusing if you expect reactions to run
-automatically when you issue commands.
-
-For synchronous, all-or-nothing execution use [`Sourced::Unit`](#sourcedunit),
-which runs the full command → event → reaction chain inside a single transaction.
+By default, tables are prefixed with `sourced_` (e.g. `sourced_messages`, `sourced_consumer_groups`). Pass a `prefix:` to `Store.new` to customise this — for example when running multiple Sourced stores in the same database:
```ruby
-# Synchronous execution with Unit
-unit = Sourced::Unit.new(Chat, backend: Sourced.config.backend)
-results = unit.handle(SendMessage.new(stream_id: 'chat-123', payload: { content: query }))
-results.events_for(Chat) # => [MessageSent, ...]
+store = Sourced::Store.new(db, prefix: 'billing')
+store.install!
+# Creates: billing_messages, billing_key_pairs, billing_consumer_groups, ...
```
-If you're using the `Sourced::CommandMethods` mixin directly (without a Unit),
-note that it persists events but does not trigger reactions. You'd need to
-explicitly call `#react` after issuing commands.
-
-```ruby
-chat = Sourced.load(Chat, 'chat-123')
-# Persists but does not call reactions
-_cmd, events = chat.send_message!(content: query)
-# Have to react manually
-commands = chat.react(events)
-```
+The prefix is carried through to exported migrations automatically.
+#### Using the Installer directly
-## Installation
+The installer is also available as a standalone object, which is useful for Rake tasks or setup scripts:
-Install the gem and add to the application's Gemfile by executing:
-
- $ bundle add sourced
-
-**Note**: this gem is under active development, so you probably want to install from Github:
-In your Gemfile:
-
- $ gem 'sourced', github: 'ismasan/sourced'
-
-## Development
-
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
-
-## Contributing
-
-Bug reports and pull requests are welcome on GitHub at https://github.com/ismasan/sourced.
+```ruby
+installer = Sourced::Installer.new(db, logger: Logger.new($stdout), prefix: 'sourced')
+installer.install # create tables
+installer.installed? # check if tables exist
+installer.uninstall # drop tables (test env only)
+installer.copy_migration_to('db/migrations')
+```
diff --git a/examples/cart.rb b/examples/cart.rb
deleted file mode 100644
index baa51d3f..00000000
--- a/examples/cart.rb
+++ /dev/null
@@ -1,232 +0,0 @@
-# frozen_string_literal: true
-
-require 'bundler'
-Bundler.setup(:test)
-
-require 'sourced'
-require 'sequel'
-
-# ActiveRecord::Base.establish_connection(adapter: 'postgresql', database: 'decider')
-unless ENV['backend_configured']
- puts 'aggregate config'
- Sourced.configure do |config|
- config.backend = Sequel.postgres('sourced_development')
- end
- Sourced.config.backend.install
- ENV['backend_configured'] = 'true'
-end
-
-# A cart Actor/Aggregate
-# Example:
-# cart = Cart.new('cart-1')
-# cart.add_item(name: 'item1', price: 100)
-# cart.place
-# cart.events
-#
-# The above sends a Cart::Place command
-# which produces a Cart::Placed event
-class Cart < Sourced::Actor
- State = Struct.new(:status, :notified, :items, :mailer_id) do
- def total = items.sum(&:price)
- end
-
- state do |id|
- State.new(:open, false, [], nil)
- end
-
- ItemAdded = Sourced::Message.define('cart.item_added') do
- attribute :name, String
- attribute :price, Integer
- end
-
- Placed = Sourced::Message.define('cart.placed')
- Notified = Sourced::Message.define('cart.notified') do
- attribute :mailer_id, String
- end
-
- # Defines a Cart::AddItem command struct
- command :add_item, name: String, price: Integer do |cart, cmd|
- event(ItemAdded, cmd.payload.to_h)
- end
-
- # Defines a Cart::Place command struct
- command :place do |_, cmd|
- event(Placed)
- end
-
- # Defines a Cart::Notify command struct
- command :notify, mailer_id: String do |_, cmd|
- puts "#{self.class.name} #{cmd.stream_id} NOTIFY"
- event(Notified, mailer_id: cmd.payload.mailer_id)
- end
-
- def self.on_exception(exception, _message, group)
- if group.error_context[:retry_count] < 3
- later = 5 + 5 * group.error_context[:retry_count]
- group.retry(later)
- else
- group.stop(exception)
- end
- end
-
- event ItemAdded do |cart, event|
- cart.items << event.payload
- end
-
- event Placed do |cart, _event|
- cart.status = :placed
- end
-
- event Notified do |cart, event|
- cart.notified = true
- cart.mailer_id = event.payload.mailer_id
- end
-
- # This block will run
- # in the same transaction as appending
- # new events to the store.
- # So if either fails, eveything is rolled back.
- # ergo, strong consistency.
- sync do |command:, events:, state:|
- puts "#{self.class.name} #{events.last.seq} SYNC"
- end
-
- # Or register a Reactor interface to react to events
- # synchronously
- # sync CartListings
-end
-
-class Mailer < Sourced::Actor
- EmailSent = Sourced::Message.define('mailer.email_sent') do
- attribute :cart_id, String
- end
-
- state do |id|
- []
- end
-
- command :send_email, cart_id: String do |_, cmd|
- # Send email here, emit EmailSent if successful
- event(EmailSent, cart_id: cmd.payload.cart_id)
- end
-
- event EmailSent do |list, event|
- list << event
- end
-end
-
-# A Saga that orchestrates the flow between Cart and Mailer
-class CartEmailsSaga < Sourced::Actor
- # Listen for Cart::Placed events and
- # send command to Mailer
- reaction Cart::Placed do |event|
- dispatch(Mailer::SendEmail, cart_id: event.stream_id).to("mailer-#{event.stream_id}")
- end
-
- # Listen for Mailer::EmailSent events and
- # send command to Cart
- reaction Mailer::EmailSent do |event|
- dispatch(Cart::Notify, mailer_id: event.stream_id).to(event.payload.cart_id)
- end
-end
-
-# A projector
-# "reacts" to events registered with .evolve
-class CartListings < Sourced::Actor
- class << self
- def handled_events = self.handled_events_for_evolve
-
- # The Reactor interface
- # @param events [Array]
- def handle_events(events)
- # For this type of event sourced projections
- # that load current state from events
- # then apply "new" events
- # TODO: the current state already includes
- # the new events, so we need to load upto events.first.seq
- instance = load(events.first.stream_id, upto: events.first.seq - 1)
- instance.handle_events(events)
- end
- end
-
- def handle_events(events)
- evolve(state, events)
- save
- [] # no commands
- end
-
- def initialize(id, **_args)
- super
- FileUtils.mkdir_p('examples/carts')
- @path = "./examples/carts/#{id}.json"
- end
-
- private def save
- backend.transaction do
- run_sync_blocks(state, nil, [])
- end
- end
-
- def init_state(id)
- { id:, items: [], status: :open, seq: 0, seqs: [] }
- end
-
- sync do |cart, _command, _events|
- File.write(@path, JSON.pretty_generate(cart))
- end
-
- # Register all events from Cart
- # So that before_evolve runs before all cart events
- evolve_all Cart.handled_commands
- evolve_all Cart
-
- before_evolve do |cart, event|
- cart[:seq] = event.seq
- cart[:seqs] << event.seq
- end
-
- event Cart::Placed do |cart, event|
- cart[:status] = :placed
- end
-
- event Cart::ItemAdded do |cart, event|
- cart[:items] << event.payload.to_h
- end
-end
-
-class LoggingReactor
- extend Sourced::Consumer
-
- class << self
- # Register as a Reactor that cares about these events
- # The workers will use this to fetch the right events
- # and ACK offsets after processing
- #
- # @return [Array]
- def handled_events = [Cart::Placed, Cart::ItemAdded]
-
- # Workers pass available events to this method
- # in order, with exactly-once semantics
- # If a list of commands is returned,
- # workers will send them to the router
- # to be dispatched to the appropriate command handlers.
- #
- # @param events [Array]
- # @option replaying [Boolean] whether this is a replay of events
- # @return [Array [reaction] Order placed! #{state.items.size} items, total: #{state.total}"
- end
-end
-
-# =============================================================================
-# Demo script
-# =============================================================================
-puts "=== LiteCart Demo (SQLite backend) ===\n\n"
-
-# Create a cart instance with a specific stream ID.
-cart, evts = Sourced.load(LiteCart, 'cart-1')
-# cart = LiteCart.new(id: 'cart-1')
-
-# Use bang methods (add_item!) to persist events to the SQLite backend.
-puts "--- Adding items ---"
-_cmd, events = cart.add_item!(name: 'Notebook', price: 1200)
-puts " Added Notebook (1200) -> #{events.map(&:type)}"
-
-_cmd, events = cart.add_item!(name: 'Pen', price: 300)
-puts " Added Pen (300) -> #{events.map(&:type)}"
-
-_cmd, events = cart.add_item!(name: 'Eraser', price: 50)
-puts " Added Eraser (50) -> #{events.map(&:type)}"
-
-puts "\n--- Removing an item ---"
-_cmd, events = cart.remove_item!(name: 'Eraser')
-puts " Removed Eraser -> #{events.map(&:type)}"
-
-puts "\n--- Placing order ---"
-_cmd, events = cart.place_order!
-puts " Order placed -> #{events.map(&:type)}"
-
-puts "\n--- Current cart state (in-memory) ---"
-puts " status: #{cart.state.status}"
-puts " items: #{cart.state.items}"
-puts " total: #{cart.state.total}"
-
-# Read the full event stream back from the database.
-puts "\n--- Event stream for cart-1 ---"
-events = Sourced.config.backend.read_stream('cart-1')
-events.each do |e|
- puts " seq:#{e.seq} #{e.type.ljust(30)} #{e.payload.to_h}"
-end
-
-# Reload a fresh cart from persisted events to show state is rebuilt.
-puts "\n--- Reload cart from events ---"
-reloaded, _events = Sourced.load(LiteCart, 'cart-1')
-puts " status: #{reloaded.state.status}"
-puts " items: #{reloaded.state.items}"
-puts " total: #{reloaded.state.total}"
-
-puts "\nDone!"
diff --git a/examples/pub.rb b/examples/pub.rb
deleted file mode 100644
index 4e1bd917..00000000
--- a/examples/pub.rb
+++ /dev/null
@@ -1,10 +0,0 @@
-require 'sequel'
-require 'sqlite3'
-require 'sourced/pubsub/sqlite'
-
-PUB = Sourced::PubSub::SQLite.new(
- db: Sequel.sqlite('./sourced_pubsub.db'),
- serializer: Sourced::PubSub::SQLite::JSONSerializer.new,
-)
-
-CH = PUB.subscribe('foo')
diff --git a/examples/socket.rb b/examples/socket.rb
deleted file mode 100644
index f1a27ce9..00000000
--- a/examples/socket.rb
+++ /dev/null
@@ -1,18 +0,0 @@
-require 'sourced'
-require 'sourced/pubsub/socket'
-
-Msg = Sourced::Event.define('test') do
- attribute :name, String
-end
-
-# Setup
-PUB = Sourced::PubSub::Socket.new(socket_dir: '/tmp/sourced_sockets')
-
-# Publishing
-# pubsub.publish('events', some_event)
-
-# Subscribing
-CH = PUB.subscribe('events')
-# channel.start do |event, ch|
-# # handle event
-# end
diff --git a/examples/workers.rb b/examples/workers.rb
deleted file mode 100644
index 43b4458a..00000000
--- a/examples/workers.rb
+++ /dev/null
@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-require_relative './cart'
-
-Sourced::Supervisor.start
diff --git a/lib/sourced.rb b/lib/sourced.rb
index da90d7d1..f6709636 100644
--- a/lib/sourced.rb
+++ b/lib/sourced.rb
@@ -1,41 +1,20 @@
# frozen_string_literal: true
require_relative 'sourced/version'
+require 'sourced/types'
+require 'sourced/injector'
-require 'securerandom'
-require 'sourced/message'
-
-# Sourced is an Event Sourcing / CQRS library for Ruby built around the "Decide, Evolve, React" pattern.
-# It provides eventual consistency by default with an actor-like execution model for building
-# event-sourced applications.
-#
-# @example Basic setup with Sequel backend
-# Sourced.configure do |config|
-# config.backend = Sequel.connect('postgres://localhost/mydb')
-# end
-#
-# @example Register actors and projectors
-# Sourced.register(MyActor)
-# Sourced.register(MyProjector)
-#
-# @example Start background workers
-# Sourced::Supervisor.start(count: 10)
-#
-# @see https://github.com/ismasan/sourced
+# Sourced is an event-sourcing library for Ruby built around stream-less,
+# partition-based consistency. Events go into a flat, globally-ordered log and
+# consistency context is assembled dynamically by querying relevant facts via
+# key-value pairs extracted from event payloads.
module Sourced
# Base error class for all Sourced-specific exceptions
class Error < StandardError; end
-
- # Raised when concurrent writes to the same stream are detected
+
ConcurrentAppendError = Class.new(Error)
-
- # Raised when concurrent acknowledgments of the same event are detected
- ConcurrentAckError = Class.new(Error)
-
- # Raised when an invalid reactor is registered
- InvalidReactorError = Class.new(Error)
-
- BackendError = Class.new(Error)
+ UnknownMessageError = Class.new(ArgumentError)
+ PastMessageDateError = Class.new(ArgumentError)
# Raised when a batch is partially processed before a message raises.
# Carries the action_pairs for successfully processed messages,
@@ -51,190 +30,70 @@ def initialize(action_pairs, failed_message, cause)
end
end
- class InvalidMessageError < Error
- attr_reader :message
-
- def initialize(message)
- @message = message
- super <<~ERR
- Invalid message #{message.class} ('#{message.type}')
- Errors: #{message.errors.inspect}
- ERR
- end
+ # @return [Configuration] the global Sourced configuration instance
+ def self.config
+ @config ||= Configuration.new
end
- # Generate a new unique stream identifier, optionally with a prefix.
- # Stream IDs define concurrency boundaries - events for the same stream ID
- # are processed sequentially, while different stream IDs can be processed concurrently.
- #
- # @param prefix [String, nil] Optional prefix for the stream ID
- # @return [String] A new UUID-based stream ID
- # @example Generate a simple stream ID
- # Sourced.new_stream_id #=> "123e4567-e89b-12d3-a456-426614174000"
- # @example Generate a prefixed stream ID
- # Sourced.new_stream_id("cart") #=> "cart-123e4567-e89b-12d3-a456-426614174000"
- def self.new_stream_id(prefix = nil)
- uuid = SecureRandom.uuid
- prefix ? "#{prefix}-#{uuid}" : uuid
+ # Configure the Sourced module. Stores the block for re-running after fork
+ # (see {.setup!}), then runs it immediately.
+ # @yieldparam config [Configuration]
+ def self.configure(&block)
+ @configure_block = block
+ setup!
end
- # Access the global Sourced configuration instance.
- #
- # @return [Configuration] The current configuration instance
- def self.config
- @config ||= Configuration.new
+ # Run (or re-run) the configure block on a fresh Configuration.
+ # Safe to call after a process fork to re-establish database connections.
+ def self.setup!
+ @config = Configuration.new
+ @configure_block&.call(@config)
+ @config.setup!
+ @config.freeze
end
- # Configure Sourced with backend, error handling, and other settings.
- # The configuration is frozen after the block executes to prevent
- # accidental modification during runtime.
- #
- # @yield [config] Yields the configuration object for setup
- # @yieldparam config [Configuration] The configuration instance to configure
- # @return [Configuration] The frozen configuration
- # @example Basic configuration with Sequel
- # Sourced.configure do |config|
- # config.backend = Sequel.connect(ENV['DATABASE_URL'])
- # config.logger = Logger.new(STDOUT)
- # end
- # @example Configuration with error handling
- # Sourced.configure do |config|
- # config.backend = Sequel.connect(ENV['DATABASE_URL'])
- # config.error_strategy do |s|
- # s.retry(times: 3, after: 5)
- # s.on_stop { |e, msg| Sentry.capture_exception(e) }
- # end
- # end
- def self.configure(&)
- yield config if block_given?
+ # Register a reactor class with the global router.
+ def self.register(reactor)
config.setup!
- config.freeze
- config
+ config.router.register(reactor)
+ @topology = nil
end
- # Register an Actor or Projector class to make it available for background processing.
- # Registered reactors can handle commands and react to events asynchronously.
- #
- # @param reactor [Class] Actor or Projector class that implements the reactor interface
- # @return [void]
- # @raise [InvalidReactorError] if the reactor doesn't implement required interface methods
- # @example Register an actor
- # Sourced.register(CartActor)
- # @example Register a projector
- # Sourced.register(CartListingsProjector)
- # @see Actor
- # @see Projector
- def self.register(reactor)
- Router.register(reactor)
+ # @return [Sourced::Store]
+ def self.store
+ config.setup!
+ config.store
end
- # @return [Boolean]
- def self.registered?(reactor)
- Router.registered?(reactor)
+ # @return [Sourced::Router]
+ def self.router
+ config.setup!
+ config.router
end
- # Append messages (probably commands) to a stream
- # auto-incrementing the sequence number
- # Raises if the message is invalid
- # This is meant for an app's front-end to dispatch commands into the system
- # so it's defensive and raises on error
- # helpers upstream of this can first validate the message
- # and surface errors back to the UI
- # TODO: in future we might want to restrict what messages
- # can be publicly dispatched. Whether that lives here, or
- # somewhere else, I'm not sure.
- #
- # @param message [Message] the mesagge to append
- # @raise [InvalidMessageError] if !message.valid?
- # @return [Message]
- # @example
- # command = CreateCart.new(stream_id: 'cart-123', payload: {}),
- # Sourced.dispatch(command)
- def self.dispatch(message)
- raise InvalidMessageError.new(message) unless message.valid?
-
- appended = config.backend.append_next_to_stream(message.stream_id, [message])
- raise BackendError, "Backend #{config.backend}#append_next_to_stream failed with message #{message.inspect}" unless appended
-
- message
+ def self.stop_consumer_group(reactor_or_id, message = nil)
+ config.router.stop_consumer_group(reactor_or_id, message)
end
- class Loader
- def initialize(backend: Sourced.config.backend)
- @backend = backend
- end
-
- def load(actor, after: nil, upto: nil)
- after ||= actor.seq
- events = @backend.read_stream(actor.id, after:, upto:)
- actor.evolve(events)
- [actor, events]
- end
+ def self.reset_consumer_group(reactor_or_id)
+ config.router.reset_consumer_group(reactor_or_id)
end
- # Load or catch up an Actor from its event history
- # @example
- # actor = MyActor.new(id: '123')
- # Sourced.load(actor)
- # actor.seq # Integer
- #
- # Actor must implement:
- # #id() => String
- # #seq() => Integer
- # #evolve(events)
- #
- # It also supports passing a Reactor class (Actor, Evolver)
- # and a stream_id
- # @example
- # actor, events = Sourced.load(MyActor, 'order-123')
- # actor, events = Sourced.load(MyActor, 'order-123', after: 20)
- def self.load(*args)
- reactor, options = case args
- in [ReactorInterface => r, String => stream_id, Hash => opts]
- [r.new(id: stream_id), opts]
- in [ReactorInterface => r, String => stream_id]
- [r.new(id: stream_id), {}]
- in [Evolve => r, Hash => opts]
- [r, opts]
- in [Evolve => r]
- [r, {}]
- else
- raise ArgumentError, "expected a Reactor class and stream_id, or a Reactor instance, but got #{args.inspect}"
- end
-
- backend = options.delete(:backend) || config.backend
- Loader.new(backend:).load(reactor, **options)
+ def self.start_consumer_group(reactor_or_id)
+ config.router.start_consumer_group(reactor_or_id)
end
- # Load history for a reactor, or a stream id string
- # @example
- # history = Sourced.history_for('order-123')
- # history = Sourced.history_for('order-123', upto: 20)
- # history = Sourced.history_for(order_actor)
- #
- # @param stream_id [String, #id, #stream_id]
- # @option after [nil, Integer] load messages after this sequence number
- # @option upto [nil, Integer] load messsages upto this sequence number
- # @return [Enumerable]
- def self.history_for(stream_id, after: nil, upto: nil)
- stream_id = if stream_id.respond_to?(:stream_id)
- stream_id.stream_id
- elsif stream_id.respond_to?(:id)
- stream_id.id
- else
- stream_id
- end
-
- config.backend.read_stream(stream_id, after:, upto:)
+ # Reset the global configuration. For test teardown.
+ def self.reset!
+ @config = nil
+ @configure_block = nil
+ @topology = nil
end
- # Build the topology graph from all registered async reactors.
- # Returns a flat array of CommandNode, EventNode, and AutomationNode structs.
- # The result is memoized; call Sourced.reset_topology to clear.
- #
- # @return [Array]
+ # Build and cache the topology graph from all reactors registered with
+ # the global {.router}.
def self.topology
- @topology ||= Topology.build(Router.instance.async_reactors)
+ @topology ||= Topology.build(router.reactors)
end
def self.reset_topology
@@ -242,56 +101,92 @@ def self.reset_topology
end
# Generate a standardized method name for message handlers.
- # Used internally to create consistent handler method names.
- #
- # @param prefix [String] The handler type prefix (e.g., 'command', 'event')
- # @param name [String] The message class name
- # @return [String] The generated method name
# @api private
def self.message_method_name(prefix, name)
"__handle_#{prefix}_#{name.split('::').map(&:downcase).join('_')}"
end
- # @!group Type Interfaces
-
- # Interface that command handlers (Deciders) must implement.
- # @!attribute [r] handled_commands
- # @return [Array] Command classes this decider handles
- # @!attribute [r] handle_command
- # @return [Method] Method to handle incoming commands
- # @!attribute [r] on_exception
- # @return [Method] Method to handle exceptions during command processing
- DeciderInterface = Types::Interface[:handled_commands, :handle_command, :on_exception]
-
- # Interface that event handlers (Reactors) must implement.
- # @!attribute [r] consumer_info
- # @return [Sourced::Consumer::ConsumerInfo] Consumer group information for this reactor
- # @!attribute [r] handled_messages
- # @return [Array] Message classes this reactor handles
- # @!attribute [r] handle
- # @return [Method] Method to handle incoming events
- # @!attribute [r] on_exception
- # @return [Method] Method to handle exceptions during event processing
- ReactorInterface = Types::Interface[:handle_batch, :consumer_info, :handled_messages, :on_exception]
+ # Returned by {.handle!} with command, reactor instance, and new events.
+ HandleResult = Data.define(:command, :reactor, :events) do
+ def to_ary = [command, reactor, events]
+ end
+
+ # Handle a command synchronously: validate, load history, decide, append, ACK.
+ def self.handle!(reactor_class, command, store: nil)
+ partition_attrs = extract_partition_attrs(command, reactor_class)
+ values = reactor_class.partition_keys.map { |k| partition_attrs[k]&.to_s }
+
+ unless command.valid?
+ return HandleResult.new(command: command, reactor: reactor_class.new(values), events: [])
+ end
+
+ store ||= self.store
+ needs_history = Injector.resolve_args(reactor_class, :handle_claim).include?(:history)
+ if needs_history
+ instance, read_result = load(reactor_class, store: store, **partition_attrs)
+ else
+ instance = reactor_class.new(values)
+ end
+
+ raw_events = instance.decide(command)
+ correlated_events = raw_events.map { |e| command.correlate(e) }
+
+ guard = read_result&.guard
+ to_append = [command] + correlated_events
+ last_position = store.append(to_append, guard: guard)
+
+ advance_registered_offsets(store, reactor_class, partition_attrs, last_position)
+
+ HandleResult.new(command: command, reactor: instance, events: correlated_events)
+ end
+
+ # Load a reactor instance from its event history using AND-filtered partition reads.
+ # Pass +upto:+ (partition-local rank) to evolve over at most the first N matching
+ # messages in the partition — useful for time-travel, deterministic replay, or debugging.
+ def self.load(reactor_class, store: nil, upto: nil, **values)
+ store ||= self.store
+ partition_attrs = reactor_class.partition_keys.to_h { |k| [k, values[k]] }
+ handled_types = reactor_class.handled_messages_for_evolve.map(&:type).uniq
+ read_result = store.read_partition(partition_attrs, handled_types:, upto: upto)
+ instance = reactor_class.new(values)
+
+ instance.evolve(read_result.messages)
+
+ [instance, read_result]
+ end
+
+ private_class_method def self.extract_partition_attrs(command, reactor_class)
+ reactor_class.partition_keys.each_with_object({}) do |key, h|
+ value = command.payload&.respond_to?(key) ? command.payload.send(key) : nil
+ h[key] = value if value
+ end
+ end
+
+ private_class_method def self.advance_registered_offsets(store, reactor_class, partition_attrs, position)
+ return unless config.router&.reactors&.include?(reactor_class)
+
+ store.advance_offset(
+ reactor_class.group_id,
+ partition: partition_attrs.transform_keys(&:to_s),
+ position: position
+ )
+ end
end
-require 'sourced/consumer'
+require 'sourced/configuration'
+require 'sourced/message'
require 'sourced/actions'
+require 'sourced/consumer'
require 'sourced/evolve'
require 'sourced/react'
require 'sourced/sync'
-require 'sourced/configuration'
-require 'sourced/router'
-require 'sourced/message'
-require 'sourced/actor'
-require 'sourced/handler'
+require 'sourced/decider'
require 'sourced/projector'
-require 'sourced/work_queue'
-require 'sourced/inline_notifier'
-require 'sourced/catchup_poller'
+require 'sourced/router'
+require 'sourced/worker'
+require 'sourced/stale_claim_reaper'
require 'sourced/dispatcher'
-require 'sourced/supervisor'
require 'sourced/command_context'
-require 'sourced/unit'
require 'sourced/topology'
-# require 'sourced/rails/railtie' if defined?(Rails::Railtie)
+require 'sourced/supervisor'
+require 'sourced/durable_workflow'
diff --git a/lib/sourced/actions.rb b/lib/sourced/actions.rb
index 6e87bc65..b62d3a44 100644
--- a/lib/sourced/actions.rb
+++ b/lib/sourced/actions.rb
@@ -1,154 +1,119 @@
# frozen_string_literal: true
module Sourced
- # Actions represent the side effects produced by command/event handlers.
- # Each persistable action class implements {#execute}, which correlates messages
- # against a source message and persists them via the backend.
- # This provides a single place for the correlate-then-persist logic,
- # used by both backend reactors and {Sourced::Unit} synchronous processing.
+ # Action builders and executable action types for reactors.
module Actions
- # Split a list of messages into
- # {AppendNext} or {Schedule} actions
- # based on their +created_at+ relative to now.
+ OK = :ok
+ RETRY = :retry
+
+ # Split produced messages into immediate append actions and delayed schedule actions.
#
- # @param messages [Array]
- # @return [Array]
- def self.build_for(messages)
+ # @param messages [Sourced::Message, Array] messages produced by a reactor
+ # @param guard [ConsistencyGuard, nil] optional concurrency guard for immediate appends
+ # @param source [Sourced::Message, nil] source message used for correlation when executing
+ # @return [Array] executable actions in append/schedule groups
+ def self.build_for(messages, guard: nil, source: nil)
actions = []
+ messages = Array(messages)
return actions if messages.empty?
- # TODO: I really need a uniform Clock object
+ # TODO: review use of Time.now
now = Time.now
- to_schedule, to_append = messages.partition { |e| e.created_at > now }
- actions << AppendNext.new(to_append) if to_append.any?
- to_schedule.group_by(&:created_at).each do |at, msgs|
- actions << Schedule.new(msgs, at:)
+ to_schedule, to_append = messages.partition { |message| message.created_at > now }
+
+ actions << Append.new(to_append, guard:, source:) if to_append.any?
+ to_schedule.group_by(&:created_at).each do |at, scheduled_messages|
+ actions << Schedule.new(scheduled_messages, at:, source:)
end
actions
end
- RETRY = :retry
- OK = :ok
-
- # ACK an arbitrary message ID
- Ack = Data.define(:message_id)
-
- # Append messages to the event store using Backend#append_next_to_stream,
- # which auto-increments stream sequence numbers.
- # Messages may target different streams and will be grouped by stream_id.
- class AppendNext
- include Enumerable
-
- # @return [Array]
- attr_reader :messages
-
- # @param messages [Array]
- def initialize(messages)
- @messages = messages
- freeze
- end
-
- def ==(other)
- other.is_a?(self.class) && messages == other.messages
- end
-
- # @yield [stream_id, message]
- # @yieldparam stream_id [String]
- # @yieldparam message [Sourced::Message]
- # @return [Enumerator] if no block given
- def each(&block)
- return enum_for(:each) unless block_given?
-
- messages.each do |message|
- block.call(message.stream_id, message)
- end
+ # Append messages to the store with optional consistency guard.
+ # Auto-correlates messages at execution time.
+ #
+ # When +source:+ is provided, it overrides the runtime's source_message
+ # for correlation (e.g. reactions correlated with the event, not the command).
+ class Append
+ attr_reader :messages, :guard, :source
+
+ # @param messages [Sourced::Message, Array] messages to append
+ # @param guard [ConsistencyGuard, nil] optional optimistic concurrency guard
+ # @param source [Sourced::Message, nil] explicit correlation source
+ def initialize(messages, guard: nil, source: nil)
+ @messages = Array(messages)
+ @guard = guard
+ @source = source
end
- # Correlate messages against the source, then persist via backend.
- #
- # @param backend [#append_next_to_stream] the storage backend
- # @param source_message [Sourced::Message] message that caused this action
- # @return [Array] correlated messages that were persisted
- def execute(backend, source_message)
- correlated = messages.map { |m| source_message.correlate(m) }
- correlated.group_by(&:stream_id).each do |stream_id, stream_messages|
- backend.append_next_to_stream(stream_id, stream_messages)
- end
- correlated
+ # @param store [Sourced::Store]
+ # @param source_message [Sourced::Message] default message to correlate from
+ # @return [Array] correlated messages that were appended
+ def execute(store, source_message)
+ correlate_from = @source || source_message
+ to_append = messages.map { |m| correlate_from.correlate(m) }
+ store.append(to_append, guard:)
+ to_append
end
end
- # Append messages to a specific stream in the event store,
- # expecting messages to be in order and with correct sequence numbers.
- # The backend will raise {Sourced::ConcurrentAppendError} if messages
- # with the same sequence already exist (optimistic concurrency control).
- class AppendAfter
- # @return [String]
- attr_reader :stream_id
- # @return [Array]
- attr_reader :messages
-
- # @param stream_id [String]
- # @param messages [Array]
- def initialize(stream_id, messages)
- @stream_id = stream_id
- @messages = messages
+ # Schedule messages for future promotion into the main log.
+ class Schedule
+ attr_reader :messages, :at, :source
+
+ # @param messages [Sourced::Message, Array] messages to schedule
+ # @param at [Time] when the messages should become available for promotion
+ # @param source [Sourced::Message, nil] explicit correlation source
+ def initialize(messages, at:, source: nil)
+ @messages = Array(messages)
+ @at = at
+ @source = source
end
- # Correlate messages against the source, then persist via backend.
- #
- # @param backend [#append_to_stream] the storage backend
- # @param source_message [Sourced::Message] message that caused this action
- # @return [Array] correlated messages that were persisted
- def execute(backend, source_message)
- correlated = messages.map { |m| source_message.correlate(m) }
- backend.append_to_stream(stream_id, correlated)
- correlated
+ # @param store [Sourced::Store]
+ # @param source_message [Sourced::Message] default message to correlate from
+ # @return [Array] correlated messages that were scheduled
+ def execute(store, source_message)
+ correlate_from = @source || source_message
+ to_schedule = messages.map { |m| correlate_from.correlate(m) }
+ store.schedule_messages(to_schedule, at: at)
+ to_schedule
end
end
- # Schedule messages for future delivery at a specific time.
- class Schedule
- # @return [Array]
- attr_reader :messages
- # @return [Time]
- attr_reader :at
-
- # @param messages [Array]
- # @param at [Time] when the messages should become available
- def initialize(messages, at:)
- @messages, @at = messages, at
+ # Execute a synchronous side effect within the current transaction.
+ class Sync
+ # @param work [#call] callable to execute
+ def initialize(work)
+ @work = work
end
- # Correlate messages against the source, then schedule via backend.
- #
- # @param backend [#schedule_messages] the storage backend
- # @param source_message [Sourced::Message] message that caused this action
- # @return [Array] correlated messages that were scheduled
- def execute(backend, source_message)
- correlated = messages.map { |m| source_message.correlate(m) }
- backend.schedule_messages(correlated, at: at)
- correlated
+ # @return [Object] the callable's return value
+ def call = @work.call
+
+ # @param _store [Object] unused
+ # @param _source_message [Object] unused
+ # @return [nil]
+ def execute(_store, _source_message)
+ call
+ nil
end
end
- # Execute a synchronous side effect (e.g. cache write, HTTP call)
- # within the current transaction. Does not persist messages.
- class Sync
+ # Execute a side effect after the transaction commits.
+ class AfterSync
# @param work [#call] callable to execute
def initialize(work)
@work = work
end
- def call(...) = @work.call(...)
+ # @return [Object] the callable's return value
+ def call = @work.call
- # Execute the work block. Backend and source_message are ignored.
- #
- # @param _backend [Object] unused
+ # @param _store [Object] unused
# @param _source_message [Object] unused
# @return [nil]
- def execute(_backend, _source_message)
+ def execute(_store, _source_message)
call
nil
end
diff --git a/lib/sourced/actor.rb b/lib/sourced/actor.rb
deleted file mode 100644
index c7d445ed..00000000
--- a/lib/sourced/actor.rb
+++ /dev/null
@@ -1,386 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- class Actor
- include Evolve
- include React
- include Sync
- extend Consumer
-
- PREFIX = 'decide'
-
- UndefinedMessageError = Class.new(KeyError)
- Error = Class.new(StandardError)
-
- class DualMessageRegistrationError < Error
- def initialize(msg_type, handled_by)
- msg = if handled_by == :reaction
- <<~MSG
- Message #{msg_type} is already registered to be handled by .reaction(#{msg_type}).
- Sourced::Actor classes can only handle the same message type either as a reaction, or a command.
- MSG
- else
- <<~MSG
- Message #{msg_type} is already registered to be handled by .command(#{msg_type}).
- Sourced::Actor classes can only handle the same message type either as a reaction, or a command.
- MSG
- end
- super msg
- end
- end
-
- class DifferentStreamError < Error
- def initialize(actor, event)
- super <<~MSG
- Actor instance #{actor.inspect} was initialized with id = '#{actor.id}',
- but it was evolved with an event for stream_id '#{event.stream_id}',
-
- The event is #{event.inspect}
- MSG
- end
- end
-
- class SmallerSequenceError < Error
- def initialize(actor, event)
- super <<~MSG
- Actor instance #{actor.inspect} is currently at sequence number #{actor.seq},
- but it was evolved with an event at sequence #{event.seq}.
-
- The event is #{event.inspect}
- MSG
- end
- end
-
- # An Actor class has its own Command and Event
- # subclasses that are used to define inine commands and events.
- # These classes serve as message registry for the Actor's inline messages.
- class Command < Sourced::Command; end
- class Event < Sourced::Event; end
-
- BLANK_HISTORY = [].freeze
-
- class << self
- def inherited(subclass)
- super
- subclass.const_set(:Command, Class.new(const_get(:Command)))
- subclass.const_set(:Event, Class.new(const_get(:Event)))
- handled_commands.each do |cmd_type|
- subclass.handled_commands << cmd_type
- end
- end
-
- # Access a Actor's Command or Event classes by name (e.g. :some_command or :some_event)
- # @param message_name [Symbol]
- # @return [Class]
- # @raise [ArgumentError] if the message is not defined
- def resolve_message_class(message_name)
- message_type = __message_type(message_name)
- msg_class = self::Event.registry[message_type] || self::Command.registry[message_type]
-
- raise UndefinedMessageError, "Message not found: #{message_name}" unless msg_class
- msg_class
- end
-
- alias [] resolve_message_class
-
- # Interface expected by React::StreamDispatcher
- # so that this works in reaction blocks
- # @example
- # stream = stream_for(SomeActor)
- # stream.command :do_something
- #
- # @return [String]
- def stream_id
- Sourced.new_stream_id
- end
-
- def handled_commands
- @handled_commands ||= []
- end
-
- # Register as a Reactor
- # @return [Array]
- def handled_messages = self.handled_commands + self.handled_messages_for_react
-
- # Define a command class, register a command handler
- # and define a method to send the command
- # Example:
- # command :add_item, name: String do |state, cmd|
- # event(ItemAdded, item_id: SecureRandom.uuid, name: cmd.payload.name)
- # end
- #
- # # The exmaple above will define a command class `AddItem` in the current namespace:
- # AddItem = Message.define('namespace.add_item', payload_schema: { name: String })
- #
- # Payload schema is a Plumb Hash schema.
- # See: https://github.com/ismasan/plumb#typeshash
- #
- def command(*args, &block)
- raise ArgumentError, 'command block expects signature (state, command)' unless block.arity == 2
-
- case args
- in [Symbol => cmd_name, Hash => payload_schema]
- __register_named_command_handler(cmd_name, payload_schema, &block)
- in [Symbol => cmd_name]
- __register_named_command_handler(cmd_name, &block)
- in [Class => cmd_type] if cmd_type < Sourced::Message
- __register_class_command_handler(cmd_type, &block)
- else
- raise ArgumentError, "Invalid arguments for #{self}.command"
- end
- end
-
- # Support defining event handlers with a symbol and a payload schema
- # Or a class.
- #
- # @example
- #
- # event SomethingHappened, field1: String do |state, event|
- # state[:status] = 'done'
- # end
- #
- # event :something_happened, field1: String do |state, event|
- # state[:status] = 'done'
- # end
- #
- def event(*args, &block)
- case args
- in [Symbol => event_name, Hash => payload_schema]
- __register_named_event_handler(event_name, payload_schema).tap do |event_class|
- super(event_class, &block)
- end
- in [Symbol => event_name]
- __register_named_event_handler(event_name).tap do |event_class|
- super(event_class, &block)
- end
- in [Class => foo]
- super
- else
- raise ArgumentError, "event expects a Symbol or Event class. Got: #{args.inspect}"
- end
- end
-
- # The Reactor interface
- # Message can be:
- # - A command, present in .handled_commands
- # - A reaction, present in .handled_messages_for_react
- # @param message [Sourced::Message]
- # @option history [Enumerable] past messages in the stream
- # @return [Sourced::Actions]
- def handle(message, history: [], replaying: false)
- instance = new(id: identity_from(message))
- instance.handle(message, history:, replaying:)
- end
-
- # Batch processing for actors. Per-message iteration internally.
- # Replay messages return OK immediately. Live messages create instance and handle.
- # @param batch [Array<[Message, Boolean]>] array of [message, replaying] pairs
- # @param history [Array] full stream history
- # @return [Array<[actions, source_message]>] action pairs
- def handle_batch(batch, history: BLANK_HISTORY)
- each_with_partial_ack(batch) do |message, replaying|
- if replaying
- [Actions::OK, message]
- else
- instance = new(id: identity_from(message))
- actions = instance.handle(message, history:)
- [actions, message]
- end
- end
- end
-
- def __message_type(msg_name)
- [__message_type_prefix, msg_name].join('.').downcase
- end
-
- # Override this in subclasses
- # to make an actor take it's @id from an arbitrary
- # field in the message
- # TODO: the danger here is that not all messages might
- # support this arbitrary field.
- # it would be nice if we could statically
- # analyse messages handled by the Actor, and raise an error
- # if not all of them support the specified field
- #
- # @param message [Sourced::Message]
- # @return [Object]
- def identity_from(message) = message.stream_id
-
- private
-
- def __register_named_command_handler(cmd_name, payload_schema = nil, &block)
- cmd_class = self::Command.define(__message_type(cmd_name), payload_schema:)
- klass_name = cmd_name.to_s.split('_').map(&:capitalize).join
- const_set(klass_name, cmd_class)
- __register_class_command_handler(cmd_class, &block)
- end
-
- def __register_class_command_handler(cmd_type, &block)
- raise DualMessageRegistrationError.new(cmd_type, :reaction) if handled_messages_for_react.include?(cmd_type)
-
- handled_commands << cmd_type
- define_method(Sourced.message_method_name(PREFIX, cmd_type.name), &block)
- end
-
- def __register_named_event_handler(event_name, payload_schema = nil)
- klass_name = event_name.to_s.split('_').map(&:capitalize).join
- event_class = self::Event.define(__message_type(event_name), payload_schema:)
- const_set(klass_name, event_class)
- end
-
- # Override this method defined in React mixin.
- # Given a symbol for a message type, resolve the message class
- # @return [Class, nil]
- def __resolve_message_class(message_symbol)
- self::Event.registry[__message_type(message_symbol)]
- end
-
- # Override the default namespace for commands and events
- # defined inline
- # @example
- #
- # def message_namespace = 'my_app.messages.'
- #
- # @return [String]
- def message_namespace
- Types::ModuleToMessageType.parse(name.to_s)
- end
-
- def __message_type_prefix
- @__message_type_prefix ||= message_namespace
- end
-
- # Override the no-op hook in Sourced::React
- # We want to make sure that a message handled as a command
- # cannot also be registered as a reaction.
- # These are the command/event semantics that Actor adds on top
- # of the underlying messaging infrastructure.
- def __validate_message_for_reaction!(event_class)
- raise DualMessageRegistrationError.new(event_class, :command) if handled_commands.include?(event_class)
- end
- end
-
- # Instance methods
-
- attr_reader :id, :seq, :uncommitted_events
-
- def initialize(id: Sourced.new_stream_id, logger: Sourced.config.logger)
- @id = id
- @seq = 0
- @uncommitted_events = []
- @logger = logger
- @__current_command = Sourced::Command.new(stream_id: id)
- end
-
- def inspect
- %(<#{self.class} id:#{id} seq:#{seq}>)
- end
-
- def handle(message, history:, replaying: false)
- return Actions::OK if replaying
-
- evolve(history)
- if handles_command?(message)
- events = decide(message)
- actions = [Actions::AppendAfter.new(id, events)]
- actions + sync_actions_with(command: message, events:, state:)
- elsif reacts_to?(message)
- Actions.build_for(react(message))
- else
- Actions::OK
- end
- end
-
- # Does this actor handle this message as a command?
- # TODO: ATM I'm just doing .handled_commands.include?
- # it would be more efficient to have an O(1) lookup
- def handles_command?(message)
- self.class.handled_commands.include?(message.class)
- end
-
- # Route a command to its defined command handler, and run it.
- # @param command [Sourced::Command]
- # @return [Array]
- def decide(command)
- command = __set_current_command(command)
- send(Sourced.message_method_name(PREFIX, command.class.name), state, command)
- @__current_command = nil
- @uncommitted_events.slice!(0..)
- end
-
- # Apply an event from within a command handler
- # @example
- #
- # command DoSomething do |state, cmd|
- # event SomethingHappened, field1: 'foo', field2: 'bar'
- # end
- #
- # Or, with symbol pointing to an event class defined with .event
- # command DoSomething do |state, cmd|
- # event :something_happened, field1: 'foo', field2: 'bar'
- # end
- #
- # @param event_name [Symbol, Class] the event name or class
- # @param payload [Hash] the event payload
- # @return [Any] the
- def event(event_name, payload = {})
- return apply(event_name, payload) unless event_name.is_a?(Symbol)
-
- event_class = Event.registry[self.class.__message_type(event_name)]
- raise ArgumentError, "Event not found: #{event_name}" unless event_class
-
- apply(event_class, payload)
- end
-
- private
-
- attr_reader :__current_command, :logger
-
- # Override Evolve#__update_on_evolve
- def __update_on_evolve(event)
- raise DifferentStreamError.new(self, event) if id != event.stream_id
- raise SmallerSequenceError.new(self, event) if seq >= event.seq
-
- @seq = event.seq
- end
-
- # TODO: in the new arch, commands
- # already exist in the event stream
- # so we don't append them again as part of uncommitted_events
- # and we don't increment @seq
- # However, when handling commands asynchronously,
- # we DO also want to append the command with the produced events
- # Think about this later.
- def __set_current_command(command)
- @__current_command = command
- end
-
- def __next_sequence
- @seq + 1
- end
-
- # Instantiate an event class and apply it to the state
- # by running registered evolve blocks.
- # Also store the event in the uncommitted events list,
- # and keep track of the sequence number.
- # To be used inside a .decide block.
- # @example
- #
- # apply SomeEvent, field1: 'foo', field2: 'bar'
- #
- # @param event_class [Sourced::Event]
- # @param payload [Hash] the event payload
- # @return [Any] the new state
- def apply(event_class, payload = {})
- evt = __current_command.follow_with_attributes(
- event_class,
- attrs: { seq: __next_sequence },
- # TODO: the infra sets this now
- # metadata: { producer: self.class.consumer_info.group_id },
- payload:
- )
- uncommitted_events << evt
- evolve([evt])
- end
- end
-end
diff --git a/lib/sourced/backends/pg_backend.rb b/lib/sourced/backends/pg_backend.rb
deleted file mode 100644
index 2f3fc63b..00000000
--- a/lib/sourced/backends/pg_backend.rb
+++ /dev/null
@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-require 'sourced/backends/sequel_backend'
-
-module Sourced
- module Backends
- # Explicit PostgreSQL backend. Inherits all behaviour from SequelBackend
- # and overrides only the adapter setup to skip auto-detection.
- #
- # @example
- # db = Sequel.postgres('myapp')
- # backend = Sourced::Backends::PGBackend.new(db)
- class PGBackend < SequelBackend
- private
-
- def setup_adapter
- @notifier = PGNotifier.new(db: @db)
- end
- end
- end
-end
diff --git a/lib/sourced/backends/sequel_backend.rb b/lib/sourced/backends/sequel_backend.rb
deleted file mode 100644
index be3ad385..00000000
--- a/lib/sourced/backends/sequel_backend.rb
+++ /dev/null
@@ -1,978 +0,0 @@
-# frozen_string_literal: true
-
-require 'sequel'
-require 'json'
-require 'sourced/message'
-require 'sourced/inline_notifier'
-
-Sequel.extension :pg_json if defined?(PG)
-
-module Sourced
- module Backends
- # Production backend implementation using Sequel ORM for PostgreSQL and SQLite.
- # This backend provides persistent storage for messages, and consumer state
- # with support for concurrency control, pub/sub notifications, and transactional processing.
- #
- # The SequelBackend handles:
- # - Message storage and retrieval with ordering guarantees
- # - Message scheduling and dispatching
- # - Consumer group management and offset tracking
- # - Concurrency control via database locks and claims
- # - Pub/sub notifications for real-time message processing
- #
- # @example Basic setup with PostgreSQL
- # db = Sequel.connect('postgres://localhost/myapp')
- # backend = Sourced::Backends::SequelBackend.new(db)
- # backend.install unless backend.installed?
- #
- # @example Configuration in Sourced
- # Sourced.configure do |config|
- # config.backend = Sequel.connect(ENV['DATABASE_URL'])
- # end
- class SequelBackend
- # Consumer group status indicating active processing
- ACTIVE = 'active'
- # Consumer group status indicating stopped processing
- STOPPED = 'stopped'
- # Pre-allocated return for empty/failed batch claims
- NO_BATCH = [nil, nil].freeze
-
- # Initialize a new Sequel backend instance.
- # Automatically sets up database connection, table names, and pub/sub system.
- #
- # @param db [Sequel::Database, String] Database connection or connection string
- # @param logger [Object] Logger instance for backend operations (defaults to configured logger)
- # @param prefix [String] Table name prefix for Sourced tables (defaults to 'sourced')
- # @param schema [String, nil] Optional PostgreSQL schema name. When set, all tables are
- # created and queried within this schema (e.g. "my_app".sourced_messages).
- # The schema is created automatically on #install if it doesn't exist.
- # Defaults to nil (uses the database's default search_path, typically "public").
- # @example Initialize with existing connection
- # backend = SequelBackend.new(Sequel.connect('postgres://localhost/mydb'))
- # @example Initialize with custom prefix
- # backend = SequelBackend.new(db, prefix: 'my_app')
- # @example Initialize with a PostgreSQL schema
- # backend = SequelBackend.new(db, schema: 'my_app')
- # @example Combine schema and prefix
- # backend = SequelBackend.new(db, schema: 'my_app', prefix: 'evt')
- # # => tables: my_app.evt_messages, my_app.evt_streams, etc.
- def initialize(db, logger: Sourced.config.logger, prefix: 'sourced', schema: nil)
- @db = connect(db)
- @logger = logger
- @prefix = prefix
- @schema = schema
- setup_adapter
- setup_tables
- @setup = false
- logger.info("Connected to #{@db}")
- end
-
- # Data structure for backend statistics and monitoring information.
- # @!attribute [r] stream_count
- # @return [Integer] Total number of message streams
- # @!attribute [r] max_global_seq
- # @return [Integer] Highest global sequence number (latest message)
- # @!attribute [r] groups
- # @return [Array] Consumer group information with processing stats
- Stats = Data.define(:stream_count, :max_global_seq, :groups)
-
- # Get comprehensive statistics about the backend state.
- # Useful for monitoring, debugging, and understanding system health.
- #
- # @return [Stats] Statistics object with stream counts, sequences, and group info
- # @example Get backend statistics
- # stats = backend.stats
- # puts "Total streams: #{stats.stream_count}"
- # puts "Latest message: #{stats.max_global_seq}"
- # stats.groups.each { |g| puts "Group #{g[:id]}: #{g[:status]}" }
- def stats
- stream_count = db[streams_table].count
- max_global_seq = db[messages_table].max(:global_seq)
- groups = db.fetch(sql_for_consumer_stats).all
- Stats.new(stream_count, max_global_seq, groups)
- end
-
- # Data structure representing a stream with its metadata.
- # @!attribute [r] stream_id
- # @return [String] Unique identifier for the stream
- # @!attribute [r] seq
- # @return [Integer] Latest sequence number in the stream
- # @!attribute [r] updated_at
- # @return [Time] Timestamp of the most recent message in the stream
- Stream = Data.define(:stream_id, :seq, :updated_at)
-
- # Retrieve a list of recently active streams, ordered by most recent activity.
- # This method is useful for diagnostics, monitoring, and debugging to understand
- # which streams have been most active in the system.
- #
- # The query is optimized with a database index on updated_at for efficient sorting
- # and uses LIMIT to minimize data transfer.
- #
- # @param limit [Integer] Maximum number of streams to return (defaults to 10, must be >= 0)
- # @return [Array] Array of Stream objects ordered by updated_at descending
- # @raise [ArgumentError] if limit is negative
- # @example Get the 5 most recently active streams
- # recent = backend.recent_streams(limit: 5)
- # recent.each do |stream|
- # puts "Stream #{stream.stream_id}: seq #{stream.seq} at #{stream.updated_at}"
- # end
- # @example Monitor system activity
- # streams = backend.recent_streams(limit: 20)
- # active_count = streams.count { |s| s.updated_at > 1.hour.ago }
- # puts "#{active_count} streams active in last hour"
- def recent_streams(limit: 10)
- # Input validation
- return [] if limit == 0
- raise ArgumentError, "limit must be a positive integer" if limit < 0
-
- # Optimized query with index on updated_at
- query = db[streams_table]
- .select(:stream_id, :seq, :updated_at)
- .reverse(:updated_at) # More idiomatic Sequel than order(Sequel.desc(:updated_at))
- .limit(limit)
-
- query.map do |row|
- Stream.new(**row)
- end
- end
-
- # Returns the backend's notifier for real-time message dispatch.
- # Returns a {PGNotifier} for PostgreSQL, {InlineNotifier} otherwise.
- # Initialized eagerly in the constructor for thread and CoW safety.
- #
- # @return [PGNotifier, InlineNotifier]
- attr_reader :notifier
-
- def transaction(&)
- db.transaction(&)
- end
-
- def schedule_messages(messages, at:)
- return false if messages.empty?
-
- now = Time.now
- rows = messages.map do |m|
- message_data = m.to_h
- message_data[:metadata] = message_data[:metadata].merge(scheduled_at: now)
- { created_at: now, available_at: at, message: JSON.dump(message_data) }
- end
-
- transaction do
- db[scheduled_messages_table].multi_insert(rows)
- end
-
- true
- end
-
- def update_schedule!
- now = Time.now
-
- transaction do
- rows = db[scheduled_messages_table]
- .where { available_at <= now }
- .order(:id)
- .limit(100)
- .for_update
- .skip_locked
- .all
-
- return 0 if rows.empty?
-
- # Process all messages
- messages = rows.map do |r|
- data = JSON.parse(r[:message], symbolize_names: true)
- data[:created_at] = now
- Message.from(data)
- end
-
- # Append messages to each stream
- messages.group_by(&:stream_id).each do |stream_id, stream_messages|
- append_next_to_stream(stream_id, stream_messages)
- end
-
- # Batch delete all processed messages
- row_ids = rows.map { |m| m[:id] }
- db[scheduled_messages_table].where(id: row_ids).delete
-
- rows.size
- end
- end
-
- # Record heartbeats for a set of worker IDs.
- # Accepts a String or an Array. Returns number of rows touched.
- def worker_heartbeat(worker_ids, at: Time.now)
- ids = Array(worker_ids).uniq
- return 0 if ids.empty?
-
- rows = ids.map { |id| { id:, last_seen: at } }
- db.transaction do
- db[workers_table]
- .insert_conflict(target: :id, update: { last_seen: at })
- .multi_insert(rows)
- end
- ids.size
- end
-
- # Release stale claims for workers that have not heartbeated within ttl_seconds.
- # Returns the number of offsets rows updated.
- def release_stale_claims(ttl_seconds: 120)
- cutoff = Time.now - ttl_seconds
-
- db.transaction do
- stale_workers = db[workers_table]
- .where { last_seen <= cutoff }
- .select(:id)
-
- ds = db[offsets_table]
- .where(claimed: true)
- .where { claimed_at <= cutoff }
- .where(claimed_by: stale_workers)
-
- ds.update(claimed: false, claimed_at: nil, claimed_by: nil)
- end
- end
-
- # Append one or more messages to a stream, creating the stream if it doesn't exist.
- # Also automatically increments the sequence number for the stream and messages,
- # without relying on the client passing the right sequence numbers.
- # This is used for appending messages in order without client-side optimistic concurrency control.
- # For example commands that will be picked up and handled by a reactor, later.
- # @param stream_id [String] Unique identifier for the message stream
- # @param messages [Sourced::Message, Array] Message(s) to append to the stream
- # @option max_retries [Integer] Maximum number of retries on unique constraint violation (default: 3)
- def append_next_to_stream(stream_id, messages, max_retries: 3)
- # Handle both single message and array of messages
- messages_array = Array(messages)
- return true if messages_array.empty?
-
- retries = 0
- begin
- db.transaction do
- # Update or create a stream.
- # If updating, increment seq by the number of messages being added.
- messages_count = messages_array.size
- stream_record = db[streams_table]
- .insert_conflict(
- target: :stream_id,
- update: {
- seq: Sequel.qualify(streams_table, :seq) + messages_count,
- updated_at: Time.now
- }
- )
- .returning(:id, :seq)
- .insert(stream_id:, seq: messages_count)
- .first
-
- # Calculate the starting sequence number for this batch
- # If stream existed, seq will be the new max seq after increment
- # If stream was created, seq will be messages_count
- starting_seq = stream_record[:seq] - messages_count + 1
-
- # Prepare rows for bulk insert with incrementing sequence numbers
- rows = messages_array.map.with_index do |msg, index|
- row = serialize_message(msg, stream_record[:id])
- row[:seq] = starting_seq + index
- row
- end
-
- # Bulk insert all messages
- db[messages_table].multi_insert(rows)
-
- # NOTIFY delivered on commit — atomically correct
- notifier.notify_new_messages(messages_array.map(&:type))
- end
-
- true
- rescue Sequel::UniqueConstraintViolation => e
- retries += 1
- if retries <= max_retries
- sleep(0.001 * retries) # Brief backoff
- retry
- else
- raise Sourced::ConcurrentAppendError, e.message
- end
- end
- end
-
- def append_to_stream(stream_id, messages)
- # Handle both single message and array of messages
- messages_array = Array(messages)
- return false if messages_array.empty?
-
- if messages_array.map(&:stream_id).uniq.size > 1
- raise ArgumentError, 'Messages must all belong to the same stream'
- end
-
- db.transaction do
- seq = messages_array.last.seq
- id = db[streams_table].insert_conflict(target: :stream_id, update: { seq:, updated_at: Time.now }).insert(stream_id:, seq:)
- rows = messages_array.map { |e| serialize_message(e, id) }
- db[messages_table].multi_insert(rows)
-
- notifier.notify_new_messages(messages_array.map(&:type))
- end
-
- true
- rescue Sequel::UniqueConstraintViolation => e
- raise Sourced::ConcurrentAppendError, e.message
- end
-
- # Reserve next message(s) for a reactor, based on the reactor's #handled_messages list.
- # This fetches the next un-acknowledged message(s) for the reactor, and processes them in a transaction
- # which acquires a lock on the message's stream_id and the reactor's group_id.
- # So that no other reactor instance in the same group can process the same stream concurrently.
- # This way, messages for the same stream are guaranteed to be processed sequentially for each reactor group.
- #
- # When batch_size > 1, fetches multiple messages from the same stream in a single
- # lock cycle, reducing per-message overhead for catch-up scenarios.
- #
- # The block is yielded once with the full batch and optional history.
- # All-or-nothing ACK semantics: on success all actions are executed and the last
- # message is ACKed; on RETRY the offset is released without ACK.
- #
- # @example
- # backend.reserve_next_for_reactor(reactor, batch_size: 50) do |batch, history|
- # # batch is Array of [message, replaying] pairs
- # reactor.handle_batch(batch, history:)
- # end
- #
- # @param reactor [Sourced::ReactorInterface]
- # @param batch_size [Integer] Number of messages to fetch per lock cycle (default: 1)
- # @param with_history [Boolean] Whether to fetch full stream history
- # @param worker_id [String]
- # @yieldparam batch [Array<[Sourced::Message, Boolean]>] array of [message, replaying] pairs
- # @yieldparam history [Array, nil] full stream history if with_history is true
- # @yieldreturn [Array<[actions, source_message]>, Sourced::Actions::RETRY] action pairs or RETRY
- def reserve_next_for_reactor(reactor, batch_size: 1, with_history: false, worker_id: nil, &block)
- worker_id ||= [Process.pid, Thread.current.object_id, Fiber.current.object_id].join('-')
- group_id = reactor.consumer_info.group_id
- handled_messages = reactor.handled_messages.map(&:type).uniq
- now = Time.now
-
- bootstrap_offsets_for(group_id)
-
- start_from = reactor.consumer_info.start_from.call
- rows, history = claim_and_fetch_batch(group_id, handled_messages, now, start_from, worker_id, batch_size, with_history:)
- return unless rows
-
- reserve_batch(group_id, rows, history:, &block)
- end
-
- private def release_offset(offset_id)
- db[offsets_table].where(id: offset_id).update(claimed: false, claimed_at: nil, claimed_by: nil)
- end
-
- # Process a batch of claimed messages through the reactor and handle the result.
- #
- # This is the core processing loop for batch message handling. It follows a
- # yield-once contract: the caller (Router) receives the full batch and returns
- # action pairs describing what side effects to execute.
- #
- # ## Flow
- #
- # 1. Deserialize raw DB rows into [Message, replaying] pairs
- # 2. Yield the batch + optional history to the caller (reactor.handle_batch)
- # 3. Handle the return value:
- # - Actions::RETRY → release the claimed offset, no ACK (message will be retried)
- # - Array of [actions, source_message] pairs → execute side effects and ACK
- # - Empty array → release offset (nothing to do, avoid stuck claim)
- #
- # ## ACK semantics
- #
- # All-or-nothing per batch. On success, all action pairs are executed in a single
- # DB transaction and the offset is advanced to the last message in the batch.
- # ack_message() both advances the offset AND releases the claim in one update.
- #
- # Action pairs don't map 1:1 to batch messages. For example, a Projector returns
- # one [sync_actions, last_msg] pair for the whole batch, plus separate
- # [reaction_actions, triggering_msg] pairs. We ACK based on the last row in the
- # batch regardless of which action pair triggered it.
- #
- # ## Error handling
- #
- # Any exception releases the claimed offset (so the batch can be retried by
- # another worker) and re-raises for the Router to handle via on_exception.
- #
- # @param group_id [String] consumer group ID
- # @param rows [Array] raw DB rows from claim_and_fetch_batch
- # @param history [Array, nil] deserialized stream history, if requested
- # @yield [batch, history] yields once to the caller for processing
- # @return [Message, nil] the first message in the batch (used by Router for logging)
- private def reserve_batch(group_id, rows, history: nil, &block)
- first_row = rows.first
-
- begin
- batch = rows.map { |row| [deserialize_message(row), row[:replaying]] }
-
- action_pairs = yield(batch, history)
-
- # RETRY: release offset without ACK. The batch will be picked up again.
- if action_pairs == Actions::RETRY
- release_offset(first_row[:offset_id])
- return batch.first&.first
- end
-
- # Execute all side effects and ACK in a single transaction.
- # This ensures actions (AppendNext, Sync, etc.) and the ACK are atomic.
- db.transaction do
- last_ack_row = nil
- action_pairs.each do |(actions, source_message)|
- should_ack = execute_actions(group_id, actions, source_message)
- if should_ack
- # Find the row matching this action_pair's source_message.
- # For partial batches (PartialBatchError), this ACKs up to the
- # last successfully processed message rather than the last batch row.
- last_ack_row = rows.find { |r| r[:id] == source_message.id } || rows.last
- end
- end
-
- if last_ack_row
- # Advance offset to last successfully processed message and release the claim
- ack_message(group_id, last_ack_row[:stream_id_fk], last_ack_row[:global_seq])
- else
- # No action pair triggered an ACK (e.g. empty action_pairs).
- # Release the claim so the batch isn't stuck.
- release_offset(first_row[:offset_id])
- end
- end
-
- batch.first&.first
-
- rescue StandardError
- # Release claim on any error so another worker can pick up the batch
- release_offset(first_row[:offset_id])
- raise
- end
- end
-
- # Atomically claim an offset and fetch up to batch_size messages in a single CTE query.
- # The CTE finds the first matching message (to identify the stream), claims the offset,
- # and fetches all pending messages from that stream in one round-trip.
- private def claim_and_fetch_batch(group_id, handled_messages, now, start_from, worker_id, batch_size, with_history: false)
- sql = if start_from.is_a?(Time)
- sql_for_claim_and_fetch_batch(handled_messages, group_id:, now:, worker_id:, batch_size:, start_from:, with_history:)
- else
- sql_for_claim_and_fetch_batch(handled_messages, group_id:, now:, worker_id:, batch_size:, with_history:)
- end
-
- all_rows = db.fetch(sql).all
- return NO_BATCH if all_rows.empty?
-
- if with_history
- batch_rows = all_rows.select { |r| r[:in_batch] }
- if batch_rows.empty?
- # Claimed but no batch messages — release the claim
- release_offset(all_rows.first[:offset_id])
- return NO_BATCH
- end
- # Deserialize all rows as history (also parses JSON for batch rows since they share objects)
- history = all_rows.map { |row| deserialize_message(row) }
- [batch_rows, history]
- else
- [all_rows, nil]
- end
- rescue Sequel::UniqueConstraintViolation
- logger.debug "Batch claim for group #{group_id} already exists, skipping"
- NO_BATCH
- end
-
- # Execute action side effects without ACKing.
- # Returns true if the message should be ACKed by the caller.
- private def execute_actions(group_id, actions, message)
- should_ack = false
- actions = [actions] unless actions.is_a?(Array)
- return true if actions.empty? # empty = implicit ACK
-
- actions.each do |action|
- case action
- when nil, Actions::OK
- should_ack = true
-
- when Actions::Ack
- ack_on(group_id, action.message_id)
-
- when Actions::RETRY
- # Should not reach here (filtered by batch loop)
-
- else
- action.execute(self, message)
- should_ack = true
- end
- end
-
- should_ack
- end
-
- # Insert missing offsets for a consumer group and all streams.
- # This runs in advance of claiming messages for processing
- # so that the claim query relies on existing offsets and FOR UPDATE SKIP LOCKED
- # to prevent concurrent claims on the same stream by different workers.
- private def bootstrap_offsets_for(group_id)
- now = Time.now
- db.transaction do
- group = db[consumer_groups_table].where(group_id:, status: ACTIVE).first
- return if group.nil?
-
- # Find streams that don't have offsets for this group
- missing_streams = db[streams_table]
- .left_join(offsets_table,
- stream_id: :id,
- group_id: group[:id])
- .where(Sequel[offsets_table][:id] => nil)
- .select(Sequel[streams_table][:id], Sequel[streams_table][:stream_id])
- .limit(100)
-
- # Prepare offset records for bulk insert
- offset_records = missing_streams.map do |stream|
- {
- group_id: group[:id],
- stream_id: stream[:id],
- global_seq: 0,
- claimed: false,
- created_at: now
- }
- end
-
- if offset_records.any?
- db[offsets_table].insert_conflict.multi_insert(offset_records)
- end
-
- offset_records.size
- end
- end
-
-
- def register_consumer_group(group_id)
- db[consumer_groups_table]
- .insert_conflict(target: :group_id, update: nil)
- .insert(group_id:, status: ACTIVE)
- end
-
- # Fetch and update a consumer group in a transaction
- # Used by Router to stop / retry consumer groups
- # when handing exceptions.
- #
- # @example
- # backend.updating_consumer_group(group_id) do |group|
- # group.stop('some reason')
- # end
- #
- # @param group_id [String]
- # @yield [GroupUpdater]
- def updating_consumer_group(group_id, &)
- dataset = db[consumer_groups_table].where(group_id:)
- group_row = dataset.for_update.first
- raise ArgumentError, "Consumer group #{group_id} not found" unless group_row
-
- ctx = group_row[:error_context] ? parse_json(group_row[:error_context]) : {}
- group_row[:error_context] = ctx
- group = GroupUpdater.new(group_id, group_row, logger)
- yield group
- updates = group.updates
- updates[:error_context] = JSON.dump(updates[:error_context])
- dataset.update(updates)
- end
-
- # Start a consumer group that has been stopped.
- # Signals the notifier so workers pick up the reactor immediately.
- #
- # @param group_id [String]
- def start_consumer_group(group_id)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- dataset = db[consumer_groups_table].where(group_id: group_id)
- dataset.update(status: ACTIVE, retry_at: nil, error_context: nil)
- notifier.notify_reactor_resumed(group_id)
- end
-
- # @param group_id [String]
- # @param reason [#inspect, NilClass]
- def stop_consumer_group(group_id, reason = nil)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- updating_consumer_group(group_id) do |group|
- group.stop(reason)
- end
- end
-
- # Reset offsets for all streams tracked by a consumer group.
- # If the consumer group is active, this will make it re-process all messages
- #
- # @param group_id [String]
- # @return [Boolean]
- def reset_consumer_group(group_id)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- db.transaction do
- row = db[consumer_groups_table].where(group_id:).select(:id).first
- return unless row
-
- id = row[:id]
- db[offsets_table].where(group_id: id).delete
- end
-
- true
- end
-
- def ack_on(group_id, message_id, &)
- db.transaction do
- row = db.fetch(sql_for_ack_on, group_id, message_id).first
- raise Sourced::ConcurrentAckError, "Stream for message #{message_id} is being concurrently processed by #{group_id}" unless row
-
- yield if block_given?
-
- ack_message(group_id, row[:stream_id_fk], row[:global_seq])
- end
- end
-
- private def ack_message(group_id, stream_id, global_seq)
- db.transaction do
- # We could use an upsert here, but
- # we create consumer groups on registration, or
- # after the first update.
- # So by this point, the consumer groups table
- # will always have a record for the group_id.
- # So we assume it's there and update it directly.
- # If the group_id doesn't exist, it will be created below,
- # but this should only happen once per group.
- update_result = db[consumer_groups_table]
- .where(group_id:)
- .returning(:id)
- .update(
- updated_at: Sequel.function(:now),
- highest_global_seq: Sequel.function(
- :greatest,
- :highest_global_seq,
- global_seq
- )
- )
-
- # Here we do issue a separate INSERT, but this should only happen
- # if the group record doesn't exist yet, for some reason.
- if update_result.empty? # No rows were updated (record doesn't exist)
- # Only then INSERT
- update_result = db[consumer_groups_table]
- .returning(:id)
- .insert(group_id:, highest_global_seq: global_seq)
- end
-
- group_id = update_result[0][:id]
-
- # Upsert offset
- # NOTE: when using #reserve_next_for_reactor,
- # an offset will always exist for an message
- # but the synchronous ack_on method can be used with an message
- # for whose group and stream no offset exists yet.
- db[offsets_table]
- .insert_conflict(
- target: [:group_id, :stream_id],
- update: { claimed: false, claimed_at: nil, claimed_by: nil, global_seq: }
- )
- .insert(
- group_id:,
- stream_id:,
- global_seq:
- )
- end
- end
-
- private def base_messages_query
- db[messages_table]
- .select(
- Sequel[messages_table][:id],
- Sequel[streams_table][:stream_id],
- Sequel[messages_table][:seq],
- Sequel[messages_table][:global_seq],
- Sequel[messages_table][:type],
- Sequel[messages_table][:created_at],
- Sequel[messages_table][:causation_id],
- Sequel[messages_table][:correlation_id],
- Sequel[messages_table][:metadata],
- Sequel[messages_table][:payload],
- )
- .join(streams_table, id: :stream_id)
- end
-
- def read_correlation_batch(message_id)
- correlation_subquery = db[messages_table]
- .select(:correlation_id)
- .where(id: message_id)
-
- query = base_messages_query
- .where(Sequel[messages_table][:correlation_id] => correlation_subquery)
- .order(Sequel[messages_table][:global_seq])
-
- query.map do |row|
- deserialize_message(row)
- end
- end
-
- def read_stream(stream_id, after: nil, upto: nil)
- _messages_table = messages_table # need local variable for Sequel block
-
- query = base_messages_query.where(Sequel[streams_table][:stream_id] => stream_id)
-
- query = query.where { Sequel[_messages_table][:seq] > after } if after
- query = query.where { Sequel[_messages_table][:seq] <= upto } if upto
- query.order(Sequel[_messages_table][:global_seq]).map do |row|
- deserialize_message(row)
- end
- end
-
- # For tests only
- def clear!
- raise 'Not in test environment' unless ENV['ENVIRONMENT'] == 'test'
- # Truncate and restart global_seq increment first
- db[messages_table].truncate(cascade: true, only: true, restart: true)
- db[messages_table].delete
- db[consumer_groups_table].delete
- db[offsets_table].delete
- db[streams_table].delete
- db[scheduled_messages_table].delete
- db[workers_table].delete
- end
-
- # Called after Sourced.configure
- def setup!(config)
- return if @setup
- if config.executor.is_a?(Sourced::AsyncExecutor)
- Sequel.extension :fiber_concurrency
- end
- @setup = true
- end
-
- # Check if all required database tables exist.
- # This verifies that the backend has been properly installed with all necessary schema.
- #
- # @return [Boolean] true if all required tables exist, false otherwise
- # @see #install
- def installed?
- installer.installed?
- end
-
- def uninstall
- installer.uninstall
- self
- end
-
- def install
- installer.install
- self
- end
-
- def copy_migration_to(dir = nil, &block)
- installer.copy_migration_to(dir, &block)
- end
-
- private
-
- # Override in subclasses to configure the notifier.
- # Default: PGNotifier for PG, InlineNotifier otherwise.
- def setup_adapter
- @notifier = @db.adapter_scheme == :postgres ? PGNotifier.new(db: @db) : InlineNotifier.new
- end
-
- # Assign table name ivars and their literal SQL representations.
- def setup_tables
- @workers_table = table_name(:workers)
- @scheduled_messages_table = table_name(:scheduled_messages)
- @streams_table = table_name(:streams)
- @offsets_table = table_name(:offsets)
- @consumer_groups_table = table_name(:consumer_groups)
- @messages_table = table_name(:messages)
- # Literal SQL strings for use in raw SQL interpolation.
- # Sequel qualified identifiers don't produce valid SQL via #to_s,
- # so we pre-compute literal versions for the raw SQL methods.
- @messages_table_literal = @db.literal(@messages_table)
- @streams_table_literal = @db.literal(@streams_table)
- @offsets_table_literal = @db.literal(@offsets_table)
- @consumer_groups_table_literal = @db.literal(@consumer_groups_table)
- end
-
- attr_reader :db, :logger, :prefix, :schema, :messages_table, :streams_table, :offsets_table, :consumer_groups_table, :scheduled_messages_table, :workers_table
-
- # Override in subclasses to use a different migration template.
- def migration_template_name
- '001_create_sourced_tables.rb.erb'
- end
-
- def installer
- @installer ||= Installer.new(
- @db,
- logger:,
- schema:,
- prefix:,
- migration_template: migration_template_name
- )
- end
-
- # CTE-based SQL that atomically claims an offset and fetches batch messages in one statement.
- # 1. first_match: finds the first pending message to identify the stream/offset (FOR UPDATE SKIP LOCKED)
- # 2. claim: claims the offset in the same statement (UPDATE ... RETURNING)
- # 3. outer SELECT: fetches up to batch_size messages from the claimed stream
- #
- # When with_history: true, adds a `batch` CTE to identify batch rows, and the outer SELECT
- # fetches ALL messages from the stream (for history), marking batch rows with `in_batch`.
- # This saves a separate read_stream query for reactors that need history.
- def sql_for_claim_and_fetch_batch(handled_messages, group_id:, now:, worker_id:, batch_size:, start_from: nil, with_history: false)
- message_types = handled_messages.map { |e| "'#{e}'" }
- now_literal = db.literal(now)
- group_id_literal = db.literal(group_id)
- worker_id_literal = db.literal(worker_id)
- batch_size_literal = db.literal(batch_size)
- message_types_sql = message_types.any? ? " AND e.type IN(#{message_types.join(',')})" : ''
- time_window_sql = start_from ? " AND e.created_at > #{db.literal(start_from)}" : ''
-
- # Common CTEs: find first matching message, claim its offset
- first_match_and_claim = <<~SQL
- WITH first_match AS (
- SELECT
- so.id as offset_id, ss.stream_id, e.stream_id as stream_id_fk,
- cg.id as group_id_fk,
- cg.highest_global_seq,
- so.global_seq as offset_seq
- FROM #{@messages_table_literal} e
- JOIN #{@streams_table_literal} ss ON e.stream_id = ss.id
- JOIN #{@consumer_groups_table_literal} cg ON cg.group_id = #{group_id_literal}
- JOIN #{@offsets_table_literal} so ON cg.id = so.group_id AND ss.id = so.stream_id
- WHERE e.global_seq > so.global_seq
- AND so.claimed = FALSE
- AND cg.status = 'active'
- AND (cg.retry_at IS NULL OR cg.retry_at <= #{now_literal})
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- FOR UPDATE OF so SKIP LOCKED
- LIMIT 1
- ),
- claim AS (
- UPDATE #{@offsets_table_literal}
- SET claimed = TRUE, claimed_at = #{now_literal}, claimed_by = #{worker_id_literal}
- FROM first_match
- WHERE #{@offsets_table_literal}.id = first_match.offset_id
- RETURNING first_match.offset_id, first_match.stream_id, first_match.stream_id_fk,
- first_match.group_id_fk, first_match.highest_global_seq, first_match.offset_seq
- )
- SQL
-
- if with_history
- # Fetch ALL stream messages + mark which are in the batch via LEFT JOIN.
- # The batch CTE identifies the batch rows (after offset, matching types, limited).
- # The outer SELECT returns every message in the stream for history.
- first_match_and_claim + <<~SQL
- ,
- batch AS (
- SELECT e.global_seq
- FROM claim
- JOIN #{@messages_table_literal} e ON e.stream_id = claim.stream_id_fk
- WHERE e.global_seq > claim.offset_seq
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- LIMIT #{batch_size_literal}
- )
- SELECT e.global_seq, e.id, claim.stream_id, e.stream_id as stream_id_fk,
- e.seq, e.type, e.created_at, e.causation_id, e.correlation_id,
- e.metadata, e.payload, claim.offset_id, claim.group_id_fk,
- (e.global_seq <= claim.highest_global_seq) AS replaying,
- claim.highest_global_seq,
- (batch.global_seq IS NOT NULL) AS in_batch
- FROM claim
- JOIN #{@messages_table_literal} e ON e.stream_id = claim.stream_id_fk
- LEFT JOIN batch ON batch.global_seq = e.global_seq
- ORDER BY e.global_seq ASC;
- SQL
- else
- first_match_and_claim + <<~SQL
- SELECT e.global_seq, e.id, claim.stream_id, e.stream_id as stream_id_fk,
- e.seq, e.type, e.created_at, e.causation_id, e.correlation_id,
- e.metadata, e.payload, claim.offset_id, claim.group_id_fk,
- (e.global_seq <= claim.highest_global_seq) AS replaying,
- claim.highest_global_seq
- FROM claim
- JOIN #{@messages_table_literal} e ON e.stream_id = claim.stream_id_fk
- WHERE e.global_seq > claim.offset_seq
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- LIMIT #{batch_size_literal};
- SQL
- end
- end
-
- def sql_for_ack_on
- @sql_for_ack_on ||= <<~SQL
- WITH candidate_rows AS (
- SELECT
- e.global_seq,
- e.stream_id AS stream_id_fk,
- pg_try_advisory_xact_lock(hashtext(?::text), hashtext(s.id::text)) as lock_obtained
- FROM #{@messages_table_literal} e
- JOIN #{@streams_table_literal} s ON e.stream_id = s.id
- WHERE e.id = ?
- )
- SELECT *
- FROM candidate_rows
- WHERE lock_obtained = true
- LIMIT 1;
- SQL
- end
-
- def sql_for_consumer_stats
- @sql_for_consumer_stats ||= <<~SQL
- SELECT
- r.group_id,
- r.status,
- r.retry_at,
- COALESCE(MIN(o.global_seq), 0) AS oldest_processed,
- COALESCE(MAX(o.global_seq), 0) AS newest_processed,
- COUNT(o.id) AS stream_count
- FROM #{@consumer_groups_table_literal} r
- LEFT JOIN #{@offsets_table_literal} o ON o.group_id = r.id AND o.global_seq > 0
- GROUP BY r.id, r.group_id, r.status, r.retry_at
- ORDER BY r.group_id;
- SQL
- end
-
- def table_name(name)
- t = [prefix, name].join('_').to_sym
- schema ? Sequel[schema.to_sym][t] : t
- end
-
- def parse_json(json)
- return json unless json.is_a?(String)
-
- JSON.parse(json, symbolize_names: true)
- end
-
- def upsert_consumer_group(group_id, status: ACTIVE)
- db[consumer_groups_table]
- .insert_conflict(target: :group_id, update: { status:, updated_at: Time.now })
- .insert(group_id:, status:)
- end
-
- def serialize_message(message, stream_id)
- row = message.to_h
- row[:stream_id] = stream_id
- row[:metadata] = JSON.dump(row[:metadata]) if row[:metadata]
- row[:payload] = JSON.dump(row[:payload]) if row[:payload]
- row
- end
-
- def deserialize_message(row)
- row[:payload] = parse_json(row[:payload]) if row[:payload]
- row[:metadata] = parse_json(row[:metadata]) if row[:metadata]
- Message.from(row)
- end
-
- def connect(db)
- case db
- when Sequel::Database
- db
- when String, Hash
- Sequel.connect(db)
- else
- raise ArgumentError, "Invalid database connection: #{db.inspect}"
- end
- end
- end
- end
-end
-
-require 'sourced/backends/sequel_backend/installer'
-require 'sourced/backends/sequel_backend/group_updater'
-require 'sourced/backends/sequel_backend/pg_notifier'
diff --git a/lib/sourced/backends/sequel_backend/group_updater.rb b/lib/sourced/backends/sequel_backend/group_updater.rb
deleted file mode 100644
index d668c32d..00000000
--- a/lib/sourced/backends/sequel_backend/group_updater.rb
+++ /dev/null
@@ -1,34 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Backends
- class SequelBackend
- class GroupUpdater
- attr_reader :group_id, :updates, :error_context
-
- def initialize(group_id, row, logger)
- @group_id = group_id
- @row = row
- @logger = logger
- @error_context = row[:error_context]
- @updates = { error_context: @error_context.dup }
- end
-
- def stop(reason = nil)
- @logger.error "stopping consumer group #{group_id}"
- @updates[:status] = STOPPED
- @updates[:retry_at] = nil
- @updates[:updated_at] = Time.now
- @updates[:error_context][:reason] = reason if reason
- end
-
- def retry(time, ctx = {})
- @logger.warn "retrying consumer group #{group_id} at #{time}"
- @updates[:updated_at] = Time.now
- @updates[:retry_at] = time
- @updates[:error_context].merge!(ctx)
- end
- end
- end
- end
-end
diff --git a/lib/sourced/backends/sequel_backend/installer.rb b/lib/sourced/backends/sequel_backend/installer.rb
deleted file mode 100644
index bedac1d4..00000000
--- a/lib/sourced/backends/sequel_backend/installer.rb
+++ /dev/null
@@ -1,88 +0,0 @@
-# frozen_string_literal: true
-
-require 'sequel'
-require 'sequel/extensions/migration'
-require 'erb'
-
-module Sourced
- module Backends
- class SequelBackend
- class Installer
- def initialize(db, logger:, schema: nil, prefix: 'sourced', migration_template: '001_create_sourced_tables.rb.erb')
- raise ArgumentError, "invalid prefix: #{prefix}" unless prefix.match?(/\A[a-zA-Z_]\w*\z/)
- raise ArgumentError, "invalid schema: #{schema}" if schema && !schema.match?(/\A[a-zA-Z_]\w*\z/)
-
- @db = db
- @logger = logger
- @schema = schema
- @prefix = prefix
- @migration_template = migration_template
- end
-
- # Eval the rendered migration and apply :up directly.
- # No Migrator, no schema_info tracking table.
- def install
- migration.apply(db, :up)
- logger.info("Sourced tables installed (prefix: #{prefix}, schema: #{schema || 'default'})")
- end
-
- # Check that all expected tables exist.
- def installed?
- all_table_names.all? { |t| db.table_exists?(t) }
- end
-
- # Apply :down on the migration to drop tables.
- def uninstall
- raise 'Not in test environment' unless ENV['ENVIRONMENT'] == 'test'
-
- migration.apply(db, :down)
- if schema
- db.run("DROP SCHEMA IF EXISTS #{db.literal(schema.to_sym)}")
- end
- end
-
- # Render the migration to a file for use with the host app's Sequel::Migrator.
- #
- # installer.copy_migration_to("db/migrations")
- # installer.copy_migration_to { "db/migrations/#{Time.now.strftime('%Y%m%d%H%M%S')}_create_sourced_tables.rb" }
- #
- def copy_migration_to(dir = nil, &block)
- path = block ? block.call : File.join(dir, '001_create_sourced_tables.rb')
- File.write(path, rendered_migration)
- logger.info("Copied Sourced migration to #{path}")
- path
- end
-
- private
-
- attr_reader :db, :logger, :schema, :prefix
-
- def migration
- @migration ||= eval(rendered_migration) # rubocop:disable Security/Eval
- end
-
- def rendered_migration
- @rendered_migration ||= begin
- template_path = File.join(__dir__, 'migrations', @migration_template)
- ERB.new(File.read(template_path)).result(binding)
- end
- end
-
- # Used in ERB template to produce table name expressions.
- # Returns Ruby source code strings, e.g. `:sourced_streams` or `Sequel[:my_schema][:sourced_streams]`
- def table_name(name)
- t = :"#{prefix}_#{name}"
- schema ? "Sequel[:#{schema}][:#{t}]" : ":#{t}"
- end
-
- # Returns actual Sequel identifiers for use in installed? checks.
- def all_table_names
- %i[streams consumer_groups offsets messages scheduled_messages workers].map do |name|
- t = :"#{prefix}_#{name}"
- schema ? Sequel[schema.to_sym][t] : t
- end
- end
- end
- end
- end
-end
diff --git a/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables.rb.erb b/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables.rb.erb
deleted file mode 100644
index ff0e3e92..00000000
--- a/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables.rb.erb
+++ /dev/null
@@ -1,83 +0,0 @@
-# frozen_string_literal: true
-
-Sequel.migration do
- change do
-<% if schema %>
- run 'CREATE SCHEMA IF NOT EXISTS "<%= schema %>"'
-<% end %>
-
- create_table(<%= table_name(:streams) %>) do
- primary_key :id
- String :stream_id, null: false, unique: true
- Time :updated_at, null: false, default: Sequel.function(:now)
- Bignum :seq, null: false
-
- index :updated_at, name: 'idx_<%= prefix %>_streams_updated_at'
- end
-
- create_table(<%= table_name(:consumer_groups) %>) do
- primary_key :id
- String :group_id, null: false, unique: true
- Bignum :highest_global_seq, null: false, default: 0
- String :status, null: false, default: 'active', index: true
- column :error_context, :jsonb
- Time :retry_at, null: true, index: true
- Time :created_at, null: false, default: Sequel.function(:now)
- Time :updated_at, null: false, default: Sequel.function(:now)
-
- index :group_id, unique: true
- end
-
- create_table(<%= table_name(:offsets) %>) do
- primary_key :id
- foreign_key :group_id, <%= table_name(:consumer_groups) %>, on_delete: :cascade
- foreign_key :stream_id, <%= table_name(:streams) %>, on_delete: :cascade
- Bignum :global_seq, null: false
- Time :created_at, null: false, default: Sequel.function(:now)
- TrueClass :claimed, null: false, default: false
- Time :claimed_at, null: true
- String :claimed_by, null: true
-
- index %i[group_id stream_id], unique: true
- index :claimed, where: { claimed: false }, name: 'idx_<%= prefix %>_offsets_unclaimed'
- index [:claimed, :claimed_by, :claimed_at], where: { claimed: true }, name: 'idx_<%= prefix %>_offsets_claimed_claimer'
- index %i[group_id global_seq], name: 'idx_<%= prefix %>_offsets_group_seq_covering'
- end
-
- create_table(<%= table_name(:messages) %>) do
- primary_key :global_seq, type: :Bignum
- column :id, :uuid, unique: true
- foreign_key :stream_id, <%= table_name(:streams) %>
- Bignum :seq, null: false
- String :type, null: false
- Time :created_at, null: false
- column :causation_id, :uuid, index: true
- column :correlation_id, :uuid, index: true
- column :metadata, :jsonb
- column :payload, :jsonb
-
- index %i[stream_id seq], unique: true
- index :type
- index :created_at
- index %i[type global_seq]
- index %i[stream_id global_seq]
- end
-
- create_table(<%= table_name(:scheduled_messages) %>) do
- primary_key :id
- Time :created_at, null: false
- Time :available_at, null: false
- column :message, :jsonb
-
- index :available_at
- end
-
- create_table(<%= table_name(:workers) %>) do
- String :id, primary_key: true, null: false
- Time :last_seen, null: false, index: true
- String :pid, null: true
- String :host, null: true
- column :info, :jsonb
- end
- end
-end
diff --git a/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables_sqlite.rb.erb b/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables_sqlite.rb.erb
deleted file mode 100644
index 8398c6b2..00000000
--- a/lib/sourced/backends/sequel_backend/migrations/001_create_sourced_tables_sqlite.rb.erb
+++ /dev/null
@@ -1,79 +0,0 @@
-# frozen_string_literal: true
-
-Sequel.migration do
- change do
- create_table(<%= table_name(:streams) %>) do
- primary_key :id
- String :stream_id, null: false, unique: true
- Time :updated_at, null: false, default: Sequel::CURRENT_TIMESTAMP
- Bignum :seq, null: false
-
- index :updated_at, name: 'idx_<%= prefix %>_streams_updated_at'
- end
-
- create_table(<%= table_name(:consumer_groups) %>) do
- primary_key :id
- String :group_id, null: false, unique: true
- Bignum :highest_global_seq, null: false, default: 0
- String :status, null: false, default: 'active', index: true
- String :error_context
- Time :retry_at, null: true, index: true
- Time :created_at, null: false, default: Sequel::CURRENT_TIMESTAMP
- Time :updated_at, null: false, default: Sequel::CURRENT_TIMESTAMP
-
- index :group_id, unique: true
- end
-
- create_table(<%= table_name(:offsets) %>) do
- primary_key :id
- foreign_key :group_id, <%= table_name(:consumer_groups) %>, on_delete: :cascade
- foreign_key :stream_id, <%= table_name(:streams) %>, on_delete: :cascade
- Bignum :global_seq, null: false
- Time :created_at, null: false, default: Sequel::CURRENT_TIMESTAMP
- TrueClass :claimed, null: false, default: false
- Time :claimed_at, null: true
- String :claimed_by, null: true
-
- index %i[group_id stream_id], unique: true
- index :claimed, name: 'idx_<%= prefix %>_offsets_unclaimed'
- index [:claimed, :claimed_by, :claimed_at], name: 'idx_<%= prefix %>_offsets_claimed_claimer'
- index %i[group_id global_seq], name: 'idx_<%= prefix %>_offsets_group_seq_covering'
- end
-
- create_table(<%= table_name(:messages) %>) do
- primary_key :global_seq, type: :Bignum
- String :id, unique: true
- foreign_key :stream_id, <%= table_name(:streams) %>
- Bignum :seq, null: false
- String :type, null: false
- Time :created_at, null: false
- String :causation_id, index: true
- String :correlation_id, index: true
- String :metadata
- String :payload
-
- index %i[stream_id seq], unique: true
- index :type
- index :created_at
- index %i[type global_seq]
- index %i[stream_id global_seq]
- end
-
- create_table(<%= table_name(:scheduled_messages) %>) do
- primary_key :id
- Time :created_at, null: false
- Time :available_at, null: false
- String :message
-
- index :available_at
- end
-
- create_table(<%= table_name(:workers) %>) do
- String :id, primary_key: true, null: false
- Time :last_seen, null: false, index: true
- String :pid, null: true
- String :host, null: true
- String :info
- end
- end
-end
diff --git a/lib/sourced/backends/sequel_backend/pg_notifier.rb b/lib/sourced/backends/sequel_backend/pg_notifier.rb
deleted file mode 100644
index a5b97c88..00000000
--- a/lib/sourced/backends/sequel_backend/pg_notifier.rb
+++ /dev/null
@@ -1,96 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Backends
- class SequelBackend
- # PostgreSQL LISTEN/NOTIFY pub/sub transport for real-time message dispatch.
- #
- # Events are multiplexed over a single PG channel. The wire format is
- # +event_name:value+ (split on the first colon).
- #
- # Implements the same interface as {Sourced::InlineNotifier}.
- #
- # @example Wiring in a Dispatcher
- # pg_notifier = PGNotifier.new(db: Sequel.postgres('mydb'))
- # pg_notifier.subscribe(queuer)
- #
- # # In a dedicated fiber:
- # pg_notifier.start # blocks, listening for notifications
- #
- # # From the append path (inside a transaction):
- # pg_notifier.notify_new_messages(['orders.created', 'orders.shipped'])
- #
- # # From start_consumer_group:
- # pg_notifier.notify_reactor_resumed('OrderReactor')
- class PGNotifier
- # @return [String] PG NOTIFY channel name
- CHANNEL = 'sourced_new_messages'
-
- # @param db [Sequel::Database] a PostgreSQL Sequel database connection.
- # The LISTEN connection should be separate from the one used for writes
- # to avoid blocking.
- def initialize(db:)
- @db = db
- @subscribers = []
- @listening = false
- end
-
- # Register a subscriber to receive all published events.
- #
- # @param callable [#call] receives +(event_name, value)+ where both are Strings
- # @return [void]
- def subscribe(callable)
- @subscribers << callable
- end
-
- # Publish an event via PG NOTIFY. Should be called inside a transaction
- # so the notification is delivered atomically on commit.
- # Wire format: +event_name:value+.
- #
- # @param event_name [String] event name
- # @param value [String] event payload
- # @return [void]
- def publish(event_name, value)
- @db.run(Sequel.lit("SELECT pg_notify('#{CHANNEL}', ?)", "#{event_name}:#{value}"))
- end
-
- # Notify that new messages were appended.
- #
- # @param types [Array] message type strings
- # @return [void]
- def notify_new_messages(types)
- publish('messages_appended', types.uniq.join(','))
- end
-
- # Notify that a stopped reactor has been resumed.
- #
- # @param group_id [String] consumer group ID of the resumed reactor
- # @return [void]
- def notify_reactor_resumed(group_id)
- publish('reactor_resumed', group_id)
- end
-
- # Block on PG LISTEN, dispatching events to all subscribers.
- # Loops with a 2-second timeout so {#stop} is checked periodically.
- #
- # @return [void]
- def start
- @listening = true
- while @listening
- @db.listen(CHANNEL, timeout: 2) do |_ch, _pid, payload|
- event_name, value = payload.split(':', 2)
- @subscribers.each { |s| s.call(event_name, value) }
- end
- end
- end
-
- # Signal the LISTEN loop to exit after the current timeout cycle.
- #
- # @return [void]
- def stop
- @listening = false
- end
- end
- end
- end
-end
diff --git a/lib/sourced/backends/sqlite_backend.rb b/lib/sourced/backends/sqlite_backend.rb
deleted file mode 100644
index 82e7602f..00000000
--- a/lib/sourced/backends/sqlite_backend.rb
+++ /dev/null
@@ -1,454 +0,0 @@
-# frozen_string_literal: true
-
-require 'sourced/backends/sequel_backend'
-
-module Sourced
- module Backends
- # SQLite-specific backend. Inherits shared behaviour from {SequelBackend}
- # and overrides methods that rely on PG-specific features:
- #
- # - No CTEs with +UPDATE ... RETURNING+ — uses separate SELECT/UPDATE steps
- # - No +FOR UPDATE SKIP LOCKED+ — relies on SQLite's transaction-level write lock
- # - No advisory locks — +ack_on+ runs inside a plain transaction
- # - No +TRUNCATE+ — +clear!+ uses +DELETE+
- # - No +greatest()+ — uses +max()+ instead
- #
- # Uses {InlineNotifier} for synchronous worker dispatch (no PG +LISTEN/NOTIFY+).
- # The {CatchUpPoller} provides a safety-net poll for startup catch-up and missed signals.
- #
- # Best suited for single-process deployments, development, scripts, and tests.
- #
- # @example File-based database
- # db = Sequel.sqlite('myapp.db')
- # backend = Sourced::Backends::SQLiteBackend.new(db)
- # backend.install unless backend.installed?
- #
- # @example In-memory database (useful for scripts and tests)
- # db = Sequel.sqlite
- # backend = Sourced::Backends::SQLiteBackend.new(db)
- #
- # @see SequelBackend
- class SQLiteBackend < SequelBackend
- # Move scheduled messages whose +available_at+ has passed into the main message log.
- #
- # Unlike the PG version, this does not use +FOR UPDATE SKIP LOCKED+ to prevent
- # concurrent workers from selecting the same rows. Instead, SQLite's database-level
- # write lock serializes concurrent callers.
- #
- # @return [Integer] number of scheduled messages promoted
- def update_schedule!
- now = Time.now
-
- transaction do
- rows = db[scheduled_messages_table]
- .where { available_at <= now }
- .order(:id)
- .limit(100)
- .all
-
- return 0 if rows.empty?
-
- messages = rows.map do |r|
- data = JSON.parse(r[:message], symbolize_names: true)
- data[:created_at] = now
- Message.from(data)
- end
-
- messages.group_by(&:stream_id).each do |stream_id, stream_messages|
- append_next_to_stream(stream_id, stream_messages)
- end
-
- row_ids = rows.map { |m| m[:id] }
- db[scheduled_messages_table].where(id: row_ids).delete
-
- rows.size
- end
- end
-
- # Append messages to a stream, auto-incrementing sequence numbers.
- #
- # SQLite doesn't support +RETURNING+ on +INSERT ... ON CONFLICT+, so the stream
- # record is fetched with a separate +SELECT+ after the upsert. Retries on
- # +UniqueConstraintViolation+ with linear backoff up to +max_retries+.
- #
- # @param stream_id [String] the stream to append to
- # @param messages [Message, Array] messages to append
- # @param max_retries [Integer] retry limit for constraint violations (default: 3)
- # @return [true]
- # @raise [Sourced::ConcurrentAppendError] after exhausting retries
- def append_next_to_stream(stream_id, messages, max_retries: 3)
- messages_array = Array(messages)
- return true if messages_array.empty?
-
- retries = 0
- begin
- db.transaction do
- messages_count = messages_array.size
-
- # Upsert stream — SQLite doesn't support RETURNING on INSERT
- db[streams_table]
- .insert_conflict(
- target: :stream_id,
- update: {
- seq: Sequel.qualify(streams_table, :seq) + messages_count,
- updated_at: Time.now
- }
- )
- .insert(stream_id:, seq: messages_count)
-
- stream_record = db[streams_table].where(stream_id:).select(:id, :seq).first
-
- starting_seq = stream_record[:seq] - messages_count + 1
-
- rows = messages_array.map.with_index do |msg, index|
- row = serialize_message(msg, stream_record[:id])
- row[:seq] = starting_seq + index
- row
- end
-
- db[messages_table].multi_insert(rows)
-
- notifier.notify_new_messages(messages_array.map(&:type))
- end
-
- true
- rescue Sequel::UniqueConstraintViolation => e
- retries += 1
- if retries <= max_retries
- sleep(0.001 * retries)
- retry
- else
- raise Sourced::ConcurrentAppendError, e.message
- end
- end
- end
-
- # Append messages to a stream with optimistic locking (sequence check).
- #
- # Messages must carry pre-assigned +seq+ values. A +UniqueConstraintViolation+
- # on the +[stream_id, seq]+ index signals a concurrent append.
- #
- # @param stream_id [String] the stream to append to
- # @param messages [Message, Array] messages with pre-assigned sequences
- # @return [true]
- # @raise [Sourced::ConcurrentAppendError] on sequence conflict
- def append_to_stream(stream_id, messages)
- messages_array = Array(messages)
- return false if messages_array.empty?
-
- if messages_array.map(&:stream_id).uniq.size > 1
- raise ArgumentError, 'Messages must all belong to the same stream'
- end
-
- db.transaction do
- seq = messages_array.last.seq
- db[streams_table]
- .insert_conflict(target: :stream_id, update: { seq:, updated_at: Time.now })
- .insert(stream_id:, seq:)
-
- # Separate SELECT — SQLite insert_conflict doesn't reliably return the id
- stream_row = db[streams_table].where(stream_id:).select(:id).first
- id = stream_row[:id]
-
- rows = messages_array.map { |e| serialize_message(e, id) }
- db[messages_table].multi_insert(rows)
-
- notifier.notify_new_messages(messages_array.map(&:type))
- end
-
- true
- rescue Sequel::UniqueConstraintViolation => e
- raise Sourced::ConcurrentAppendError, e.message
- end
-
- # Yield a {GroupUpdater} for the given consumer group inside a transaction.
- #
- # The PG version uses +FOR UPDATE+ to row-lock the group. SQLite relies on its
- # database-level transaction lock instead.
- #
- # @param group_id [String] consumer group identifier
- # @yield [GroupUpdater] group updater for setting retry_at, stop, etc.
- # @raise [ArgumentError] if the consumer group is not found
- def updating_consumer_group(group_id, &)
- db.transaction do
- dataset = db[consumer_groups_table].where(group_id:)
- group_row = dataset.first
- raise ArgumentError, "Consumer group #{group_id} not found" unless group_row
-
- ctx = group_row[:error_context] ? parse_json(group_row[:error_context]) : {}
- group_row[:error_context] = ctx
- group = GroupUpdater.new(group_id, group_row, logger)
- yield group
- updates = group.updates
- updates[:error_context] = JSON.dump(updates[:error_context])
- dataset.update(updates)
- end
- end
-
- # Acknowledge a message for a consumer group, running an optional block
- # inside the same transaction.
- #
- # The PG version uses +pg_try_advisory_xact_lock+ to prevent concurrent
- # processing of the same stream by the same group. SQLite relies on the
- # database-level write lock instead, so a second concurrent caller will
- # block until the first commits.
- #
- # @param group_id [String] consumer group identifier
- # @param message_id [String] UUID of the message to acknowledge
- # @yield optional block to run inside the transaction before ACKing
- # @raise [Sourced::ConcurrentAckError] if the message is not found
- def ack_on(group_id, message_id, &)
- db.transaction do
- row = db[messages_table]
- .join(streams_table, id: :stream_id)
- .where(Sequel[messages_table][:id] => message_id)
- .select(
- Sequel[messages_table][:global_seq],
- Sequel[messages_table][:stream_id].as(:stream_id_fk)
- )
- .first
-
- raise Sourced::ConcurrentAckError, "Message #{message_id} not found for group #{group_id}" unless row
-
- yield if block_given?
-
- ack_message(group_id, row[:stream_id_fk], row[:global_seq])
- end
- end
-
- # Clear all data. For tests only.
- #
- # Uses +DELETE+ instead of +TRUNCATE+ (not supported by SQLite).
- # Also resets the +sqlite_sequence+ table so +global_seq+ auto-increment
- # restarts from 1.
- #
- # @raise [RuntimeError] if +ENV['ENVIRONMENT']+ is not +'test'+
- # @return [void]
- def clear!
- raise 'Not in test environment' unless ENV['ENVIRONMENT'] == 'test'
- db[offsets_table].delete
- db[messages_table].delete
- db[consumer_groups_table].delete
- db[streams_table].delete
- db[scheduled_messages_table].delete
- db[workers_table].delete
- # Reset auto-increment counters so global_seq starts from 1
- db.run("DELETE FROM sqlite_sequence") if db.table_exists?(:sqlite_sequence)
- end
-
- private
-
- # Configure SQLite-specific PRAGMAs and set up the {InlineNotifier}.
- #
- # - +foreign_keys = ON+ — enforce FK constraints (off by default in SQLite)
- # - +journal_mode = WAL+ — allow concurrent reads during writes
- # - +busy_timeout = 5000+ — wait up to 5 s for a write lock before raising
- #
- # @return [void]
- def setup_adapter
- @notifier = InlineNotifier.new
- @db.run('PRAGMA foreign_keys = ON')
- @db.run('PRAGMA journal_mode = WAL')
- @db.run('PRAGMA busy_timeout = 5000')
- end
-
- private
-
- # @return [String] filename of the SQLite-specific migration ERB template
- def migration_template_name
- '001_create_sourced_tables_sqlite.rb.erb'
- end
-
- # Acknowledge a message by advancing the consumer group's offset.
- #
- # Uses +max()+ instead of PG's +greatest()+ to update +highest_global_seq+.
- # Falls back to +INSERT+ if the consumer group row doesn't exist yet.
- #
- # @param group_id [String] consumer group identifier
- # @param stream_id [Integer] stream FK (internal integer ID, not the string stream_id)
- # @param global_seq [Integer] global sequence number of the acknowledged message
- # @return [void]
- def ack_message(group_id, stream_id, global_seq)
- db.transaction do
- # Update consumer group — use max() instead of greatest()
- updated = db[consumer_groups_table]
- .where(group_id:)
- .update(
- updated_at: Time.now,
- highest_global_seq: Sequel.function(:max, :highest_global_seq, global_seq)
- )
-
- if updated == 0
- db[consumer_groups_table].insert(group_id:, highest_global_seq: global_seq)
- end
-
- group_id_fk = db[consumer_groups_table].where(group_id:).select(:id).first[:id]
-
- # Upsert offset
- db[offsets_table]
- .insert_conflict(
- target: [:group_id, :stream_id],
- update: { claimed: false, claimed_at: nil, claimed_by: nil, global_seq: }
- )
- .insert(
- group_id: group_id_fk,
- stream_id:,
- global_seq:
- )
- end
- end
-
- # Decompose the PG CTE (claim + fetch in one statement) into
- # separate SELECT, UPDATE, SELECT steps within a transaction.
- #
- # SQLite has no +FOR UPDATE SKIP LOCKED+, so two concurrent workers
- # contend for the database-level write lock rather than skipping locked rows.
- #
- # @param group_id [String] consumer group identifier
- # @param handled_messages [Array] message type strings to filter
- # @param now [Time] current time for claim timestamps and retry_at checks
- # @param start_from [Time, nil] optional time window lower bound
- # @param worker_id [String] identifier for the claiming worker
- # @param batch_size [Integer] maximum messages to fetch
- # @param with_history [Boolean] whether to include full stream history
- # @return [Array(Array, Array?), NO_BATCH] batch rows and optional history
- def claim_and_fetch_batch(group_id, handled_messages, now, start_from, worker_id, batch_size, with_history: false)
- message_types_sql = handled_messages.any? ? " AND e.type IN(#{handled_messages.map { |e| db.literal(e) }.join(',')})" : ''
- time_window_sql = start_from.is_a?(Time) ? " AND e.created_at > #{db.literal(start_from)}" : ''
-
- db.transaction do
- # Step 1: Find first unclaimed offset with a matching pending message
- first_match = db.fetch(<<~SQL).first
- SELECT
- so.id as offset_id, ss.stream_id, e.stream_id as stream_id_fk,
- cg.id as group_id_fk,
- cg.highest_global_seq,
- so.global_seq as offset_seq
- FROM #{@messages_table_literal} e
- JOIN #{@streams_table_literal} ss ON e.stream_id = ss.id
- JOIN #{@consumer_groups_table_literal} cg ON cg.group_id = #{db.literal(group_id)}
- JOIN #{@offsets_table_literal} so ON cg.id = so.group_id AND ss.id = so.stream_id
- WHERE e.global_seq > so.global_seq
- AND so.claimed = 0
- AND cg.status = 'active'
- AND (cg.retry_at IS NULL OR cg.retry_at <= #{db.literal(now)})
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- LIMIT 1
- SQL
-
- return NO_BATCH unless first_match
-
- # Step 2: Claim the offset
- claimed = db[offsets_table]
- .where(id: first_match[:offset_id], claimed: false)
- .update(claimed: true, claimed_at: now, claimed_by: worker_id)
-
- return NO_BATCH if claimed == 0
-
- # Step 3: Fetch messages
- if with_history
- fetch_batch_with_history(first_match, message_types_sql, time_window_sql, batch_size)
- else
- fetch_batch(first_match, message_types_sql, time_window_sql, batch_size)
- end
- end
- rescue Sequel::UniqueConstraintViolation
- logger.debug "Batch claim for group #{group_id} already exists, skipping"
- NO_BATCH
- end
-
- # Fetch a batch of messages for the claimed offset.
- #
- # @param first_match [Hash] the claimed offset row
- # @param message_types_sql [String] SQL fragment filtering message types
- # @param time_window_sql [String] SQL fragment filtering by time
- # @param batch_size [Integer] maximum messages to fetch
- # @return [Array(Array, nil), NO_BATCH]
- def fetch_batch(first_match, message_types_sql, time_window_sql, batch_size)
- rows = db.fetch(<<~SQL).all
- SELECT e.global_seq, e.id, #{db.literal(first_match[:stream_id])} as stream_id,
- e.stream_id as stream_id_fk,
- e.seq, e.type, e.created_at, e.causation_id, e.correlation_id,
- e.metadata, e.payload, #{db.literal(first_match[:offset_id])} as offset_id,
- #{db.literal(first_match[:group_id_fk])} as group_id_fk,
- (e.global_seq <= #{db.literal(first_match[:highest_global_seq])}) AS replaying,
- #{db.literal(first_match[:highest_global_seq])} as highest_global_seq
- FROM #{@messages_table_literal} e
- WHERE e.stream_id = #{db.literal(first_match[:stream_id_fk])}
- AND e.global_seq > #{db.literal(first_match[:offset_seq])}
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- LIMIT #{db.literal(batch_size)}
- SQL
-
- return NO_BATCH if rows.empty?
-
- coerce_booleans!(rows)
- [rows, nil]
- end
-
- # Fetch a batch of messages along with full stream history.
- #
- # @param first_match [Hash] the claimed offset row
- # @param message_types_sql [String] SQL fragment filtering message types
- # @param time_window_sql [String] SQL fragment filtering by time
- # @param batch_size [Integer] maximum messages to fetch
- # @return [Array(Array, Array), NO_BATCH]
- def fetch_batch_with_history(first_match, message_types_sql, time_window_sql, batch_size)
- # Fetch all stream messages for history
- all_rows = db.fetch(<<~SQL).all
- SELECT e.global_seq, e.id, #{db.literal(first_match[:stream_id])} as stream_id,
- e.stream_id as stream_id_fk,
- e.seq, e.type, e.created_at, e.causation_id, e.correlation_id,
- e.metadata, e.payload, #{db.literal(first_match[:offset_id])} as offset_id,
- #{db.literal(first_match[:group_id_fk])} as group_id_fk,
- (e.global_seq <= #{db.literal(first_match[:highest_global_seq])}) AS replaying,
- #{db.literal(first_match[:highest_global_seq])} as highest_global_seq
- FROM #{@messages_table_literal} e
- WHERE e.stream_id = #{db.literal(first_match[:stream_id_fk])}
- ORDER BY e.global_seq ASC
- SQL
-
- return NO_BATCH if all_rows.empty?
-
- # Identify which rows are in the batch
- batch_seqs = db.fetch(<<~SQL).map { |r| r[:global_seq] }
- SELECT e.global_seq
- FROM #{@messages_table_literal} e
- WHERE e.stream_id = #{db.literal(first_match[:stream_id_fk])}
- AND e.global_seq > #{db.literal(first_match[:offset_seq])}
- #{message_types_sql}#{time_window_sql}
- ORDER BY e.global_seq ASC
- LIMIT #{db.literal(batch_size)}
- SQL
-
- batch_set = batch_seqs.to_set
- all_rows.each { |r| r[:in_batch] = batch_set.include?(r[:global_seq]) }
-
- coerce_booleans!(all_rows)
-
- batch_rows = all_rows.select { |r| r[:in_batch] }
- if batch_rows.empty?
- release_offset(first_match[:offset_id])
- return NO_BATCH
- end
-
- history = all_rows.map { |row| deserialize_message(row) }
- [batch_rows, history]
- end
-
- # Convert SQLite integer booleans (0/1) to Ruby booleans.
- #
- # SQLite boolean expressions like +(expr) AS replaying+ return +0+ or +1+
- # rather than +true+/+false+. This normalizes the +:replaying+ field so
- # downstream code works with Ruby booleans.
- #
- # @param rows [Array] rows to mutate in place
- # @return [void]
- def coerce_booleans!(rows)
- rows.each { |r| r[:replaying] = (r[:replaying] == 1 || r[:replaying] == true) }
- end
- end
- end
-end
diff --git a/lib/sourced/backends/test_backend.rb b/lib/sourced/backends/test_backend.rb
deleted file mode 100644
index df2b26f0..00000000
--- a/lib/sourced/backends/test_backend.rb
+++ /dev/null
@@ -1,276 +0,0 @@
-# frozen_string_literal: true
-
-require 'thread'
-require 'sourced/inline_notifier'
-
-module Sourced
- module Backends
- class TestBackend
- ACTIVE = 'active'
- STOPPED = 'stopped'
-
- def initialize
- clear!
- @mutex = Mutex.new
- @in_tx = false
- @tx_id = nil
- end
-
- def messages = @state.messages
-
- def inspect
- %(<#{self.class} messages:#{messages.size} streams:#{@state.messages_by_stream_id.size}>)
- end
-
- def clear!
- @state = State.new
- @notifier = InlineNotifier.new
- end
-
- # Returns the backend's notifier for real-time message dispatch.
- # Always returns an {InlineNotifier} since TestBackend has no PG connection.
- # Eagerly initialized in {#clear!} for thread and CoW safety.
- #
- # @return [InlineNotifier]
- attr_reader :notifier
-
- def installed? = true
-
- def handling_reactor_exceptions(_reactor, &)
- yield
- end
-
- def reserve_next_for_reactor(reactor, batch_size: 1, with_history: false, worker_id: Process.pid.to_s, &)
- group_id = reactor.consumer_info.group_id
- start_from = reactor.consumer_info.start_from.call
- transaction do
- group = @state.groups[group_id]
- if group.active? && (group.retry_at.nil? || group.retry_at <= Time.now)
- group.reserve_next(reactor.handled_messages, start_from, method(:process_actions), batch_size:, with_history:, &)
- end
- end
- end
-
- private def process_actions(group_id, actions, ack, event, offset)
- should_ack = false
- actions = [actions] unless actions.is_a?(Array)
- # Empty actions is assumed to be an ACK
- return ack.() if actions.empty?
-
- actions.each do |action|
- case action
- when nil, Actions::OK
- should_ack = true
-
- when Actions::Ack
- offset.locked = false
- ack_on(group_id, action.message_id)
-
- when Actions::RETRY
- # Don't ack
-
- else
- action.execute(self, event)
- should_ack = true
- end
- end
-
- ack.() if should_ack
- end
-
- def ack_on(group_id, event_id, &)
- transaction do
- group = @state.groups[group_id]
- group.ack_on(event_id, &)
- end
- end
-
- def register_consumer_group(group_id)
- transaction do
- @state.groups[group_id]
- end
- end
-
- def updating_consumer_group(group_id, &)
- transaction do
- group = @state.groups[group_id]
- yield group
- end
- end
-
- # Start a consumer group that has been stopped.
- # Signals the notifier so workers pick up the reactor immediately.
- #
- # @param group_id [String]
- def start_consumer_group(group_id)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- transaction do
- group = @state.groups[group_id]
- group.error_context = {}
- group.status = ACTIVE
- group.retry_at = nil
- end
- notifier.notify_reactor_resumed(group_id)
- end
-
- def stop_consumer_group(group_id, error = nil)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- transaction do
- group = @state.groups[group_id]
- group.stop(error)
- end
- end
-
- def reset_consumer_group(group_id)
- group_id = group_id.consumer_info.group_id if group_id.respond_to?(:consumer_info)
- transaction do
- group = @state.groups[group_id]
- group.reset!
- end
- true
- end
-
- def schedule_messages(messages, at: Time.now)
- @state.schedule_messages(messages, at:)
- true
- end
-
- def update_schedule!
- count = 0
- transaction do
- @state.next_scheduled_messages do |scheduled_messages|
- scheduled_messages.group_by(&:stream_id).each do |stream_id, stream_messages|
- append_next_to_stream(stream_id, stream_messages)
- end
- count = scheduled_messages.size
- end
- count
- end
- end
-
- Stats = Data.define(:stream_count, :max_global_seq, :groups)
-
- def stats
- stream_count = @state.messages_by_stream_id.size
- max_global_seq = messages.size
- groups = @state.groups.values.map(&:to_h)
- Stats.new(stream_count, max_global_seq, groups)
- end
-
- # Retrieve a list of recently active streams, ordered by most recent activity.
- # This is the in-memory implementation that maintains stream metadata during testing.
- #
- # @param limit [Integer] Maximum number of streams to return (defaults to 10)
- # @return [Array] Array of Stream objects ordered by updated_at descending
- # @see SequelBackend#recent_streams
- def recent_streams(limit: 10)
- # Input validation (consistent with SequelBackend)
- return [] if limit == 0
- raise ArgumentError, "limit must be a positive integer" if limit < 0
-
- @state.streams.values.sort_by(&:updated_at).reverse.take(limit)
- end
-
- def transaction(&)
- if @in_tx
- yield
- else
- @mutex.synchronize do
- @in_tx = true
- @state_snapshot = @state.copy
- result = yield
- @in_tx = false
- @state_snapshot = nil
- result
- end
- end
- rescue StandardError => e
- @in_tx = false
- @state = @state_snapshot if @state_snapshot
- raise
- end
-
- # @param stream_id [String] Unique identifier for the event stream
- # @param messages [Sourced::Message, Array] Event(s) to append to the stream
- # @option max_retries [Integer] Not used in this backend, but kept for interface compatibility
- def append_next_to_stream(stream_id, messages, max_retries: 3)
- # Handle both single event and array of messages
- messages_array = Array(messages)
- return true if messages_array.empty?
-
- transaction do
- last_message = @state.messages_by_stream_id[stream_id].last
- last_seq = last_message ? last_message.seq : 0
-
- messages_with_seq = messages_array.map.with_index do |message, index|
- message.with(seq: last_seq + index + 1)
- end
-
- append_to_stream(stream_id, messages_with_seq)
- end
- end
-
- def append_to_stream(stream_id, messages)
- # Handle both single event and array of events
- messages_array = Array(messages)
- return false if messages_array.empty?
-
- transaction do
- check_unique_seq!(messages_array)
-
- messages_array.each do |message|
- @state.messages_by_correlation_id[message.correlation_id] << message
- @state.messages_by_stream_id[stream_id] << message
- @state.messages << message
- @state.stream_id_seq_index[seq_key(stream_id, message)] = true
- @state.upsert_stream(stream_id, message.seq)
- end
- end
- @state.groups.each_value(&:reindex)
- notifier.notify_new_messages(messages_array.map(&:type))
- true
- end
-
- def read_correlation_batch(message_id)
- message = @state.messages.find { |e| e.id == message_id }
- return [] unless message
- @state.messages_by_correlation_id[message.correlation_id]
- end
-
- def read_stream(stream_id, after: nil, upto: nil)
- messages = @state.messages_by_stream_id[stream_id]
- messages = messages.select { |e| e.seq > after } if after
- messages = messages.select { |e| e.seq <= upto } if upto
- messages
- end
-
- # No-op heartbeats for test backend
- def worker_heartbeat(worker_ids, at: Time.now)
- Array(worker_ids).size
- end
-
- # No-op stale claim release for test backend
- def release_stale_claims(ttl_seconds: 120)
- 0
- end
-
- private
-
- def check_unique_seq!(messages)
- duplicate = messages.find do |message|
- @state.stream_id_seq_index[seq_key(message.stream_id, message)]
- end
- if duplicate
- raise Sourced::ConcurrentAppendError, "Duplicate stream_id/seq: #{duplicate.stream_id}/#{duplicate.seq}"
- end
- end
-
- def seq_key(stream_id, message)
- [stream_id, message.seq]
- end
- end
- end
-end
-
-require 'sourced/backends/test_backend/group'
-require 'sourced/backends/test_backend/state'
diff --git a/lib/sourced/backends/test_backend/group.rb b/lib/sourced/backends/test_backend/group.rb
deleted file mode 100644
index 80913cc9..00000000
--- a/lib/sourced/backends/test_backend/group.rb
+++ /dev/null
@@ -1,172 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Backends
- class TestBackend
- class Group
- attr_reader :group_id
- attr_accessor :status, :error_context, :retry_at
-
- Offset = Struct.new(:stream_id, :index, :locked)
-
- def initialize(group_id, backend)
- @group_id = group_id
- @backend = backend
- @status = :active
- @error_context = {}
- @retry_at = nil
- @highest_index = -1
- reset!
- end
-
- def active? = @status == :active
-
- def stop(reason = nil)
- @error_context[:reason] = reason if reason
- @status = :stopped
- end
-
- def reset!
- @offsets = {}
- reindex
- end
-
- def retry(time, ctx = {})
- @error_context.merge!(ctx)
- @retry_at = time
- end
-
- def to_h
- active_offsets = @offsets.values.select { |o| o.index >= 0 }
- oldest_processed = (active_offsets.min_by(&:index)&.index || -1) + 1
- newest_processed = (active_offsets.max_by(&:index)&.index || -1) + 1
- stream_count = active_offsets.size
-
- {
- group_id:,
- status: @status.to_s,
- oldest_processed:,
- newest_processed:,
- stream_count:,
- retry_at:
- }
- end
-
- def reindex
- backend.messages.each do |e|
- @offsets[e.stream_id] ||= Offset.new(e.stream_id, -1, false)
- end
- end
-
- def ack_on(message_id, &)
- global_seq = backend.messages.find_index { |e| e.id == message_id }
- return unless global_seq
-
- evt = backend.messages[global_seq]
- offset = @offsets[evt.stream_id]
- if offset.locked
- raise Sourced::ConcurrentAckError, "Stream for message #{message_id} is being concurrently processed by #{group_id}"
- else
- offset.locked = true
- yield if block_given?
- offset.index = global_seq
- @highest_index = global_seq if global_seq > @highest_index
- offset.locked = false
- end
- end
-
- NOOP_FILTER = ->(_) { true }
-
- def reserve_next(handled_messages, time_window, process_actions, batch_size: 1, with_history: false, &block)
- time_filter = time_window.is_a?(Time) ? ->(e) { e.created_at > time_window } : NOOP_FILTER
- evt = nil
- offset = nil
- index = -1
-
- backend.messages.each.with_index do |e, idx|
- offset = @offsets[e.stream_id]
- if offset.locked # stream locked by another consumer in the group
- next
- elsif idx > offset.index && handled_messages.include?(e.class) && time_filter.call(e) # new message for the stream
- evt = e
- offset.locked = true
- index = idx
- break
- else # messages already consumed
- end
- end
-
- return unless evt
-
- reserve_batch(evt, index, offset, handled_messages, time_filter, process_actions, batch_size, with_history:, &block)
- end
-
- private
-
- def reserve_batch(first_evt, first_index, offset, handled_messages, time_filter, process_actions_callback, batch_size, with_history: false, &block)
- stream_id = first_evt.stream_id
- raw_batch = [[first_evt, first_index]]
-
- # Find additional messages from same stream
- backend.messages.each.with_index do |e, idx|
- break if raw_batch.size >= batch_size
- next if idx <= first_index
- next unless e.stream_id == stream_id
- next unless handled_messages.include?(e.class)
- next unless time_filter.call(e)
-
- raw_batch << [e, idx]
- end
-
- # Build history if requested: all messages from this stream
- history = if with_history
- backend.messages.select { |e| e.stream_id == stream_id }
- end
-
- # Build batch of [message, replaying] pairs
- batch = raw_batch.map { |msg, idx| [msg, @highest_index >= idx] }
-
- # Yield batch + history once, get back action_pairs or RETRY
- action_pairs = yield(batch, history)
-
- if action_pairs == Actions::RETRY
- offset.locked = false
- return first_evt
- end
-
- # Execute all action pairs
- if action_pairs.empty?
- # Nothing to process — release the claim so another worker can retry
- offset.locked = false
- return first_evt
- end
-
- noop_ack = -> {}
- action_pairs.each do |actions, source_message|
- process_actions_callback.(group_id, actions, noop_ack, source_message, offset)
- end
-
- # ACK once for the last successfully processed message.
- # Find the raw_batch entry matching the last action_pair's source_message,
- # so partial batches ACK up to the last successful message (not the last in the batch).
- last_source_message = action_pairs.last&.last
- last_entry = raw_batch.find { |msg, _idx| msg.id == last_source_message&.id } || raw_batch.last
- last_idx = last_entry[1]
- ack(offset, last_idx) if last_idx > offset.index
-
- offset.locked = false
- first_evt
- end
-
- def ack(offset, index)
- # ACK reactor/message
- offset.index = index
- @highest_index = index if index > @highest_index
- end
-
- attr_reader :backend
- end
-
- end
- end
-end
diff --git a/lib/sourced/backends/test_backend/state.rb b/lib/sourced/backends/test_backend/state.rb
deleted file mode 100644
index c6371ff8..00000000
--- a/lib/sourced/backends/test_backend/state.rb
+++ /dev/null
@@ -1,86 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Backends
- class TestBackend
- class State
- attr_reader :messages, :groups, :messages_by_correlation_id, :messages_by_stream_id, :stream_id_seq_index, :streams, :scheduled_messages
-
- def initialize(
- messages: [],
- groups: Hash.new { |h, k| h[k] = Group.new(k, self) },
- messages_by_correlation_id: Hash.new { |h, k| h[k] = [] },
- messages_by_stream_id: Hash.new { |h, k| h[k] = [] },
- stream_id_seq_index: {},
- streams: {},
- scheduled_messages: []
- )
-
- @messages = messages
- @groups = groups
- @messages_by_correlation_id = messages_by_correlation_id
- @messages_by_stream_id = messages_by_stream_id
- @stream_id_seq_index = stream_id_seq_index
- @streams = streams
- @scheduled_messages = scheduled_messages
- end
-
- ScheduledMessageRecord = Data.define(:message, :at, :position) do
- def <=>(other)
- self.position <=> other.position
- end
- end
-
- Stream = Data.define(:stream_id, :seq, :updated_at) do
- def hash = stream_id
- def eql?(other) = other.is_a?(Stream) && stream_id == other.stream_id
- end
-
- def upsert_stream(stream_id, seq)
- str = Stream.new(stream_id, seq, Time.now)
- @streams[stream_id] = str
- end
-
- def schedule_messages(messages, at: Time.now)
- counter = @scheduled_messages.size
- records = messages.map do |a|
- counter += 1
- ScheduledMessageRecord.new(a, at, [at, counter])
- end
- @scheduled_messages += records
- @scheduled_messages.sort!
- end
-
- def next_scheduled_messages(&)
- now = Time.now
- next_records, @scheduled_messages = @scheduled_messages.partition do |r|
- r.at <= now
- end
- next_messages = next_records.map(&:message)
- yield next_messages if next_messages.any?
- next_messages
- end
-
- def copy
- self.class.new(
- messages: messages.dup,
- groups: deep_dup(groups),
- messages_by_correlation_id: deep_dup(messages_by_correlation_id),
- messages_by_stream_id: deep_dup(messages_by_stream_id),
- stream_id_seq_index: deep_dup(stream_id_seq_index),
- streams: streams.dup,
- scheduled_messages: scheduled_messages.dup
- )
- end
-
- private
-
- def deep_dup(hash)
- hash.each.with_object(hash.dup.clear) do |(k, v), new_hash|
- new_hash[k] = v.dup
- end
- end
- end
- end
- end
-end
diff --git a/lib/sourced/command_context.rb b/lib/sourced/command_context.rb
index 003d40f9..b831413d 100644
--- a/lib/sourced/command_context.rb
+++ b/lib/sourced/command_context.rb
@@ -3,65 +3,112 @@
require 'sourced/types'
module Sourced
- # A command factory to instantiate commands from Hash attributes
- # including extra metadata.
- # @example
+ # Builds commands instances with shared default metadata.
#
- # ctx = Sourced::CommandContext.new(
- # stream_id: params[:stream_id],
- # metadata: {
- # user_id: session[:user_id]
- # }
- # )
+ # @example Build a command from a type string
+ # ctx = CommandContext.new(metadata: { user_id: 10 })
+ # cmd = ctx.build(type: 'orders.place', payload: { item: 'hat' })
+ # cmd.metadata[:user_id] # => 10
#
- # # params[:command] should be a Hash with { type: String, payload: Hash | nil }
- #
- # cmd = ctx.build(params[:command])
- # cmd.stream_id # String
- # cmd.metadata[:user_id] # == session[:user_id]
- #
- # Passing a command subclass will scope command lookup to subclasses of that class.
- # Useful for restricting clients to a specific set of commands.
- #
- # @example
- #
- # ctx = Sourced::CommandContext.new(scope: PublicCommand)
- #
- # cmd = ctx.build(type: 'do_something', payload: { foo: 'bar' })
- #
- # # Or with class and attrs
- # cmd = ctx.build(SomeCommand, stream_id: '111', payload: { foo: 'bar' })
- #
- # Attempting to build a command not in the scope will raise an error.
+ # @example Scope to a custom command subclass
+ # scope = Class.new(Sourced::Command)
+ # MyCmd = scope.define('my.cmd') { attribute :name, String }
+ # ctx = CommandContext.new(scope: scope)
+ # cmd = ctx.build(type: 'my.cmd', payload: { name: 'hello' })
class CommandContext
- # @option stream_id [String]
- # @option metadata [Hash] metadata to add to commands built by this context
- # @option scope [Sourced::Message] Message class to use as command registry
- def initialize(stream_id: nil, metadata: Plumb::BLANK_HASH, scope: Sourced::Command)
- @defaults = {
- stream_id:,
- metadata:
- }.freeze
+ class << self
+ # Register a block to run when building specific command type(s).
+ # The block receives the app scope and the command, and must return the (possibly modified) command.
+ #
+ # @param message_classes [Class] one or more command classes to match
+ # @yield [app, cmd] transformation block
+ # @return [void]
+ def on(*message_classes, &block)
+ message_classes.each { |klass| (message_blocks[klass] ||= []) << block }
+ end
+
+ # Register a block to run for all commands.
+ # The block receives the app scope and the command, and must return the (possibly modified) command.
+ #
+ # @yield [app, cmd] transformation block
+ # @return [void]
+ def any(&block)
+ any_blocks << block
+ end
+
+ # @api private
+ def message_blocks
+ @message_blocks ||= {}
+ end
+
+ # @api private
+ def any_blocks
+ @any_blocks ||= []
+ end
+
+ # @api private
+ def inherited(subclass)
+ super
+ message_blocks.each { |k, v| subclass.message_blocks[k] = v.dup }
+ any_blocks.each { |blk| subclass.any_blocks << blk }
+ end
+ end
+
+ # @param metadata [Hash] default metadata merged into every command built by this context
+ # @param scope [Class] message class whose registry is used to resolve type strings (default: {Sourced::Command})
+ # @param app [Object, nil] request-scoped object passed to callback blocks
+ #
+ # @example
+ # ctx = CommandContext.new(metadata: { user_id: 42 }, app: rack_app)
+ def initialize(metadata: Plumb::BLANK_HASH, scope: Sourced::Command, app: nil)
+ @defaults = { metadata: }.freeze
@scope = scope
+ @app = app
end
- # @param attrs [Hash] attributes to lookup and buils a scope from.
- # @return [Sourced::Message]
+ # Build a command instance, merging in default metadata.
+ #
+ # @overload build(attrs)
+ # Resolve the command class from the +type+ key in +attrs+ via the scope's registry.
+ # @param attrs [Hash] must include +:type+ and +:payload+ keys
+ # @return [Sourced::Message]
+ # @example
+ # ctx.build(type: 'orders.place', payload: { item: 'hat' })
+ #
+ # @overload build(klass, attrs)
+ # Use the given command class directly.
+ # @param klass [Class] a Sourced::Command subclass
+ # @param attrs [Hash] must include +:payload+ key
+ # @return [Sourced::Message]
+ # @example
+ # ctx.build(PlaceOrder, payload: { item: 'hat' })
+ #
+ # @raise [ArgumentError] if arguments don't match either form
+ # @raise [Sourced::UnknownMessageError] if the type string is not registered in the scope
def build(*args)
- case args
- in [Class => klass, Hash => attrs]
- attrs = defaults.merge(Types::SymbolizedHash.parse(attrs))
- klass.parse(attrs)
- in [Hash => attrs]
- attrs = defaults.merge(Types::SymbolizedHash.parse(attrs))
- scope.from(attrs)
- else
- raise ArgumentError, "Invalid arguments: #{args.inspect}"
- end
+ cmd = case args
+ in [Class => klass, Hash => attrs]
+ attrs = defaults.merge(Types::SymbolizedHash.parse(attrs))
+ klass.parse(attrs)
+ in [Hash => attrs]
+ attrs = defaults.merge(Types::SymbolizedHash.parse(attrs))
+ scope.from(attrs)
+ else
+ raise ArgumentError, "Invalid arguments: #{args.inspect}"
+ end
+ run_pipeline(cmd)
end
private
- attr_reader :defaults, :scope
+ EMPTY_ARRAY = [].freeze
+
+ attr_reader :defaults, :scope, :app
+
+ def run_pipeline(cmd)
+ blocks = self.class.message_blocks[cmd.class] || EMPTY_ARRAY
+ steps = blocks + self.class.any_blocks
+ steps.reduce(cmd) { |c, st| instance_exec(app, c, &st) }
+ end
end
end
diff --git a/lib/sourced/command_methods.rb b/lib/sourced/command_methods.rb
deleted file mode 100644
index a5286b44..00000000
--- a/lib/sourced/command_methods.rb
+++ /dev/null
@@ -1,151 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- # Provides command invocation methods for actors.
- #
- # CommandMethods automatically generates instance methods from command definitions,
- # allowing you to invoke commands in two ways:
- #
- # 1. **In-memory version** (e.g., `actor.start(name: 'Joe')`)
- # - Validates the command and executes the decision handler
- # - Returns a tuple of [cmd, new_events]
- # - Does NOT persist events to backend
- #
- # 2. **Durable version** (e.g., `actor.start!(name: 'Joe')`)
- # - Same as in-memory, but also appends events to backend
- # - Raises {FailedToAppendMessagesError} if backend fails
- #
- # ## Usage
- #
- # Include the module in an Actor and define commands normally:
- #
- # class MyActor < Sourced::Actor
- # include Sourced::CommandMethods
- #
- # command :create_item, name: String do |state, cmd|
- # event :item_created, cmd.payload
- # end
- # end
- #
- # actor = MyActor.new(id: 'actor-123')
- # cmd, events = actor.create_item(name: 'Widget') # In-memory
- # cmd, events = actor.create_item!(name: 'Widget') # Persists to backend
- #
- # ## Method Naming
- #
- # Methods are created based on your command name:
- # - Symbol commands like `:create_item` → `create_item` and `create_item!` methods
- # - Class commands like `CreateItem` → `create_item` and `create_item!` methods
- #
- module CommandMethods
- # Raised when the durable command (bang method) fails to append events to the backend.
- #
- # @see #__issue_command
- class FailedToAppendMessagesError < StandardError
- def initialize(cmd, events)
- super <<~MSG
- Failed to append events to backend.
- Command is #{cmd.inspect}
- Events are #{events.inspect}
- MSG
- end
- end
-
- def self.included(base)
- base.send :extend, ClassMethods
- end
-
- # Issues a command without persisting to the backend.
- #
- # This is the core logic used by the generated command methods. It validates
- # the command and executes the decision handler if valid.
- #
- # @private
- # @param cmd_class [Class] A subclass of {Sourced::Message} representing the command
- # @param payload [Hash] The command payload data
- # @return [Array<(Sourced::Message, Array)>] A tuple of [command, events]
- # where command is the validated command object and events are the produced events
- # (empty array if command was invalid)
- #
- # @example
- # cmd, events = __issue_command(MyActor::CreateItem, name: 'Widget')
- # puts cmd.valid? # => true
- # puts events.length # => 1
- private def __issue_command(cmd_class, payload = {})
- cmd = cmd_class.new(stream_id: id, payload:)
- return [cmd, React::EMPTY_ARRAY] unless cmd.valid?
-
- [cmd, decide(cmd)]
- end
-
- # @private
- # Class methods that hook into the command definition and create instance methods.
- module ClassMethods
- # Hooks into the command definition to automatically generate instance methods.
- #
- # This method is called automatically when you define a command in your actor.
- # It creates both in-memory and durable (bang) versions of the command method.
- #
- # @see Sourced::Actor.command
- def command(*args, &block)
- super.tap do |_|
- case args
- in [Symbol => cmd_name, *_]
- klass_name = cmd_name.to_s.split('_').map(&:capitalize).join
- cmd_class = const_get(klass_name)
- __command_methods_define(cmd_name, cmd_class)
- in [Class => cmd_class] if cmd_class < Sourced::Message
- cmd_name = Sourced::Types::ModuleToMethodName.parse(cmd_class.name.split('::').last)
- __command_methods_define(cmd_name, cmd_class)
- end
- end
- end
-
- # Creates in-memory and durable command invocation methods.
- #
- # Defines two methods on the actor instance:
- # - `method_name(**payload)` - In-memory version, validates and decides without persisting
- # - `method_name!(**payload)` - Durable version, also appends events to backend
- #
- # @private
- # @param cmd_name [Symbol] The command method name (e.g., :create_item)
- # @param cmd_class [Class] The command message class
- #
- # @example Generated methods
- # class MyActor < Sourced::Actor
- # include Sourced::CommandMethods
- # command :create_item, name: String do |state, cmd|
- # event :item_created, cmd.payload
- # end
- # end
- #
- # actor = MyActor.new(id: 'a1')
- #
- # # In-memory version
- # cmd, events = actor.create_item(name: 'Widget')
- # # => Returns [cmd, events] without touching backend
- # # => If invalid: [invalid_cmd, []]
- #
- # # Durable version
- # cmd, events = actor.create_item!(name: 'Widget')
- # # => Returns [cmd, events] if valid and appended successfully
- # # => Raises FailedToAppendMessagesError if backend fails
- # # => If invalid: [invalid_cmd, []]
- private def __command_methods_define(cmd_name, cmd_class)
- define_method(cmd_name) do |**payload|
- __issue_command(cmd_class, payload)
- end
-
- define_method("#{cmd_name}!") do |**payload|
- cmd_events = __issue_command(cmd_class, payload)
- return cmd_events unless cmd_events.first.valid?
-
- success = Sourced.config.backend.append_to_stream(id, cmd_events.last)
- raise FailedToAppendMessagesError.new(*cmd_events) unless success
-
- cmd_events
- end
- end
- end
- end
-end
diff --git a/lib/sourced/configuration.rb b/lib/sourced/configuration.rb
index 328578ca..1a19bae4 100644
--- a/lib/sourced/configuration.rb
+++ b/lib/sourced/configuration.rb
@@ -1,184 +1,77 @@
# frozen_string_literal: true
-require 'console' # comes with async gem
-require 'sourced/types'
-require 'sourced/backends/test_backend'
-require 'sourced/pubsub/test'
+require 'logger'
require 'sourced/error_strategy'
require 'sourced/async_executor'
module Sourced
- # Configure a Sourced app.
- # @example
- #
- # Sourced.configure do |config|
- # config.backend = Sequel.Postgres('postgres://localhost/mydb')
- # config.logger = Logger.new(STDOUT)
- # end
- #
class Configuration
- # Backends must expose these methods
- BackendInterface = Types::Interface[
+ StoreInterface = Types::Interface[
:installed?,
- :reserve_next_for_reactor,
- :append_to_stream,
- :read_correlation_batch,
- :read_stream,
- :transaction,
- :stats,
- :updating_consumer_group,
+ :install!,
+ :append,
+ :read,
+ :read_partition,
+ :claim_next,
+ :ack,
+ :release,
:register_consumer_group,
- :start_consumer_group,
- :stop_consumer_group,
- :reset_consumer_group
+ :worker_heartbeat,
+ :release_stale_claims,
+ :notifier
]
- PubSubInterface = Types::Interface[:publish, :subscribe]
+ attr_accessor :logger, :worker_count, :batch_size,
+ :catchup_interval, :max_drain_rounds,
+ :claim_ttl_seconds, :housekeeping_interval,
+ :executor
- # Interface that all executors must implement
- # @see AsyncExecutor
- # @see ThreadExecutor
- ExecutorInterface = Types::Interface[
- :start
- ]
-
- attr_accessor :logger
- # House-keeping configuration
- # interval: main loop tick for housekeeping (seconds)
- # heartbeat interval: how often to record heartbeats (seconds)
- # claim_ttl_seconds: how long before a worker is considered dead for claim release
- attr_accessor(
- :worker_count,
- :worker_batch_size,
- :housekeeping_count,
- :housekeeping_interval,
- :housekeeping_heartbeat_interval,
- :housekeeping_claim_ttl_seconds,
- :catchup_interval,
- :max_drain_rounds
- )
-
- attr_reader :backend, :executor, :pubsub
+ attr_reader :store, :router
def initialize
- @logger = Console
- @backend = Backends::TestBackend.new
- @pubsub = PubSub::Test.new
- @error_strategy = ErrorStrategy.new
- @executor = AsyncExecutor.new
- @setup = false
- # Worker and house-keeping defaults
+ @logger = Logger.new($stdout)
@worker_count = 2
- @worker_batch_size = 50
- @housekeeping_count = 1
- @housekeeping_interval = 3
- @housekeeping_heartbeat_interval = 5
- @housekeeping_claim_ttl_seconds = 120
+ @batch_size = 50
@catchup_interval = 5
@max_drain_rounds = 10
- @subscribers = []
- end
-
- def subscribe(callable = nil, &block)
- callable ||= block
- @subscribers << callable
- self
- end
-
- def setup!
- return if @setup
-
- @backend.setup!(self) if @backend.respond_to?(:setup!)
- @subscribers.each { |s| s.call(self) }
- @setup = true
+ @claim_ttl_seconds = 120
+ @housekeeping_interval = 30
+ @executor = AsyncExecutor.new
+ @store = nil
+ @router = nil
+ @error_strategy = ErrorStrategy.new
+ @setup = false
end
- # Configure the backend for the app.
- # Defaults to in-memory TestBackend
- # Also auto-sets pubsub when backend is a PG database.
- # @param bnd [BackendInterface]
- def backend=(bnd)
- @backend = case bnd.class.name
- when 'Sequel::Postgres::Database'
- require 'sourced/backends/pg_backend'
- require 'sourced/pubsub/pg'
- @pubsub = PubSub::PG.new(db: bnd, logger: @logger)
- Sourced::Backends::PGBackend.new(bnd)
+ # Accepts a Sourced::Store instance, a Sequel::SQLite::Database connection
+ # (auto-wrapped in Sourced::Store.new(db)), or any object implementing StoreInterface.
+ def store=(s)
+ @store = case s.class.name
when 'Sequel::SQLite::Database'
- require 'sourced/backends/sqlite_backend'
- Sourced::Backends::SQLiteBackend.new(bnd)
- else
- BackendInterface.parse(bnd)
+ require 'sourced/store'
+ Store.new(s)
+ else StoreInterface.parse(s)
end
end
- # Configure the pubsub implementation for the app.
- # Defaults to in-memory PubSub::Test.
- # Automatically set when backend is a PG database.
- # Can be overridden with any object implementing `subscribe` and `publish`.
- # @param ps [#subscribe, #publish]
- def pubsub=(ps)
- @pubsub = PubSubInterface.parse(ps)
- end
-
- # Configure the executor for the app.
- # Supports both symbol shortcuts and executor instances.
- # Defaults to AsyncExecutor for fiber-based concurrency.
- #
- # @param ex [Symbol, Object] The executor to use
- # @option ex [Symbol] :async Use AsyncExecutor (fiber-based, default)
- # @option ex [Symbol] :thread Use ThreadExecutor (thread-based)
- # @option ex [ExecutorInterface] Custom executor instance
- # @raise [ArgumentError] if executor doesn't implement ExecutorInterface
- #
- # @example Using symbol shortcuts
- # config.executor = :async # Default fiber-based
- # config.executor = :thread # Thread-based for CPU-intensive work
- #
- # @example Using custom executor
- # config.executor = MyCustomExecutor.new
- def executor=(ex)
- @executor = case ex
- when :async
- AsyncExecutor.new
- when :thread
- require 'sourced/thread_executor'
- ThreadExecutor.new
- when ExecutorInterface
- ex
- else
- raise ArgumentError, "executor=(e) must support interface #{ExecutorInterface.inspect}"
- end
- end
-
- # Assign an error strategy
- # @param strategy [ErrorStrategy, #call(Exception, Sourced::Message, Group)]
- # @raise [ArgumentError] if strategy does not respond to #call
def error_strategy=(strategy)
- raise ArgumentError, 'Must respond to #call(Exception, Sourced::Message, Group)' unless strategy.respond_to?(:call)
+ raise ArgumentError, 'Must respond to #call' unless strategy.respond_to?(:call)
@error_strategy = strategy
end
- # Configure a built-in Sourced::ErrorStrategy
- # @example
- # config.error_strategy do |s|
- # s.retry(times: 30, after: 50, backoff: ->(retry_after, retry_count) { retry_after * retry_count })
- #
- # s.on_retry do |n, exception, message, later|
- # puts "Retrying #{n} times" }
- # end
- #
- # s.on_stop do |exception, message|
- # Sentry.capture_exception(exception)
- # end
- # end
- #
- # @yieldparam s [ErrorStrategy]
- def error_strategy(&blk)
- return @error_strategy unless block_given?
+ attr_reader :error_strategy
+
+ def setup!
+ return if @setup
- self.error_strategy = ErrorStrategy.new(&blk)
+ unless @store
+ require 'sourced/store'
+ @store = Store.new(Sequel.sqlite)
+ end
+ @store.install!
+ @router ||= Router.new(store: @store)
+ @setup = true
end
end
end
diff --git a/lib/sourced/consumer.rb b/lib/sourced/consumer.rb
index 6b4e01d7..d6d3caf9 100644
--- a/lib/sourced/consumer.rb
+++ b/lib/sourced/consumer.rb
@@ -1,122 +1,164 @@
# frozen_string_literal: true
module Sourced
- # This mixin provides consumer info configuration
- # and a .consumer_info method to access it.
- # @example
- #
- # class MyConsumer
- # extend Sourced::Consumer
- #
- # consumer do |c|
- # # consumer group
- # c.group_id = 'my-group'
- #
- # # Start consuming events from the beginning of history
- # c.start_from = :beginning
- # end
- # end
- #
- # MyConsumer.consumer_info.group_id # => 'my-group'
- #
+ # Accumulates mutations to a consumer group row for atomic persistence.
+ # Accumulates consumer group mutations for atomic persistence.
+ # Used by {Store#updating_consumer_group}.
+ class GroupUpdater
+ attr_reader :group_id, :updates, :error_context
+
+ def initialize(group_id, row, logger)
+ @group_id = group_id
+ @logger = logger
+ @error_context = row[:error_context]
+ @updates = { error_context: @error_context.dup }
+ end
+
+ def stop(message: nil)
+ @logger.error "Sourced: stopping consumer group #{group_id}"
+ @updates[:status] = Store::STOPPED
+ @updates[:retry_at] = nil
+ @updates[:updated_at] = Time.now.iso8601
+ @updates[:error_context][:message] = message if message
+ end
+
+ def fail(exception: nil)
+ @logger.error "Sourced: failing consumer group #{group_id}. #{exception&.class}: #{exception&.message}"
+ @updates[:status] = Store::FAILED
+ @updates[:retry_at] = nil
+ @updates[:updated_at] = Time.now.iso8601
+ if exception
+ @updates[:error_context][:exception_class] = exception.class.to_s
+ @updates[:error_context][:exception_message] = exception.message
+ end
+ end
+
+ def retry(time, **ctx)
+ @logger.warn "Sourced: retrying consumer group #{group_id} at #{time}"
+ @updates[:updated_at] = Time.now.iso8601
+ @updates[:retry_at] = time.iso8601
+ @updates[:error_context].merge!(ctx)
+ end
+ end
+
+ # Shared consumer configuration for reactors.
+ # Extended (not included) onto reactor classes.
module Consumer
- class ConsumerInfo < Types::Data
- ToBlock = Types::Any.transform(Proc) { |v| -> { v } }
- StartFromBeginning = Types::Value[:beginning] >> Types::Static[nil] >> ToBlock
- StartFromNow = Types::Value[:now] >> Types::Static[-> { Time.now - 5 }.freeze]
- StartFromTime = Types::Interface[:call].check('must return a Time') { |v| v.call.is_a?(Time) }
+ def self.extended(base)
+ super
+ base.extend ClassMethods
+ end
- StartFrom = (
- StartFromBeginning | StartFromNow | StartFromTime
- ).default { -> { nil } }
+ def partition_keys
+ @partition_keys ||= []
+ end
- attribute :group_id, Types::String.present, writer: true
- attribute :start_from, StartFrom, writer: true
- attribute :batch_size, Types::Integer.nullable.default(nil), writer: true
+ def partition_by(*keys)
+ @partition_keys = keys.flatten.map(&:to_sym)
end
- def consumer_info
- @consumer_info ||= ConsumerInfo.new(group_id: name, start_from: :beginning)
+ def group_id
+ @group_id ||= name
end
- def consumer(&)
- return consumer_info unless block_given?
+ def consumer_group(id)
+ @group_id = id
+ end
- info = ConsumerInfo.new(group_id: name)
- yield info
- raise Plumb::ParseError, info.errors unless info.valid?
+ # Message types this consumer evolves from. Used by {#context_for}
+ # to build query conditions for history reads.
+ # Defaults to empty; overridden by Sourced::Evolve mixin.
+ def handled_messages_for_evolve
+ @handled_messages_for_evolve ||= []
+ end
- @consumer_info = info
+ # Build query conditions from partition attributes and handled evolve types.
+ # Override in reactor for custom per-command conditions.
+ def context_for(partition_attrs)
+ handled_messages_for_evolve.flat_map { |klass|
+ klass.to_conditions(**partition_attrs)
+ }
end
- # Implement this in your reactors
- # to manage exception handling in eventually-consistent workflows
- # @example retry with exponential back off
- #
- # def self.on_exception(exception, _message, group)
- # retry_count = group.error_context[:retry_count] || 0
- # if retry_count < 3
- # later = 5 + 5 * retry_count
- # group.retry(later, retry_count: retry_count + 1)
- # else
- # group.stop(exception)
- # end
- # end
- #
- # @param exception [Exception] the exception raised
- # @param message [Sourced::Message] the event or command being handled
- # @param group [#stop, #retry] consumer group object to update state, ie. for retries
def on_exception(exception, message, group)
Sourced.config.error_strategy.call(exception, message, group)
end
- # Default handle_batch implementation that wraps per-message .handle calls.
- # Returns array of [actions, source_message] pairs.
- # Reactors with optimized batch processing (Projector, Actor) override this.
+ # Called by {Router#stop_consumer_group} after the group is marked as stopped.
+ # Override in reactor classes to run cleanup logic on stop.
#
- # @param batch [Array<[Message, Boolean]>] array of [message, replaying] pairs
- # @return [Array<[actions, source_message]>] action pairs
- def handle_batch(batch)
- each_with_partial_ack(batch) do |message, replaying|
- kargs = {}
- kargs[:replaying] = replaying if handle_kargs.include?(:replaying)
- kargs[:logger] = Sourced.config.logger if handle_kargs.include?(:logger)
- actions = handle(message, **kargs)
- [actions, message]
- end
+ # @param message [String, nil] optional reason for stopping
+ # @return [void]
+ def on_stop(message = nil)
+ # no-op by default
end
- private
-
- # Iterate batch with per-message error handling.
- # Collects [actions, message] pairs returned by the block.
- # On mid-batch failure, raises PartialBatchError with pairs collected so far,
- # allowing the backend to ACK up to the last successful message.
- # If the first message fails, raises the original error (no partial ACK possible).
+ # Called by {Router#reset_consumer_group} after the group's offsets are cleared.
+ # Override in reactor classes to run cleanup logic on reset
+ # (e.g. clearing caches or projections).
#
- # Used by Consumer (default handle_batch), Actor, and Projector (reaction loop)
- # to provide partial ACK across all reactor types.
+ # @return [void]
+ def on_reset
+ # no-op by default
+ end
+
+ # Called by {Router#start_consumer_group} after the group is marked as active.
+ # Override in reactor classes to run setup logic on start.
#
- # @param batch [Array<[Message, Boolean]>] array of [message, replaying] pairs
- # @yield [message, replaying] called for each message in the batch
- # @yieldreturn [Array(actions, Message), nil] action pair, or nil to skip
- # @return [Array<[actions, source_message]>] action pairs
- def each_with_partial_ack(batch)
+ # @return [void]
+ def on_start
+ # no-op by default
+ end
+
+ # Iterate messages collecting [actions, message] pairs.
+ # On mid-batch failure, raises PartialBatchError with pairs collected so far.
+ # If the first message fails, re-raises the original error.
+ def each_with_partial_ack(messages)
results = []
- batch.each do |message, replaying|
- pair = yield(message, replaying)
+ messages.each do |msg|
+ pair = yield(msg)
results << pair if pair
rescue StandardError => e
raise e if results.empty?
- raise PartialBatchError.new(results, message, e)
+ raise Sourced::PartialBatchError.new(results, msg, e)
end
results
end
- # Lazily resolved keyword argument names for the reactor's .handle method.
- # Cached as a class instance variable.
- def handle_kargs
- @handle_kargs ||= Injector.resolve_args(self, :handle)
+ module ClassMethods
+ # Resolve a messages class from a symbol or type-like string.
+ #
+ # Symbols are normalized by replacing dots with underscores before
+ # matching against registered message types. For example,
+ # +:course_created+ matches "course.created" and
+ # "course_created".
+ #
+ # @param message_symbol [Symbol, String] symbolic message identifier
+ # @return [Class, nil] matching messages class, or +nil+ if none found
+ #
+ # @example
+ # CourseDecider[:courses_created]
+ # # => CourseCreated
+ def [](message_symbol)
+ normalized = message_symbol.to_s.tr('.', '_')
+ find_registered_message_class(normalized)
+ end
+
+ private
+
+ def find_registered_message_class(normalized_name, base = Sourced::Message)
+ base.registry.keys.each do |type|
+ klass = base.registry[type]
+ return klass if type.tr('.', '_') == normalized_name
+ end
+
+ base.subclasses.each do |subclass|
+ klass = find_registered_message_class(normalized_name, subclass)
+ return klass if klass
+ end
+
+ nil
+ end
end
end
end
diff --git a/lib/sourced/decider.rb b/lib/sourced/decider.rb
new file mode 100644
index 00000000..e92208cd
--- /dev/null
+++ b/lib/sourced/decider.rb
@@ -0,0 +1,179 @@
+# frozen_string_literal: true
+
+module Sourced
+ # Reactor base class for command-handling workflows in Sourced.
+ class Decider
+ include Sourced::Evolve
+ include Sourced::React
+ include Sourced::Sync
+ extend Sourced::Consumer
+
+ PREFIX = 'sourced_decide'
+
+ class << self
+ # @return [Array] command message classes handled by this decider
+ def handled_commands
+ @handled_commands ||= []
+ end
+
+ # Messages to claim: commands to decide on + events to react to.
+ # Evolve types are NOT included — they are only for context_for().
+ #
+ # @return [Array] command and reaction message classes
+ def handled_messages
+ handled_commands + handled_messages_for_react
+ end
+
+ # Register a command handler.
+ #
+ # @param message_class [Class] commands class to handle
+ # @yield [state, message] command handler block
+ # @return [void]
+ def command(message_class, &block)
+ handled_commands << message_class
+ define_method(Sourced.message_method_name(PREFIX, message_class.to_s), &block)
+ end
+
+ # Build executable actions for a batch of claimed messages.
+ #
+ # Each message in +new_messages+ routes through one of three branches:
+ #
+ # 1. **Command** — if the message class is in {.handled_commands}, the
+ # decider runs {Decider#decide}, produces new events, and wraps them
+ # in an {Actions::Append} (guarded by +history.guard+). Per-message
+ # +sync+ / +after_sync+ actions are collected.
+ #
+ # 2. **Reaction-triggering event** — if the decider
+ # {React#reacts_to? reacts to} this message (and the batch is not
+ # replaying), its state is evolved with the event, {React#react} is
+ # invoked, and any produced messages are wrapped in
+ # {Actions::Append} / {Actions::Schedule} with +source: msg+ so
+ # the infra layer correlates reaction messages against the event.
+ #
+ # 3. **Anything else** — yields +[Actions::OK, msg]+ (the message is
+ # acked with no side effects). In practice this is unreachable
+ # because claim filtering uses {.handled_messages}, but it's kept
+ # as a safety net.
+ #
+ # ### Reactions are deferred
+ #
+ # Reactions do **not** run inline with the command that produced the
+ # triggering event. A command's batch only appends its events; the
+ # decider's own subscription (via +handled_messages_for_react+) picks
+ # those events up on the next claim cycle and runs the reaction in a
+ # separate +handle_batch+ invocation.
+ #
+ # Consequences:
+ #
+ # - The originating command's +after_sync+ fires as soon as its
+ # events commit — no longer blocked by slow reactions.
+ # - Command and its reactions are in **separate transactions**. A
+ # failing reaction does not roll back the command.
+ # - There is one extra claim-cycle of latency before reactions run
+ # (sub-ms via +InlineNotifier+; up to +catchup_interval+ as
+ # fallback).
+ # - +sync+ / +after_sync+ blocks also fire on reaction claims
+ # (+messages: [evt]+, +events: []+) — inspectors that assume the
+ # primary message is always a command will see different shapes.
+ #
+ # @param partition_values [Hash{Symbol => String}] partition key-value pairs
+ # @param new_messages [Array] claimed messages to process
+ # @param history [ReadResult] prior partition history (evolved into state)
+ # @param replaying [Boolean] when +true+, the reaction branch is skipped
+ # @return [Array, PositionedMessage)>] action/source pairs
+ def handle_batch(partition_values, new_messages, history:, replaying: false)
+ instance = new(partition_values)
+ instance.evolve(history.messages)
+
+ each_with_partial_ack(new_messages) do |msg|
+ if handled_commands.include?(msg.class)
+ raw_events = instance.decide(msg)
+ actions = Actions.build_for(raw_events, guard: history.guard, source: msg)
+ actions += instance.collect_actions(
+ state: instance.state, messages: [msg], events: raw_events
+ )
+
+ [actions, msg]
+ elsif !replaying && instance.reacts_to?(msg)
+ instance.evolve([msg])
+ reaction_msgs = Array(instance.react(msg))
+ actions = Actions.build_for(reaction_msgs, source: msg)
+ actions += instance.collect_actions(
+ state: instance.state, messages: [msg], events: []
+ )
+
+ [actions, msg]
+ else
+ [Actions::OK, msg]
+ end
+ end
+ end
+
+ # Build executable actions for a claimed batch.
+ #
+ # @param claim [ClaimResult] claimed partition batch
+ # @param history [ReadResult] event history for the partition
+ # @return [Array, PositionedMessage)>] action/source pairs
+ def handle_claim(claim, history:)
+ values = partition_keys.to_h { |k| [k, claim.partition_value[k.to_s]] }
+ handle_batch(values, claim.messages, history:, replaying: claim.replaying)
+ end
+
+ # Copy registered command handlers into subclasses.
+ #
+ # @param subclass [Class] subclass being created
+ # @return [void]
+ def inherited(subclass)
+ super
+ handled_commands.each do |cmd_class|
+ subclass.handled_commands << cmd_class
+ end
+ end
+ end
+
+ attr_reader :partition_values
+
+ # @param partition_values [Hash{Symbol => String}] partition key-value pairs
+ def initialize(partition_values = {})
+ @partition_values = partition_values
+ @uncommitted_events = []
+ end
+
+ # Decide a command against the decider's current in-memory state.
+ #
+ # @param command [Sourced::Message] command to handle
+ # @return [Array] newly produced events
+ def decide(command)
+ @uncommitted_events = []
+ method_name = Sourced.message_method_name(PREFIX, command.class.to_s)
+ send(method_name, state, command) if respond_to?(method_name)
+ @uncommitted_events.dup
+ end
+
+ # Produce a new event from within a command handler and apply it
+ # to the decider's in-memory state immediately.
+ #
+ # Accepts either a messages class or a symbol resolved via .[].
+ #
+ # @param event_class [Class, Symbol] event class or symbolic event name
+ # @param payload [Hash] payload attributes for the event
+ # @return [Sourced::Message] the newly built event
+ #
+ # @example Produce by class
+ # command RegisterDevice do |_state, cmd|
+ # event DeviceRegistered, device_id: cmd.payload.device_id
+ # end
+ #
+ # @example Produce by symbol
+ # command RegisterDevice do |_state, cmd|
+ # event :device_registered, device_id: cmd.payload.device_id
+ # end
+ def event(event_class, payload = {})
+ event_class = self.class[event_class] if event_class.is_a?(Symbol)
+ evt = event_class.new(payload: payload)
+ @uncommitted_events << evt
+ evolve([evt])
+ evt
+ end
+ end
+end
diff --git a/lib/sourced/dispatcher.rb b/lib/sourced/dispatcher.rb
index bdd4d979..651b6618 100644
--- a/lib/sourced/dispatcher.rb
+++ b/lib/sourced/dispatcher.rb
@@ -1,41 +1,37 @@
# frozen_string_literal: true
require 'sourced/work_queue'
-require 'sourced/worker'
require 'sourced/catchup_poller'
+require 'sourced/worker'
+require 'sourced/scheduled_message_poller'
+require 'sourced/stale_claim_reaper'
module Sourced
# Orchestrator that wires together the signal-driven dispatch pipeline:
- # {WorkQueue}, {NotificationQueuer}, {CatchUpPoller}, backend notifier, and {Worker}s.
+ # {WorkQueue}, {NotificationQueuer}, {CatchUpPoller}, store notifier, and {Worker}s.
#
- # Does not own the process lifecycle — the caller ({Supervisor} or Falcon service)
- # provides the task/fiber context via {#spawn_into}, and triggers shutdown
- # via {#stop}.
+ # Does not own the process lifecycle — the caller provides the task/fiber
+ # context via {#spawn_into}, and triggers shutdown via {#stop}.
#
- # @example Usage with Supervisor
- # dispatcher = Dispatcher.new(router: Sourced::Router, worker_count: 4)
+ # @example Usage with a task runner
+ # dispatcher = Sourced::Dispatcher.new(router: router, worker_count: 4)
# executor.start do |task|
# dispatcher.spawn_into(task)
# end
- # # On shutdown:
# dispatcher.stop
#
# @example With custom queue for testing
# queue = WorkQueue.new(max_per_reactor: 2, queue: Queue.new)
- # dispatcher = Dispatcher.new(router: router, work_queue: queue)
+ # dispatcher = Sourced::Dispatcher.new(router: router, work_queue: queue)
class Dispatcher
- # Subscriber for the backend notifier pub/sub. Routes events to the {WorkQueue}
+ # Subscriber for the store notifier. Routes events to the {WorkQueue}
# by resolving message types or group IDs to reactor classes.
#
# Handles two events:
- # - +'messages_appended'+ — value is comma-separated type strings;
+ # - +'messages_appended'+ — comma-separated type strings;
# maps types to interested reactors and pushes them
- # - +'reactor_resumed'+ — value is a consumer group ID;
+ # - +'reactor_resumed'+ — a consumer group ID;
# looks up the reactor and pushes it directly
- #
- # @example
- # queuer = NotificationQueuer.new(work_queue: queue, reactors: [OrderReactor])
- # backend.notifier.subscribe(queuer)
class NotificationQueuer
MESSAGES_APPENDED = 'messages_appended'
REACTOR_RESUMED = 'reactor_resumed'
@@ -82,29 +78,45 @@ def build_type_lookup(reactors)
# @return [Hash{String => Class}] mapping from group_id to reactor class
def build_group_id_lookup(reactors)
reactors.each_with_object({}) do |reactor, lookup|
- lookup[reactor.consumer_info.group_id] = reactor
+ lookup[reactor.group_id] = reactor
end
end
end
- # @return [Array] worker instances managed by this dispatcher
+ # @return [Array] worker instances managed by this dispatcher
attr_reader :workers
- # @param router [Router] the router providing async_reactors and backend
+ def self.spawn_into(task)
+ config = Sourced.config
+ dispatcher = Sourced::Dispatcher.new(
+ router: Sourced.router,
+ worker_count: config.worker_count,
+ batch_size: config.batch_size,
+ max_drain_rounds: config.max_drain_rounds,
+ catchup_interval: config.catchup_interval,
+ housekeeping_interval: config.housekeeping_interval,
+ claim_ttl_seconds: config.claim_ttl_seconds,
+ logger: config.logger
+ ).spawn_into(task)
+ end
+
+ # @param router [Sourced::Router] the router providing reactors and store
# @param worker_count [Integer] number of worker fibers to spawn (default 2)
- # @param batch_size [Integer] messages per backend fetch (default 1)
- # @param max_drain_rounds [Integer] max drain iterations before a worker
- # re-enqueues a reactor (default 10)
+ # @param batch_size [Integer] max messages per claim (default 50)
+ # @param max_drain_rounds [Integer] max drain iterations before re-enqueue (default 10)
# @param catchup_interval [Numeric] seconds between catch-up polls (default 5)
- # @param work_queue [WorkQueue, nil] optional pre-built queue (useful for testing
- # with a plain +Queue.new+ to avoid Async dependency)
+ # @param housekeeping_interval [Numeric] seconds between heartbeat/reap cycles (default 30)
+ # @param claim_ttl_seconds [Integer] stale claim age threshold in seconds (default 120)
+ # @param work_queue [WorkQueue, nil] optional pre-built queue (useful for testing)
# @param logger [Object] logger instance
def initialize(
router:,
worker_count: 2,
- batch_size: 1,
+ batch_size: 50,
max_drain_rounds: 10,
catchup_interval: 5,
+ housekeeping_interval: 30,
+ claim_ttl_seconds: 120,
work_queue: nil,
logger: Sourced.config.logger
)
@@ -114,37 +126,49 @@ def initialize(
return if worker_count.zero?
- reactors = router.async_reactors.select { |r| r.handled_messages.any? }.to_a
+ reactors = router.reactors.select { |r| r.handled_messages.any? }.to_a
@work_queue = work_queue || WorkQueue.new(max_per_reactor: worker_count)
@workers = worker_count.times.map do |i|
Worker.new(
work_queue: @work_queue,
- router: router,
+ router:,
name: "worker-#{i}",
- batch_size: batch_size,
- max_drain_rounds: max_drain_rounds,
- logger: logger
+ batch_size:,
+ max_drain_rounds:,
+ logger:
)
end
notification_queuer = NotificationQueuer.new(work_queue: @work_queue, reactors: reactors)
- @backend_notifier = router.backend.notifier
- @backend_notifier.subscribe(notification_queuer)
+ @store_notifier = router.store.notifier
+ @store_notifier.subscribe(notification_queuer)
@catchup_poller = CatchUpPoller.new(
work_queue: @work_queue,
- reactors: reactors,
+ reactors:,
+ interval: catchup_interval,
+ logger:
+ )
+
+ @scheduled_message_poller = ScheduledMessagePoller.new(
+ store: router.store,
interval: catchup_interval,
- logger: logger
+ logger:
+ )
+
+ @stale_claim_reaper = StaleClaimReaper.new(
+ store: router.store,
+ interval: housekeeping_interval,
+ ttl_seconds: claim_ttl_seconds,
+ worker_ids_provider: -> { @workers.map(&:name) },
+ logger:
)
end
# Spawn all component fibers into the caller's task context.
- # Spawns: backend notifier (LISTEN), catch-up poller, and N workers.
- #
- # Supports both the executor's Task (+#spawn+) and Async::Task (+#async+).
+ # Spawns: store notifier (e.g. PG LISTEN), catch-up poller, and N workers.
#
# @param task [Object] an executor task or Async::Task to spawn fibers into
# @return [void]
@@ -153,32 +177,40 @@ def spawn_into(task)
s = task.respond_to?(:spawn) ? :spawn : :async
- # Backend notifier (PG LISTEN fiber — no-op for non-PG)
- task.send(s) { @backend_notifier.start }
+ # Store notifier (start — no-op for InlineNotifier)
+ task.send(s) { @store_notifier.start }
# CatchUp poller
task.send(s) { @catchup_poller.run }
+ # Scheduled message poller
+ task.send(s) { @scheduled_message_poller.run }
+
+ # Stale claim reaper
+ task.send(s) { @stale_claim_reaper.run }
+
# Workers
@workers.each do |w|
task.send(s) { w.run }
end
+
+ self
end
# Stop all components and close the work queue.
- # Stops in order: backend notifier, catch-up poller, workers, then
- # pushes shutdown sentinels into the queue to unblock any waiting workers.
#
# @return [void]
def stop
return if @workers.empty?
- @logger.info "Dispatcher: stopping #{@workers.size} workers"
- @backend_notifier.stop
+ @logger.info "Sourced::Dispatcher: stopping #{@workers.size} workers"
+ @store_notifier.stop
@catchup_poller.stop
+ @scheduled_message_poller.stop
+ @stale_claim_reaper.stop
@workers.each(&:stop)
@work_queue.close(@workers.size)
- @logger.info 'Dispatcher: all components stopped'
+ @logger.info 'Sourced::Dispatcher: all components stopped'
end
end
end
diff --git a/lib/sourced/durable_workflow.rb b/lib/sourced/durable_workflow.rb
index a01a1f4b..c66edf13 100644
--- a/lib/sourced/durable_workflow.rb
+++ b/lib/sourced/durable_workflow.rb
@@ -1,38 +1,64 @@
# frozen_string_literal: true
+require 'securerandom'
+
module Sourced
+ # Durable workflow base class.
+ #
+ # A workflow instance is identified by a +workflow_id+ string, which doubles
+ # as the partition key. All lifecycle events (WorkflowStarted, StepStarted,
+ # StepFailed, StepComplete, ContextUpdated, WaitStarted, WaitEnded,
+ # WorkflowComplete, WorkflowFailed) carry +workflow_id+ as their first
+ # payload attribute so {Sourced::Message#extracted_keys} indexes them for
+ # partition queries.
+ #
+ # The step-memoisation mechanism (@lookup + catch(:halt))
+ # allows +durable+ / +wait+ / +context+ / +execute+ to be re-entered safely.
class DurableWorkflow
extend Sourced::Consumer
+ include Sourced::Evolve
+
+ partition_by :workflow_id
UnknownMessageError = Class.new(StandardError)
+ # Stable hash-based key for a given (method, args) pair.
def self.step_key(step_name, args)
[step_name, args].hash.to_s
end
def self.inherited(child)
+ super
+ child.partition_by(:workflow_id)
cname = child.name.to_s.gsub(/::/, '.')
.gsub(/([A-Z]+)([A-Z][a-z])/, '\1_\2')
.gsub(/([a-z\d])([A-Z])/, '\1_\2')
- .tr("-", "_")
+ .tr('-', '_')
.downcase
child.const_set(:WorkflowStarted, Sourced::Event.define("#{cname}.workflow.started") do
+ attribute :workflow_id, String
attribute :args, Sourced::Types::Array.default([].freeze)
end)
child.const_set(:ContextUpdated, Sourced::Event.define("#{cname}.context.updated") do
+ attribute :workflow_id, String
attribute :context, Sourced::Types::Any
end)
child.const_set(:WorkflowComplete, Sourced::Event.define("#{cname}.workflow.complete") do
+ attribute :workflow_id, String
attribute :output, Sourced::Types::Any
end)
- child.const_set(:WorkflowFailed, Sourced::Event.define("#{cname}.workflow.failed"))
+ child.const_set(:WorkflowFailed, Sourced::Event.define("#{cname}.workflow.failed") do
+ attribute :workflow_id, String
+ end)
child.const_set(:StepStarted, Sourced::Event.define("#{cname}.step.started") do
+ attribute :workflow_id, String
attribute :key, String
attribute :step_name, Sourced::Types::Lax::Symbol
attribute :args, Sourced::Types::Array.default([].freeze)
end)
child.const_set(:StepFailed, Sourced::Event.define("#{cname}.step.failed") do
+ attribute :workflow_id, String
attribute :key, String
attribute :step_name, Sourced::Types::Lax::Symbol
attribute :error_message, String
@@ -40,71 +66,46 @@ def self.inherited(child)
attribute :backtrace, Sourced::Types::Array[String]
end)
child.const_set(:StepComplete, Sourced::Event.define("#{cname}.step.complete") do
+ attribute :workflow_id, String
attribute :key, String
attribute :step_name, Sourced::Types::Lax::Symbol
attribute :output, Sourced::Types::Any
end)
child.const_set(:WaitStarted, Sourced::Event.define("#{cname}.wait.started") do
+ attribute :workflow_id, String
attribute :count, Integer
attribute :at, Sourced::Types::Forms::Time
end)
- child.const_set(:WaitEnded, Sourced::Event.define("#{cname}.wait.ended"))
- end
+ child.const_set(:WaitEnded, Sourced::Event.define("#{cname}.wait.ended") do
+ attribute :workflow_id, String
+ end)
- def self.handled_messages
+ # Register all event classes so:
+ # - Router claims them on our consumer group (`handled_messages`).
+ # - `context_for(workflow_id:)` builds OR conditions for the partition
+ # read (one per event type, via `Message.to_conditions`).
[
- self::WorkflowStarted,
- self::WorkflowComplete,
- self::StepStarted,
- self::StepFailed,
- self::StepComplete,
- self::WaitStarted,
- self::WaitEnded
- ]
- end
-
- def self.handle(message, history:, logger: Sourced.config.logger)
- from(history, logger:).__handle(message)
- end
-
- class Waiter
- attr_reader :stream_id, :instance
-
- def initialize(reactor, stream_id, backend: Sourced.config.backend, logger: Sourced.config.logger)
- @reactor, @stream_id, @backend = reactor, stream_id, backend
- @instance = @reactor.new(logger:)
- @value = nil
- end
-
- def wait
- while instance.status != :complete && instance.status != :failed
- sleep 0.1
- load
- end
- instance
- end
-
- def load
- history = @backend.read_stream(@stream_id)
- instance.__from(history)
+ child::WorkflowStarted, child::ContextUpdated, child::WorkflowComplete,
+ child::WorkflowFailed, child::StepStarted, child::StepFailed,
+ child::StepComplete, child::WaitStarted, child::WaitEnded
+ ].each do |klass|
+ child.handled_messages_for_evolve << klass unless child.handled_messages_for_evolve.include?(klass)
end
end
- def self.from(history, logger: nil)
- new(logger:).__from(history)
- end
-
- def self.execute(*args)
- stream_id = "workflow-#{SecureRandom.uuid}"
- evt = self::WorkflowStarted.parse(stream_id:, payload: { args: })
- Sourced.config.backend.append_next_to_stream(stream_id, evt)
- Waiter.new(self, stream_id, backend: Sourced.config.backend)
+ # Message types this consumer claims. Same set as evolve types because
+ # every workflow event both advances state and re-triggers the workflow.
+ def self.handled_messages
+ handled_messages_for_evolve
end
+ # Define the initial context hash. Block receives no arguments.
def self.context(&block)
define_method :initial_context, &block
end
+ # Wrap a method so the runtime memoises its result across workflow
+ # re-entries.
def self.durable(method_name, retries: nil)
source_method = :"__durable_source_#{method_name}"
alias_method source_method, method_name
@@ -115,38 +116,37 @@ def self.durable(method_name, retries: nil)
case cached&.status
when :complete
cached.output
- when :started # ready to call.
+ when :started
begin
output = send(source_method, *args)
- @new_events << self.class::StepComplete.parse(
- stream_id: id,
- payload: { key:, step_name: method_name, output: }
+ @new_events << self.class::StepComplete.new(
+ payload: { workflow_id: id, key:, step_name: method_name, output: }
)
throw :halt
rescue StandardError => e
- # TODO: this catches NameError?
- # Syntax errors should immediatly stop the workflow, not retry
- @new_events << self.class::StepFailed.parse(
- stream_id: id,
- payload: { key:, step_name: method_name, error_message: e.inspect, error_class: e.class.to_s, backtrace: e.backtrace }
+ @new_events << self.class::StepFailed.new(
+ payload: {
+ workflow_id: id,
+ key:,
+ step_name: method_name,
+ error_message: e.inspect,
+ error_class: e.class.to_s,
+ backtrace: e.backtrace
+ }
)
if retries && cached.attempts == retries
- @new_events << self.class::WorkflowFailed.parse(stream_id: id)
+ @new_events << self.class::WorkflowFailed.new(payload: { workflow_id: id })
end
-
throw :halt
end
- when :failed # retry. Exponential backoff, etc
- @new_events << self.class::StepStarted.parse(
- stream_id: id,
- payload: { key:, step_name: method_name, args: }
+ when :failed
+ @new_events << self.class::StepStarted.new(
+ payload: { workflow_id: id, key:, step_name: method_name, args: }
)
-
throw :halt
- when nil # first call. Schedule StepStarted
- @new_events << self.class::StepStarted.parse(
- stream_id: id,
- payload: { key:, step_name: method_name, args: }
+ when nil
+ @new_events << self.class::StepStarted.new(
+ payload: { workflow_id: id, key:, step_name: method_name, args: }
)
throw :halt
end
@@ -176,12 +176,106 @@ def complete_with(output)
end
end
- attr_reader :id, :seq, :context, :args, :output, :status
+ # Kick off a new workflow instance. Appends a WorkflowStarted event and
+ # returns a {Waiter} that can poll for completion.
+ #
+ # @param args [Array] positional args passed to the workflow's #execute
+ # @param store [Sourced::Store] defaults to Sourced.store
+ # @return [Waiter]
+ def self.execute(*args, store: Sourced.store)
+ workflow_id = "workflow-#{SecureRandom.uuid}"
+ evt = self::WorkflowStarted.new(payload: { workflow_id:, args: })
+ store.append([evt])
+ Waiter.new(self, workflow_id, store:)
+ end
+
+ # Router entry point. Drops claim.messages from the read history (the
+ # router's +store.read(conditions)+ returns the full partition, including
+ # messages being claimed) and delegates to {.handle_batch}.
+ def self.handle_claim(claim, history:)
+ claim_positions = claim.messages.map { |m| m.position if m.respond_to?(:position) }.compact.to_set
+ prior = history.messages.reject { |m| m.respond_to?(:position) && claim_positions.include?(m.position) }
+ prior_history = ReadResult.new(messages: prior, guard: history.guard)
+ values = claim.partition_value.transform_keys(&:to_sym)
+ handle_batch(values, claim.messages, history: prior_history)
+ end
+
+ # GWT-compatible entry point. +history.messages+ must be disjoint from
+ # +new_messages+ — the caller owns that distinction.
+ def self.handle_batch(partition_values, new_messages, history:, replaying: false)
+ workflow_id = partition_values[:workflow_id]
+ instance = new([workflow_id])
+ instance.__replay(history.messages)
+
+ each_with_partial_ack(new_messages) do |msg|
+ instance.__evolve(msg)
+ actions = instance.__handle(msg, guard: history.guard)
+ [actions, msg]
+ end
+ end
+
+ # Direct handler used by unit tests and by {Waiter}. Mirrors
+ # {Sourced::DurableWorkflow.handle}: +history+ should already contain
+ # +message+ as its last element.
+ def self.handle(message, history:)
+ from(history).__handle(message)
+ end
+
+ # Rebuild a workflow instance by replaying +history+.
+ def self.from(history)
+ new.__replay(history)
+ end
+
+ # Load a workflow instance for +workflow_id+ from the store.
+ def self.load(workflow_id, store: Sourced.store)
+ _inst, _rr = Sourced.load(self, store:, workflow_id: workflow_id)
+ end
- def initialize(logger: nil)
- @id = nil
- @seq = 0
- @logger = logger
+ # Polls the store for terminal workflow events.
+ class Waiter
+ attr_reader :workflow_id, :instance
+
+ def initialize(klass, workflow_id, store: Sourced.store)
+ @klass = klass
+ @workflow_id = workflow_id
+ @store = store
+ @instance = klass.new([workflow_id])
+ end
+
+ def wait(timeout: nil)
+ deadline = timeout ? Time.now + timeout : nil
+ until @instance.status == :complete || @instance.status == :failed
+ raise 'DurableWorkflow wait timed out' if deadline && Time.now > deadline
+
+ sleep 0.05
+ load
+ end
+ @instance
+ end
+
+ def load
+ handled_types = @klass.handled_messages_for_evolve.map(&:type).uniq
+ result = @store.read_partition({ workflow_id: @workflow_id }, handled_types:)
+ @instance = @klass.new([@workflow_id])
+ @instance.__replay(result.messages)
+ @instance
+ end
+ end
+
+ attr_reader :id, :context, :args, :output, :status
+
+ # +partition_values+ may be:
+ # - an Array like +['wf-id']+ (from {.handle_claim})
+ # - a Hash like +{ workflow_id: 'wf-id' }+ (from {Sourced.load})
+ # - a String +'wf-id'+
+ # - nil
+ def initialize(partition_values = nil)
+ @id = case partition_values
+ when Array then partition_values.first
+ when Hash then partition_values[:workflow_id]
+ when String then partition_values
+ else nil
+ end
@status = :new
@args = []
@output = nil
@@ -189,27 +283,28 @@ def initialize(logger: nil)
@new_events = []
@wait_count = 0
@waiters = []
- @initial_context = nil
@context = initial_context
end
def initial_context = nil
- def __from(history)
- history.each do |event|
- __evolve(event)
- end
+ def __replay(history)
+ Array(history).each { |m| __evolve(m) }
self
end
- def __evolve(event)
- @id = event.stream_id
- @seq = event.seq
+ # Override Sourced::Evolve#evolve so {Sourced.load} (which calls +instance.evolve+)
+ # applies workflow events via our manual dispatcher.
+ def evolve(messages)
+ __replay(messages)
+ end
+ def __evolve(event)
case event
when self.class::ContextUpdated
@context = deep_dup(event.payload.context)
when self.class::WorkflowStarted
+ @id ||= event.payload.workflow_id
@args = event.payload.args
@status = :started
when self.class::WorkflowFailed
@@ -233,18 +328,16 @@ def __evolve(event)
end
end
- def __handle(message)
- return Sourced::Actions::OK if @status == :complete || @status == :failed
+ # Decide the next action given +message+. State is assumed to already
+ # reflect +message+ (caller replayed it).
+ def __handle(message, guard: nil)
+ return Actions::OK if @status == :complete || @status == :failed
if message.is_a?(self.class::WaitStarted)
- evt = self.class::WaitEnded.parse(stream_id: id)
- return Sourced::Actions::Schedule.new([evt], at: message.payload.at)
+ evt = self.class::WaitEnded.new(payload: { workflow_id: id })
+ return Actions::Schedule.new([evt], at: message.payload.at)
end
- # Raise if @waiting ?
- # we don't allow handling any new messages until wait has ended.
-
- # TODO: all this deep-duping is not efficient.
@initial_context = deep_dup(@context)
completed = false
@@ -255,47 +348,48 @@ def __handle(message)
completed = true
end
- # If any step updated context, even on failure
- # make sure to log a ContextUpdated event to keep track of that state
- @new_events << self.class::ContextUpdated.parse(
- stream_id: id,
- payload: { context: deep_dup(@context) }
- ) if @context != @initial_context
-
- @new_events << self.class::WorkflowComplete.parse(
- stream_id: id,
- payload: { output: }
- ) if completed
-
- last_seq = @seq
- events = @new_events.map { |e| e.with(seq: last_seq += 1 )}
- Sourced::Actions::AppendAfter.new(id, events)
+ if @context != @initial_context
+ @new_events << self.class::ContextUpdated.new(
+ payload: { workflow_id: id, context: deep_dup(@context) }
+ )
+ end
+
+ if completed
+ @new_events << self.class::WorkflowComplete.new(
+ payload: { workflow_id: id, output: }
+ )
+ end
+
+ events = @new_events
+ @new_events = []
+ return Actions::OK if events.empty?
+
+ Actions::Append.new(events, guard: guard)
end
private
- attr_reader :logger
-
def wait(seconds)
@wait_count += 1
- if @waiters[@wait_count] # we're already waiting for this method call
- return seconds
- else # first time we call this method
- @new_events << self.class::WaitStarted.parse(
- stream_id: id,
- payload: { count: @wait_count, at: Time.now + seconds }
+ if @waiters[@wait_count]
+ seconds
+ else
+ @new_events << self.class::WaitStarted.new(
+ payload: { workflow_id: id, count: @wait_count, at: Time.now + seconds }
)
-
throw :halt
end
end
- def deep_dup(hash)
- return hash.dup unless hash.is_a?(Hash)
-
- hash.each.with_object({}) do |(k, v), new_hash|
- new_hash[k] = deep_dup(v)
+ def deep_dup(value)
+ case value
+ when Hash
+ value.each.with_object({}) { |(k, v), h| h[k] = deep_dup(v) }
+ when Array
+ value.map { |v| deep_dup(v) }
+ else
+ value.dup rescue value
end
end
end
diff --git a/lib/sourced/error_strategy.rb b/lib/sourced/error_strategy.rb
index 2d1ff632..57a4fbb5 100644
--- a/lib/sourced/error_strategy.rb
+++ b/lib/sourced/error_strategy.rb
@@ -3,9 +3,9 @@
module Sourced
# Built-in configurable error strategy
# for handling exceptions raised during processing messages (commands or events)
- # By default it stops the consumer group immediately.
+ # By default it marks the consumer group as failed immediately.
# It can be configured to retry a number of times with a delay between retries.
- # It can also register callbacks to be called on retry and on stop.
+ # It can also register callbacks to be called on retry and on failure.
#
# @example retry with exponential back off and callbacks
# strategy = Sourced::ErrorStrategy.new do |s|
@@ -15,7 +15,7 @@ module Sourced
# LOGGER.info("Retrying #{n} times")
# end
#
- # s.on_stop do |exception, message|
+ # s.on_fail do |exception, _message|
# Sentry.capture_exception(exception)
# end
# end
@@ -32,7 +32,7 @@ def initialize(&setup)
@retry_after = RETRY_AFTER
@backoff = BACKOFF
@on_retry = NOOP_CALLBACK
- @on_stop = NOOP_CALLBACK
+ @on_fail = NOOP_CALLBACK
yield(self) if block_given?
freeze
@@ -53,15 +53,15 @@ def on_retry(callable = nil, &blk)
@on_retry = callable || blk
end
- def on_stop(callable = nil, &blk)
- @on_stop = callable || blk
+ def on_fail(callable = nil, &blk)
+ @on_fail = callable || blk
end
# The Error Strategy interface
#
# @param exception [Exception]
# @param message [Sourced::Message]
- # @param group [#retry, #stop]
+ # @param group [#retry, #fail]
def call(exception, message, group)
retry_count = group.error_context[:retry_count] || 1
if retry_count <= max_retries
@@ -71,8 +71,8 @@ def call(exception, message, group)
retry_count += 1
group.retry(later, retry_count:)
else
- @on_stop.call(exception, message)
- group.stop(exception:, message:)
+ @on_fail.call(exception, message)
+ group.fail(exception:)
end
end
diff --git a/lib/sourced/evolve.rb b/lib/sourced/evolve.rb
index af0a963e..f99a4c0c 100644
--- a/lib/sourced/evolve.rb
+++ b/lib/sourced/evolve.rb
@@ -1,94 +1,44 @@
# frozen_string_literal: true
module Sourced
- # This mixin provides an .event macro
- # to register event handlers for a class
- # These event handlers are "evolvers", ie. they evolve
- # a piece of state based on events.
- # More here: https://ismaelcelis.com/posts/decide-evolve-react-pattern-in-ruby/#2-evolve
- #
- # Example:
- #
- # class Projector
- # include Sourced::Evolve
- #
- # state do
- # { status: 'new' }
- # end
- #
- # event SomethingHappened do |state, event|
- # state[:status] = 'done'
- # end
- # end
- #
- # pr = Projector.new
- # state = pr.evolve([SomethingHappened.new])
- # state[:status] # => 'done'
- #
- # From the outside, this mixin exposes .handled_messages_for_evolve
- #
- # .handled_messages_for_evolve() Array
- #
- # It also provides a .before_evolve and .evolve_all macros
- # See comments in code for details.
+ # Evolve mixin for reactors.
+ # State evolution from a history of messages.
+ # State block receives partition values hash instead of stream id.
module Evolve
- PREFIX = 'evolution'
- NOOP_HANDLER = ->(*_) { nil }
+ PREFIX = 'sourced_evolution'
def self.included(base)
super
base.extend ClassMethods
end
- # Initialize in-memory state for this evolver.
- # Override this method to provide a custom initial state.
- # @return [Any] the initial state
- def init_state(_id)
+ def init_state(_partition_values)
nil
end
def state
- @state ||= init_state(id)
+ @state ||= init_state(partition_values)
end
- # Override this in host class
- def id = nil
+ def partition_values
+ @partition_values ||= {}
+ end
- # Apply a list of events to a piece of state
- # by running event handlers registered in this class
- # via the .event macro.
- #
- # @param events [Array]
- # @return [Object]
- def evolve(events)
- Array(events).each do |event|
- method_name = Sourced.message_method_name(Evolve::PREFIX, event.class.to_s)
- # We might be evolving old events in history
- # even if we don't have handlers for them anymore
- # we still need to increment seq
- __update_on_evolve(event)
- if respond_to?(method_name)
- before_evolve(state, event)
- send(method_name, state, event)
- end
+ # Apply messages to state via registered handlers.
+ # Skips messages without a registered handler.
+ def evolve(messages)
+ Array(messages).each do |msg|
+ method_name = Sourced.message_method_name(PREFIX, msg.class.to_s)
+ send(method_name, state, msg) if respond_to?(method_name)
end
-
state
end
- private def before_evolve(*_)
- nil
- end
-
- private def __update_on_evolve(event)
- # Noop
- end
-
module ClassMethods
def inherited(subclass)
super
- handled_messages_for_evolve.each do |evt_type|
- subclass.handled_messages_for_evolve << evt_type
+ handled_messages_for_evolve.each do |klass|
+ subclass.handled_messages_for_evolve << klass
end
end
@@ -96,67 +46,15 @@ def handled_messages_for_evolve
@handled_messages_for_evolve ||= []
end
- # Define an initial state factory for this evolver.
- # @example
- #
- # state do
- # { status: 'new' }
- # end
- #
+ # Define initial state factory. Block receives partition values hash.
def state(&blk)
define_method(:init_state, &blk)
end
- # This module only accepts registering event handlers
- # with qualified event classes
- # Decider overrides this method to allow
- # defining event handlers with symbols
- # which are registered as Event classes in the decider namespace.
- # @example
- #
- # event SomethingHappened do |state, event|
- # state[:status] = 'done'
- # end
- #
- # @param event_class [Sourced::Message]
- # @return [void]
- def event(event_class, &block)
- unless event_class.is_a?(Class) && event_class < Sourced::Message
- raise ArgumentError,
- "Invalid argument #{event_class.inspect} for #{self}.event"
- end
-
- handled_messages_for_evolve << event_class
- block = NOOP_HANDLER unless block_given?
- define_method(Sourced.message_method_name(Evolve::PREFIX, event_class.to_s), &block)
- end
-
- # Run this block before any of the registered event handlers
- # Example:
- # before_evolve do |state, event|
- # state.udpated_at = event.created_at
- # end
- def before_evolve(&block)
- define_method(:before_evolve, &block)
- end
-
- # Example:
- # # With an Array of event types
- # evolve_all [:event_type1, :event_type2] do |state, event|
- # state.updated_at = event.created_at
- # end
- #
- # # From another Evolver that responds to #handled_messages_for_evolve
- # evolve_all CartAggregate do |state, event|
- # state.updated_at = event.created_at
- # end
- #
- # @param event_list [Array, #handled_messages_for_evolve() [Array}]
- def evolve_all(event_list, &block)
- event_list = event_list.handled_messages_for_evolve if event_list.respond_to?(:handled_messages_for_evolve)
- event_list.each do |event_type|
- event(event_type, &block)
- end
+ # Register an evolve handler for a Sourced::Message subclass.
+ def evolve(message_class, &block)
+ handled_messages_for_evolve << message_class
+ define_method(Sourced.message_method_name(PREFIX, message_class.to_s), &block)
end
end
end
diff --git a/lib/sourced/falcon/environment.rb b/lib/sourced/falcon/environment.rb
index 22ff606a..f30a77a3 100644
--- a/lib/sourced/falcon/environment.rb
+++ b/lib/sourced/falcon/environment.rb
@@ -2,13 +2,15 @@
module Sourced
module Falcon
- # Environment mixin for configuring a combined Falcon web server + Sourced workers service.
+ # Environment mixin for configuring a combined Falcon web server + workers service.
#
- # Include this module in a Falcon service definition to get Sourced worker defaults
- # alongside the standard Falcon server environment.
+ # Include this module in a Falcon service definition to get workers defaults
+ # alongside the standard Falcon server environment. All settings are read from
+ # {Sourced.config} — no per-service config methods needed.
#
- # Housekeeping settings are read from Sourced.config by default (configured in your
- # boot/environment file). They can be overridden per-service in falcon.rb if needed.
+ # The Service automatically calls {Sourced.setup!} at the start of +run+ to
+ # re-establish database connections after Falcon forks (SQLite connections
+ # are not fork-safe). This replays the block passed to {Sourced.configure}.
#
# @example falcon.rb
# #!/usr/bin/env falcon-host
@@ -20,38 +22,11 @@ module Falcon
# include Falcon::Environment::Rackup
#
# url "http://[::]:9292"
- # sourced_worker_count 4
# end
module Environment
include ::Falcon::Environment::Server
- def service_class
- Sourced::Falcon::Service
- end
-
- # Number of Sourced worker fibers to spawn per container process.
- # @return [Integer]
- def sourced_worker_count = Sourced.config.worker_count
-
- # Number of housekeeper fibers to spawn per container process.
- # @return [Integer]
- def sourced_housekeeping_count = Sourced.config.housekeeping_count
-
- # Seconds between housekeeper scheduling cycles.
- # @return [Numeric]
- def sourced_housekeeping_interval = Sourced.config.housekeeping_interval
-
- # Seconds between worker heartbeats.
- # @return [Numeric]
- def sourced_housekeeping_heartbeat_interval = Sourced.config.housekeeping_heartbeat_interval
-
- # Seconds before a claim is considered stale and can be reaped.
- # @return [Numeric]
- def sourced_housekeeping_claim_ttl_seconds = Sourced.config.housekeeping_claim_ttl_seconds
-
- # Number of messages to fetch per lock cycle for batch processing.
- # @return [Integer]
- def sourced_worker_batch_size = Sourced.config.worker_batch_size
+ def service_class = Sourced::Falcon::Service
end
end
end
diff --git a/lib/sourced/falcon/service.rb b/lib/sourced/falcon/service.rb
index d5e099fc..0135d0a2 100644
--- a/lib/sourced/falcon/service.rb
+++ b/lib/sourced/falcon/service.rb
@@ -1,44 +1,26 @@
# frozen_string_literal: true
require 'sourced/dispatcher'
-require 'sourced/house_keeper'
module Sourced
module Falcon
# A Falcon service that runs both the web server and Sourced background workers
# as sibling fibers within the same Async reactor.
#
- # Uses a Dispatcher for signal-driven worker dispatch instead of blind polling.
- # Lifecycle is managed by Falcon's container — no need for a separate Supervisor
- # process or signal handling.
+ # Uses a Sourced::Dispatcher for signal-driven worker dispatch. The Dispatcher
+ # already embeds the StaleClaimReaper, so no separate HouseKeeper is needed
+ # (unlike {Sourced::Falcon::Service}).
+ #
+ # All configuration is read from {Sourced.config}.
class Service < ::Falcon::Service::Server
def run(instance, evaluator)
- server = evaluator.make_server(@bound_endpoint)
-
- @dispatcher = Sourced::Dispatcher.new(
- router: Sourced::Router,
- worker_count: evaluator.sourced_worker_count,
- batch_size: evaluator.sourced_worker_batch_size,
- logger: Sourced.config.logger
- )
+ Sourced.setup!
- housekeepers = evaluator.sourced_housekeeping_count.times.map do |i|
- Sourced::HouseKeeper.new(
- backend: Sourced.config.backend,
- name: "falcon-hk-#{i}",
- interval: evaluator.sourced_housekeeping_interval,
- heartbeat_interval: evaluator.sourced_housekeeping_heartbeat_interval,
- claim_ttl_seconds: evaluator.sourced_housekeeping_claim_ttl_seconds,
- worker_ids_provider: -> { @dispatcher.workers.map(&:name) }
- )
- end
-
- @sourced_housekeepers = housekeepers
+ server = evaluator.make_server(@bound_endpoint)
Async do |task|
server.run
- housekeepers.each { |hk| task.async { hk.work } }
- @dispatcher.spawn_into(task)
+ Sourced::Dispatcher.spawn_into(task)
task.children.each(&:wait)
end
@@ -47,7 +29,6 @@ def run(instance, evaluator)
def stop(...)
@dispatcher&.stop
- @sourced_housekeepers&.each(&:stop)
super
end
end
diff --git a/lib/sourced/handler.rb b/lib/sourced/handler.rb
deleted file mode 100644
index 0123bc72..00000000
--- a/lib/sourced/handler.rb
+++ /dev/null
@@ -1,84 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Handler
- PREFIX = 'handle'
-
- def self.included(base)
- base.send :extend, Consumer
- base.send :extend, ClassMethods
- end
-
- def handle(message, history:, replaying: false)
- handler_name = Sourced.message_method_name(PREFIX, message.class.name)
- return Actions::OK unless respond_to?(handler_name)
-
- args = {history:, replaying:}
- expected_args = self.class.__args_lookup[handler_name]
- handler_args = expected_args.each.with_object({}) do |key, memo|
- memo[key] = args[key]
- end
-
- send(handler_name, message, **handler_args)
- end
-
- module ClassMethods
- def handled_messages
- @handled_messsages ||= []
- end
-
- def on(*args, &block)
- case args
- in [Symbol => msg_name, Hash => payload_schema]
- __register_named_message_handler(msg_name, payload_schema, &block)
- in [Symbol => msg_name]
- __register_named_message_handler(msg_name, &block)
- in [Class => msg_type] if msg_type < Sourced::Message
- __register_class_message_handler(msg_type, &block)
- else
- args.each do |arg|
- on(*arg, &block)
- end
- end
- end
-
- def handle(message, history: [], replaying: false)
- results = new.handle(message, history:, replaying:)
- Actions.build_for(results)
- end
-
- def __args_lookup
- @__args_lookup ||= {}
- end
-
- private
-
- def __register_named_message_handler(msg_name, payload_schema = nil, &block)
- msg_class = Sourced::Message.define(__message_type(msg_name), payload_schema:)
- klass_name = msg_name.to_s.split('_').map(&:capitalize).join
- const_set(klass_name, msg_class)
- __register_class_message_handler(msg_class, &block)
- end
-
- def __register_class_message_handler(msg_type, &block)
- handled_messages << msg_type
- handler_name = Sourced.message_method_name(PREFIX, msg_type.name)
- __args_lookup[handler_name] = Sourced::Injector.resolve_args(block)
- define_method(handler_name, &block)
- end
-
- # TODO: these are in Actor too
- def __message_type(msg_name)
- [__message_type_prefix, msg_name].join('.').downcase
- end
-
- def message_namespace
- Types::ModuleToMessageType.parse(name.to_s)
- end
-
- def __message_type_prefix
- @__message_type_prefix ||= message_namespace
- end
- end
- end
-end
diff --git a/lib/sourced/house_keeper.rb b/lib/sourced/house_keeper.rb
deleted file mode 100644
index b1b6a938..00000000
--- a/lib/sourced/house_keeper.rb
+++ /dev/null
@@ -1,76 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- class HouseKeeper
- attr_reader :name
-
- def initialize(
- logger: Sourced.config.logger,
- interval: 3,
- heartbeat_interval: 5,
- claim_ttl_seconds: 120,
- backend:,
- name:,
- worker_ids_provider: nil
- )
- @logger = logger
- @interval = interval
- @heartbeat_interval = heartbeat_interval
- @claim_ttl_seconds = claim_ttl_seconds
- @backend = backend
- @name = name
- @running = false
- @worker_ids_provider = worker_ids_provider || -> { [] }
- end
-
- def work
- @running = true
-
- # On start wait for a random period
- # in order to space out multiple house-keepers
- sleep rand(5)
- logger.info "HouseKeeper #{name}: starting"
-
- # Reap stale claims on startup (from previous runs where workers were killed)
- released = backend.release_stale_claims(ttl_seconds: @claim_ttl_seconds)
- logger.info "HouseKeeper #{name}: startup cleanup released #{released} stale claims" if released && released > 0
-
- last_heartbeat = Time.at(0)
- last_stale_reaping = Time.now
- while @running
- sleep @interval
-
- # 1) Schedule due messages
- schcount = backend.update_schedule!
- logger.info "HouseKeeper #{name}: appended #{schcount} scheduled messages" if schcount > 0
-
- now = Time.now
-
- # 2) Heartbeat alive workers (bulk upsert)
- if now - last_heartbeat >= @heartbeat_interval
- ids = Array(@worker_ids_provider.call).uniq
- hb = backend.worker_heartbeat(ids)
- logger.debug "HouseKeeper #{name}: heartbeated #{hb} workers" if hb > 0
- last_heartbeat = now
- end
-
- # 3) Reap stale claims (only every claim_ttl_seconds, since claims can't be stale until then)
- if now - last_stale_reaping >= @claim_ttl_seconds
- released = backend.release_stale_claims(ttl_seconds: @claim_ttl_seconds)
- logger.info "HouseKeeper #{name}: released #{released} stale claims" if released && released > 0
- last_stale_reaping = now
- end
- end
-
- logger.info "HouseKeeper #{name}: stopped"
- end
-
- def stop
- @running = false
- end
-
- private
-
- attr_reader :logger, :backend, :interval
- end
-end
diff --git a/lib/sourced/installer.rb b/lib/sourced/installer.rb
new file mode 100644
index 00000000..f1675abc
--- /dev/null
+++ b/lib/sourced/installer.rb
@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require 'sequel'
+require 'sequel/extensions/migration'
+require 'erb'
+
+module Sourced
+ class Installer
+ TABLE_SUFFIXES = %i[messages key_pairs message_key_pairs scheduled_messages consumer_groups offsets offset_key_pairs workers].freeze
+
+ attr_reader :messages_table, :key_pairs_table, :message_key_pairs_table,
+ :scheduled_messages_table, :consumer_groups_table, :offsets_table,
+ :offset_key_pairs_table, :workers_table
+
+ def initialize(db, logger:, prefix: 'sourced', migration_template: '001_create_sourced_tables.rb.erb')
+ raise ArgumentError, "invalid prefix: #{prefix}" unless prefix.match?(/\A[a-zA-Z_]\w*\z/)
+
+ @db = db
+ @logger = logger
+ @prefix = prefix
+ @migration_template = migration_template
+
+ TABLE_SUFFIXES.each do |suffix|
+ instance_variable_set(:"@#{suffix}_table", :"#{prefix}_#{suffix}")
+ end
+ end
+
+ # Eval the rendered migration and apply :up directly.
+ def install
+ migration.apply(db, :up)
+ logger.info("Sourced tables installed (prefix: #{prefix})")
+ end
+
+ # Check that all expected tables exist.
+ def installed?
+ all_table_names.all? { |t| db.table_exists?(t) }
+ end
+
+ # Apply :down on the migration to drop tables.
+ def uninstall
+ raise 'Not in test environment' unless ENV['ENVIRONMENT'] == 'test'
+
+ migration.apply(db, :down)
+ end
+
+ # Render the migration to a file for use with the host app's Sequel::Migrator.
+ #
+ # installer.copy_migration_to("db/migrations")
+ # installer.copy_migration_to { "db/migrations/#{Time.now.strftime('%Y%m%d%H%M%S')}_create_ccc_tables.rb" }
+ #
+ def copy_migration_to(dir = nil, &block)
+ path = block ? block.call : File.join(dir, '001_create_sourced_tables.rb')
+ File.write(path, rendered_migration)
+ logger.info("Copied Sourced migration to #{path}")
+ path
+ end
+
+ private
+
+ attr_reader :db, :logger, :prefix
+
+ def migration
+ @migration ||= eval(rendered_migration) # rubocop:disable Security/Eval
+ end
+
+ def rendered_migration
+ @rendered_migration ||= begin
+ template_path = File.join(__dir__, 'migrations', @migration_template)
+ ERB.new(File.read(template_path)).result(binding)
+ end
+ end
+
+ # Returns actual symbols for use in installed? checks.
+ def all_table_names
+ TABLE_SUFFIXES.map { |suffix| instance_variable_get(:"@#{suffix}_table") }
+ end
+ end
+end
diff --git a/lib/sourced/message.rb b/lib/sourced/message.rb
index b5d7789d..c0b1e492 100644
--- a/lib/sourced/message.rb
+++ b/lib/sourced/message.rb
@@ -2,84 +2,65 @@
require 'sourced/types'
-# A superclass and registry to define event types
-# for example for an event-driven or event-sourced system.
-# All events have an "envelope" set of attributes,
-# including unique ID, stream_id, type, timestamp, causation ID,
-# event subclasses have a type string (ex. 'users.name.updated') and an optional payload
-# This class provides a `.define` method to create new event types with a type and optional payload struct,
-# a `.from` method to instantiate the correct subclass from a hash, ex. when deserializing from JSON or a web request.
-# and a `#follow` method to produce new events based on a previous event's envelope, where the #causation_id and #correlation_id
-# are set to the parent event
-# @example
-#
-# # Define event struct with type and payload
-# UserCreated = Message.define('users.created') do
-# attribute :name, Types::String
-# attribute :email, Types::Email
-# end
-#
-# # Instantiate a full event with .new
-# user_created = UserCreated.new(stream_id: 'user-1', payload: { name: 'Joe', email: '...' })
-#
-# # Use the `.from(Hash) => Message` factory to lookup event class by `type` and produce the right instance
-# user_created = Message.from(type: 'users.created', stream_id: 'user-1', payload: { name: 'Joe', email: '...' })
-#
-# # Use #follow(payload Hash) => Message to produce events following a command or parent event
-# create_user = CreateUser.new(...)
-# user_created = create_user.follow(UserCreated, name: 'Joe', email: '...')
-# user_created.causation_id == create_user.id
-# user_created.correlation_id == create_user.correlation_id
-# user_created.stream_id == create_user.stream_id
-#
-# ## Message registries
-# Each Message class has its own registry of sub-classes.
-# You can use the top-level Sourced::Message.from(hash) to instantiate all message types.
-# You can also scope the lookup by sub-class.
-#
-# @example
-#
-# class PublicCommand < Sourced::Message; end
-#
-# DoSomething = PublicCommand.define('commands.do_something')
-#
-# # Use .from scoped to PublicCommand subclass
-# # to ensure that only PublicCommand subclasses are accessible.
-# cmd = PublicCommand.from(type: 'commands.do_something', payload: { ... })
-#
-# ## JSON Schemas
-# Plumb data structs support `.to_json_schema`, so you can document all events in the registry with something like
-#
-# Message.subclasses.map(&:to_json_schema)
-#
module Sourced
- UnknownMessageError = Class.new(ArgumentError)
- PastMessageDateError = Class.new(ArgumentError)
-
+ # A query condition for reading messages from the store.
+ # Matches on (message_type AND all attrs key-value pairs).
+ # Multiple conditions are OR'd when passed to {Store#read}.
+ QueryCondition = Data.define(:message_type, :attrs)
+
+ # Returned by {Store#read} and {Store#claim_next} for optimistic concurrency.
+ # Pass to {Store#append} via +guard:+ to detect conflicting writes.
+ ConsistencyGuard = Data.define(:conditions, :last_position)
+
+ # Base message class. Messages have no stream_id or seq — they go into a
+ # flat, globally-ordered log.
+ #
+ # Supports +causation_id+ and +correlation_id+ for tracing causal chains.
+ #
+ # Define message types via {.define}:
+ #
+ # CourseCreated = Sourced::Message.define('course.created') do
+ # attribute :course_name, String
+ # end
+ #
class Message < Types::Data
+ EMPTY_ARRAY = [].freeze
+
attribute :id, Types::AutoUUID
- attribute :stream_id, Types::String.present
attribute :type, Types::String.present
- attribute :created_at, Types::Forms::Time.default { Time.now } #Types::JSON::AutoUTCTime
attribute? :causation_id, Types::UUID::V4
attribute? :correlation_id, Types::UUID::V4
- attribute :seq, Types::Integer.default(1)
+ attribute :created_at, Types::Forms::Time.default { Time.now }
attribute :metadata, Types::Hash.default(Plumb::BLANK_HASH)
attribute :payload, Types::Static[nil]
+ # Lookup table mapping type strings to message subclasses.
class Registry
+ # @param message_class [Class] the root message class for this registry
def initialize(message_class)
@message_class = message_class
@lookup = {}
end
+ # @return [Array] registered type strings
def keys = @lookup.keys
+
+ # @return [Array] direct subclasses of the root message class
def subclasses = message_class.subclasses
+ # Register a message class under a type string.
+ #
+ # @param key [String] message type string
+ # @param klass [Class] message subclass
def []=(key, klass)
@lookup[key] = klass
end
+ # Look up a message class by type string.
+ # Searches this registry first, then recurses into subclass registries.
+ #
+ # @param key [String] message type string
+ # @return [Class, nil]
def [](key)
klass = lookup[key]
return klass if klass
@@ -91,8 +72,15 @@ def [](key)
nil
end
- def inspect
- %(<#{self.class}:#{object_id} #{lookup.size} keys, #{subclasses.size} child registries>)
+ # All registered message classes across this registry and subclass registries.
+ #
+ # @return [Enumerator] if no block given
+ # @yield [Class] each registered message class
+ def all(&block)
+ return enum_for(:all) unless block
+
+ lookup.each_value(&block)
+ subclasses.each { |c| c.registry.all(&block) }
end
private
@@ -100,124 +88,166 @@ def inspect
attr_reader :lookup, :message_class
end
+ # @return [Registry] the message type registry for this class
def self.registry
@registry ||= Registry.new(self)
end
+ # Base class for typed message payloads.
class Payload < Types::Data
+ # @param key [Symbol] attribute name
+ # @return [Object] attribute value
def [](key) = attributes[key]
+
+ # @see Hash#fetch
def fetch(...) = to_h.fetch(...)
end
- def self.define(type_str, payload_schema: nil, &payload_block)
+ # Define a new message type. Registers it in the {.registry} and
+ # optionally defines a typed payload.
+ #
+ # @param type_str [String] unique message type identifier (e.g. 'course.created')
+ # @yield optional block to define payload attributes via +attribute+ DSL
+ # @return [Class] the new message subclass
+ #
+ # @example
+ # UserJoined = Sourced::Message.define('user.joined') do
+ # attribute :course_name, String
+ # attribute :user_id, String
+ # end
+ def self.define(type_str, &payload_block)
type_str.freeze unless type_str.frozen?
- if registry[type_str]
- Sourced.config.logger.warn("Message '#{type_str}' already defined")
- end
registry[type_str] = Class.new(self) do
def self.node_name = :data
define_singleton_method(:type) { type_str }
attribute :type, Types::Static[type_str]
- if payload_schema
- const_set(:Payload, Payload[payload_schema])
- attribute :payload, self::Payload
- elsif block_given?
- const_set(:Payload, Class.new(Payload, &payload_block))
- attribute :payload, self::Payload
+ if block_given?
+ payload_class = Class.new(Payload, &payload_block)
+ const_set(:Payload, payload_class)
+ attribute :payload, payload_class
+ names = payload_class._schema.to_h.keys.map(&:to_sym).freeze
+ define_singleton_method(:payload_attribute_names) { names }
end
end
end
+ # Instantiate the correct message subclass from a hash with a +:type+ key.
+ #
+ # @param attrs [Hash] must include +:type+ matching a registered type string
+ # @return [Message] instance of the appropriate subclass
+ # @raise [Sourced::UnknownMessageError] if the type string is not registered
def self.from(attrs)
klass = registry[attrs[:type]]
- raise UnknownMessageError, "Unknown event type: #{attrs[:type]}" unless klass
+ raise Sourced::UnknownMessageError, "Unknown message type: #{attrs[:type]}" unless klass
klass.new(attrs)
end
- def self.build(stream_id, payload = nil)
- attrs = {stream_id:}
- attrs[:payload] = payload if payload
- parse(attrs)
- end
-
def initialize(attrs = {})
- unless attrs[:payload]
- attrs = attrs.merge(payload: {})
- end
+ attrs = attrs.merge(payload: {}) unless attrs[:payload]
super(attrs)
end
- def with_metadata(meta = {})
- return self if meta.empty?
+ # Identity implementation of the +to_message+ contract — see
+ # {.===} and {Sourced::PositionedMessage#to_message}.
+ def to_message = self
- attrs = metadata.merge(meta)
- with(metadata: attrs)
- end
+ # Make +case/when+ transparent to {Sourced::PositionedMessage} (or any
+ # wrapper implementing +#to_message+). Ruby's default +Module#===+
+ # is implemented in C and ignores +is_a?+ overrides, so wrapped
+ # messages would otherwise fall through the +else+ branch.
+ def self.===(other)
+ return true if super
+ return false unless other.respond_to?(:to_message)
- def follow(event_class, payload_attrs = nil)
- follow_with_attributes(
- event_class,
- payload: payload_attrs
- )
+ unwrapped = other.to_message
+ !unwrapped.equal?(other) && super(unwrapped)
end
- def follow_with_seq(event_class, seq, payload_attrs = nil)
- follow_with_attributes(
- event_class,
- attrs: { seq: },
- payload: payload_attrs
- )
+ def with_metadata(meta = {})
+ return self if meta.empty?
+
+ with(metadata: metadata.merge(meta))
end
- def follow_with_stream_id(event_class, stream_id, payload_attrs = nil)
- follow_with_attributes(
- event_class,
- attrs: { stream_id: },
- payload: payload_attrs
- )
+ def with_payload(attrs = {})
+ hash = to_h
+ (hash[:payload] ||= {}).merge!(attrs)
+ self.class.new(hash)
end
- def follow_with_attributes(event_class, attrs: {}, payload: nil, metadata: nil)
- meta = self.metadata
- meta = meta.merge(metadata) if metadata
- attrs = { stream_id:, causation_id: id, correlation_id:, metadata: meta }.merge(attrs)
- attrs[:payload] = payload.to_h if payload
- event_class.parse(attrs)
+ def at(datetime)
+ if datetime < created_at
+ raise Sourced::PastMessageDateError, "Message #{type} can't be delayed to a date in the past"
+ end
+
+ with(created_at: datetime)
end
+ # Set causation and correlation IDs on another message, establishing
+ # a causal link from this message to +message+. Merges metadata.
+ #
+ # @param message [Message] the message to correlate
+ # @return [Message] a copy of +message+ with causation/correlation set
+ #
+ # @example
+ # caused = source_event.correlate(SomeCommand.new(payload: { ... }))
+ # caused.causation_id # => source_event.id
+ # caused.correlation_id # => source_event.correlation_id
def correlate(message)
attrs = {
causation_id: id,
- correlation_id:,
+ correlation_id: correlation_id,
metadata: metadata.merge(message.metadata || Plumb::BLANK_HASH)
}
message.with(attrs)
end
- # A copy of a message with a new stream_id
- # @param stream_id [String, #stream_id]
- # @return [Message]
- def to(stream_id)
- stream_id = stream_id.stream_id if stream_id.respond_to?(:stream_id)
- with(stream_id:)
+ # Returns the declared payload attribute names for this message class.
+ # Subclasses created via {.define} override this with a cached frozen array.
+ #
+ # @return [Array] attribute names (e.g. +[:course_name, :user_id]+)
+ def self.payload_attribute_names = EMPTY_ARRAY
+
+ # Build a {QueryCondition} for the intersection of this message's declared
+ # attributes and the given key-value pairs. Attributes not declared on this
+ # message class are silently ignored. Returns an array with a single condition
+ # containing all matching attrs, or an empty array if none match.
+ #
+ # @param attrs [Hash{Symbol => String}] partition attribute values
+ # @return [Array]
+ #
+ # @example
+ # CourseCreated.to_conditions(course_name: 'Algebra', user_id: 'joe')
+ # # => [QueryCondition('course.created', { course_name: 'Algebra' })]
+ # # user_id ignored — CourseCreated doesn't declare it
+ def self.to_conditions(**attrs)
+ supported = payload_attribute_names
+ matched = attrs.select { |key, _| supported.include?(key) }
+ .transform_values(&:to_s)
+ return [] if matched.empty?
+
+ [QueryCondition.new(message_type: type, attrs: matched)]
end
- def at(datetime)
- if datetime < created_at
- raise PastMessageDateError, "Message #{type} can't be delayed to a date in the past"
- end
- with(created_at: datetime)
- end
+ # Auto-extract key-value pairs from all top-level payload attributes.
+ # Used by {Store#append} to index messages for querying.
+ #
+ # @return [Array] pairs of [name, value], skipping nils
+ def extracted_keys
+ return [] unless payload
- def to_json(*)
- to_h.to_json(*)
+ payload.to_h.filter_map { |k, v|
+ [k.to_s, v.to_s] unless v.nil?
+ }
end
private
+ # Hook called by Plumb after schema parsing, when +:id+ has been resolved.
+ # Defaults +causation_id+ and +correlation_id+ to the message's own +id+.
def prepare_attributes(attrs)
attrs[:correlation_id] = attrs[:id] unless attrs[:correlation_id]
attrs[:causation_id] = attrs[:id] unless attrs[:causation_id]
diff --git a/lib/sourced/migrations/001_create_sourced_tables.rb.erb b/lib/sourced/migrations/001_create_sourced_tables.rb.erb
new file mode 100644
index 00000000..70ec29d4
--- /dev/null
+++ b/lib/sourced/migrations/001_create_sourced_tables.rb.erb
@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+ up do
+ create_table?(:<%= messages_table %>) do
+ primary_key :position
+ String :message_id, null: false, unique: true
+ String :message_type, null: false
+ String :causation_id
+ String :correlation_id
+ String :payload, null: false
+ String :metadata
+ String :created_at, null: false
+
+ index :message_type, name: 'idx_<%= prefix %>_ccc_message_type'
+ index :correlation_id, name: 'idx_<%= prefix %>_ccc_correlation_id'
+ end
+
+ create_table?(:<%= key_pairs_table %>) do
+ primary_key :id
+ String :name, null: false
+ String :value, null: false
+
+ index %i[name value], unique: true, name: 'idx_<%= prefix %>_ccc_key_pair_nv'
+ end
+
+ create_table?(:<%= message_key_pairs_table %>) do
+ foreign_key :message_position, :<%= messages_table %>, key: :position
+ foreign_key :key_pair_id, :<%= key_pairs_table %>
+ primary_key %i[message_position key_pair_id]
+
+ index %i[key_pair_id message_position], name: 'idx_<%= prefix %>_ccc_mkp_key'
+ end
+
+ create_table?(:<%= scheduled_messages_table %>) do
+ primary_key :id
+ String :created_at, null: false
+ String :available_at, null: false
+ String :message, null: false
+
+ index :available_at, name: 'idx_<%= prefix %>_ccc_scheduled_available_at'
+ end
+
+ create_table?(:<%= consumer_groups_table %>) do
+ primary_key :id
+ String :group_id, null: false, unique: true
+ String :status, null: false, default: 'active'
+ Integer :highest_position, null: false, default: 0
+ Integer :discovery_position, null: false, default: 0
+ Integer :last_nil_types_max_pos, null: false, default: 0
+ String :partition_by
+ String :error_context
+ String :retry_at
+ String :created_at, null: false
+ String :updated_at, null: false
+ end
+
+ create_table?(:<%= offsets_table %>) do
+ primary_key :id
+ foreign_key :consumer_group_id, :<%= consumer_groups_table %>, on_delete: :cascade
+ String :partition_key, null: false
+ Integer :last_position, null: false, default: 0
+ Integer :claimed, null: false, default: 0
+ String :claimed_at
+ String :claimed_by
+
+ index %i[consumer_group_id partition_key], unique: true, name: 'idx_<%= prefix %>_ccc_offsets_cg_pk'
+ index %i[consumer_group_id claimed], name: 'idx_<%= prefix %>_ccc_offsets_cg_claimed'
+ end
+
+ create_table?(:<%= offset_key_pairs_table %>) do
+ foreign_key :offset_id, :<%= offsets_table %>, on_delete: :cascade
+ foreign_key :key_pair_id, :<%= key_pairs_table %>
+ primary_key %i[offset_id key_pair_id]
+ end
+
+ create_table?(:<%= workers_table %>) do
+ String :id, primary_key: true, null: false
+ String :last_seen, null: false
+ end
+ end
+
+ down do
+ drop_table?(:<%= offset_key_pairs_table %>)
+ drop_table?(:<%= offsets_table %>)
+ drop_table?(:<%= consumer_groups_table %>)
+ drop_table?(:<%= scheduled_messages_table %>)
+ drop_table?(:<%= message_key_pairs_table %>)
+ drop_table?(:<%= key_pairs_table %>)
+ drop_table?(:<%= messages_table %>)
+ drop_table?(:<%= workers_table %>)
+ end
+end
diff --git a/lib/sourced/projector.rb b/lib/sourced/projector.rb
index a8f329f1..07214e4f 100644
--- a/lib/sourced/projector.rb
+++ b/lib/sourced/projector.rb
@@ -1,253 +1,84 @@
# frozen_string_literal: true
module Sourced
- # Projectors react to events
- # and update views of current state somewhere (a DB, files, etc)
+ # Reactor base class for Sourced read-model projectors.
class Projector
- include React
- include Evolve
- include Sync
- extend Consumer
-
- REACTION_WITH_STATE_PREFIX = 'reaction_with_state'
- BLANK_ARRAY = [].freeze
+ include Sourced::Evolve
+ include Sourced::React
+ include Sourced::Sync
+ extend Sourced::Consumer
class << self
- # The Reactor interface
- def handled_messages = handled_messages_for_evolve
-
- # Override this check defined in React mixin
- private def __validate_message_for_reaction!(event_class)
- if event_class && !handled_messages_for_evolve.include?(event_class)
- raise ArgumentError, '.reaction only works with event types handled by this class via .event(event_type)'
- end
- end
-
- # The Ractor Interface
- # @param message [Sourced::Message]
- # @option replaying [Boolean]
- # @option history [Enumerable]
- # @return [Array]
- def handle(message, replaying:, history: BLANK_ARRAY)
- new(id: identity_from(message)).handle(message, replaying:, history:)
- end
-
- # Override this in subclasses
- # to make an actor take its @id from an arbitrary
- # field in the message
+ # Projectors claim events they evolve from + events they react to.
#
- # @param message [Sourced::Message]
- # @return [Object]
- def identity_from(message) = message.stream_id
+ # @return [Array] evolved and reacted-to message classes
+ def handled_messages
+ (handled_messages_for_evolve + handled_messages_for_react).uniq
+ end
private
- # Shared batch finalization for both StateStored and EventSourced projectors.
- # Runs reaction handlers first (which may issue 3rd party requests),
- # then builds deferred sync actions last (executed in backend DB transaction).
- #
- # Unlike Actors (which create a new instance per message), Projectors use an
- # evolve-all-sync-once optimization: all batch messages are evolved into a
- # single instance's state before reactions run. This means on partial failure,
- # the sync must be rebuilt for only the successfully processed messages.
- # The block is called with the partial message list to produce correct sync_actions
- # (e.g. StateStored rebuilds a fresh instance with partial evolve).
- #
- # @param instance [Projector] the projector instance (already evolved)
- # @param batch [Array<[Message, Boolean]>] original batch pairs
- # @param messages [Array] extracted messages from batch
- # @param all_replaying [Boolean] whether all messages in the batch are replaying
- # @yield [partial_messages] called on reaction error to build partial sync_actions
- # @yieldparam partial_messages [Array] messages successfully processed
- # @yieldreturn [Array] sync actions for partial state
- # @return [Array<[actions, source_message]>] action pairs
- def sync_and_react(instance, batch, messages, all_replaying, &on_partial_sync)
- reaction_pairs = if all_replaying
- BLANK_ARRAY
+ def build_action_pairs(instance, messages, replaying:)
+ sync_actions = instance.collect_actions(
+ state: instance.state, messages:, replaying:
+ )
+
+ reaction_pairs = if replaying
+ []
else
- each_with_partial_ack(batch) do |msg, replaying|
- next if replaying
+ each_with_partial_ack(messages) do |msg|
next unless instance.reacts_to?(msg)
-
- reaction_cmds = instance.react(msg)
- reaction_cmds.any? ? [Actions.build_for(reaction_cmds), msg] : nil
+ reaction_msgs = Array(instance.react(msg))
+ actions = Actions.build_for(reaction_msgs)
+ actions.any? ? [actions, msg] : nil
end
end
- # All reactions succeeded. Build deferred sync for all messages (runs last in backend transaction).
- sync_actions = instance.sync_actions_with(
- state: instance.state, events: messages, replaying: all_replaying
- )
-
reaction_pairs + [[sync_actions, messages.last]]
- rescue PartialBatchError => e
- # Augment partial reaction results with sync for the successfully processed messages.
- fail_idx = messages.index { |m| m.id == e.failed_message.id }
- partial_messages = messages[0...fail_idx]
- if on_partial_sync && partial_messages.any?
- sync_actions = on_partial_sync.call(partial_messages)
- e.action_pairs << [sync_actions, partial_messages.last] if sync_actions.any?
- end
- raise
end
end
- attr_reader :id, :seq
-
- def initialize(id:, logger: Sourced.config.logger)
- @id = id
- @seq = 0
- @logger = logger
- end
+ attr_reader :partition_values
- def inspect
- %(<#{self.class} id:#{id} seq:#{seq}>)
+ # @param partition_values [Hash{Symbol => String}] partition key-value pairs
+ def initialize(partition_values = {})
+ @partition_values = partition_values
end
- def handle(message, replaying:)
- raise NotImplementedError, 'implement me in subclasses'
- end
-
- private
-
- attr_reader :logger
-
- # Override Evolve#__update_on_evolve
- def __update_on_evolve(event)
- @seq = event.seq
- end
-
- # A StateStored projector fetches initial state from
- # storage somewhere (DB, files, API)
- # And then after reacting to events and updating state,
- # it can save it back to the same or different storage.
- # @example
- #
- # class CartListings < Sourced::Projector::StateStored
- # # Fetch listing record from DB, or new one.
- # state do |id|
- # CartListing.find_or_initialize(id)
- # end
- #
- # # Evolve listing record from events
- # evolve Carts::ItemAdded do |listing, event|
- # listing.total += event.payload.price
- # end
- #
- # # Sync listing record back to DB
- # sync do |state:, events:, replaying:|
- # state.save!
- # end
- # end
+ # Projector variant that evolves only the claimed messages on top of stored state.
class StateStored < self
class << self
- # State-stored version doesn't load :history
- def handle(message, replaying: false)
- new(id: identity_from(message)).handle(message, replaying:)
- end
-
- # Optimized batch processing: one state load, one sync for entire batch.
- # On partial failure, rebuilds a fresh instance with only the successfully
- # processed messages so the sync persists correct partial state.
- # @param batch [Array<[Message, Boolean]>] array of [message, replaying] pairs
- # @return [Array<[actions, source_message]>] action pairs
- def handle_batch(batch)
- first_msg, _ = batch.first
- instance = new(id: identity_from(first_msg))
- messages = batch.map(&:first)
- all_replaying = batch.all? { |_, r| r }
-
- instance.state
- instance.evolve(messages)
- sync_and_react(instance, batch, messages, all_replaying) do |partial_messages|
- partial = new(id: identity_from(first_msg))
- partial.state
- partial.evolve(partial_messages)
- partial.sync_actions_with(state: partial.state, events: partial_messages, replaying: all_replaying)
- end
+ def handle_batch(partition_values, new_messages, history: nil, replaying: false)
+ instance = new(partition_values)
+ instance.evolve(new_messages)
+ build_action_pairs(instance, new_messages, replaying:)
end
- end
-
- def handle(message, replaying:)
- # Load state from storage
- state
- # Evolve new message
- evolve(message)
- # Collect sync actions
- actions = sync_actions_with(state:, events: [message], replaying:)
- # Replaying? Just return sync action
- return actions if replaying
- # Not replaying. Also run reactions
- if reacts_to?(message)
- actions += Actions.build_for(react(message))
+ # @param claim [ClaimResult] claimed partition batch
+ # @return [Array, PositionedMessage)>] action/source pairs
+ def handle_claim(claim)
+ values = partition_keys.to_h { |k| [k, claim.partition_value[k.to_s]] }
+ handle_batch(values, claim.messages, replaying: claim.replaying)
end
-
- actions
end
end
- # An EventSourced projector fetches initial state from
- # past events in the event store.
- # And then after reacting to events and updating state,
- # it can save it to a DB table, a file, etc.
- # @example
- #
- # class CartListings < Sourced::Projector::EventSourced
- # # Initial in-memory state
- # state do |id|
- # { id:, total: 0 }
- # end
- #
- # # Evolve listing record from events
- # evolve Carts::ItemAdded do |listing, event|
- # listing[:total] += event.payload.price
- # end
- #
- # # Sync listing record to a file
- # sync do |state:, events:, replaying:|
- # File.write("/listings/#{state[:id]}.json", JSON.dump(state))
- # end
- # end
+ # Projector variant that rebuilds state from full history each time.
class EventSourced < self
- BLANK_HISTORY = [].freeze
-
class << self
- # Optimized batch processing: one history evolve, one sync for entire batch.
- # On partial failure, reuses the already-evolved instance for sync since
- # EventSourced state is rebuilt from history on every invocation (idempotent).
- # @param batch [Array<[Message, Boolean]>] array of [message, replaying] pairs
- # @param history [Array] full stream history
- # @return [Array<[actions, source_message]>] action pairs
- def handle_batch(batch, history: BLANK_HISTORY)
- first_msg, _ = batch.first
- instance = new(id: identity_from(first_msg))
- messages = batch.map(&:first)
- all_replaying = batch.all? { |_, r| r }
-
- instance.evolve(history)
- sync_and_react(instance, batch, messages, all_replaying) do |partial_messages|
- # EventSourced state is rebuilt from history, so sync is idempotent.
- # Reuse the already-evolved instance but sync only partial_messages.
- instance.sync_actions_with(state: instance.state, events: partial_messages, replaying: all_replaying)
- end
+ def handle_batch(partition_values, new_messages, history:, replaying: false)
+ instance = new(partition_values)
+ instance.evolve(history.messages)
+ build_action_pairs(instance, new_messages, replaying:)
end
- end
- def handle(message, replaying:, history:)
- # Evolve new message from history
- evolve(history)
- # Collect sync actions
- actions = sync_actions_with(state:, events: [message], replaying:)
- # Replaying? Just return sync action
- return actions if replaying
-
- # Not replaying. Also run reactions
- if reacts_to?(message)
- actions += Actions.build_for(react(message))
+ # @param claim [ClaimResult] claimed partition batch
+ # @param history [ReadResult] full partition history
+ # @return [Array, PositionedMessage)>] action/source pairs
+ def handle_claim(claim, history:)
+ values = partition_keys.to_h { |k| [k, claim.partition_value[k.to_s]] }
+ handle_batch(values, claim.messages, history:, replaying: claim.replaying)
end
-
- actions
end
end
end
diff --git a/lib/sourced/pubsub/pg.rb b/lib/sourced/pubsub/pg.rb
deleted file mode 100644
index ea23b6b5..00000000
--- a/lib/sourced/pubsub/pg.rb
+++ /dev/null
@@ -1,253 +0,0 @@
-# frozen_string_literal: true
-
-require 'sequel'
-require 'thread'
-require 'json'
-
-module Sourced
- module PubSub
- # a PubSub implementation using Postgres' LISTEN/NOTIFY
- #
- # This class provides a publish-subscribe mechanism using Postgres LISTEN/NOTIFY,
- # with connection pooling - each channel name reuses a single database
- # listener per process, allowing multiple subscribers to share the same listener thread / fiber.
- # Relies on Sourced's configured Executor to use Threads or Fibers for concurrency
- #
- # @example Publishing a message
- # pubsub = Sourced.config.pubsub
- # event = MyEvent.new(stream_id: '111', payload: 'hello')
- # pubsub.publish('my_channel', event)
- #
- # @example Subscribing to messages
- # pubsub = Sourced.config.pubsub
- # channel = pubsub.subscribe('my_channel')
- # channel.start do |event, ch|
- # case event
- # when MyEvent
- # puts "Received: #{event}"
- # end
- # end
- #
- # @example Multiple subscribers to the same channel
- # # Both threads listen to the same channel but receive all messages
- # Thread.new do
- # ch1 = pubsub.subscribe('events')
- # ch1.start { |event| puts "Subscriber 1: #{event}" }
- # end
- #
- # Thread.new do
- # ch2 = pubsub.subscribe('events')
- # ch2.start { |event| puts "Subscriber 2: #{event}" }
- # end
- #
- class PG
- # Listener holds a single DB connection per channel name/process
- # Channel instances with the same name instantiated in different threads/fibers but the same process
- # share a Listener instance
- # The Listener receives messages and dispatches them to all channels.
- class Listener
- # @param db [Sequel::Database] the database connection for listening
- # @param channel_name [String] the name of the Postgres channel to listen on
- # @param timeout [Integer] the timeout in seconds for listen operations (default: 2)
- # @param logger [Logger]
- def initialize(db:, channel_name:, timeout: 2, logger:)
- @db = db
- @channel_name = channel_name
- @logger = logger
- @channels = {}
- @timeout = timeout
- @queue = Sourced.config.executor.new_queue
- @running = true
- @info = "[#{[self.class.name, @channel_name, Process.pid, object_id].join(' ')}]"
- end
-
- # Subscribe a channel to this listener
- #
- # @param channel [Channel] the channel to subscribe
- # @return [self]
- def subscribe(channel)
- start
- @queue << [:subscribe, channel]
- self
- end
-
- # Unsubscribe a channel from this listener
- #
- # @param channel [Channel] the channel to unsubscribe
- # @return [self]
- def unsubscribe(channel)
- @queue << [:unsubscribe, channel]
- self
- end
-
- # Start the listener threads for database listening and message dispatching
- #
- # @return [self]
- def start
- return if @control
-
- @control = Sourced.config.executor.start(wait: false) do |t|
- t.spawn do
- while (msg = @queue.pop)
- case msg
- in :stop
- @running = false
- # Stop all channels?
- @logger.info "#{@info} stopping"
- @channels.values.each(&:stop)
- in [:unsubscribe, channel]
- if @channels.delete(channel.object_id)
- @logger.info { "#{@info} unsubscribe channel #{channel.object_id}" }
- if @channels.empty?
- @logger.info { "#{@info} all channels unsubscribed." }
- end
- end
- in [:subscribe, channel]
- @logger.info { "#{@info} subscribe channel #{channel.object_id}" }
- @channels[channel.object_id] ||= channel
- in [:dispatch, message]
- @channels.values.each { |ch| ch << message }
- end
- end
-
- @logger.info "#{@info} Stopped"
- end
-
- t.spawn do
- @db.listen(@channel_name, timeout: @timeout, loop: true) do |_channel, _pid, payload|
- break unless @running
-
- message = parse(payload)
- @queue << [:dispatch, message]
- end
- end
- end
-
- @logger.info { "#{@info} Started" }
- end
-
- # Stop the listener threads and unsubscribe all channels
- #
- # @return [self]
- def stop
- @queue << :stop
- @control&.wait
- @control = nil
- @logger.info "#{@info} Stopped"
- self
- end
-
- private def parse(payload)
- data = JSON.parse(payload, symbolize_names: true)
- Sourced::Message.from(data)
- end
- end
-
- # Initialize a new PubSub instance with database and logger
- #
- # @param db [Sequel::Database] the database connection for publishing and listening
- # @param logger [Logger] the logger instance for recording events
- def initialize(db:, logger:)
- @db = db
- @logger = logger
- @mutex = Mutex.new
- @listeners ||= {}
- end
-
- # Subscribe to messages on a channel
- #
- # Creates or reuses a listener for the given channel name and returns a new channel object
- # that can be used to receive messages. Multiple subscribers to the same channel share a
- # single database listener per process.
- #
- # @param channel_name [String] the name of the channel to subscribe to
- # @return [Channel] a new channel object for receiving messages
- def subscribe(channel_name)
- @mutex.synchronize do
- listener = @listeners[channel_name] ||= Listener.new(db: @db, channel_name:, logger: @logger)
- ch = Channel.new(name: channel_name, listener:)
- listener.subscribe(ch)
- ch
- end
- end
-
- # Publish a message to a channel
- #
- # Sends an event to all subscribers on the given channel via Postgres NOTIFY.
- #
- # @param channel_name [String] the name of the channel to publish to
- # @param event [Sourced::Message] the message to publish
- # @return [self]
- def publish(channel_name, event)
- event_data = JSON.dump(event.to_h)
- @db.run(Sequel.lit('SELECT pg_notify(?, ?)', channel_name, event_data))
- self
- end
- end
-
- class Channel
- NOTIFY_CHANNEL = 'sourced-scheduler-ch'
-
- attr_reader :name
-
- # Initialize a new channel for receiving messages
- #
- # @param name [String] the name of the channel (default: NOTIFY_CHANNEL)
- # @param listener [Listener] the listener managing this channel
- def initialize(name: NOTIFY_CHANNEL, listener:)
- @name = name
- @running = false
- @listener = listener
- @queue = Sourced.config.executor.new_queue
- end
-
- # Add a message to this channel's queue
- #
- # @param message [Sourced::Message] the message to queue
- # @return [self]
- def <<(message)
- @queue << message
- self
- end
-
- # Start listening to incoming events on this channel
- #
- # Blocks until the channel is stopped. Messages are passed to either the provided
- # handler or the given block.
- #
- # @param handler [#call, nil] a callable object to use as an event handler
- # @yieldparam message [Sourced::Message] the incoming message
- # @yieldparam channel [Channel] this channel instance
- # @return [self]
- def start(handler: nil, &block)
- return self if @running
-
- @running = true
-
- handler ||= block
-
- while (msg = @queue.pop)
- handler.call(msg, self)
- end
-
- @running = false
-
- self
- end
-
- # Stop listening on this channel
- #
- # Unsubscribes from the listener and marks the channel to stop processing messages.
- # The stop takes effect on the next iteration of the message loop.
- #
- # @return [self]
- def stop
- return self unless @running
-
- @listener.unsubscribe self
- @queue << nil
- self
- end
- end
- end
-end
diff --git a/lib/sourced/pubsub/test.rb b/lib/sourced/pubsub/test.rb
deleted file mode 100644
index 8a6a48d5..00000000
--- a/lib/sourced/pubsub/test.rb
+++ /dev/null
@@ -1,86 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module PubSub
- # An in-memory pubsub implementation for testing and development.
- # Thread/fiber-safe. Each subscriber gets its own queue,
- # so a slow consumer won't block other subscribers.
- class Test
- def initialize
- @mutex = Mutex.new
- @subscribers = {}
- end
-
- # @param channel_name [String]
- # @return [Channel]
- def subscribe(channel_name)
- channel = Channel.new(name: channel_name, pubsub: self)
- @mutex.synchronize do
- @subscribers[channel_name] = (@subscribers[channel_name] || []) + [channel]
- end
- channel
- end
-
- # Remove a channel from the subscriber list.
- # @param channel [Channel]
- def unsubscribe(channel)
- @mutex.synchronize do
- arr = @subscribers[channel.name]
- @subscribers[channel.name] = arr - [channel] if arr
- end
- end
-
- # @param channel_name [String]
- # @param event [Sourced::Message]
- # @return [self]
- def publish(channel_name, event)
- channels = @subscribers[channel_name]
- return self unless channels
-
- channels.each { |ch| ch << event }
- self
- end
-
- class Channel
- attr_reader :name
-
- def initialize(name:, pubsub:)
- @name = name
- @pubsub = pubsub
- @queue = Sourced.config.executor.new_queue
- end
-
- # Push a message into this channel's queue.
- # @param message [Sourced::Message]
- # @return [self]
- def <<(message)
- @queue << message
- self
- end
-
- # Block and process messages from the queue.
- # @param handler [#call, nil]
- # @yieldparam message [Sourced::Message]
- # @yieldparam channel [Channel]
- # @return [self]
- def start(handler: nil, &block)
- handler ||= block
-
- while (msg = @queue.pop)
- handler.call(msg, self)
- end
-
- self
- end
-
- # Stop processing and unsubscribe from the pubsub.
- # @return [self]
- def stop
- @pubsub.unsubscribe(self)
- @queue << nil
- self
- end
- end
- end
- end
-end
diff --git a/lib/sourced/rails/install_generator.rb b/lib/sourced/rails/install_generator.rb
deleted file mode 100644
index 8f9a0ea3..00000000
--- a/lib/sourced/rails/install_generator.rb
+++ /dev/null
@@ -1,57 +0,0 @@
-# frozen_string_literal: true
-
-require 'rails/generators'
-require 'rails/generators/active_record'
-
-module Sourced
- module Rails
- class InstallGenerator < ::Rails::Generators::Base
- include ActiveRecord::Generators::Migration
-
- source_root File.expand_path('templates', __dir__)
-
- class_option :prefix, type: :string, default: 'sourced'
-
- def copy_initializer_file
- create_file 'config/initializers/sourced.rb' do
- <<~CONTENT
- # frozen_string_literal: true
-
- require 'sourced'
- require 'sourced/backends/active_record_backend'
-
- # This table prefix is used to generate the initial database migrations.
- # If you change the table prefix here,
- # make sure to migrate your database to the new table names.
- Sourced::Backends::ActiveRecordBackend.table_prefix = '#{table_prefix}'
-
- # Configure Sors to use the ActiveRecord backend
- Sourced.configure do |config|
- config.backend = Sourced::Backends::ActiveRecordBackend.new
- config.logger = Rails.logger
- end
- CONTENT
- end
- end
-
- def copy_bin_file
- copy_file 'bin_sourced', 'bin/sourced'
- chmod 'bin/sourced', 0o755
- end
-
- def create_migration_file
- migration_template 'create_sourced_tables.rb.erb', File.join(db_migrate_path, 'create_sourced_tables.rb')
- end
-
- private
-
- def migration_version
- "[#{ActiveRecord::VERSION::STRING.to_f}]"
- end
-
- def table_prefix
- options['prefix']
- end
- end
- end
-end
diff --git a/lib/sourced/rails/railtie.rb b/lib/sourced/rails/railtie.rb
deleted file mode 100644
index 4eb069d7..00000000
--- a/lib/sourced/rails/railtie.rb
+++ /dev/null
@@ -1,16 +0,0 @@
-# frozen_string_literal: true
-
-module Sourced
- module Rails
- class Railtie < ::Rails::Railtie
- # TODO: review this.
- # Workers use Async, so this is needed
- # but not sure this can be safely used with non Async servers like Puma.
- # config.active_support.isolation_level = :fiber
-
- generators do
- require 'sourced/rails/install_generator'
- end
- end
- end
-end
diff --git a/lib/sourced/rails/templates/bin_sors b/lib/sourced/rails/templates/bin_sors
deleted file mode 100644
index 4400e39b..00000000
--- a/lib/sourced/rails/templates/bin_sors
+++ /dev/null
@@ -1,8 +0,0 @@
-#!/usr/bin/env ruby
-
-require_relative "../config/environment"
-require "sourced"
-
-ActiveRecord::Base.logger = nil
-
-Sourced::Supervisor.start
diff --git a/lib/sourced/rails/templates/create_sors_tables.rb.erb b/lib/sourced/rails/templates/create_sors_tables.rb.erb
deleted file mode 100644
index 211ccdb7..00000000
--- a/lib/sourced/rails/templates/create_sors_tables.rb.erb
+++ /dev/null
@@ -1,55 +0,0 @@
-# frozen_string_literal: true
-
-class CreateSorsTables < ActiveRecord::Migration<%= migration_version %>
- def change
- # Uncomment for Postgres v12 or earlier to enable gen_random_uuid() support
- # enable_extension 'pgcrypto'
-
- if connection.class.name == 'ActiveRecord::ConnectionAdapters::SQLite3Adapter'
- create_table :<%= table_prefix %>_events, id: false do |t|
- t.string :id, null: false, index: { unique: true }
- t.bigint :global_seq, primary_key: true
- t.bigint :seq
- t.string :stream_id, null: false, index: true
- t.string :type, null: false
- t.datetime :created_at
- t.string :producer
- t.string :causation_id, index: true
- t.string :correlation_id
- t.text :payload
- end
- else
- create_table :<%= table_prefix %>_events, id: :uuid do |t|
- t.bigserial :global_seq, index: true
- t.bigint :seq
- t.string :stream_id, null: false, index: true
- t.string :type, null: false
- t.datetime :created_at
- t.string :producer
- t.uuid :causation_id, index: true
- t.uuid :correlation_id
- t.jsonb :payload
- end
- end
-
- add_index :<%= table_prefix %>_events, %i[stream_id seq], unique: true
-
- create_table :<%= table_prefix %>_streams do |t|
- t.text :stream_id, null: false, index: { unique: true }
- t.boolean :locked, default: false, null: false
- end
-
- create_table :<%= table_prefix %>_commands do |t|
- t.string :stream_id, null: false
- if t.class.name == 'ActiveRecord::ConnectionAdapters::SQLite3::TableDefinition'
- t.text :data, null: false
- t.datetime :scheduled_at, null: false, default: -> { 'CURRENT_TIMESTAMP' }
- else
- t.jsonb :data, null: false
- t.datetime :scheduled_at, null: false, default: -> { 'NOW()' }
- end
- end
-
- add_foreign_key :<%= table_prefix %>_commands, :<%= table_prefix %>_streams, column: :stream_id, primary_key: :stream_id
- end
-end
diff --git a/lib/sourced/react.rb b/lib/sourced/react.rb
index 1e19cc76..72adb3fb 100644
--- a/lib/sourced/react.rb
+++ b/lib/sourced/react.rb
@@ -1,39 +1,13 @@
# frozen_string_literal: true
+require 'set'
+
module Sourced
- # This mixin provides a .react macro to register
- # message handlers for a class
- # These message handlers are "reactions", ie. they react to
- # messages by producing new commands which will initiate new Decider flows.
- # More here: https://ismaelcelis.com/posts/decide-evolve-react-pattern-in-ruby/#3-react
- #
- # Example:
- #
- # class Saga
- # include Sourced::React
- #
- # # Host class must implement a #state method
- # # which will be passed to reaction handlers
- # attr_reader :state
- #
- # def initialize(id:)
- # @state = { id: }
- # end
- #
- # # React to an event and return a new command.
- # # This command will be scheduled for processing by a Decider.
- # # Using Sourced::Event#follow copies over metadata from the event
- # # including causation and correlation IDs.
- # reaction SomethingHappened do |state, event|
- # event.follow(DoSomethingElse, field1: 'value1')
- # end
- # end
- #
- # saga = Saga.new(id: '123')
- # commands = saga.react([something_happened])
- #
+ # React mixin for reactors.
+ # Supports the same dispatch-based reaction DSL as Sourced::React,
+ # adapted to Sourced's stream-less messages.
module React
- PREFIX = 'reaction'
+ PREFIX = 'sourced_reaction'
EMPTY_ARRAY = [].freeze
def self.included(base)
@@ -41,51 +15,47 @@ def self.included(base)
base.extend ClassMethods
end
- # @param events [Array]
- # @return [Array]
- def react(events)
- __handling_reactions(Array(events)) do |event|
- method_name = Sourced.message_method_name(React::PREFIX, event.class.to_s)
+ # Run reaction handlers for one or more messages.
+ # Supports both explicit message returns and dispatch(...) calls.
+ def react(messages)
+ __handling_reactions(Array(messages)) do |message|
+ method_name = Sourced.message_method_name(PREFIX, message.class.to_s)
if respond_to?(method_name)
- Array(send(method_name, state, event)).compact
+ Array(send(method_name, state, message)).compact
else
EMPTY_ARRAY
end
end
end
- # TODO: O(1) lookup
def reacts_to?(message)
self.class.handled_messages_for_react.include?(message.class)
end
private
- def __handling_reactions(events, &)
- @__stream_dispatchers = []
- events.each do |event|
- @__event_for_reaction = event
- yield event
+ def __handling_reactions(messages)
+ messages.flat_map do |message|
+ @__reaction_dispatchers = []
+ @__message_for_reaction = message
+ explicit = Array(yield(message)).compact.reject { |value| value.is_a?(Dispatcher) }
+ dispatched = @__reaction_dispatchers.map(&:message)
+ explicit + dispatched
end
- cmds = @__stream_dispatchers.map(&:message)
- @__stream_dispatchers.clear
- cmds
+ ensure
+ @__reaction_dispatchers = []
+ @__message_for_reaction = nil
end
class Dispatcher
attr_reader :message
- def initialize(msg)
- @message = msg
+ def initialize(message)
+ @message = message
end
def inspect = %(<#{self.class} #{@message}>)
- def to(stream_id)
- @message = @message.to(stream_id)
- self
- end
-
def at(datetime)
@message = @message.at(datetime)
self
@@ -97,28 +67,47 @@ def with_metadata(attrs = {})
end
end
- def dispatch(command_class, payload = {})
- command_class = self.class[command_class] if command_class.is_a?(Symbol)
- cmd = @__event_for_reaction
- .follow(command_class, payload)
- .with_metadata(producer: self.class.consumer_info.group_id)
-
- dispatcher = Dispatcher.new(cmd)
- @__stream_dispatchers << dispatcher
+ # Queue a follow-up message from within a reaction block.
+ #
+ # The returned {Dispatcher} can be chained to delay the message
+ # or add metadata before it is appended.
+ #
+ # @param message_class [Class, Symbol] messages class, or a symbol
+ # resolved via .[]
+ # @param payload [Hash] message payload attributes
+ # @return [Dispatcher] chainable wrapper around the dispatched message
+ #
+ # @example Dispatch by class
+ # reaction StudentEnrolled do |_state, event|
+ # dispatch(NotifyStudent, student_id: event.payload.student_id)
+ # end
+ #
+ # @example Dispatch by symbol with delay and metadata
+ # reaction StudentEnrolled do |_state, event|
+ # dispatch(:notify_student, student_id: event.payload.student_id)
+ # .with_metadata(channel: 'email')
+ # .at(Time.now + 300)
+ # end
+ def dispatch(message_class, payload = {})
+ message_class = self.class[message_class] if message_class.is_a?(Symbol)
+ message = @__message_for_reaction
+ .correlate(message_class.new(payload: payload))
+ .with_metadata(producer: self.class.group_id)
+
+ dispatcher = Dispatcher.new(message)
+ @__reaction_dispatchers << dispatcher
dispatcher
end
module ClassMethods
def inherited(subclass)
super
- handled_messages_for_react.each do |evt_type|
- subclass.handled_messages_for_react << evt_type
+ handled_messages_for_react.each do |klass|
+ subclass.handled_messages_for_react << klass
+ end
+ catch_all_react_events.each do |klass|
+ subclass.catch_all_react_events << klass
end
- end
-
- # Override this with extend Sourced::Consumer
- def consumer_info
- Sourced::Consumer::ConsumerInfo.new(group_id: name)
end
def handled_messages_for_react
@@ -129,77 +118,47 @@ def catch_all_react_events
@catch_all_react_events ||= Set.new
end
- # Define a reaction to an event
- # @example
- # reaction SomethingHappened do |state, event|
- # stream = stream_for(event)
- # # stream = stream_for("new-stream-id")
- # stream.command DoSomethingElse
- # end
+ # Register a reaction handler for one or more messages types.
#
- # The host class is expected to define a #state method
- # These handlers will load the decider's state from past events, and yield the state and the event to the block.
- # @example
- # reaction SomethingHappened do |state, event|
- # if state[:count] % 3 == 0
- # steam_for(event).command DoSomething
- # end
- # end
+ # Accepts message classes, symbols resolved via .[],
+ # multiple arguments, or no arguments for a catch-all reaction across
+ # all evolve types without an explicit handler.
#
- # If no event class given, the handler is registered for all events
- # set to evolve in .handled_messaged_for_evolve, unless
- # specific reactions have already been registered for them
- # The host class is expected to support .handled_messaged_for_evolve
- # see Evolve mixin
- # @example
- # reaction do |state, event|
- # LOGGER.info state
+ # @example React to a specific event class
+ # reaction StudentEnrolled do |state, event|
+ # dispatch(NotifyStudent, student_id: event.payload.student_id)
# end
#
- # @overload reaction do |state, event|
- # @overload reaction(event_symbol) do |state, event|
- # @param event_symbol [Symbol] Symbolised message name
- # @overload reaction(event_class) do |state, event|
- # @param event_class [Class] Must be subclass of Sourced::Message
- # @overload reaction(*events) do |state, event|
- # @param *events [Array