feat: add durable metadata and single-node recovery

CHANGELOG.md

@@ -8,6 +8,14 @@ Notable changes to Loafer are documented here. This project follows
 
 ### Added
 
+- Versioned SQLite/PostgreSQL metadata migrations for immutable pipeline versions, runs, stages,
+  partitions, batches, checkpoints, events, artifacts, schedules, and transactional outbox rows.
+- Explicit run, stage, and batch state machines; idempotent run/schedule/cancel commands; leases,
+  monotonic fencing tokens, heartbeats, cooperative cancellation, and classified retries.
+- Filesystem and in-memory object-storage adapters behind a shared port for logs, documents,
+  generated artifacts, and replayable temporary batch output.
+- A separately runnable durable worker plus `loafer enqueue` and `loafer worker` commands, with
+  batch-artifact replay from the last committed source position after a worker crash.
 - A framework-independent application service with strict, JSON-roundtrippable contracts for run
   requests, execution plans, batch envelopes, events, snapshots, and results.
 - Runtime ports and local adapters for cancellation, checkpoints, secret resolution, event
@@ -23,6 +31,11 @@ Notable changes to Loafer are documented here. This project follows
 
 ### Changed
 
+- Scheduled callbacks now create durable idempotent run commands; pipeline execution happens only
+  in a separately started worker process.
+- Bounded durable runs stage each transformed batch as an immutable object and commit its metadata,
+  checkpoint, event, and outbox record under the active fencing token before attempt-local target
+  writes and final publication.
 - The CLI, scheduler, and legacy Python runner now share the same application boundary while core
   execution orchestration remains independent of client frameworks.
 - Durable application contracts now exclude credentials, connector instances, iterators, provider
@@ -44,6 +57,12 @@ Notable changes to Loafer are documented here. This project follows
 
 ### Known limitations
 
+- SQLite metadata is restricted to the embedded profile with one scheduler and one worker; it does
+  not advertise high availability or distributed claims. PostgreSQL is the authoritative platform
+  profile. The bundled object-storage adapter is local filesystem storage, not a distributed blob
+  store.
+- Durable checkpoint replay currently applies to declared row-local, single-partition runs with a
+  stable offset-ordered source. Materialized/global transforms still restart as whole runs.
 - MongoDB row-local runs remain rejected until a tested staging/merge protocol replaces direct
   partial batch effects. PostgreSQL append is intentionally at-least-once across an ambiguous
   target-commit/checkpoint gap; keyed upsert is the replay-safe merge mode.

PRODUCTION_READINESS.md

@@ -432,6 +432,25 @@ Exit gate:
 - migrations work from an empty database and the previous supported schema;
 - SQLite and PostgreSQL contract tests pass for the capabilities each profile advertises.
 
+**Current status:** implementation and single-node recovery verification complete for declared
+row-local, single-partition runs. Versioned migrations persist immutable pipeline versions, runs,
+stages, partitions, batches, events, artifacts, checkpoints, schedules, and outbox records behind
+one SQLAlchemy metadata interface. PostgreSQL uses row locks for concurrent claims and is the
+authoritative platform profile; SQLite is explicitly restricted to one scheduler and one worker
+without HA or NATS claims. Run, stage, and batch state machines reject impossible transitions,
+events allocate contiguous per-run sequences, run creation and schedule firing are idempotent, and
+every worker mutation requires its active lease and fencing token.
+
+The bounded data plane now stages transformed batches in object storage before atomically
+committing batch metadata, checkpoint, event, and outbox state. Worker-kill tests terminate after
+each of three batch commit boundaries and prove that a newly fenced worker replays committed
+artifacts, skips the durable source offset, and publishes the exact output. Empty-database,
+previous-schema upgrade, rollback/re-apply, constraint, cancellation, schedule, and stale-token
+contracts pass on SQLite; the migration, event, claim, and fencing contracts also pass against a
+live PostgreSQL 16 database. Recovery remains limited to stable offset-ordered, row-local,
+single-partition execution. Materialized/global transforms still restart as whole runs, and the
+bundled filesystem object adapter is not a distributed object store.
+
 ### Phase 4 — Build authentication, tenancy, and the control-plane API
 
 **Goal:** expose safe multi-tenant application use cases without running data work in HTTP

README.md

@@ -7,9 +7,10 @@ Define a source, transformation, and target; validate the pipeline; then run it
 or from a scheduler. Transformations can use SQL, custom Python, multi-step pipelines, or optional
 LLM-generated artifacts.
 
-> **Project status:** Loafer currently ships as a CLI engine with a local scheduler/daemon. The
-> multi-tenant API, distributed workers, web operations dashboard, and terminal dashboard are under
-> active development. The `/studio` web route is a product preview, not a connected control plane.
+> **Project status:** Loafer ships as a CLI engine with durable single-node scheduling and worker
+> recovery. The multi-tenant API, distributed queue/workers, web operations dashboard, and terminal
+> dashboard are under active development. The `/studio` web route is a product preview, not a
+> connected control plane.
 
 ## What works today
 
@@ -20,6 +21,8 @@ LLM-generated artifacts.
 - PostgreSQL and MongoDB upserts
 - Cursor-based incremental extraction with local state
 - Local scheduling, daemon management, run summaries, and logs
+- SQLite/PostgreSQL run metadata, fenced worker leases, durable batch checkpoints, replayable
+  temporary output, and monotonic run events
 - Optional Gemini, OpenAI, Claude, and Qwen providers
 - Resource-limited Python transform subprocesses on Linux and macOS
 - Declared row-local ETL with bounded batches, per-batch validation, schema policies,
@@ -247,6 +250,8 @@ implemented; `ocr_applied` remains `false` in provenance.
 
 ```text
 loafer run <pipeline.yaml>
+loafer enqueue <pipeline.yaml> --command-key <idempotency-key>
+loafer worker [--once]
 loafer validate <pipeline.yaml>
 loafer connectors
 loafer schedule <pipeline.yaml>
@@ -260,6 +265,11 @@ loafer init
 
 Use `loafer <command> --help` for command-specific options.
 
+`loafer schedule` and the scheduler daemon only enqueue durable run commands; start `loafer worker`
+as a separate process to execute them. The embedded profile defaults to SQLite under `~/.loafer`
+and supports one scheduler and one worker. Set `LOAFER_METADATA_URL` to a PostgreSQL URL for the
+authoritative platform store and `LOAFER_OBJECTS_PATH` to choose the local artifact root.
+
 ## Self-hosted platform direction
 
 The production architecture separates clients, control plane, and data plane:
@@ -275,9 +285,10 @@ The web dashboard and planned terminal dashboard will use the same API, permissi
 metrics, and logs. Workers will run independently so startups can deploy the stack on one host
 while larger installations can scale and isolate worker pools.
 
-The full stack is not shipped yet. Until the API, durable queue, tenant authorization, worker
-leases, and recovery tests exist, use the CLI/Docker path for bounded workloads and do not expose
-Studio as a production operations surface.
+The full stack is not shipped yet. Durable metadata, leases, fencing, outbox records, and
+single-node bounded-batch recovery are implemented; the authenticated API, distributed queue,
+tenant authorization, and distributed object store are not. Use the CLI/Docker path for bounded
+workloads and do not expose Studio as a production operations surface.
 
 The planned web source uses Crawlee for Python with HTTP/Parsel and Playwright execution profiles.
 It will support bounded crawling, authorized authenticated sessions, JavaScript rendering, and