feat: add durable metadata and single-node recovery

[lupppig]

Summary

• add versioned SQLite/PostgreSQL metadata for immutable pipeline versions, runs, stages,
  partitions, batches, events, artifacts, checkpoints, schedules, and transactional outbox records
• enforce run, stage, and batch state machines with idempotent commands, monotonic event sequences,
  leases, heartbeats, fencing tokens, cooperative cancellation, and retry categories
• separate scheduled command creation from worker execution with loafer enqueue and loafer worker
• stage bounded batch output in object storage and recover from the last durable checkpoint after
  worker termination
• update the changelog, readiness roadmap, README, and architecture reference

Why

The bounded data plane could publish atomically, but its run metadata and checkpoints were not
durable enough to observe or resume work after a process failure. This change establishes the
single-node recovery contract before distributed transport is introduced.

Recovery contract

Durable replay is advertised for stable offset-ordered, row-local, single-partition runs. SQLite
is restricted to one scheduler and one worker; PostgreSQL is the authoritative platform profile.
Materialized/global transforms still restart as whole runs.

Validation

• uv run pytest -q --tb=short: 752 passed, 56 skipped, 10 pre-existing stress-test warnings
• live PostgreSQL 16 metadata contracts: 2 passed
• worker-kill recovery after batch commit boundaries 1, 2, and 3
• uv run ruff check loafer tests
• pre-push Ruff lint and format checks

Summary by CodeRabbit

• New Features

  • Added durable single-node pipeline scheduling, run tracking, worker leases, checkpoints, events, cancellation, and crash recovery.
  • Added enqueue and worker CLI commands for separate scheduling and execution.
  • Added local filesystem and in-memory artifact storage.
  • Added idempotent commands, protected state transitions, and recovery of committed batches after worker interruption.

• Documentation

  • Updated the README, changelog, and architecture documentation with current capabilities and limitations.

• Tests

  • Added coverage for recovery, migrations, scheduling, storage, state transitions, fencing, and PostgreSQL integration.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b2aba589-38c0-4588-8ce4-1f2c05679af4

📥 Commits

Reviewing files that changed from the base of the PR and between 75e9448 and 17ed4b9.

📒 Files selected for processing (27)

• CHANGELOG.md
• PRODUCTION_READINESS.md
• README.md
• loafer/adapters/metadata.py
• loafer/adapters/metadata_schema.py
• loafer/adapters/object_storage.py
• loafer/adapters/runtime.py
• loafer/application/__init__.py
• loafer/application/durable.py
• loafer/application/service.py
• loafer/cli.py
• loafer/core/run_state.py
• loafer/data_plane.py
• loafer/engine.py
• loafer/exceptions.py
• loafer/metadata.py
• loafer/ports/metadata.py
• loafer/ports/object_storage.py
• loafer/scheduler.py
• loafer/worker.py
• skills/loafer-engineering/references/architecture.md
• tests/e2e/test_durable_recovery.py
• tests/integration/test_metadata_store.py
• tests/unit/test_metadata_store.py
• tests/unit/test_object_storage.py
• tests/unit/test_run_state.py
• tests/unit/test_scheduler.py

---

📝 Walkthrough

Walkthrough

The PR adds durable single-node execution with versioned metadata, fenced leases, idempotent scheduling, object-storage batch staging, worker processes, and checkpoint-based recovery after worker termination.

Changes

Durable execution

Layer / File(s)	Summary
Metadata contracts and state transitions loafer/metadata.py, loafer/ports/*, loafer/core/run_state.py, loafer/exceptions.py, loafer/adapters/metadata_schema.py, tests/unit/test_run_state.py	Defines immutable metadata records, durable protocols, lifecycle state machines, metadata errors, and versioned migrations.
Metadata and object-storage persistence loafer/adapters/metadata.py, loafer/adapters/object_storage.py, tests/unit/test_metadata_store.py, tests/integration/test_metadata_store.py, tests/unit/test_object_storage.py	Adds SQLAlchemy persistence for runs, leases, events, schedules, batches, checkpoints, and outbox records. Adds filesystem and memory object-storage adapters with checksum and path validation.
Enqueue, scheduling, and worker orchestration loafer/application/*, loafer/scheduler.py, loafer/cli.py, loafer/worker.py, README.md, CHANGELOG.md, PRODUCTION_READINESS.md, skills/loafer-engineering/references/architecture.md	Adds durable pipeline enqueueing, separate scheduler and worker commands, lease-based worker execution, retry and cancellation handling, and updated project documentation.
Durable batch recovery path loafer/adapters/runtime.py, loafer/data_plane.py, loafer/engine.py, loafer/application/service.py, tests/e2e/test_durable_recovery.py	Restores committed artifacts, skips recovered source rows, stages new batches before target publication, and persists checkpoints through the recovery port. The end-to-end test verifies recovery after simulated worker termination.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant MetadataStore
  participant Worker
  participant ObjectStorage
  participant Target
  Scheduler->>MetadataStore: enqueue idempotent run command
  Worker->>MetadataStore: claim run lease
  Worker->>ObjectStorage: stage batch artifact
  Worker->>MetadataStore: commit batch and checkpoint
  Worker->>Target: publish staged batch
  Worker->>MetadataStore: record events and final state

Loading

Possibly related PRs

• lupppig/loafer#17: Extends the application service with durable recovery, enqueueing, worker execution, and related CLI commands.
• lupppig/loafer#18: Provides the bounded-batch data-plane behavior extended by durable checkpoint recovery.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/durable-run-recovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}