diff --git a/docs/jobcoordinator.md b/docs/jobcoordinator.md new file mode 100644 index 00000000..cb24b8c2 --- /dev/null +++ b/docs/jobcoordinator.md @@ -0,0 +1,143 @@ +# Job Coordinator (database-backed) + +The job coordinator turns **any executable** into a distributed worker. It is a +database-backed sibling of the file-based grid wrapper: instead of +`.LOCKED/.PROCESSED/.FAILED` marker files on a shared filesystem, the queue and +its coordination state live in a database (SQLite or PostgreSQL). + +A **job** is a whole batch/run; a **task** is one unit of work within it +(typically a file path). Two commands make up the workflow: + +| Command | Purpose | +| --- | --- | +| `claimed propagate_jobs` | Populate a job with tasks from a glob/directory (status `pending`). | +| `claimed work_jobs` | Claim the job's tasks one-by-one and run a worker script per task. | + +Because the database is the single source of truth and claiming is atomic, many +workers — on one machine or across many nodes — cooperate safely and never +process the same task twice. + +## Concepts + +Each task is one row in the `claimed_jobs` table, identified by its **job** (the +batch namespace) plus its **task_name** (the file path emitted by +`propagate_jobs`). The job is a mandatory namespace: it lets many independent +batches share one database, and workers only ever claim tasks from the job they +were pointed at. Uniqueness is per `(job, task_name)`, so the same file path may +appear in more than one job. A task moves through: + +``` +pending ── claim ──▶ processing ── worker exit 0 ──▶ succeeded + └── worker exit ≠0 ──▶ failed +``` + +The table also records `worker_id`, `attempts`, timestamps, and (on failure) an +`error` message. + +## `propagate_jobs` + +```bash +claimed propagate_jobs --db --job +``` + +- `--db ` — a SQLite file path (e.g. `jobs.db`, `/tmp/jobs.db`) or a full + database URL. Bare paths and `*.db` / `*.sqlite` become SQLite; `sqlite:///…` + and `postgresql://…` (also the legacy `postgres://…`) are used as given. +- `--job ` — **mandatory** namespace for this batch. Use a distinct job + name per independent run so several runs can share one database. +- `` — a glob (with `**` recursion) or a directory. **Quote it** so the + shell does not expand the glob before CLAIMED sees it. + +The command is **idempotent**: re-running only inserts `(job, task_name)` pairs +that do not already exist (`INSERT … ON CONFLICT DO NOTHING`), so you can grow +the queue incrementally. + +```bash +claimed propagate_jobs --db /tmp/jobs.db --job run-2026 'data/**/*.tif' +# Inserted 42 tasks into job 'run-2026' (0 already existed). Total pending: 42 +``` + +## `work_jobs` + +```bash +claimed work_jobs --db --job --worker