Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ CORPUSKIT_API_BIND_HOST=127.0.0.1
# Prometheus metrics are served at /internal/metrics. Staging and production
# require a distinct 32-512 character opaque bearer token.
CORPUSKIT_METRICS_ENABLED=true
CORPUSKIT_METRICS_BEARER_TOKEN=
# CORPUSKIT_METRICS_BEARER_TOKEN=<32-512-character-opaque-token>
# Authenticated, tenant-scoped fixed-window limits are shared through PostgreSQL.
# Staging/production API processes require this to remain enabled.
CORPUSKIT_API_RATE_LIMIT_ENABLED=false
Expand Down Expand Up @@ -64,7 +64,7 @@ CORPUSKIT_TEMPORAL_NAMESPACE=default
CORPUSKIT_TEMPORAL_TASK_QUEUE=batch-cpu
# Staging/production require TLS. Set an API key only with TLS enabled.
CORPUSKIT_TEMPORAL_TLS=false
CORPUSKIT_TEMPORAL_API_KEY=
# CORPUSKIT_TEMPORAL_API_KEY=<provider-api-key>
CORPUSKIT_TEMPORAL_CONNECT_TIMEOUT_SECONDS=10
CORPUSKIT_TEMPORAL_ACTIVITY_HEARTBEAT_SECONDS=5
CORPUSKIT_TEMPORAL_MAX_CONCURRENT_ACTIVITIES=8
Expand Down Expand Up @@ -93,13 +93,13 @@ CORPUSKIT_WORKER_MODEL_CACHE_MOUNT_READ_ONLY=false
# CORPUSKIT_WORKER_DATG_INDEX_PUBLISH_ROOT=/datg-index-publish
CORPUSKIT_WORKER_DATG_CACHE_MOUNT_READ_ONLY=false
CORPUSKIT_WORKER_PHON_RL_CACHE_ROOTS={}
CORPUSKIT_DISPATCHER_ID=
# CORPUSKIT_DISPATCHER_ID=corpuskit-local-dispatcher
CORPUSKIT_DISPATCHER_BATCH_SIZE=20
CORPUSKIT_DISPATCHER_POLL_SECONDS=1

# Production OIDC. Required when CORPUSKIT_AUTH_MODE=oidc.
CORPUSKIT_OIDC_ISSUER=
CORPUSKIT_OIDC_AUDIENCE=
# CORPUSKIT_OIDC_ISSUER=https://identity.example.com/
# CORPUSKIT_OIDC_AUDIENCE=corpuskit-api
CORPUSKIT_OIDC_ALGORITHMS=["RS256"]
CORPUSKIT_OIDC_JWKS_CACHE_SECONDS=300
CORPUSKIT_OIDC_REFRESH_COOLDOWN_SECONDS=10
Expand All @@ -109,9 +109,9 @@ CORPUSKIT_OIDC_ORGANIZATION_CLAIM=org_id
CORPUSKIT_OIDC_ROLE_CLAIM=role

# Optional server-managed provider credentials. Never expose these to the web app.
CORPUSKIT_PROVIDER_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
# CORPUSKIT_PROVIDER_API_KEY=<provider-key>
# OPENAI_API_KEY=<provider-key>
# ANTHROPIC_API_KEY=<provider-key>

# Frontend server/BFF. This value is never exposed to browser JavaScript.
CORPUSKIT_API_INTERNAL_URL=http://127.0.0.1:8000/api/v1
Expand All @@ -130,8 +130,8 @@ CORPUSKIT_WEB_LOGIN_TRANSACTION_SECONDS=600
CORPUSKIT_WEB_OIDC_TIMEOUT_SECONDS=5
CORPUSKIT_WEB_REDIS_COMMAND_TIMEOUT_MS=2000
CORPUSKIT_WEB_REDIS_PREFIX=corpuskit:web
CORPUSKIT_WEB_SESSION_REDIS_URL=
CORPUSKIT_WEB_OIDC_CLIENT_ID=
CORPUSKIT_WEB_OIDC_CLIENT_SECRET=
CORPUSKIT_WEB_OIDC_REDIRECT_URI=
# CORPUSKIT_WEB_SESSION_REDIS_URL=rediss://sessions.example.com:6379/0
# CORPUSKIT_WEB_OIDC_CLIENT_ID=corpuskit-web
# CORPUSKIT_WEB_OIDC_CLIENT_SECRET=<confidential-client-secret>
# CORPUSKIT_WEB_OIDC_REDIRECT_URI=https://app.example.com/auth/callback
CORPUSKIT_WEB_OIDC_SCOPES=openid profile offline_access
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ All notable changes to CorpusKit will be documented here. The project follows

### Added

- A comprehensive Docker-first getting-started guide, copyable direct-development environment
examples, first-use workflow, lifecycle and troubleshooting commands, and an enforced local
startup documentation contract.
- Append-only corpus version creation from manual sentences or bounded UTF-8 TXT, CSV, and JSON
imports, with parent lineage, atomic quota accounting, audit evidence, API contracts, and an
accessible project-workbench flow.
Expand Down
64 changes: 48 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,29 @@ reproducible user workflows.
> The repository contains a hardened release-candidate pipeline, but no registry publication or
> production promotion is claimed until its external acceptance gates are recorded.

## Quick start

For an isolated local website, install Git and a running Docker Engine/Desktop with Docker
Compose v2, then run:

```text
git clone https://github.com/jemsbhai/corpuskit.git
cd corpuskit
docker compose --profile web up --build --detach --wait
```

Open <http://127.0.0.1:3000/projects>. No host Python, Node.js, npm, eSpeak NG, or `.env` file is
needed for this Docker path. The first run builds the locked API and web images, migrates
PostgreSQL, initializes private MinIO storage, and provisions the pinned PHOIBLE snapshot.

> **Do not publish this demo:** every browser receives the same local owner identity and the
> checked-in development secrets are deterministic. Keep it on loopback. Shared deployments
> require OIDC and the production topology.

The comprehensive [getting-started guide](docs/getting-started.md) covers verification, first
corpus creation, ports, logs, updates, persistence and reset, common failures, durable local jobs,
Windows/PowerShell commands, direct source development, and the production boundary.

## Product scope

CorpusKit is designed for TTS/ASR dataset builders, speech researchers, clinical
Expand Down Expand Up @@ -160,16 +183,19 @@ the [release](https://github.com/jemsbhai/corpuskit/blob/main/docs/operations/re

## Development prerequisites

- Python 3.12
The [getting-started guide](docs/getting-started.md) is the authoritative first-run path. Use the
Docker quick start unless you are changing application source.

- Python >=3.12,<3.13
- [`uv`](https://docs.astral.sh/uv/)
- Node.js 24 LTS and npm 11
- Node.js >=24.18.1,<25 and npm 11.16.0
- eSpeak NG for real G2P workflows
- Docker with Compose for the full local stack

Install locked dependencies after lockfiles have been generated:
The lockfiles are committed. Install them without re-resolving dependencies:

```bash
uv sync --all-groups
uv sync --frozen --all-groups
npm ci
```

Expand All @@ -182,17 +208,24 @@ uv run corpuskit-phoible provision --json
uv run corpuskit-phoible status --json
```

Run the API and web application:
For direct source development, copy the separate API and Next.js demo environment examples and
prepare the local database:

```bash
mkdir -p data
export CORPUSKIT_DATABASE_URL=sqlite+aiosqlite:///./data/corpuskit.db
uv run corpuskit-phoible provision
cp .env.example .env
cp apps/web/.env.example apps/web/.env.local
uv run corpuskit-phoible provision --json
uv run corpuskit-phoible status --json
CORPUSKIT_DATABASE_URL=sqlite+aiosqlite:///./data/corpuskit.db \
uv run corpuskit-db upgrade
uv run corpuskit-api
npm run dev
```

Then run `uv run corpuskit-api` and `npm run dev` in **two separate terminals** from the
repository root. PowerShell equivalents and eSpeak installation notes are in the
[getting-started guide](docs/getting-started.md). The root `.env` is loaded by Python; Next.js
loads `apps/web/.env.local` and does not consume the root file.

Every Python service also requires an exact process posture. `corpuskit-api` uses the default
`CORPUSKIT_RUNTIME_ROLE=api`; direct dispatcher, worker, and maintenance processes must set
`dispatcher`, `worker`, and `maintenance`, respectively. A mismatched entry point fails before
Expand All @@ -201,13 +234,12 @@ the complete role/command map. Direct API installs bind only to `127.0.0.1` by d
set `CORPUSKIT_API_BIND_HOST=0.0.0.0` explicitly behind the Compose loopback publication or the
Helm Service, ingress, and NetworkPolicy boundary.

PowerShell uses `$env:CORPUSKIT_DATABASE_URL =
"sqlite+aiosqlite:///./data/corpuskit.db"` after `New-Item -ItemType Directory -Force data`.
The migration CLI deliberately requires the database URL in the process environment; it
does not infer a production target. Run migrations explicitly before every direct API
startup after pulling schema changes. Docker Compose performs the same development-only
ordering through its one-shot `migrate` and `provision-phoible` services. The latter owns
write access to the shared PHOIBLE volume; API and worker consumers mount it read-only. See
The migration CLI deliberately requires an explicit database URL in the process environment; it
does not load `.env` or infer a production target. Run migrations before direct API startup after
pulling schema changes. The API itself loads the root `.env`. Docker Compose performs the same
development-only ordering through its one-shot `migrate` and `provision-phoible` services. The
latter owns write access to the shared PHOIBLE volume; API and worker consumers mount it
read-only. See
the [PHOIBLE provisioning runbook](https://github.com/jemsbhai/corpuskit/blob/main/docs/operations/phoible-provisioning.md) for air-gapped
installation, verification, recovery, and release-job guidance.

Expand Down
9 changes: 9 additions & 0 deletions apps/web/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Development-only settings for `npm run dev` from the CorpusKit repository root.
# Copy this file to apps/web/.env.local. Never reuse these deterministic values in
# a shared, staging, or production deployment.
CORPUSKIT_ENVIRONMENT=development
CORPUSKIT_API_INTERNAL_URL=http://127.0.0.1:8000/api/v1
CORPUSKIT_WEB_AUTH_MODE=demo
CORPUSKIT_WEB_STATE_SECRET=AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE
CORPUSKIT_WEB_SESSION_ENCRYPTION_KEYS=["AgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI"]
CORPUSKIT_WEB_ALLOWED_RETURN_PATHS=["/","/projects","/evaluate","/analysis","/capabilities","/g2p","/inventory","/coverage","/selection","/generation","/advanced","/jobs","/artifacts"]
17 changes: 14 additions & 3 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,19 @@ services:
entrypoint: ["/bin/sh", "-ec"]
command:
- >-
mc alias set local http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}";
mc mb --ignore-existing "local/$${MINIO_BUCKET}";
mc anonymous set none "local/$${MINIO_BUCKET}"
attempt=0;
until
mc alias set local http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}" &&
mc mb --ignore-existing "local/$${MINIO_BUCKET}" &&
mc anonymous set none "local/$${MINIO_BUCKET}";
do
attempt=$$((attempt + 1));
if [ "$${attempt}" -ge 20 ]; then
echo "MinIO initialization failed after $${attempt} attempts" >&2;
exit 1;
fi;
sleep 1;
done
environment:
MC_CONFIG_DIR: /tmp/.mc
MINIO_ROOT_USER: ${CORPUSKIT_MINIO_ROOT_USER:-corpuskit-local}
Expand Down Expand Up @@ -280,6 +290,7 @@ services:
- "127.0.0.1:${CORPUSKIT_WEB_PORT:-3000}:3000"
tmpfs:
- /tmp:rw,noexec,nosuid,size=64m
- /app/apps/web/.next/cache:rw,noexec,nosuid,size=64m,uid=1000,gid=1000,mode=0700
cap_drop:
- ALL
security_opt:
Expand Down
Loading
Loading