CICD Blueprint — self-hosted GitLab in k8s + CI/CD, end to end

📖 Full documentation site: https://bruj0.github.io/k8s-cicd-gitlab/

Read this first — the diagrams explain how the pieces fit together far better than the prose in this README.

The site contains:

C4 architecture docs — system context, container, and component diagrams for the bootstrap package, Phase 2 GitLab install pipeline, Phase 3 app onboarding pipeline, and the OpenTofu IaC layer
Per-phase runbooks — step-by-step instructions for Phase 1 (cluster prep), Phase 2 (GitLab stack install), Phase 3 (app onboarding), and the secrets inventory
Workflow overview — sequence diagrams for the control plane and data plane, showing what happens during a normal install vs. a destroy/apply cycle

What is this?

One repo, one machine, agentic driven. A 5-node kind cluster hosts a self-managed GitLab (chart-bundled Envoy Gateway terminates *.local.example.net), a registered Runner, and OpenBao for secret injection. Push a commit, get a running workload on https://<app>.local.example.net.

This repo implements the GitLab + Gilab Runner + OpenBao + CI/CD templates provisioned in a local Kubernetes cluster with OpenTofu. The templates build a GitLab CI pipeline that builds, test and deploys apps to the self-hosted Gitlab.

End-to-end narrative

flowchart TD
  dev["Developer<br/>(this repo)"] -->|git push| gl["Self-hosted GitLab<br/>gitlab.local.example.net"]
  gl -->|webhook| ci["GitLab CI runner job"]
  ci -->|tofu validate| ok1["IaC OK"]
  ci -->|helm lint / template| ok2["Helm OK"]
  ci -->|helm install --set image.tag=$CI_COMMIT_SHA| k8s["kind cluster (5 nodes)"]
  k8s -->|apps-https listener<br/>+ HTTPRoute| app["https://app.apps.local.example.net"]
  gl -. secret ref .-> openbao["OpenBao"]
  ci -. read at job time .-> openbao

How the blueprint implements this:

Phase 1 — Cluster. OpenTofu creates a 5-node kind cluster, with a shared hostPath mount under infra/data/shared/ (chart-managed PVCs from Phase 2 land here too).
Phase 2 — Stack. Bootstrap installs Gateway API CRDs, OpenBao, GitLab CE (which sub-installs Envoy Gateway and mints a self-signed wildcard cert for *.local.example.net via a cfssl Job), and a Runner registered against the cluster-internal GitLab Service URL.
Phase 3 — App. A sample app (the guestbook Helm chart in apps/guestbook/) is built and pushed to the in-cluster registry. A .gitlab-ci.yml validates the IaC + Helm on every push; on main, it deploys the chart to the cluster. Required secrets are read at job time from OpenBao.

What you get after running this

What	Where to find it
5-node local Kubernetes cluster (1 control-plane + 4 workers)	`kubectl get nodes` after Phase 1
CloudNativePG operator + single-instance PG cluster (`postgresql-cnpg-1`), SCRAM-SHA-256 auth, chart-bundled (chart 10.x drops the bundled PG)	`kubectl -n postgresql get cluster`
Self-hosted GitLab CE (chart-bundled OpenBao subchart for GitLab Secrets Manager)	`https://gitlab.local.example.net`
GitLab Registry / KAS / MinIO object storage (LFS / artifacts / packages)	`https://{registry,kas,minio}.local.example.net`
Bootstrap-installed OpenBao (standalone, chart-bundled PVC, PG-backed)	`https://openbao.local.example.net`
Self-registered GitLab Runner (Kubernetes executor, registers against in-cluster `gitlab-webservice-default.gitlab.svc:8181` Service URL)	`Admin → CI/CD → Runners` after Phase 2
Stable PV/PVC pairs for CloudNativePG/Redis/MinIO/OpenBao/Gitaly (each bound by exact PVC name + CNPG annotations on the postgresql one)	`infra/data/shared/stable/<service>/` (hostPath-backed, `Retain` policy). The chart PVCs bind to these by name so `tofu destroy && tofu apply` keeps the data — opt-in via `var.preserve_stateful_data = true` (paired with `bootstrap --destroy --preserve-data`); default is destructive (`tofu destroy` wipes everything via the `null_resource.wipe_data` provisioner + chart-managed `rm -rf` teardown). See `docs/phase-2.md § Stable storage` for the contract on PVC naming and CNPG annotations.
Wildcard cert + CA for `.local.example.net` (infra tier) + `.apps.local.example.net` (apps tier), both signed by the same self-signed CA	`infra/tls/wildcard/` (10-year self-signed, regenerated by Phase 2's cfssl Job; `--destroy` wipes it for a full reset). The two certs descend from one CA so a single trust anchor covers both hostname families — see `docs/accessing-apps.md` for the two-listener / two-cert rationale.
Chart-managed Secrets snapshot	`infra/secrets/gitlab-runtime-secrets.yaml` (chart-minted postgres/redis/minio/rails/gitaly/kas passwords, mode 0600) + `infra/secrets/cnpg-role-passwords.json` (bootstrap-minted gitlab/openbao PG roles, mode 0600) — both written at the end of every successful Phase 2 install. See `docs/secrets.md` for how each secret is sourced and restored.
One-shot teardown (destructive)	`uv run blueprint-bootstrap --destroy [--yes] [--dry-run]` (cluster + stable/ + cert + secrets + OpenBao init JSON, with a privileged-container fallback for pod-UID-owned dirs). Add `--preserve-data` to keep the host-side stateful dirs intact (legacy 2026-06 contract; see `infra/tofu/variables.tf:preserve_stateful_data`).
Sample workload (guestbook) packaged as a Helm chart	`apps/guestbook/helm-chart/`. The chart reads `REDIS_MASTER_HOST` / `REDIS_REPLICA_HOST` env vars (with FQDN defaults) so it can reach redis in a different namespace — see `docs/accessing-apps.md` for the cross-namespace wiring.
GitLab CI pipeline that validates and deploys	Phase 3 — `uv run blueprint-phase3`. Creates the `blueprint-apps` GitLab group + four projects (`shared-code`, `guestbook`, `redis`, `redis-slave`), renders the per-project `.gitlab-ci.yml` from `apps/shared-code/templates/`, sets CI/CD variables (`CI_KUBECONFIG_B64`, `CI_REGISTRY_IMAGE`, …), and triggers a smoke pipeline per project. The bootstrap mints a fresh admin PAT via `gitlab-rails runner` (execed in the toolbox pod) so the whole provisioning step survives a `tofu destroy && apply && bootstrap --phase 2`. See `docs/phase-3.md` + `.agents/skills/provision-phase-3/` for the runbook + iteration loop.
3rd-party / in-tree-chart app publishing	`uv run blueprint-new-app <name>` scaffolds the source-of-truth files (`apps/<name>/{deploy-info.yaml,values.yaml,README.md,helm-chart/}` + a bearer token in OpenBao) and `uv run blueprint-publish-app <name>` syncs to GitLab, renders the per-`AppKind` CI, sets CI variables, commits + pushes, and triggers a pipeline. The build pods use a build-pod-friendly kubeconfig (`infra/logs/ci-deployer.kubeconfig`, server: `https://kubernetes.default.svc`) so they can talk to the in-cluster apiserver. The full end-user runbook is in `docs/gitops-3rd-party-apps.md`; the agent-driven flow in `.agents/skills/provision-new-3p-app/`.

The eventual pipeline is the demo: a push to a sample app triggers a runner job that validates the OpenTofu code (tofu validate), validates the Helm chart (helm template / helm lint), and on main deploys the chart into the cluster, exposing the pod through the chart's apps-https listener on <project>.apps.local.example.net. Phases 1 + 2 build the platform; Phase 3 wires the app into it.

Prerequisites

See docs/prereqs.md. The short list: Linux or macOS, kind 0.27+, kubectl, helm ≥3.16, tofu 1.6+, uv 0.4+, podman or docker, ~10 GB free RAM for the cluster + GitLab.

Quick start

1. Get the code

git clone https://github.com/bruj0/k8s-cicd-gitlab.git
cd k8s-cicd-gitlab

2a. Hand it to an agent (recommended)

The skills under .agents/skills/ follow the agentskills.io open standard, which several AI coding agents (GitHub Copilot, Cursor, Claude Code, Codex) read directly as background context the moment the repo is opened. For agents that don't auto-discover, paste the skill file into the chat as a reference. The recommended loop:

Open the k8s-cicd-gitlab/ folder in your AI coding agent. If the agent auto-loads skills (<agent> skills is a thing — see your agent's docs), the three skills are already in context. Otherwise paste .agents/skills/provision-phase-1/SKILL.md, .agents/skills/provision-phase-2/SKILL.md, and .agents/skills/provision-phase-3/SKILL.md into the first message.
Prompt the agent:

Run provision-phase-1 end to end and report when the cluster's 5 nodes are Ready. Then run provision-phase-2 and report when GitLab is reachable. Then run provision-phase-3 and report when the 4 GitLab projects exist and the smoke pipelines are green. Don't run anything that needs sudo — print the command and ask me to run it. Follow each skill's Smoke tests section before declaring green.
The agent reads each skill's Pre-flight → Install → Smoke tests → Iteration loop in order, runs the matching uv run blueprint-bootstrap --phase N (or blueprint-phase3) command, and reads back the smoke-test invariants. When something fails, the skill's Iteration loop already maps the failure to the exact installer class to fix.
Review the diff (the agent should have touched only infra/scripts/bootstrap/phase<N>/ or infra/scripts/bootstrap/VERSIONS.json, or infra/scripts/bootstrap/phase3/, or apps/shared-code/templates/) and commit.

Why this works: the skill files are plain Markdown, they follow the canonical ten-section template, and the agent uses them as a deterministic runbook (Install section → one-liner) plus a checklist (Smoke tests section → invariants) plus a recovery manual (Iteration loop + Common pitfalls → where to look first). That keeps the agent on-rails even when a step fails.

2b. Or run it by hand

The full cluster + stack is up in four commands once prereqs are met (see docs/prereqs.md):

# 1. Install the bootstrap's Python deps into .venv/ (the committed
#    uv.lock makes this reproducible). Idempotent.
uv sync

# 2. Bootstrap the working tree (prereqs check, tofu init, helm
#    chart cache). Also idempotent. Prints the next commands.
uv run blueprint-bootstrap --phase 1

# 3. You apply — the bootstrap never does (per spec, OpenTofu is
#    run by a person). Cluster comes up; Headlamp URL is printed.
tofu -chdir=infra/tofu apply -auto-approve

# 4. Install GitLab + Runner + OpenBao + chart-managed Envoy
#    Gateway. End-to-end takes ~10 min on a beefy laptop.
uv run blueprint-bootstrap --phase 2

3. Post-install host-side steps

Three steps the bootstrap can't do for you (all of them are outside the cluster):

Trust the chart's wildcard CA — the GitLab chart's pre-install cfssl Job mints *.local.example.net and stores the CA in the gitlab-wildcard-tls-ca Secret; export it to disk and add it to the host trust store:
```
kubectl -n gitlab get secret gitlab-wildcard-tls-ca \
  -o jsonpath='{.data.cfssl_ca}' | base64 -d > infra/tls/public/ca.crt
sudo trust anchor infra/tls/public/ca.crt
```

Map the wildcard to 127.0.0.1 so the browser reaches Envoy on the kind node:

echo "127.0.0.1 gitlab.local.example.net registry.local.example.net \
             kas.local.example.net minio.local.example.net \
             openbao.local.example.net" | sudo tee -a /etc/hosts

Read OpenBao secrets via the blueprint-secrets CLI (auto-port-forwards 127.0.0.1:8200, so no kubectl port-forward is needed):

uv run blueprint-secrets read gitlab initial_root_password   # GitLab root pw
uv run blueprint-secrets ui                                  # OpenBao UI

Reach in-cluster services without remembering the kubectl port-forward incantation. The same CLI grew a generic dispatcher — picks the right kubeconfig, namespace, service, and port for you, prints what to do next, and tears down the forward on Ctrl-C:
```
uv run blueprint-secrets port-forward --list                # show menu
uv run blueprint-secrets port-forward openbao               # http://127.0.0.1:8200/ui
uv run blueprint-secrets port-forward gitlab-registry       # docker login 127.0.0.1:5000
uv run blueprint-secrets port-forward minio                 # S3 API + console hints
uv run blueprint-secrets port-forward gitlab-webservice     # curl http://127.0.0.1:8181/-/health
```
The cluster's intended user-facing path is the Gateway/NodePort pair (browser → https://gitlab.<domain>) — see step 2 above. port-forward is the cluster-side workhorse for things the Gateway doesn't (yet) expose (e.g. Container Registry's plain HTTP v2 endpoint on 127.0.0.1:5000).

Full details for each phase are in docs/phase-1.md, docs/phase-2.md, and docs/phase-3.md. At that point you have a working cluster, GitLab, Runner, and OpenBao — the platform Phase 3 will deploy the sample app into.

4. Provision Phase 3 (GitLab-side app onboarding)

The Phase 1 + Phase 2 commands leave you with a cluster + GitLab + Runner + OpenBao. Phase 3 wires the three demo apps (guestbook, redis, redis-slave) into GitLab as projects, each with a CI pipeline that builds the container, runs the unit tests, and (on main) deploys the Helm chart into the cluster.

# 5. Bootstrap GitLab projects (4 of them: shared-code, guestbook,
#    redis, redis-slave). The bootstrap mints a fresh admin PAT via
#    gitlab-rails runner, so the step survives `tofu destroy`.
#    Idempotent: re-running resumes from the failed step.
uv run blueprint-phase3

Three (optional) variations:

# Pre-flight only — talks to GitLab + OpenBao + checks the in-cluster
# gitlab-webservice Service, but mutates nothing.
uv run blueprint-phase3 --check

# Wipe the 4 blueprint-apps projects from GitLab (keeps the group).
# Use this when iterating on the .gitlab-ci.yml templates.
uv run blueprint-phase3 --destroy

# Drop the local apps-local/ working tree + re-clone every project
# from GitLab. Useful when an `apps/` commit left the working tree
# out of sync.
uv run blueprint-phase3 --reset-clones

After blueprint-phase3 reports green, push a change to any of the four projects and watch the smoke pipeline:

# Push a trivial commit to the guestbook repo to kick a build.
cd apps-local/blueprint-apps/guestbook
echo "# touched at $(date -Iseconds)" >> README.md
git commit -am "smoke: touch README"
git push origin main

# Watch the build from the GitLab UI, or from the CLI:
glab pipeline ci view --live          # or:
glab api projects/$(glab project list --search guestbook -o json \
  | jq -r '.[0].id')/pipelines?per_page=1 | jq '.[0].id,.status,.web_url'

The shipped pipeline (per .agents/skills/provision-phase-3/ Smoke tests) covers: helm lint, helm template, docker build (BuildKit), docker push to the in-cluster registry, and a helm upgrade --install against the cluster using kubeconfig from a masked CI/CD variable. Read docs/phase-3.md for the per-step contract and the wipe-and-reinstall invariant.

Manual verification (if you went the hand-driven path): run each skill's Smoke tests section step by step. If you handed the repo to an agent, the agent already did this — just read its final report and the smoke-test output it captured.

The full URLs and login credentials after each phase are maintained in the per-phase skills so they don't drift:

Phase 1 (cluster + Headlamp): .agents/skills/provision-phase-1/SKILL.md
Phase 2 (GitLab + Runner + OpenBao): .agents/skills/provision-phase-2/SKILL.md
Phase 3 (GitLab projects + CI for guestbook/redis/redis-slave): .agents/skills/provision-phase-3/SKILL.md

Interacting with the GitLab repos after Phase 3

Phase 3 lands you with two distinct working surfaces and the line between them is the single most important thing to internalise. Mis-treat either side and you end up with silent drift between GitLab and the bootstrap's local state.

blueprint/                                      (this repo, on the host)
├── apps/<name>/                                CANONICAL GitLab-side
│                                                source. Frozen after
│                                                first push; edit code
│                                                here, the bootstrap
│                                                re-pushes it.
└── apps-local/blueprint-apps/<name>/           WORKING TREE — one
                                                 `git clone` per
                                                 GitLab project.
                                                 Gitignored; re-creatable
                                                 from GitLab with
                                                 --reset-clones.

GitLab (https://gitlab.local.example.net/blueprint-apps/)
└── <name>                                      RECEIVING REPO — what
                                                 `glab` / `git push`
                                                 see. The bootstrap
                                                 syncs the canonical
                                                 source into here on
                                                 every Phase 3 run.

As an end user (GitLab is the source of truth)

If you're not iterating on the blueprint itself — i.e. you treat GitLab as the canonical source and just want to clone, edit, commit, and push — skip apps/ and blueprint-phase3 entirely. Clone from GitLab directly, push, and the Runner takes it from there.

Trust + DNS first (one-time per machine; same steps the Post-install section covers):

# 1. Trust the wildcard CA so curl/Git/docker trust https://gitlab.local.example.net.
#    Phase 2 drops a copy of the CA at infra/tls/wildcard/ca.pem
#    (alongside the cert + key, all 10-year self-signed):
sudo trust anchor --store infra/tls/wildcard/ca.pem
# macOS equivalent:
#   sudo security add-trusted-cert -d -r trustRoot \
#     -k /Library/Keychains/System.keychain \
#     infra/tls/wildcard/ca.pem

# 2. Map the wildcard to 127.0.0.1 so the URL resolves to the kind node
echo "127.0.0.1 gitlab.local.example.net registry.local.example.net \
             kas.local.example.net minio.local.example.net \
             openbao.local.example.net" | sudo tee -a /etc/hosts

Log in once at https://gitlab.local.example.net with root

the password Phase 2 wrote to OpenBao:

uv run blueprint-secrets read gitlab initial_root_password

Set up a Personal Access Token for Git-over-HTTPS + the Container Registry:

User menu → Edit profile → Access tokens
Name: laptop, Scopes: api, read_repository, write_repository, read_registry, write_registry, create_runner (last one only if you'll register a Runner).
Copy the token. Store it somewhere safe.

Tell Git to use the token instead of typing it every push (picked up by HTTPS clones):

git config --global url."https://oauth2:${GITLAB_TOKEN}@gitlab.local.example.net/".insteadOf "https://gitlab.local.example.net/"
# or, store it in your keychain:
git config --global credential.helper store       # and enter once per host
git config --global credential.https://gitlab.local.example.net.username oauth2
git config --global credential.https://gitlab.local.example.net.password <token>

Clone any of the four blueprint-apps projects and push like a normal remote:

git clone https://gitlab.local.example.net/blueprint-apps/guestbook.git
cd guestbook
$EDITOR README.md
git commit -am "docs: typo fix"
git push origin main
# → Runner picks it up, builds, deploys.

To make Git stop verifying the self-signed cert every clone, either trust the CA globally (recommended — see step 1 above) or scope it to this repo:

git config --global http.https://gitlab.local.example.net.sslCAInfo \
  infra/tls/wildcard/ca.pem

Browse to the running app (the chart exposes each project on <project>.apps.local.example.net via the chart's apps-https listener — a separate wildcard from the four GitLab-infra FQDNs that sit under *.local.example.net):

guestbook → https://guestbook.apps.local.example.net
redis / redis-slave are not exposed externally; their pods are reachable inside the cluster only (they exist to give the guestbook something to talk to). To poke at them from your laptop:
```
uv run blueprint-secrets port-forward redis-master   # redis-cli 127.0.0.1:6379
```

The two hostname families stay separate so apps can't shadow the infra cert (or vice versa) — see docs/accessing-apps.md for the full two-listener / two-cert architecture.

Push Docker images to the in-cluster registry (the same one the CI uses — useful for local builds you want to run in the cluster):

uv run blueprint-secrets port-forward gitlab-registry
# in another terminal:
docker login 127.0.0.1:5000 -u oauth2 -p "$GITLAB_TOKEN"
docker build -t 127.0.0.1:5000/blueprint-apps/guestbook:dev .
docker push  127.0.0.1:5000/blueprint-apps/guestbook:dev

# NB: the registry has NO TLS on the loopback port-forward (plain
# HTTP v2). Treat 127.0.0.1:5000 as localhost-only.

Prefer the glab CLI for everything that isn't a git operation — pipeline status, MRs, registry tags, variables — so you don't have to click through the UI:

glab auth login --hostname gitlab.local.example.net --token "$GITLAB_TOKEN"

glab ci view -p blueprint-apps/guestbook --live          # tail a pipeline
glab mr create -p blueprint-apps/guestbook \
  --title "fix: ..." --target-branch main --description "..." --remove-source-branch
glab api 'projects/$(glab project list --search guestbook -o json | jq -r .[0].id)/registry/repositories' \
  | jq '.[].id'

The canonical edit loop

The intended day-to-day loop is edit apps/<name>/ → run blueprint-phase3 → watch the pipeline:

# 1. Edit the canonical source
$EDITOR apps/guestbook/guestbook-go/main.go

# 2. Run the bootstrap — rsyncs the delta, re-renders
#    .gitlab-ci.yml, commits, pushes. Idempotent.
export KUBECONFIG=$PWD/infra/tofu/kubeconfig
uv run blueprint-phase3
# → "synced + pushed guestbook to blueprint-apps/guestbook @ <sha>"

# 3. Watch the pipeline
glab ci view -p blueprint-apps/guestbook --live
# or open https://gitlab.local.example.net/blueprint-apps/guestbook/-/pipelines

Inspecting + debugging pipelines

# All four projects, with IDs + URLs
glab api 'groups/blueprint-apps/projects?per_page=10' \
  | jq '.[] | {id, path_with_namespace, web_url, last_activity_at}'

# Recent pipelines for a project
glab api 'projects/<id>/pipelines?per_page=5' \
  | jq '.[] | {id, status, ref, web_url}'

# A specific job's full log
glab api 'projects/<id>/jobs/<job_id>/trace' | tail -80

# Image tags pushed to the in-cluster registry
glab api 'projects/<id>/registry/repositories/<repo_id>/tags?per_page=10' \
  | jq '.[] | {name: .name, created_at: .created_at}'

Local-clone loop (apps-local/)

apps-local/blueprint-apps/<name>/ is a regular git clone of the GitLab project. Use it when you need to iterate on a pipeline template or hand-edit a CI file without going through a full blueprint-phase3 run:

cd apps-local/blueprint-apps/guestbook
git checkout -b debug/fix-kaniko-cache
$EDITOR .gitlab-ci.yml
git commit -am "debug: drop cache flag to test cold build"
git push -u origin debug/fix-kaniko-cache
# Watch the branch's pipeline:
# https://gitlab.local.example.net/blueprint-apps/guestbook/-/pipelines?ref=debug/fix-kaniko-cache

The catch: a hand-edited .gitlab-ci.yml in the working tree will be overwritten the next time blueprint-phase3 re-renders. Two ways around this:

Work on a branch in apps-local/. The bootstrap always touches main; your branch stays untouched until you merge it.
Use --no-overwrite-ci to keep a hand-edited .gitlab-ci.yml on main while everything else still gets rsynced + committed. Fold the change back into apps/shared-code/templates/ when it's stable, then drop the flag.

Cheat-sheet

I want to…	Run
Sync the latest `apps/<name>/` to GitLab (bulk, all 4 in-house apps)	`uv run blueprint-phase3`
Pre-flight only, no mutations	`uv run blueprint-phase3 --check`
Wipe the working tree + re-clone everything	`uv run blueprint-phase3 --reset-clones`
Wipe the 4 GitLab projects (keep the group)	`uv run blueprint-phase3 --destroy`
Limit a run to one project	`uv run blueprint-phase3 --project guestbook`
Leave a hand-edited `.gitlab-ci.yml` alone	`uv run blueprint-phase3 --no-overwrite-ci`
Add a 3rd-party / in-tree-chart app (one-time scaffold)	`uv run blueprint-new-app <name>`
Re-publish a 3rd-party / in-tree-chart app to GitLab (idempotent)	`uv run blueprint-publish-app <name>`
Re-publish a 3rd-party app under a different group	`uv run blueprint-publish-app <name> --group Services`
Preview a publish-app run without mutations	`uv run blueprint-publish-app <name> --dry-run`
Pull remote changes (UI edits, MRs)	`cd apps-local/blueprint-apps/<name> && git pull --ff-only`
Watch the latest pipeline	`glab ci view -p blueprint-apps/<name> --live`
Read a job's full log	`glab api projects/<id>/jobs/<job_id>/trace`
List the group's projects + IDs	`glab api 'groups/blueprint-apps/projects'`

What NOT to do

Don't git push from apps/<name>/ directly. That directory is not a working tree — it's the canonical source the bootstrap owns. Pushing from it bypasses the render + commit step. Always edit apps/<name>/, then run blueprint-phase3.
Don't hand-edit files on main in apps-local/.../ and expect them to stick. The next blueprint-phase3 will rsync from apps/<name>/ and overwrite. Branch it (see above) or fold the change back into the canonical source.
Don't glab project delete or glab variable set outside the bootstrap. Per AGENTS.md § 4 rule #3, every GitLab-side change is automated by blueprint-phase3. Hand-edits create drift the bootstrap will undo on the next run.

The full version of this section (with the auth + trust sub-sections, the registry walk-through, and a deeper explanation of the working-tree/canonical-source split) is in docs/phase-3.md § 12.

Adding a 3rd-party / in-tree-chart app

Beyond the four in-house cluster apps (guestbook, redis, redis-slave, shared-code) that Phase 3 creates as a group, the blueprint also supports adding upstream helm charts (Headlamp, Kyverno, kube-prometheus, anything from a public helm repo) as GitOps-managed GitLab projects — one at a time, on demand, by humans or AI agents. The two-command flow:

# 1. (once per chart) declare the source in VERSIONS.json,
#    then scaffold the source-of-truth files.
uv run blueprint-new-app headlamp

# 2. (idempotent — re-run after every bump or values edit)
#    sync to GitLab, render CI, set variables, commit+push,
#    trigger a pipeline. Iterates on failure to surface a
#    user-action or auto-fix common issues.
uv run blueprint-publish-app headlamp

After ~2 minutes, the URL is live. blueprint-publish-app reuses the Phase 3 helpers (GitlabClient.ensure_group / ensure_project / set_variable, AppsLocalFS.clone / pull_ff / commit_and_push, ci_render.render_for) but composes them for one app at a time — so the 4 in-house apps still go through blueprint-phase3 together, while in-tree-chart apps can be (re)published on demand without booting the full 6-step pipeline.

Each app is GitOps-managed end-to-end:

The chart is vendored in apps/<name>/helm-chart/ so a rebuild of the kind cluster needs no internet and a version bump is one PR.
A per-app deploy-info.yaml is the contract between the chart and the CI (name, ingress hostname, auth via OpenBao). The chart itself is never modified.
The per-project CI runs helm diff → helm upgrade --install → bearer-token smoke, on every push to main. The build pods use a build-pod-friendly kubeconfig (infra/logs/ci-deployer.kubeconfig with server: https://kubernetes.default.svc) so they can talk to the apiserver from inside the cluster.

The full end-user runbook (every step, the iteration loop, common failure modes, design rationale) is in docs/gitops-3rd-party-apps.md. The agent-driven flow is in .agents/skills/provision-new-3p-app/SKILL.md; the two are kept in lockstep — the skill's "Run" section mirrors the manual steps.

Iteration loop

All three phases are idempotent. The bootstrap's job is to leave you in a clean "everything is verified" state — when something is off, the right move is:

Read the smoke test in the corresponding skill (provision-phase-1, provision-phase-2, provision-phase-3, or provision-new-3p-app). Each skill enumerates the checkable invariants.
Find the matching installer under infra/scripts/bootstrap/phase<N>/ (Phase 1+2) or under infra/scripts/bootstrap/phase3/ (Phase 3 + the publish_app orchestrator for the in-tree-chart lane). Each installer is a single-responsibility class with a one-line __init__ that takes the paths and version catalog it needs.
Fix the installer (or the YAML reference under bootstrap/phase2/references/, or the per-project CI template under apps/shared-code/templates/), re-run the bootstrap.
Update AGENTS.md if the rule you tripped over should be a hard rule, and commit the docs in the same PR as the code fix.

Wiping the cluster + Phase 2 state

The bootstrap ships a one-shot teardown that wipes both the infrastructure layer (the kind cluster, the kubeconfig) and the bootstrap-owned host-side state (infra/data/shared/stable/ hostPath PV backing dirs, chart-managed leftovers, the chart-managed Secrets snapshot, the wildcard TLS cert + CA, OpenBao's init JSON):

uv run blueprint-bootstrap --destroy --yes          # default: full wipe
uv run blueprint-bootstrap --destroy --preserve-data  # opt-out: keep stateful data
uv run blueprint-bootstrap --destroy --dry-run      # print only

Default contract (2026-07+): tofu destroy wipes everything. The bootstrap's --destroy runs tofu destroy (cluster lifecycle is owned by OpenTofu; see AGENTS.md § 4 rule #3), and the cluster's null_resource.wipe_data destroy provisioner (in infra/tofu/cluster.tf) sweeps infra/data/shared/ as part of tofu destroy. Paired with local-path-provisioner's default rm -rf teardown, the result is: cluster gone, every PV gone, every host-side data dir gone. A fresh bootstrap --phase 2 rebuilds from scratch.

Bidirectional mode (opt-in, paired with the cluster-apply side):

tofu -chdir=infra/tofu apply -var=preserve_stateful_data=true
uv run blueprint-bootstrap --destroy --preserve-data

The two flags must agree — tofu destroy reads the var, and the cluster's null_resource.wipe_data destroy provisioner short-circuits to a no-op when the var is true. The kind extra_mounts.propagation field also flips from Bidirectional to default HostToContainer under the same var so the host bind-source isn't propagated through on container umount. Useful for users who want to recreate the cluster but reuse the on-disk PG / Redis / MinIO / OpenBao / Gitaly data. Note: chart-minted fresh credentials won't match stale on-disk PG unless the chart-managed Secrets snapshot (infra/secrets/gitlab-runtime-secrets.yaml) is also preserved (passing --preserve-data keeps it).

If a PV dir is owned by a pod UID (e.g. openbao runs as UID 100, postgres as UID 1001) and the unprivileged rm fails, bootstrap --destroy falls back to a one-shot Docker/Podman privileged container that bind-mounts the parent dir and rm -rf's the child. Same trick the null_resource.wipe_data provisioner uses on the tofu side.

Phase 3 survives a wipe+reinstall by design. The whole Phase-3 provisioning step is one command and is fully self-contained: blueprint-phase3 re-uses an admin PAT stored in OpenBao (secret/gitlab/bootstrap/admin_token) if present, and otherwise mints a fresh one via gitlab-rails runner execed in the toolbox pod. So the canonical replication contract is:

tofu -chdir=infra/tofu destroy -auto-approve
tofu -chdir=infra/tofu apply   -auto-approve
uv run blueprint-bootstrap --phase 2    # recreates OpenBao + GitLab + Runner
uv run blueprint-phase3                 # recreates the 4 GitLab projects + CI

End state after the four commands is identical to the pre-wipe end state — same group path (blueprint-apps), same project IDs, same CI/CD variables, same per-project pipeline shape, same registry images. There is no manual glab step in between.

The only way to create the cluster is tofu -chdir=infra/tofu apply -auto-approve. --destroy only tears down; it never re-creates.

How the bootstrap + skills fit together

Two tools drive this repo, and they play different roles:

The bootstrap application (`infra/scripts/bootstrap/`)

A single uv-managed Python package that does the actual work.

infra/scripts/
├── bootstrap.py                 # thin shim → delegates to bootstrap/
└── bootstrap/                   # the package (installed by uv sync)
    ├── __main__.py              # python -m bootstrap → same entry as the console script
    ├── VERSIONS.json            # pinned versions, single source of truth
    ├── cli.py                   # click wrapper → blueprint-bootstrap entry point
    │                           # + `--destroy` / `--port-forward` / `--dry-run` / `--user`
    ├── secrets_cli.py           # click wrapper → blueprint-secrets entry point
    ├── app.py                   # composition root: BootstrapApp wires phases → installers
    ├── prereq.py                # Phase 1 prereq check + installer
    ├── helm_cache.py            # downloads charts into infra/helm-charts/
    ├── tofu.py                  # tofu init / plan / validate (read-only ops)
    ├── paths.py                 # resolved Path constants (chart dir, secrets dir, infra/tls/)
    ├── os_detect.py             # apt / dnf / pacman / brew branching
    ├── shell.py                 # idempotent subprocess wrapper (logs every command)
    ├── logger.py, versions.py, installer.py
    └── phase2/                  # Phase 2 only — 13 steps orchestrated by Phase2Pipeline
        ├── pipeline.py          # orchestrates the 13 Phase 2 steps in order
        ├── catalog.py           # Phase2Installers dataclass (bundle of every installer)
        ├── gateway.py           # Gateway API CRDs (standard v1.5.0 + chart-shipped Envoy CRDs)
        ├── local_path_provisioner.py  # local-path StorageClass + `local-path` as default
        ├── stable_storage.py    # pre-create stable PV/PVC pairs (existing_claim /
        │                        # volume_claim_template / pvc_with_volume_name); also stamps
        │                        # CNPG-specific PVC name (`<cluster>-<serial>`) + annotations
        ├── cloudnative_pg.py    # CNPG operator + Cluster/postgresql-cnpg (single instance, 8Gi)
        │                        # + bootstrap `gitlabhq_production` + `openbao` PG databases
        ├── redis.py             # bitnami/redis single-node (architecture=standalone)
        ├── minio.py             # MinIO single-node + 11 GitLab buckets + the dual-key
        │                        # `gitlab-rails-storage` Secret (Rails `connection` +
        │                        # Docker-registry native `config` for the registry s3 driver)
        ├── openbao.py           # OpenBao chart install + init + unseal (PG backend)
        ├── wildcard_certs.py    # mint self-signed CA + wildcard cert + 4 listener Secrets
        ├── persistent_secrets.py # snapshot + restore chart-managed Secrets (postgres/redis/
        │                        # minio/rails/gitaly/kas) so `tofu destroy && apply` works
        ├── gitlab.py            # GitLab chart install (chart 10.x bundles Envoy Gateway +
        │                        # OpenBao subchart; consumes external PG/Redis/MinIO)
        ├── runner.py            # Runner chart install, registers against in-cluster
        │                        # `gitlab-webservice-default.gitlab.svc:8181` over HTTP
        ├── secrets.py           # hvac client (OpenBao) + lazy shared port-forward
        └── references/          # vendored CRDs, CNPG Cluster, chart fragments

After uv sync, four console scripts land on the per-checkout $PATH:

Script	Wraps	Purpose
`blueprint-bootstrap`	`bootstrap.cli:main`	The installer (`--phase 1 \| 2`, `--check`, `--dry-run`, `--user`).
`blueprint-secrets`	`bootstrap.secrets_cli:main`	Post-install helper: `read <path> <key>`, `ui` (OpenBao), and `port-forward <service>` (any in-cluster Service — auto-locates the tofu kubeconfig + namespace + port).
`blueprint-phase3`	`bootstrap.phase3_cli:main`	Bulk GitLab-side provisioning: create the `blueprint-apps` group + 4 in-house projects (guestbook, redis, redis-slave, shared-code), render `.gitlab-ci.yml`, set CI variables, trigger smoke pipelines. Idempotent re-runs.
`blueprint-new-app`	`bootstrap.new_app_cli:main`	Scaffolder: turns a name into `apps/<name>/{values.yaml,deploy-info.yaml,README.md,helm-chart/}` + a `apps_manifest.yaml` entry, so a 3rd-party chart can be (re)published. Source-of-truth only — does not install.
`blueprint-publish-app`	`bootstrap.publish_app_cli:main`	Decoupled per-app publisher: ensure GitLab group + project, clone/pull, rsync, render `.gitlab-ci.yml`, set CI variables, commit + push, trigger a pipeline. Sibling of `blueprint-phase3` for the in-tree-chart lane — reuses Phase 3's helpers but composes them for one app at a time. Iterates on CI failure to surface a user-action or auto-fix common issues.

BootstrapApp (in app.py) is the composition root: it instantiates each installer — a single-responsibility class that takes only the paths and version catalog it needs — and runs them in order. That's the SOLID shape: appending "another installer for Phase 3" means adding one class under phase3/, one wiring line in app.py, and (if exposed via CLI) one click option in cli.py. No installer ever depends on another.

The hard constraint: the bootstrap prepares, it never applies. There is no --apply flag. The cluster comes up only when a person runs tofu -chdir=infra/tofu apply. That boundary is documented in AGENTS.md § 4 rule #3.

The skills (`.agents/skills/`)

Three per-phase SKILL.md files, one per phase. They follow the agentskills.io open standard, so they work as background context for any AI coding agent (Copilot, Claude, Cursor, etc.) and as plain runbooks for humans. Each skill has the same ten sections, in the same order:

1.  Pre-flight                       # what to verify before starting
2.  Install                          # the actual install one-liner
3.  Smoke tests                      # the checkable invariants + how to read them
4.  URLs you can reach after install # the exact hostnames + logins
5.  Iteration loop                   # when something is off, what to read + re-run
6.  Canonical (known-good) pinned versions
7.  Common pitfalls (frozen — append, don't rewrite)
8.  Rules of thumb (apply when adding Phase-N pieces)
9.  When the install is green        # the exact "you're done" output
10. How to undo                      # the symmetric teardown

The section names vary slightly per phase (the Phase 2 skill has two Rules of thumb sections because Phase 2 touches multiple charts; the Phase 1 and Phase 3 skills have one). The shape is the same, and every skill titles the smoke-test section "Smoke tests" — that's the one to read first when something is off.

The four skills shipped here:

Skill	Drives	Reads from
`.agents/skills/provision-phase-1/`	`blueprint-bootstrap --phase 1` + the six `tofu apply`-style handoff commands	`bootstrap/prereq.py`, `bootstrap/helm_cache.py`, `bootstrap/tofu.py`, Headlamp chart cache
`.agents/skills/provision-phase-2/`	`blueprint-bootstrap --phase 2` + the three post-install host-side steps (trust CA, /etc/hosts, `blueprint-secrets`)	`bootstrap/phase2/pipeline.py` + `gateway.py` / `openbao.py` / `gitlab.py` / `runner.py` / `secrets.py`
`.agents/skills/provision-phase-3/`	`blueprint-phase3` (mints admin PAT via gitlab-rails runner, creates the `blueprint-apps` group + 4 projects, renders CI from `apps/shared-code/templates/`, sets CI/CD variables, triggers smoke pipelines)	`bootstrap/phase3/pipeline.py` + `gitlab_client.py` / `apps_local.py` / `ci_render.py` / `glab_client.py`
`.agents/skills/provision-new-3p-app/`	`blueprint-new-app <name>` (scaffold the source-of-truth files for a 3rd-party / in-tree-chart app) + `blueprint-publish-app <name>` (decoupled per-app publisher: sync to GitLab, render CI, set variables, commit + push, trigger pipeline)	`bootstrap/new_app/scaffold.py` + `bootstrap/phase3/publish_app.py` (reuses `gitlab_client.py` / `apps_local.py` / `ci_render.py` from Phase 3)

How they relate

The skills are driving instructions; the bootstrap is the machine that executes them:

A skill tells what to do and how to verify — it points at the exact installer class to read, the exact smoke test to run, the exact URL to open.
uv run blueprint-bootstrap --phase N (or blueprint-phase3) does the work — it pre-flights, installs, and prints the next user command.
The skill is for AI agents and humans alike; the bootstrap is for both shells to call.

Concrete example, Phase 2:

The user (or an AI agent) reads .agents/skills/provision-phase-2/SKILL.md.
The skill's Install section says: uv run blueprint-bootstrap --phase 2.
bootstrap.cli:main parses the flag, instantiates BootstrapApp, which wires together Phase2Pipeline → GatewayCRDsInstaller → OpenBaoInstaller → GitlabInstaller → GitLabRunnerInstaller.
Each installer reads VERSIONS.json, calls the injected CommandRunner for the helm/kubectl command, and logs the output under its own logger. (CommandRunner is the idempotent subprocess Protocol in shell.py — every shell call is logged with --dry-run honoured.)
The skill's Smoke tests section walks the user through verifying each step (5 Envoy pods Running, Gateway Programmed=True, runner registered, OpenBao has secrets, GitLab UI returns 200).
The skill's Iteration loop maps a failed smoke test to the exact installer + line range to inspect, so a fix becomes "edit one class, re-run the bootstrap."

When a phase lands, it's expected to grow one new installer class under phase<N>/ (or phase3/), one new entry in the appropriate composition root (BootstrapApp or Phase3App), and (if user-facing) one new section in the skill — not a parallel class hierarchy and not a new top-level command. The same shape applies to the per-app publish lane: a new blueprint-publish-app <name> invocation extends the existing PublishApp orchestrator under phase3/publish_app.py and reuses the existing GitlabClient / AppsLocalFS / ci_render helpers, not a new top-level command.

Layout

blueprint/
├── apps/             # Canonical application source (frozen after first push to GitLab)
│   ├── shared-code/  # Cross-app CI templates — referenced via include: from the 3 apps
│   ├── guestbook/    # Demo: classic k8s guestbook (Go app + helm chart + Dockerfile)
│   ├── redis/        # Demo: redis master (helm chart; no Dockerfile → uses upstream)
│   ├── redis-slave/  # Demo: redis slave workload (helm chart; no Dockerfile)
│   ├── headlamp/     # 3rd-party / in-tree-chart example (upstream helm chart vendored)
│   └── kyverno/      # 3rd-party / in-tree-chart example
├── apps-local/       # gitignored working tree (one `git clone` per GitLab project,
│                     # under blueprint-apps/) — populated by `blueprint-phase3`
│                     # or `blueprint-publish-app`
├── docs/             # Per-phase runbooks + prereqs
│   ├── prereqs.md
│   ├── phase-1.md                          # Cluster bring-up runbook
│   ├── phase-2.md                          # GitLab + Runner + OpenBao contributor guide
│   ├── phase-3.md                          # GitLab-side app onboarding + per-project CI
│   └── gitops-3rd-party-apps.md            # 3rd-party / in-tree-chart provisioning (end-user)
├── pyproject.toml    # uv project: installs blueprint-bootstrap + blueprint-secrets
│                     # + blueprint-phase3 + blueprint-new-app + blueprint-publish-app
├── uv.lock           # committed for reproducibility
├── AGENTS.md         # Hard rules + layout map for AI agents and humans
└── infra/
    ├── helm-charts/  # Locally-cached helm charts (GitLab, Runner, OpenBao, Headlamp, …)
    ├── scripts/
    │   ├── bootstrap.py     # thin shim → delegates to bootstrap/ package
    │   ├── phase3.py        # thin shim → delegates to bootstrap/phase3_cli.py
    │   └── bootstrap/       # class-based pipeline (SOLID), packaged via pyproject.toml
    │       ├── VERSIONS.json    # Pinned versions, single source of truth
    │       ├── cli.py           # click wrapper: blueprint-bootstrap entry point
    │       ├── secrets_cli.py   # click wrapper: blueprint-secrets (post-install helper)
    │       ├── phase3_cli.py    # click wrapper: blueprint-phase3 entry point (Phase 3)
    │       ├── new_app_cli.py   # click wrapper: blueprint-new-app entry point
    │       ├── publish_app_cli.py  # click wrapper: blueprint-publish-app entry point
    │       ├── app.py          # composition root for Phase 1+2 (BootstrapApp)
    │       ├── app_phase3.py   # composition root for Phase 3 (Phase3App)
    │       ├── new_app/
    │       │   └── scaffold.py    # AppScaffolder + AppSpec (per-app source-of-truth files)
    │       ├── phase3/
    │       │   ├── publish_app.py    # PublishApp (decoupled per-app publisher)
    │       │   ├── pipeline.py       # Phase3Pipeline (bulk 4-app provisioning)
    │       │   ├── apps_local.py     # AppsLocalFS (clone / pull / rsync / commit-and-push)
    │       │   ├── apps_manifest.py  # AppSpec + AppsManifest + AppKind enum
    │       │   ├── ci_render.py      # per-app-kind template dispatch (typed, no if/else)
    │       │   ├── gitlab_client.py  # GitlabClient (ensure_group / ensure_project / set_variable)
    │       │   └── glab_client.py    # raw HTTP + glab auth helper
    │       └── phase<N>/       # Phase-specific installers
    └── tofu/         # OpenTofu configuration (kind cluster)

(infra/tls/ is fully gitignored except for .gitkeep. It's a scratch dir used as a one-shot export target for the GitLab chart's cfssl Job — Phase 2's post-install step exports the CA there before adding it to the host trust store.)

Conventions

These are the rules-of-thumb encoded in AGENTS.md. The hard rules (cluster lifecycle is tofu only, no edits to shipped charts, no commits of secrets) live there and matter for both humans and AI agents touching this repo.

Bootstrap prepares, never applies. Per spec, the bootstrap application verifies the system and provisions all the configuration so a person can run tofu apply themselves. There is no --apply flag on the bootstrap.
No shell scripts for non-trivial logic. The spec rules shell out for Python. infra/scripts/bootstrap/ is a class-based package composed of single-responsibility classes wired together by app.py (Phase 1+2) and app_phase3.py (Phase 3).
uv is the Python toolchain. pyproject.toml + uv.lock are committed; .venv/ is gitignored. Five console scripts: blueprint-bootstrap (install, phases 1+2), blueprint-secrets (post-install OpenBao helper), blueprint-phase3 (Phase 3 — bulk GitLab-side provisioning of the 4 in-house cluster apps), blueprint-new-app (per-app source-of-truth scaffolder for in-tree-chart apps), and blueprint-publish-app (the decoupled per-app publisher — reuses Phase 3's helpers but composes them for one app at a time). Don't reintroduce system-level pip install.
Versions in one place. infra/scripts/bootstrap/VERSIONS.json pins every tool, every helm chart, every chart repo URL. No class hardcodes a version — they all read from the catalog.
Helm charts are cached locally. Charts land in infra/helm-charts/<name>-<version>.tgz; the bootstrap installs them from that path so re-installs don't require a network round-trip.
No plaintext secrets in git. Phase 2 stores the OpenBao unseal keys + root token in infra/secrets/openbao-init.json (gitignored, mode 0600) and the bootstrap-minted PG role passwords in infra/secrets/cnpg-role-passwords.json. The chart-minted postgres/redis/minio/rails/gitaly/kas passwords are snapshotted to infra/secrets/gitlab-runtime-secrets.yaml. The values that the developer needs day-to-day (GitLab initial root password, Runner registration token) are pushed into OpenBao at secret/gitlab/.... Read them back via uv run blueprint-secrets read <path> <key>. The full secret inventory is in docs/secrets.md.
TLS is local-CA now, LE later. Public LE can't validate *.local.example.net. The GitLab chart's pre-install cfssl Job mints the wildcard cert for us today; once public DNS is delegated, swap to cert-manager-issued certs without changing anything about the bootstrap or the chart values.
CNPG PVC contract. The CloudNativePG operator looks up the pre-created PVC by metadata.controller UID. The PVC name MUST be <cluster>-<serial> (i.e. postgresql-cnpg-1 for a single instance). The PVC MUST carry CNPG annotations (cnpg.io/cluster, cnpg.io/instanceName, cnpg.io/instanceRole=primary, cnpg.io/nodeSerial=1, cnpg.io/pvcRole=main). bootstrap.py --phase 2 enforces both via phase2/stable_storage.py and re-stamps the ownerReference after Cluster creation so the field-index selector picks the volume up. See docs/phase-2.md § Stable storage for the full contract.
GitLab-rails-storage Secret carries two keys, not one. The Docker-registry s3 driver and the Rails-side object_store parsers want different YAML schemas for the same MinIO bucket. The chart's registry subchart reads registry.storage.key (default config) and mounts that into the storage: block of /etc/docker/registry/config.yml; the Rails-side readers (appConfig.object_store.*.connection) read the key connection. Both keys live on a single Secret named gitlab-rails-storage. Don't merge them.
The Runner registers in-cluster, not via the public hostname. Pods that need to register against GitLab use http://gitlab-webservice-default.gitlab.svc:8181, never https://gitlab.local.example.net. Port 8181 speaks plain HTTP (TLS terminates at the Gateway above it).
Templates are real YAML files, never Python string literals. No yaml.dump({...}) of a dict that hardcodes pipeline shape. Every .gitlab-ci.yml / Application / HelmRelease content lives under apps/<name>/ or apps/shared-code/templates/ as committed .yaml / .tpl files and is loaded with Path.read_text() + string.Template.safe_substitute. The bootstrap/phase3/ci_render.py module is the canonical example; substitution vars are $UPPER_SNAKE_CASE; missing vars raise KeyError from .substitute (use .safe_substitute only for optional substitutions).
glab is for the human / AI agent, not the bootstrap. The bootstrap talks to the GitLab API via python-gitlab + requests (see bootstrap/phase3/gitlab_client.py). The one exception is glab auth login (bootstrap/phase3/glab_client.py:seed_glab_auth) — it populates ~/.config/glab-cli/config.yml so the agent (and the user's shell) can glab api ... against GitLab without re-prompting. Bootstrap operations never read that file.
apps/ is canonical + frozen; apps-local/ is the runtime working tree. The bootstrap rsyncs apps/<name>/ → apps-local/blueprint-apps/<name>/, renders .gitlab-ci.yml, commits, and pushes. apps-local/ is gitignored; it is recreated on every --reset-clones run. Treat apps/ as the source of truth — edit code there; the bootstrap pushes it.

Trade-offs and what we'd do with more time

These are the calls we made deliberately — they're the questions a reviewer is most likely to ask.

Why OpenTofu and not Terraform. OpenTofu is the OSS fork and the prereqs installer covers both apt/dnf/pacman/brew.
Why kind and not minikube/k3d. kind's per-container hostPath mounts (used for the per-node + shared volumes in Phase 1) are the cleanest way to model a multi-node cluster on one host. minikube's mount story is VM-shaped; k3d is closer but shipped with fewer workers per control-plane out of the box.
Why the GitLab chart instead of GitLab Omnibus. The chart is what the GitLab project itself ships as the reference for running GitLab inside Kubernetes; the chart also bundles Envoy Gateway as gateway-helm sub-chart, so the same wildcard cert handles GitLab, the Registry, KAS, MinIO, and (Phase 3) the deployed app.
Why Phase 2 ships one Runner (executor), not a fleet. A single executor inside the cluster is enough for the validation + deploy jobs.
Why a runner-in-pod and not a shell executor. The runner is registered with kubernetes executor; jobs spin up ephemeral pods. That keeps CI workloads inside the same RBAC boundary as the workload they deploy, no host-mount leakage.
Tradeoff on local-network state. *.local.example.net resolves on the developer host via /etc/hosts, not on the cluster. Pods use Service DNS (gitlab-webservice-default.gitlab.svc:8181, openbao.openbao.svc:8200) for in-cluster traffic. The distinction is encoded as a rule in AGENTS.md so future contributors don't paper over it with a clever alias.
What we'd do with more time.
1. Move TLS to cert-manager + Let's Encrypt once a real DNS name is delegated.
2. ~~Add a dispose subcommand~~ (done: blueprint-bootstrap --destroy [--yes] [--dry-run] — wipes cluster + stable/ + chart-managed Secrets snapshot + wildcard cert + OpenBao init JSON. Privilege container fallback for pod-UID-owned PV dirs.)
3. ~~Add Phase 3 — GitLab-side app onboarding~~ (done: blueprint-phase3 mints an admin PAT via gitlab-rails runner, creates the blueprint-apps group + 4 projects, renders .gitlab-ci.yml from apps/shared-code/templates/, and triggers a smoke pipeline per project).
4. Wire the chart cache into Helm's OCI cache (helm pull --untar) so we don't re-download on every re-run when VERSIONS.json bumps.
5. Promote Phase 2's chart-managed Secrets snapshot (gitlab-runtime-secrets.yaml) from --destroy --yes reset to an opt-in --preserve mode so tofu destroy && tofu apply && bootstrap --phase 2 preserves GitLab users / repos / CI variables across the cluster recreate — the snapshot mechanism is already in place, just needs a --no-wipe-stable flag.

Navigation Menu

Repository files navigation

CICD Blueprint — self-hosted GitLab in k8s + CI/CD, end to end

📖 Full documentation site: https://bruj0.github.io/k8s-cicd-gitlab/

What is this?

Contents

End-to-end narrative

What you get after running this

Prerequisites

Quick start

1. Get the code

2a. Hand it to an agent (recommended)

2b. Or run it by hand

3. Post-install host-side steps

4. Provision Phase 3 (GitLab-side app onboarding)

Interacting with the GitLab repos after Phase 3

As an end user (GitLab is the source of truth)

The canonical edit loop

Inspecting + debugging pipelines

Local-clone loop (apps-local/)

Cheat-sheet

What NOT to do

Adding a 3rd-party / in-tree-chart app

Iteration loop

Wiping the cluster + Phase 2 state

How the bootstrap + skills fit together

The bootstrap application (`infra/scripts/bootstrap/`)

The skills (`.agents/skills/`)

How they relate

Layout

Conventions

Trade-offs and what we'd do with more time

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.agents/skills		.agents/skills
.github/workflows		.github/workflows
apps		apps
docs		docs
gitlab		gitlab
infra		infra
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
README.md		README.md
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

CICD Blueprint — self-hosted GitLab in k8s + CI/CD, end to end

📖 Full documentation site: https://bruj0.github.io/k8s-cicd-gitlab/

What is this?

Contents

End-to-end narrative

What you get after running this

Prerequisites

Quick start

1. Get the code

2a. Hand it to an agent (recommended)

2b. Or run it by hand

3. Post-install host-side steps

4. Provision Phase 3 (GitLab-side app onboarding)

Interacting with the GitLab repos after Phase 3

As an end user (GitLab is the source of truth)

The canonical edit loop

Inspecting + debugging pipelines

Local-clone loop (apps-local/)

Cheat-sheet

What NOT to do

Adding a 3rd-party / in-tree-chart app

Iteration loop

Wiping the cluster + Phase 2 state

How the bootstrap + skills fit together

The bootstrap application (infra/scripts/bootstrap/)

The skills (.agents/skills/)

How they relate

Layout

Conventions

Trade-offs and what we'd do with more time

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

The bootstrap application (`infra/scripts/bootstrap/`)

The skills (`.agents/skills/`)