Skip to content

[Stack 3/17] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs)#2510

Open
jucor wants to merge 1 commit intospr/edge/d39cf65dfrom
spr/edge/0c448343
Open

[Stack 3/17] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs)#2510
jucor wants to merge 1 commit intospr/edge/d39cf65dfrom
spr/edge/0c448343

Conversation

@jucor
Copy link
Copy Markdown
Collaborator

@jucor jucor commented Mar 30, 2026

Summary

Replace pip install with uv pip install in the delphi Dockerfile for faster dependency installation. uv pip is a drop-in replacement — same requirements.lock, same pyproject.toml, same installed packages in site-packages.

Changes

  • delphi/Dockerfile:
    • Copy uv binary from ghcr.io/astral-sh/uv:0.11.2 (pinned version, single static binary)
    • Place in /opt/uv/ in builder to avoid leaking into production image via COPY --from=builder /usr/local/bin
    • Set UV_SYSTEM_PYTHON=1 (install into system Python, not a venv)
    • Replace all pip install with uv pip install in builder and test stages
    • Update BuildKit cache mount targets from /root/.cache/pip to /root/.cache/uv
    • Test stage copies uv from builder (single source of truth)

What's NOT changed

  • Makefile — untouched, make rebuild-delphi works as before
  • docker-compose.yml / docker-compose.test.yml — untouched
  • pyproject.toml / requirements.lock — untouched, same format
  • pip-compile workflow — untouched, still used for lock file generation
  • Final/production image — no uv added, stays lean

CI Benchmark (GitHub Actions, ubuntu-latest, --no-cache)

5 pip runs (Mar 27 stack push) vs 3 uv pip runs (this PR):

Step pip (n=5) uv pip (n=3) Speedup
Docker build 264s (sd=6) 169s (sd=2) 1.56x (-94s)
Pytest run 223s (sd=4) 227s (sd=4) ~same

~94 seconds saved per CI run on the Docker build step. Pytest runtime is unchanged (same packages, same tests). Low variance in both groups confirms this is a real improvement, not noise.

Local Benchmark (Apple M1 Max, 64GB, Docker Desktop, --no-cache)

Step pip uv pip Speedup
Dependencies install 149.3s 80.4s 1.9x
Dev deps install 10.0s 2.2s 4.5x

Test plan

  • docker compose -f docker-compose.test.yml build --no-cache delphi succeeds locally
  • Built image has all expected packages (pip show diagnostic passes in build log)
  • CI passes (3 successful runs)
  • make rebuild-delphi works (Makefile untouched)

Squashed commits

  • Speed up CI by using uv pip instead of pip

commit-id:0c448343


Stack:


⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

@jucor jucor changed the title Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) [Stack 3/17] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) Mar 30, 2026
@jucor jucor force-pushed the spr/edge/0c448343 branch from e4353af to 6fd489f Compare March 30, 2026 22:39
## Summary


Replace `pip install` with `uv pip install` in the delphi Dockerfile for faster dependency installation. `uv pip` is a drop-in replacement — same `requirements.lock`, same `pyproject.toml`, same installed packages in `site-packages`.

### Changes

- **`delphi/Dockerfile`**:
  - Copy `uv` binary from `ghcr.io/astral-sh/uv:0.11.2` (pinned version, single static binary)
  - Place in `/opt/uv/` in builder to avoid leaking into production image via `COPY --from=builder /usr/local/bin`
  - Set `UV_SYSTEM_PYTHON=1` (install into system Python, not a venv)
  - Replace all `pip install` with `uv pip install` in builder and test stages
  - Update BuildKit cache mount targets from `/root/.cache/pip` to `/root/.cache/uv`
  - Test stage copies `uv` from builder (single source of truth)

### What's NOT changed

- **Makefile** — untouched, `make rebuild-delphi` works as before
- **docker-compose.yml / docker-compose.test.yml** — untouched
- **pyproject.toml / requirements.lock** — untouched, same format
- **pip-compile workflow** — untouched, still used for lock file generation
- **Final/production image** — no `uv` added, stays lean

### CI Benchmark (GitHub Actions, `ubuntu-latest`, `--no-cache`)

5 pip runs (Mar 27 stack push) vs 3 uv pip runs (this PR):

| Step | pip (n=5) | uv pip (n=3) | Speedup |
|------|-----------|--------------|---------|
| **Docker build** | **264s** (sd=6) | **169s** (sd=2) | **1.56x (-94s)** |
| Pytest run | 223s (sd=4) | 227s (sd=4) | ~same |

**~94 seconds saved per CI run** on the Docker build step. Pytest runtime is unchanged (same packages, same tests). Low variance in both groups confirms this is a real improvement, not noise.

### Local Benchmark (Apple M1 Max, 64GB, Docker Desktop, `--no-cache`)

| Step | pip | uv pip | Speedup |
|------|-----|--------|---------|
| **Dependencies install** | **149.3s** | **80.4s** | **1.9x** |
| Dev deps install | 10.0s | 2.2s | **4.5x** |

## Test plan

- [x] `docker compose -f docker-compose.test.yml build --no-cache delphi` succeeds locally
- [x] Built image has all expected packages (`pip show` diagnostic passes in build log)
- [x] CI passes (3 successful runs)
- [x] `make rebuild-delphi` works (Makefile untouched)


## Squashed commits

- Speed up CI by using `uv pip` instead of `pip`

commit-id:0c448343
@jucor jucor force-pushed the spr/edge/0c448343 branch from 6fd489f to f00accb Compare March 30, 2026 22:47
@github-actions
Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 76 76 0%
benchmarks/bench_repness.py 81 81 0%
benchmarks/bench_update_votes.py 38 38 0%
benchmarks/benchmark_utils.py 34 34 0%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1118 336 70%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 387 233 40%
database/postgres.py 305 205 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 22 91%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 52 16 69%
pca_kmeans_rep/repness.py 361 48 87%
pca_kmeans_rep/stats.py 107 22 79%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 17 91%
regression/comparer.py 887 720 19%
regression/datasets.py 103 22 79%
regression/recorder.py 36 27 25%
regression/utils.py 137 118 14%
run_math_pipeline.py 260 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 54 52%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 265 265 0%
umap_narrative/803_check_batch_status.py 175 175 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 157 157 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 477 18%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/general.py 62 41 34%
Total 10919 7646 30%

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Delphi Docker build to use uv pip instead of pip install to reduce CI Docker build time while keeping the same lockfile-driven dependency set and leaving the production image lean.

Changes:

  • Add uv (pinned tag) to the builder stage and switch dependency installs to uv pip install.
  • Update BuildKit cache mounts from pip’s cache directory to uv’s cache directory.
  • Reuse the builder’s uv binary in the test stage to install dev dependencies more quickly.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -54,8 +61,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \

# Install the project package (without dependencies - they're already installed)
# This registers entry points and installs the package in development mode
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above this step says the package is installed "in development mode", but uv pip install --no-deps . performs a standard (non-editable) install, same as pip install .. Either update the comment to avoid implying an editable/develop install, or switch to an editable install if that’s actually intended.

Suggested change
# This registers entry points and installs the package in development mode
# This registers entry points and installs the package without re-installing dependencies

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants