Skip to content

[Stack 7/27] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs)#2484

Closed
jucor wants to merge 1 commit intojc/disable-snapshot-testsfrom
jc/uv-pip-ci
Closed

[Stack 7/27] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs)#2484
jucor wants to merge 1 commit intojc/disable-snapshot-testsfrom
jc/uv-pip-ci

Conversation

@jucor
Copy link
Copy Markdown
Collaborator

@jucor jucor commented Mar 30, 2026

Summary

Stacked on #2482 (Add SKIP_GOLDEN env var to disable golden snapshot tests). Please review and merge #2482 first.
Next in stack: #2419 (Deep analysis of Python-Clojure discrepancies and fix plan)

Replace pip install with uv pip install in the delphi Dockerfile for faster dependency installation. uv pip is a drop-in replacement — same requirements.lock, same pyproject.toml, same installed packages in site-packages.

Changes

  • delphi/Dockerfile:
    • Copy uv binary from ghcr.io/astral-sh/uv:0.11.2 (pinned version, single static binary)
    • Place in /opt/uv/ in builder to avoid leaking into production image via COPY --from=builder /usr/local/bin
    • Set UV_SYSTEM_PYTHON=1 (install into system Python, not a venv)
    • Replace all pip install with uv pip install in builder and test stages
    • Update BuildKit cache mount targets from /root/.cache/pip to /root/.cache/uv
    • Test stage copies uv from builder (single source of truth)

What's NOT changed

  • Makefile — untouched, make rebuild-delphi works as before
  • docker-compose.yml / docker-compose.test.yml — untouched
  • pyproject.toml / requirements.lock — untouched, same format
  • pip-compile workflow — untouched, still used for lock file generation
  • Final/production image — no uv added, stays lean

CI Benchmark (GitHub Actions, ubuntu-latest, --no-cache)

5 pip runs (Mar 27 stack push) vs 3 uv pip runs (this PR):

Step pip (n=5) uv pip (n=3) Speedup
Docker build 264s (sd=6) 169s (sd=2) 1.56x (-94s)
Pytest run 223s (sd=4) 227s (sd=4) ~same

~94 seconds saved per CI run on the Docker build step. Pytest runtime is unchanged (same packages, same tests). Low variance in both groups confirms this is a real improvement, not noise.

Local Benchmark (Apple M1 Max, 64GB, Docker Desktop, --no-cache)

Step pip uv pip Speedup
Dependencies install 149.3s 80.4s 1.9x
Dev deps install 10.0s 2.2s 4.5x

Test plan

  • docker compose -f docker-compose.test.yml build --no-cache delphi succeeds locally
  • Built image has all expected packages (pip show diagnostic passes in build log)
  • CI passes (3 successful runs)
  • make rebuild-delphi works (Makefile untouched)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Delphi Docker build to use uv pip instead of pip to speed up dependency installation during CI builds.

Changes:

  • Adds uv into the builder stage and configures UV_SYSTEM_PYTHON=1 to install into system Python.
  • Replaces pip install with uv pip install in dependency install steps (including BuildKit cache mount updates).
  • Copies uv into the test stage to install dev dependencies with uv pip.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@jucor jucor changed the base branch from edge to jc/disable-snapshot-tests March 30, 2026 12:43
@jucor jucor changed the title Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) [Stack 6/27] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) Mar 30, 2026
@jucor jucor changed the title [Stack 6/27] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) [Stack 7/27] Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs) Mar 30, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 37 to 39
# Install dependencies from lock file (cached layer - reused unless requirements.lock changes)
# BuildKit cache mount keeps pip cache between builds for faster rebuilds
# BuildKit cache mount keeps uv cache between builds for faster rebuilds
# If USE_CPU_TORCH is true, we install CPU-specific wheels and filter them out of the lockfile
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comments in this section are not indented like the rest of the instructions in the builder stage (most lines use two-space indentation, e.g., COPY pyproject... above). Please align the indentation of these comment lines for consistency/readability within the stage.

Copilot uses AI. Check for mistakes.
@jucor jucor force-pushed the jc/disable-snapshot-tests branch from 509a831 to ec4c93e Compare March 30, 2026 16:49
@github-actions
Copy link
Copy Markdown

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 76 76 0%
benchmarks/bench_repness.py 81 81 0%
benchmarks/bench_update_votes.py 38 38 0%
benchmarks/benchmark_utils.py 34 34 0%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1118 336 70%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 387 233 40%
database/postgres.py 305 205 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 22 91%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 52 16 69%
pca_kmeans_rep/repness.py 361 48 87%
pca_kmeans_rep/stats.py 107 22 79%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 17 91%
regression/comparer.py 887 720 19%
regression/datasets.py 103 22 79%
regression/recorder.py 36 27 25%
regression/utils.py 137 118 14%
run_math_pipeline.py 260 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 54 52%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 265 265 0%
umap_narrative/803_check_batch_status.py 175 175 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 157 157 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 477 18%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/general.py 62 41 34%
Total 10919 7646 30%

This was referenced Mar 30, 2026
@jucor
Copy link
Copy Markdown
Collaborator Author

jucor commented Mar 30, 2026

Superseded by spr-managed PR stack. See the new stack starting at #2508.

@jucor jucor closed this Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants