add cuda fallback to pixi template

[mashehu]

Same as the one in the micromamba template added in 67542ce

[pditommaso]

⚠️ Blocking: the CUDA fallback is attached to a command that doesn't solve, so it never triggers

The micromamba fix (67542ce) works because the retry wraps micromamba install -f conda.yml — the command that performs the solve, and the first command in the chain. In the pixi flow the work is split across three commands, and the retry ended up on the one that doesn't solve.

Verified locally with pixi 0.40.3:

Step	Behaviour	Solve?
pixi init --import conda.yml	no pixi.lock produced	❌
pixi add conda-forge::which	creates pixi.lock — solves the entire manifest, incl. deps imported from conda.yml	✅
pixi install (lock up-to-date)	prints the lock-file is up-to-date, just installs from the lock	❌

And pixi add exits non-zero on an unsatisfiable solve:

$ pixi add "conda-forge::which==999.999.999"; echo $?
  × failed to solve the conda requirements ...
1

So on a GPU-less build host, a __cuda-requiring package in conda.yml fails the solve at pixi add, the && chain short-circuits there, and the wrapped pixi install (with CONDA_OVERRIDE_CUDA) is never reached. The template renders correctly and the tests pass, but the fallback is effectively dead code.

(I also confirmed cupy resolves with a __cuda >=13 dependency in the lock, i.e. the cuda virtual package is engaged at add-time, not install-time.)

Suggested direction

The override needs to be in effect during the solve, i.e. during the pixi add step(s). Since every pixi add re-solves the whole environment, wrapping only the first isn't sufficient (a later add would re-solve without the override and fail again). The robust shape is to run init → add(s) → install and, on a detected __cuda failure, re-run the add+install region with CONDA_OVERRIDE_CUDA exported — which is a small templating restructure since the base packages are currently emitted as separate && pixi add lines.

Also

• Branch predates fix: make which available for R (bioconda) post-links in conda v2 builds #1070 (which added && pixi add conda-forge::which), so it currently has a merge conflict and needs a rebase. Note that after rebasing, the which add guarantees a solve before pixi install, cementing the issue above.
• The fallback keys on the solver error literally containing __cuda — worth confirming pixi/rattler's actual error wording on a real Linux build once the placement is fixed.

The &&/|| retry mechanism itself is correct (retries only on __cuda, still fails the build on other errors) — it's just hooked onto the wrong command.

[pditommaso]

Note: the analysis above was AI-generated (and locally reproduced with pixi 0.40.3). We'd rather not merge without independent proof of correctness — if you could share a self-contained test script that demonstrates the CUDA fallback actually triggering on a GPU-less build (i.e. a conda.yml with a __cuda-requiring package that fails without the override and succeeds with it), that would settle it quickly. 🙏

[mashehu]

Updated to include the changes from #1070.

claude-assisted test script:

#!/usr/bin/env bash
#
# Demonstrates the conda/pixi:v1 CUDA fallback on a GPU-less build host.
#
#   1. control: solving a conda.yml with a __cuda-requiring package fails at
#      `pixi add` time, and the solver error contains the literal string
#      `__cuda` (what the template's grep keys on)
#   2. control: the same solve succeeds after
#      `pixi workspace system-requirements add cuda 99`
#   3. end-to-end: the rendered dockerfile template builds successfully, with
#      the build log proving the first solve failed and the fallback kicked in
#
# The fallback needs two mechanisms (verified with pixi 0.61):
#   - `pixi workspace system-requirements add cuda 99` fixes the SOLVE: pixi
#     lock solving is machine-independent -- it uses the manifest
#     [system-requirements], not the host's virtual packages, so
#     CONDA_OVERRIDE_CUDA alone (which micromamba honours) has no effect there
#   - `export CONDA_OVERRIDE_CUDA=99` fixes the INSTALL: once the lock contains
#     __cuda-dependent packages, `pixi install`/`pixi add` check them against
#     the machine's virtual packages and fail on a GPU-less host without it
#
# Requires docker on a host without an NVIDIA driver (any laptop/CI runner).
#
# The default test package is the cuda build of jaxlib, which has __cuda in
# its hard `depends`. Note cupy is NOT a suitable test package anymore: since
# the conda-forge cuda-version migration it carries __cuda only under
# `constrains`, which is vacuously satisfied when the virtual package is
# absent, so it solves fine on GPU-less hosts.
#
# Usage: ./pixi-cuda-fallback-test.sh [repo-root]
# Env:   PLATFORM  docker platform, e.g. linux/amd64 (default: host native)
#        PACKAGE   __cuda-requiring conda spec (default: "jaxlib =*=cuda*")

set -euo pipefail

repo=${1:-$(cd "$(dirname "$0")" && pwd)}
template="$repo/src/main/resources/templates/conda-pixi-v1/dockerfile-conda-file.txt"
[ -f "$template" ] || { echo "template not found: $template" >&2; exit 1; }

package=${PACKAGE:-jaxlib =*=cuda*}
pixi_image=public.cr.seqera.io/wave/pixi:0.61.0-noble
platform_args=()
[ -n "${PLATFORM:-}" ] && platform_args=(--platform "$PLATFORM")

workdir=$(mktemp -d)
trap 'rm -rf "$workdir"' EXIT

cat > "$workdir/conda.yml" <<EOF
channels:
  - conda-forge
dependencies:
  - $package
EOF

# same commands the template runs; --no-install keeps the controls solve-only,
# $1 optionally injects the fallback command between init and add
run_solve() {
    docker run --rm "${platform_args[@]}" \
        -v "$workdir/conda.yml:/opt/wave/conda.yml" -w /opt/wave \
        "$pixi_image" \
        sh -c "pixi init --import /opt/wave/conda.yml ${1:+&& $1} && pixi add --no-install conda-forge::which"
}

echo "== 1) solve WITHOUT the cuda system-requirement must fail and mention __cuda"
if out=$(run_solve 2>&1); then
    echo "FAIL: solve unexpectedly succeeded" >&2
    echo "      (does '$package' still have __cuda in its hard depends?)" >&2
    exit 1
fi
echo "$out" | tail -n 6
echo "$out" | grep -q __cuda \
    || { echo "FAIL: solver error does not mention __cuda - the fallback grep would not trigger" >&2; exit 1; }
echo "OK: solve failed at 'pixi add' and the error contains __cuda"

echo
echo "== 2) same solve WITH the cuda system-requirement must succeed"
run_solve "pixi workspace system-requirements add cuda 99" > /dev/null
echo "OK: solve succeeds after 'pixi workspace system-requirements add cuda 99'"

echo
echo "== 3) full template build: fallback must trigger and the build succeed"
sed -e "s|{{pixi_image}}|$pixi_image|" \
    -e "s|{{base_packages}}| conda-forge::procps-ng|g" \
    -e "s|{{base_image}}|ubuntu:24.04|" \
    "$template" > "$workdir/Dockerfile"

# build only the stage containing the fallback logic, without exporting an
# image: the final stage merely COPYies the (multi-GB cuda) environment and
# the image export duplicates it again -- both just double/triple the disk
# footprint without adding anything to what is being verified here
log="$workdir/build.log"
if ! docker build "${platform_args[@]}" --progress=plain --no-cache --target build \
        --output type=cacheonly -f "$workdir/Dockerfile" "$workdir" > "$log" 2>&1; then
    echo "FAIL: docker build failed; last lines of the log:" >&2
    tail -n 30 "$log" >&2
    exit 1
fi
grep -q __cuda "$log" \
    || { echo "FAIL: build log shows no __cuda failure - the fallback was never exercised" >&2; exit 1; }
grep -q "CONDA_LOCK_END" "$log" \
    || { echo "FAIL: build did not reach the lock output" >&2; exit 1; }
echo "build log evidence (first failure, then retry):"
grep -E "__cuda \*|system-requirements|Added " "$log" | head -n 6
echo "OK: first solve failed with __cuda, retry with the cuda system-requirement completed the build"

echo
echo "all checks passed"