CAMB speed and precision update targeting future SO

.agents/skills/isolate-accuracy-fix/SKILL.md

@@ -0,0 +1,98 @@
+---
+name: isolate-accuracy-fix
+description: "Isolate minimal efficient code changes to improve numerical accuracy or fix regression mismatches in CAMB. Use for run_tests.py failures, check_accuracy instability, step-size or switch sensitivity, low-k or recombination-era regressions, and when accuracy fixes must preserve runtime."
+argument-hint: "Describe the failing test, ini file, output mismatch, or accuracy-sensitive regime to isolate."
+user-invocable: true
+---
+
+# Isolate Minimal Accuracy Fixes
+
+Use this skill when CAMB has a numerical mismatch, stability failure, or accuracy regression and the goal is to find the smallest code change that fixes it without paying unnecessary runtime cost.
+
+This workflow is for targeted isolation, not broad tuning. Prefer a local fix to a global increase in tolerances, sampling, or step control.
+
+## When to Use
+
+- `fortran/tests/run_tests.py` fails on one or a few parameter files.
+- `camb/check_accuracy.py` shows unstable outputs or larger-than-expected boosted-reference differences.
+- A change in step size, switch timing, approximation handoff, or tolerance appears to affect results.
+- A proposed fix improves accuracy but may slow unrelated runs.
+- You need to identify which switch, redshift range, `k` range, or output family is actually sensitive.
+
+## Main Goal
+
+Find the narrowest code path and the narrowest phase-space region that controls the failing behavior, then implement the smallest performant fix that removes the mismatch.
+
+## Core Rules
+
+1. Start from a concrete failing anchor.
+   Use a failing ini file, output stem, test command, or changed routine.
+
+2. Reproduce in a clean comparison environment.
+   If the workspace is dirty, use a disposable copy under `/tmp` so A/B tests do not disturb user edits.
+
+3. Isolate the list of physically distinct failing outputs and address them one at a time.
+   E.g. low-l EE, mpk, and high-L TT failures are likely to be different issues. Focus separately on each in turn.
+
+4. Do not infer causality from a command option just because it is present.
+   Vary or remove the suspected trigger and check whether the same physical output still fails. E.g. a command with high lensing accuracy can still expose an unrelated low-l EE issue.
+
+5. a) Probe broadly first. Sweep AccuracyBoost / lAccuracyBoost / lSampleBoost
+   to confirm the issue is accuracy-controlled, not a bug.
+   The camb/check_accuracy.py script command can do this for stability under changes in accuracy paremeters, as well as providing info on which other less broad boost parameters affect the result. You may need to wait many minutes while it runs.
+   b) Then change one lever at a time when narrowing the cause, more locally that just a boost parameter.
+
+6. Prefer local overrides over global defaults.
+   If a change is only needed in a specific regime (a redshift window, k-range, or output type), test a localized change before touching global defaults. A broad change is acceptable when the precision gain is broad and the runtime cost is negligible; narrow the target when the fix is expensive.
+
+7. Treat runtime as a first-class constraint.
+   A fix is incomplete if it passes but slows transfer or matter-power runs for no precision benefit.
+
+8. Your changes should make physical sense.
+   Target changes to relevant physical time/k/other scales, rather than be ad-hoc patches that treat symptom rather than accuracy of general underlying calculation. Give physical explanation for each physically distinct change that you make, and if it makes no sense, investigate further.
+
+9. When changing a sampling grid, identify all consumers of that grid.
+   CMB source grids, transfer grids, and lensing grids can be shared with matter-power or source-window paths. Run an adjacent shared-path check before accepting the change.
+
+10. Treat index-based regression tests as grid-fragile.
+    If a test checks `array[i]`, decide whether `i` represents a physical point or just the old sampling. For grid changes, validate by interpolating to fixed physical coordinates before updating expected values.
+
+11. Separate "more accurate" from "different grid".
+    A passing boosted comparison is not enough if unrelated outputs move. Check a like-for-like physical comparison or a higher-accuracy reference before updating tests.
+
+12. Prefer direct control variables over proxy boosts.
+    If a broad boost works, identify which concrete derived quantity changed
+    (e.g. `dlnk`, `dk`, `lognum`, sample number, switch location, integration limit). Prefer changing that quantity directly rather than keeping a boost that also
+    changes unrelated ranges.
+
+## Acceptance Matrix
+
+For each proposed local fix, test:
+
+- The original failing command, suppressing unrelated output families only when needed to keep focus.
+- The same physical output with the suspected trigger removed or varied.
+- An adjacent path that shares the changed grid, switch, or tolerance.
+- The default unit/regression test covering that shared path.
+- Runtime before and after for the target run.
+- For any accepted boost-derived fix, document the concrete internal quantity
+  being changed and confirm unrelated quantities controlled by the same boost
+  are not part of the final patch.
+- Never use a local scaled boost fix where a more specific/physically-targeted fix to specific k/time/output/etc might work available.
+- No temporary or unjustified code changes remain in the working tree
+
+## Deliverable
+
+Leave the final, minimal fix applied to the working tree as unstaged changes so the user can review the diff directly.
+
+Clean up any A/B scaffolding before finishing where not likely to be useful again:
+
+- Remove disposable `/tmp` copies used for comparison.
+- Revert any exploratory edits, debug prints, or temporary parameter bumps that are not part of the final fix.
+
+Report should include (for each separate physical issue):
+
+- The minimal failing reproducer (ini + command).
+- The localized change(s) (file + lines) that resolves it
+- Before/after numbers for the failing diagnostic.
+- Runtime impact on a representative non-failing run (cpu times).
+- Physical justification for why the change targets the right scale/domain.

.devcontainer/Dockerfile

@@ -0,0 +1,50 @@
+ARG VARIANT="3.14-bookworm"
+FROM mcr.microsoft.com/devcontainers/python:${VARIANT}
+
+ENV CAMB_DEVCONTAINER_VENV=/home/vscode/.local/share/camb-devcontainer/.venv \
+    PRE_COMMIT_HOME=/home/vscode/.cache/pre-commit \
+    UV_CACHE_DIR=/home/vscode/.cache/uv \
+    UV_LINK_MODE=copy \
+    UV_PROJECT_ENVIRONMENT=/home/vscode/.local/share/camb-devcontainer/.venv
+
+RUN apt-get update \
+    && export DEBIAN_FRONTEND=noninteractive \
+    && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    g++ \
+    gfortran \
+    git \
+    libgsl-dev \
+    make \
+    pkg-config \
+    tar \
+    unzip \
+    xz-utils \
+    && curl -LsSf https://astral.sh/uv/install.sh -o /tmp/install-uv.sh \
+    && env UV_INSTALL_DIR=/usr/local/bin sh /tmp/install-uv.sh \
+    && rm -f /tmp/install-uv.sh \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir -p \
+    /home/vscode/.local/share/camb-devcontainer \
+    "${PRE_COMMIT_HOME}" \
+    "${UV_CACHE_DIR}" \
+    && chown -R vscode:vscode /home/vscode/.local /home/vscode/.cache \
+    && su vscode -c 'uv venv /home/vscode/.local/share/camb-devcontainer/.venv --python /usr/local/bin/python3' \
+    && su vscode -c 'uv pip install \
+        --upgrade \
+        --python /home/vscode/.local/share/camb-devcontainer/.venv/bin/python3 \
+        pip \
+        setuptools \
+        wheel \
+        findent \
+        fortls \
+        numpy \
+        scipy \
+        sympy \
+        packaging \
+        ruff \
+        pre-commit'

.devcontainer/README.md

@@ -0,0 +1,64 @@
+# CAMB devcontainer
+
+This devcontainer uses Python 3.14 and caps the container and default build runtime to 8 cores.
+
+The Python environment is created inside the container filesystem, not in the workspace checkout, so it does not fight with host-mounted file permissions or filesystem performance.
+
+The repository-level VS Code settings stay host-friendly and only help VS Code discover a workspace `./.venv`.
+The devcontainer sets its own interpreter path separately, so a Windows host virtual environment and the Linux
+container virtual environment can coexist cleanly.
+
+The devcontainer is designed to open either the main CAMB checkout or a linked Git worktree. It mounts the parent
+directory of the opened folder so the container can always see the repository `.git` metadata, and the startup script
+normalizes linked-worktree metadata to portable relative paths so the same checkout works from both Windows and the
+Linux container.
+
+Git inside the container is also told to ignore executable-bit changes for the mounted workspace. On Windows-backed
+bind mounts, Linux often reports regular files as executable even when the Git index records them as non-executable,
+which would otherwise produce spurious `100644 => 100755` diffs.
+
+When the container is created it:
+
+- installs the CAMB build toolchain, including `gfortran`, `g++`, `make`, and `libgsl-dev`
+- installs `uv` for locked, repeatable Python environment syncs
+- creates the container-local Python virtual environment and pre-installs the small Python dependency/tool set
+- downloads CosmoRec into `external/CosmoRec`
+- clones HYREC-2 into `external/HYREC-2`
+- patches the CosmoRec makefile to add `-fPIC`, which CAMB's Python wrapper requires
+- rebuilds `camb` with `recfast`, `cosmorec`, and `hyrec` enabled, and adds the workspace to the environment with a `.pth` file
+- configures `core.hooksPath` to use the tracked `.githooks/pre-commit` wrapper, which can use `pre-commit` from the container venv or from a host `.venv`
+- pre-installs the `pre-commit` hook environments into a container-local cache so later hook runs can work offline
+
+The devcontainer also installs `findent` and `fortls` into the container-local virtual environment so the Modern Fortran
+extension can format and index code without depending on host-side tools.
+
+The image and bootstrap intentionally avoid `pip install -e .`:
+
+- the Dockerfile uses `uv pip install ...` to install the small runtime and dev tool set directly into the container-local virtual environment
+- `post-create.sh` runs `python setup.py make` to rebuild the linked Fortran library with the optional recombination backends enabled
+- `post-create.sh` writes a small `.pth` file in the container-local environment so the workspace checkout is importable immediately without waiting on slow editable metadata generation
+
+Clean-tree CAMB builds automatically do the first Fortran sub-build with `-j1` until compiler-generated `.d` files
+exist, after which `make -j` and `python setup.py make` can safely respect any `MAKEFLAGS` you set.
+
+The first container bootstrap still needs network access to fetch the hook repositories listed in
+`.pre-commit-config.yaml`, but once bootstrap succeeds the tracked Git hook can run later without internet access.
+
+The `external/` directory stays in the workspace, so the downloaded sources are visible from Windows as normal files.
+
+Useful commands inside the container:
+
+```bash
+uv pip install --python /home/vscode/.local/share/camb-devcontainer/.venv/bin/python3 numpy scipy sympy packaging ruff pre-commit
+uv run python setup.py make
+uv run python -m unittest camb.tests.camb_test
+```
+
+On the host, keep using a normal workspace `.venv` if you want. The tracked Git hook resolves the right
+`pre-commit` executable at runtime instead of baking in one interpreter path during installation.
+
+Because the workspace may still contain a host-side `.venv` that is not usable inside Linux, tools that do not honor the
+devcontainer's selected interpreter may need to be pointed explicitly at
+`/home/vscode/.local/share/camb-devcontainer/.venv/bin/python3`.
+
+If you explicitly want a standards-based editable install after the container is up, you can still run `python -m pip install --no-build-isolation --no-deps -e .`, but that setuptools metadata phase is the part that remains slow on this bind-mounted checkout.

.devcontainer/devcontainer.json

@@ -0,0 +1,69 @@
+{
+  "name": "camb-py314",
+  "build": {
+    "dockerfile": "Dockerfile",
+    "context": "..",
+    "args": {
+      "VARIANT": "3.14-bookworm"
+    }
+  },
+  "runArgs": [
+    "--cpus=8"
+  ],
+  "workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces/camb-parent,type=bind",
+  "workspaceFolder": "/workspaces/camb-parent/${localWorkspaceFolderBasename}",
+  "mounts": [
+    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.gitconfig,target=/tmp/devcontainer-host.gitconfig,type=bind,readonly",
+    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.codex,target=/tmp/devcontainer-host-codex,type=bind,readonly",
+    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.claude,target=/tmp/devcontainer-host-claude,type=bind,readonly",
+    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.claude.json,target=/tmp/devcontainer-host.claude.json,type=bind,readonly"
+  ],
+  "initializeCommand": "git submodule update --init --recursive forutils",
+  "containerEnv": {
+    "COSMOREC_PATH": "${containerWorkspaceFolder}/external/CosmoRec/",
+    "FORUTILSPATH": "${containerWorkspaceFolder}/forutils",
+    "GIT_CONFIG_COUNT": "2",
+    "GIT_CONFIG_KEY_0": "submodule.forutils.ignore",
+    "GIT_CONFIG_VALUE_0": "all",
+    "GIT_CONFIG_KEY_1": "core.fileMode",
+    "GIT_CONFIG_VALUE_1": "false",
+    "HYREC_PATH": "${containerWorkspaceFolder}/external/HYREC-2/",
+    "OMP_NUM_THREADS": "8",
+    "OPENBLAS_NUM_THREADS": "8",
+    "RECOMBINATION_FILES": "recfast cosmorec hyrec",
+    "CAMB_DEVCONTAINER_VENV": "/home/vscode/.local/share/camb-devcontainer/.venv",
+    "PRE_COMMIT_HOME": "/home/vscode/.cache/pre-commit",
+    "UV_CACHE_DIR": "/home/vscode/.cache/uv",
+    "UV_LINK_MODE": "copy",
+    "UV_PROJECT_ENVIRONMENT": "/home/vscode/.local/share/camb-devcontainer/.venv"
+  },
+  "remoteEnv": {
+    "PATH": "/home/vscode/.local/share/camb-devcontainer/.venv/bin:${containerEnv:PATH}",
+    "VIRTUAL_ENV": "/home/vscode/.local/share/camb-devcontainer/.venv"
+  },
+  "postCreateCommand": "bash .devcontainer/scripts/post-create.sh",
+  "postStartCommand": "bash .devcontainer/scripts/post-start.sh",
+  "customizations": {
+    "vscode": {
+      "settings": {
+        "fortran.formatting.formatter": "findent",
+        "fortran.formatting.path": "/home/vscode/.local/share/camb-devcontainer/.venv/bin",
+        "python.defaultInterpreterPath": "/home/vscode/.local/share/camb-devcontainer/.venv/bin/python3",
+        "python-envs.workspaceSearchPaths": [
+          "/home/vscode/.local/share/camb-devcontainer"
+        ],
+        "python.venvPath": "/home/vscode/.local/share/camb-devcontainer",
+        "terminal.integrated.defaultProfile.linux": "bash",
+        "python-envs.terminal.autoActivationType": "off"
+      },
+      "extensions": [
+        "charliermarsh.ruff",
+        "fortran-lang.linter-gfortran",
+        "ms-python.debugpy",
+        "ms-python.python",
+        "openai.chatgpt",
+        "anthropic.claude-code"
+      ]
+    }
+  }
+}

.devcontainer/scripts/install-recombination-deps.sh

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+workspace_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+external_dir="${workspace_dir}/external"
+cosmorec_dir="${COSMOREC_PATH:-${external_dir}/CosmoRec}"
+hyrec_dir="${HYREC_PATH:-${external_dir}/HYREC-2}"
+cosmorec_url="${COSMOREC_URL:-https://www.cita.utoronto.ca/~jchluba/Recombination/_Downloads_/CosmoRec.v2.0.3b.tar.gz}"
+hyrec_repo="${HYREC_REPO:-https://github.com/nanoomlee/HYREC-2.git}"
+
+safe_git() {
+    env -u GIT_DIR -u GIT_WORK_TREE -u GIT_COMMON_DIR -u GIT_INDEX_FILE git "$@"
+}
+
+cosmorec_dir="${cosmorec_dir%/}"
+hyrec_dir="${hyrec_dir%/}"
+patched_cosmorec_flags=0
+
+mkdir -p "${external_dir}"
+
+if [[ ! -f "${cosmorec_dir}/Makefile" ]]; then
+    archive_path="${external_dir}/CosmoRec.v2.0.3b.tar.gz"
+    temp_dir="$(mktemp -d)"
+    trap 'rm -rf "${temp_dir}"' EXIT
+
+    if [[ ! -f "${archive_path}" ]]; then
+        curl --fail --show-error --location --retry 3 --retry-delay 2 "${cosmorec_url}" -o "${archive_path}"
+    fi
+
+    mkdir -p "${temp_dir}/extract"
+    tar --no-same-owner --no-same-permissions -xzf "${archive_path}" -C "${temp_dir}/extract"
+
+    extracted_root="$(find "${temp_dir}/extract" -mindepth 1 -maxdepth 1 -type d | head -n 1)"
+    if [[ -z "${extracted_root}" ]]; then
+        echo "Failed to locate extracted CosmoRec sources" >&2
+        exit 1
+    fi
+
+    rm -rf "${cosmorec_dir}"
+    mv "${extracted_root}" "${cosmorec_dir}"
+fi
+
+chmod -R u+rwX "${cosmorec_dir}" >/dev/null 2>&1 || true
+
+cosmorec_makefile_in="${cosmorec_dir}/Makefile.in"
+if [[ -f "${cosmorec_makefile_in}" ]] && ! grep -Eq '^[[:space:]]*CXXFLAGS.*-fPIC' "${cosmorec_makefile_in}"; then
+    sed -i -E '/^[[:space:]]*CXXFLAGS[[:space:]]*=/ s/$/ -fPIC/' "${cosmorec_makefile_in}"
+    patched_cosmorec_flags=1
+fi
+
+if [[ -f "${cosmorec_makefile_in}" ]]; then
+    if grep -Eq '^[[:space:]]*CXXFLAGSLOC[[:space:]]*=.*RECFASTPPPATH' "${cosmorec_makefile_in}"; then
+        sed -i -E '/^[[:space:]]*CXXFLAGSLOC[[:space:]]*=/ s|[[:space:]]+-D RECFASTPPPATH=.*$||' "${cosmorec_makefile_in}"
+        patched_cosmorec_flags=1
+    fi
+
+    if ! grep -Eq '^[[:space:]]+-D RECFASTPPPATH=' "${cosmorec_makefile_in}"; then
+        sed -i '/-D COSMORECPATH=/i\              -D RECFASTPPPATH=\\"$(PWD)/$(DEV_DIR)/Cosmology/Recfast++/\\" \\' "${cosmorec_makefile_in}"
+        patched_cosmorec_flags=1
+    fi
+fi
+
+cosmorec_makefile="${cosmorec_dir}/Makefile"
+if [[ -f "${cosmorec_makefile}" ]] && ! grep -Eq '^[[:space:]]*CXXFLAGS.*-fPIC' "${cosmorec_makefile}"; then
+    printf '\nCXXFLAGS += -fPIC\n' >> "${cosmorec_makefile}"
+    patched_cosmorec_flags=1
+fi
+
+if [[ "${patched_cosmorec_flags}" == "1" ]]; then
+    make -C "${cosmorec_dir}" tidy >/dev/null 2>&1 || true
+fi
+
+if [[ ! -d "${hyrec_dir}/.git" ]]; then
+    rm -rf "${hyrec_dir}"
+    safe_git clone --depth 1 --filter=blob:none "${hyrec_repo}" "${hyrec_dir}"
+fi
+
+chmod -R u+rwX "${hyrec_dir}" >/dev/null 2>&1 || true
+
+if [[ ! -f "${hyrec_dir}/Makefile" ]]; then
+    echo "HYREC-2 checkout at ${hyrec_dir} does not contain a Makefile" >&2
+    exit 1
+fi