Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
efb5168
test(125): add failing tests for mount infrastructure (RED phase)
azchin Mar 17, 2026
e5a28d3
feat(125): add volume mounts to templates and update renderer context
azchin Mar 17, 2026
da2f9d8
feat(125): add OSS_CRS_FUZZ_PROJ and OSS_CRS_TARGET_SOURCE env vars t…
azchin Mar 17, 2026
1370a98
test(125): add failing tests for WorkDir.get_target_source_dir and Ta…
azchin Mar 17, 2026
24ae0d5
feat(125): add WorkDir.get_target_source_dir and Target WORKDIR extra…
azchin Mar 17, 2026
77fb1be
feat(125): make OSS_CRS_TARGET_SOURCE unconditional in env_policy and…
azchin Mar 17, 2026
5a75368
feat(125): wire WORKDIR extraction into build flow and update renderers
azchin Mar 17, 2026
5bddd77
fix(125): add _has_repo and extract_workdir_to_host to test mocks
azchin Mar 17, 2026
dbd2467
feat(125): replace SourceType enum and rewrite download_source with m…
azchin Mar 18, 2026
7088446
docs(125): update libCRS.md download-source section to mount-based API
azchin Mar 18, 2026
a0e7627
docs(125): update crs-development-guide.md and libCRS/README.md for n…
azchin Mar 18, 2026
4d928e5
chore: gitignore
azchin Mar 19, 2026
eb9914c
chore: temp bump crs-codex for new download-source
azchin Mar 19, 2026
bd74086
docs: changelog
azchin Mar 19, 2026
cbaa019
chore: linter
azchin Mar 20, 2026
ff8f0d9
merge main into refactor/124-download-source (resolve litellm/postgre…
0xdkay Apr 2, 2026
66a6d3f
test(133): add edge case and cross-layer contract tests for download-…
0xdkay Apr 3, 2026
db15748
Merge remote-tracking branch 'origin/main' into refactor/124-download…
azchin Apr 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ tags
.planning
.oss-crs-workdir/
third_party/
.planning/
18 changes: 13 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ stricter subset of Keep a Changelog).
- roboduck to registry/ and example/
- fuzzing-brain to registry/ and example/ (bug-finding, C/C++, multi-provider LLM)
- buttercup-seed-gen to registry/ and example/
- `libCRS download-source fuzz-proj <dest>`: copies clean fuzz project
- `libCRS download-source target-source <dest>`: copies clean target source

### Changed
- Clarified that target env `repo_path` is the effective in-container source
Expand All @@ -25,11 +27,14 @@ stricter subset of Keep a Changelog).
- `OSS_CRS_REPO_PATH` resolution is documented as: final `WORKDIR` -> `$SRC` ->
`/src` fallback chain.
- Target build-option resolution now uses precedence:
CLI `--sanitizer` flag -> `additional_env` override (SANITIZER at CRS-entry scope)
-> `project.yaml` fallback (uses address if provided, else first)
CLI `--sanitizer` flag -> `additional_env` override (SANITIZER at CRS-entry scope)
-> `project.yaml` fallback (uses address if provided, else first)
-> framework defaults.
- `artifacts --sanitizer` is now optional; when omitted, sanitizer is resolved
using the same contract (compose/project/default) used by build/run flows.
- **Breaking:** `libCRS download-source` API replaced — `target`/`repo`
subcommands removed, use `fuzz-proj`/`target-source` instead. Python API
`download_source()` now returns `None` instead of `Path`.

### Deprecated
- Deprecated CLI aliases:
Expand All @@ -40,14 +45,17 @@ stricter subset of Keep a Changelog).

### Removed
- Removed legacy CLI alias `--target-repo-path`; use `--target-source-path`.
- Removed `libCRS download-source target` and `download-source repo` commands.
- Removed `SourceType.TARGET` and `SourceType.REPO` enum values from libCRS.
- Removed ~140 lines of fallback resolution logic from libCRS
(`_resolve_repo_source_path`, `_normalize_repo_source_path`,
`_translate_repo_hint_to_build_output`, `_resolve_downloaded_repo_path`,
`_relative_repo_hint`).

### Fixed
- The local run path now passes a `Path` compose-file object consistently into
`docker_compose_up()`, so helper-sidecar teardown classification applies on
the main local run path.
- `libCRS download-source repo` now prefers the live runtime source workspace
(`$SRC` / `/src`) over build-output snapshots and preserves workspace layout
consistently for nested `WORKDIR` targets.

### Security
- N/A
3 changes: 3 additions & 0 deletions docs/crs-development-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,8 @@ Your containers receive these environment variables automatically:
| `ARCHITECTURE` | Target architecture | `x86_64` |
| `FUZZING_LANGUAGE` | Target language | `c` |
| `OSS_CRS_SNAPSHOT_IMAGE` | Snapshot Docker image tag (set on non-builder modules that are not `run_snapshot` when a snapshot tag exists for the run) | `my-crs-snapshot-asan:latest` |
| `OSS_CRS_FUZZ_PROJ` | Fuzz project mount path (read-only) | `/OSS_CRS_FUZZ_PROJ` |
| `OSS_CRS_TARGET_SOURCE` | Target source mount path (read-only) | `/OSS_CRS_TARGET_SOURCE` |

Notes:
- `additional_env` keys are validated with pattern `[A-Za-z_][A-Za-z0-9_]*`.
Expand All @@ -337,6 +339,7 @@ Use these with any OpenAI-compatible client library. All requests are proxied th
# Build outputs
libCRS submit-build-output <src_path> <dst_path>
libCRS download-build-output <src_path> <dst_path>
libCRS download-source <type> <dst_path> # type: fuzz-proj, target-source
libCRS skip-build-output <dst_path>

# Automatic directory submission (runs as daemon)
Expand Down
21 changes: 14 additions & 7 deletions docs/design/libCRS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ libCRS relies on several environment variables injected by CRS Compose at contai
| `OSS_CRS_LOG_DIR` | Writable filesystem path for persisting CRS agent/internal logs to the host |
| `OSS_CRS_FETCH_DIR` | Read-only filesystem path for fetching inter-CRS data and bootup data (set on run containers, and on build-target builder containers when directed inputs are provided) |
| `OSS_CRS_SNAPSHOT_IMAGE` | Docker image tag of the snapshot (set on non-builder modules that are not `run_snapshot` when a snapshot tag exists for the run) |
| `OSS_CRS_FUZZ_PROJ` | Read-only mount containing the fuzz project directory (set on all CRS containers) |
| `OSS_CRS_TARGET_SOURCE` | Read-only mount containing the target source directory (set on all CRS containers) |

## Architecture

Expand Down Expand Up @@ -152,21 +154,26 @@ $ libCRS download-build-output /fuzzer /opt/fuzzer

#### `download-source` ✅

Download target source tree into the container from canonical source paths.
Download source tree into the container from mount paths.

```bash
$ libCRS download-source <target|repo> <dst_path>
$ libCRS download-source <fuzz-proj|target-source> <dst_path>
```

| Argument | Description |
|---|---|
| `target` | Source from `OSS_CRS_PROJ_PATH` (project files) |
| `repo` | Source from the live repo workspace rooted at `$SRC` / `/src`, using `OSS_CRS_REPO_PATH` as the effective workdir hint |
| `fuzz-proj` | Copy from `/OSS_CRS_FUZZ_PROJ` (fuzz project mount) |
| `target-source` | Copy from `/OSS_CRS_TARGET_SOURCE` (target source mount) |
| `dst_path` | Destination path inside the container |

**Example** — Retrieve effective source tree for patch analysis:
**Example** — Copy fuzz project files for analysis:
```bash
$ libCRS download-source repo /work/src
$ libCRS download-source fuzz-proj /work/project
```

**Example** — Copy target source tree for patch generation:
```bash
$ libCRS download-source target-source /work/src
```

#### `skip-build-output` ✅
Expand Down Expand Up @@ -518,7 +525,7 @@ libCRS submit patch /tmp/patch.diff
|---|---|---|
| `submit-build-output` | ✅ Implemented | Uses `rsync` for file copying |
| `download-build-output` | ✅ Implemented | Uses `rsync` for file copying |
| `download-source` | ✅ Implemented | Copies from `OSS_CRS_PROJ_PATH` or `OSS_CRS_REPO_PATH` |
| `download-source` | ✅ Implemented | Copies from `/OSS_CRS_FUZZ_PROJ` or `/OSS_CRS_TARGET_SOURCE` mounts |
| `skip-build-output` | ✅ Implemented | Creates `.skip` sentinel file |
| `register-submit-dir` | ✅ Implemented | Daemon with `watchdog` + batch submission |
| `register-shared-dir` | ✅ Implemented | Symlink-based sharing |
Expand Down
2 changes: 1 addition & 1 deletion libCRS/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ This installs the `libCRS` CLI at `/usr/local/bin/libCRS`.

| Category | Commands |
|---|---|
| Build output | `submit-build-output`, `download-build-output`, `skip-build-output` |
| Build output | `submit-build-output`, `download-build-output`, `download-source`, `skip-build-output` |
| Data exchange | `register-submit-dir`, `register-fetch-dir`, `submit`, `fetch` |
| Shared filesystem | `register-shared-dir` |
| Log persistence | `register-log-dir` |
Expand Down
14 changes: 5 additions & 9 deletions libCRS/libCRS/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ def dir_name(self) -> str:


class SourceType(str, Enum):
TARGET = "target"
REPO = "repo"
FUZZ_PROJ = "fuzz-proj"
TARGET_SOURCE = "target-source"

def __str__(self) -> str:
return self.value
Expand All @@ -47,16 +47,12 @@ def download_build_output(self, src_path: str, dst_path: Path) -> None:
pass

@abstractmethod
def download_source(self, source_type: SourceType, dst_path: Path) -> Path:
def download_source(self, source_type: SourceType, dst_path: Path) -> None:
"""Download source tree to local destination path.

source_type:
- target: OSS_CRS_PROJ_PATH
- repo: OSS_CRS_REPO_PATH

Returns:
The primary source directory inside ``dst_path`` that callers should
operate on after the copy completes.
- fuzz-proj: /OSS_CRS_FUZZ_PROJ (fuzz project mount)
- target-source: /OSS_CRS_TARGET_SOURCE (target source mount)
"""
pass

Expand Down
4 changes: 2 additions & 2 deletions libCRS/libCRS/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,14 +118,14 @@ def main():
# download-source command
download_source_parser = subparsers.add_parser(
"download-source",
help="Download source tree from target/repo source path to destination",
help="Download source tree from mount path to destination",
)
download_source_parser.add_argument(
"type",
type=SourceType,
choices=list(SourceType),
metavar="TYPE",
help="Source type: target, repo",
help="Source type: fuzz-proj, target-source",
)
download_source_parser.add_argument(
"dst_path",
Expand Down
179 changes: 11 additions & 168 deletions libCRS/libCRS/local.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import logging
import os
import socket
import tempfile
import time
Expand All @@ -14,6 +13,9 @@

logger = logging.getLogger(__name__)

_FUZZ_PROJ_MOUNT = Path("/OSS_CRS_FUZZ_PROJ")
_TARGET_SOURCE_MOUNT = Path("/OSS_CRS_TARGET_SOURCE")


def _get_build_out_dir() -> Path:
return Path(get_env("OSS_CRS_BUILD_OUT_DIR"))
Expand All @@ -35,176 +37,17 @@ def download_build_output(self, src_path: str, dst_path: Path) -> None:
dst = Path(dst_path)
rsync_copy(src, dst)

def download_source(self, source_type: SourceType, dst_path: Path) -> Path:
if source_type == SourceType.TARGET:
env_key = "OSS_CRS_PROJ_PATH"
try:
src = Path(get_env(env_key)).resolve()
except KeyError as exc:
raise RuntimeError(
f"download-source {source_type} requires env var {env_key}"
) from exc
if not src.exists():
raise RuntimeError(
f"download-source {source_type} source path does not exist: {src}"
)
resolved_dst = Path(dst_path).resolve()
elif source_type == SourceType.REPO:
src = self._resolve_repo_source_path()
resolved_dst = self._resolve_downloaded_repo_path(src, Path(dst_path))
def download_source(self, source_type: SourceType, dst_path: Path) -> None:
if source_type == SourceType.FUZZ_PROJ:
src = _FUZZ_PROJ_MOUNT
elif source_type == SourceType.TARGET_SOURCE:
src = _TARGET_SOURCE_MOUNT
else:
raise ValueError(f"Unsupported source type: {source_type}")

dst = Path(dst_path).resolve()

# Guard against recursive/self copy (rsync source == destination or overlaps).
if dst == src or src in dst.parents or dst in src.parents:
raise RuntimeError(
f"Refusing download-source copy due to overlapping paths: src={src}, dst={dst}"
)

rsync_copy(src, dst)
return resolved_dst

def _resolve_repo_source_path(self) -> Path:
"""Resolve the repo source tree for ``download-source repo``.

``OSS_CRS_REPO_PATH`` is treated as a hint describing the effective repo
workdir. When that hint lives under the runtime source workspace
(``$SRC`` / ``/src``), return the workspace root so callers see the same
layout regardless of nested WORKDIR usage. If the hint is stale or
missing, prefer the live runtime source workspace and only fall back to
build-output snapshots as a last resort.

The returned path preserves the framework's source-workspace layout when
the repo is rooted under ``$SRC`` so nested targets do not change the
shape of the downloaded tree depending on whether the repo hint exists.
"""
repo_hint = os.environ.get("OSS_CRS_REPO_PATH")
runtime_root = Path(os.environ.get("SRC", "/src")).resolve()
build_out_dir = os.environ.get("OSS_CRS_BUILD_OUT_DIR") or None
build_out_src = (
Path(build_out_dir).resolve() / "src" if build_out_dir is not None else None
)
if repo_hint:
hinted_src = Path(repo_hint).resolve()
if hinted_src.exists():
normalized_hint = self._normalize_repo_source_path(
hinted_src, runtime_root
)
if normalized_hint is not None:
return normalized_hint
return hinted_src
else:
hinted_src = None

search_roots: list[Path] = [runtime_root]

for search_root in search_roots:
normalized_root = self._normalize_repo_source_path(
search_root, runtime_root
)
if normalized_root is None:
continue
return normalized_root

if build_out_src is not None:
if hinted_src is not None:
translated_hint = self._translate_repo_hint_to_build_output(
hinted_src, runtime_root, build_out_src
)
if translated_hint is not None:
return translated_hint
normalized_build_out = self._normalize_repo_source_path(
build_out_src, build_out_src
)
if normalized_build_out is not None:
return normalized_build_out

if hinted_src is not None:
raise RuntimeError(
f"download-source repo source path does not exist: {hinted_src}"
)
raise RuntimeError(
"download-source repo requires OSS_CRS_REPO_PATH or a git repository "
f"under {runtime_root} or {Path(build_out_dir).resolve() / 'src' if build_out_dir else '$OSS_CRS_BUILD_OUT_DIR/src'}"
)

def _normalize_repo_source_path(
self, candidate: Path, source_root: Path
) -> Path | None:
candidate = candidate.resolve()
source_root = source_root.resolve()
if not candidate.exists() or not source_root.exists():
return None

if not candidate.is_relative_to(source_root):
return candidate if candidate.exists() else None

git_dir = next(source_root.rglob(".git"), None)
if git_dir is not None:
return source_root

if candidate != source_root:
return candidate if candidate.exists() else None

return None

def _translate_repo_hint_to_build_output(
self, hinted_src: Path, runtime_root: Path, build_out_src: Path | None
) -> Path | None:
if build_out_src is None or not build_out_src.exists():
return None

relative_hint: Path | None = None
for source_root in (runtime_root, Path("/src").resolve()):
if hinted_src.is_relative_to(source_root):
relative_hint = hinted_src.relative_to(source_root)
break
if relative_hint is None:
return None

translated = build_out_src / relative_hint
if not translated.exists():
return None

normalized = self._normalize_repo_source_path(translated, build_out_src)
if normalized is not None:
return normalized
return translated

def _resolve_downloaded_repo_path(self, resolved_src: Path, dst_path: Path) -> Path:
dst = dst_path.resolve()
repo_hint = os.environ.get("OSS_CRS_REPO_PATH")
if not repo_hint:
return dst

hinted_src = Path(repo_hint).resolve()
runtime_root = Path(os.environ.get("SRC", "/src")).resolve()
build_out_dir = os.environ.get("OSS_CRS_BUILD_OUT_DIR") or None
build_out_src = (
Path(build_out_dir).resolve() / "src" if build_out_dir is not None else None
)

relative_hint = self._relative_repo_hint(hinted_src, runtime_root)
if relative_hint is None:
return dst

source_roots = [runtime_root, Path("/src").resolve()]
if build_out_src is not None:
source_roots.append(build_out_src.resolve())

if any(resolved_src == source_root for source_root in source_roots):
return (dst / relative_hint).resolve()
return dst

def _relative_repo_hint(self, hinted_src: Path, runtime_root: Path) -> Path | None:
for source_root in (runtime_root.resolve(), Path("/src").resolve()):
if hinted_src == source_root:
return Path()
if hinted_src.is_relative_to(source_root):
return hinted_src.relative_to(source_root)
return None
if not src.exists():
raise RuntimeError(f"download-source {source_type}: {src} does not exist")
rsync_copy(src, Path(dst_path))

def submit_build_output(self, src_path: str, dst_path: Path) -> None:
src = Path(src_path)
Expand Down
Loading
Loading