Provide the ability to create non repo based datasets - #761
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Enables repository-independent dataset categories while preserving repository-backed evaluation behavior.
Changes:
- Splits dataset entries into repository-grounded and repository-independent types.
- Adds category-driven repository cloning controls to evaluation workflows.
- Decouples instruction and skill profiles from repository names.
Reviewed changes
Copilot reviewed 24 out of 26 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_custom_instructions.py |
Tests customization profiles. |
tests/test_category_command.py |
Tests repository requirements. |
tests/test_agent_skills.py |
Updates skill profile tests. |
tests/conftest.py |
Makes NL2AL fixtures repository-independent. |
src/bcbench/types.py |
Derives category repository requirements. |
src/bcbench/operations/skills_operations.py |
Resolves skills by profile. |
src/bcbench/operations/setup_operations.py |
Restricts repository setup to grounded entries. |
src/bcbench/operations/instruction_operations.py |
Resolves customizations by profile. |
src/bcbench/dataset/dataset_entry.py |
Introduces RepoGroundedEntry. |
src/bcbench/dataset/codereview.py |
Marks code review as repository-grounded. |
src/bcbench/dataset/__init__.py |
Exports the new entry type. |
src/bcbench/contamination/runner.py |
Narrows contamination inputs. |
src/bcbench/contamination/filepath_identification.py |
Requires patch-backed entries. |
src/bcbench/commands/dataset.py |
Handles repository-specific fields conditionally. |
src/bcbench/commands/contamination.py |
Validates repository-grounded entries. |
src/bcbench/commands/category.py |
Emits repository requirements. |
scripts/Setup-ContainerAndRepository.ps1 |
Adds optional clone skipping. |
EXPERIMENT.md |
Documents customization profiles. |
dataset/nl2al_quarantine.jsonl |
Removes repository-only fields. |
dataset/nl2al_challenge.jsonl |
Removes repository-only fields. |
CATEGORIES.md |
Documents requires_repo. |
.github/workflows/get-entries.yml |
Exposes repository requirements. |
.github/workflows/copilot-evaluation.yml |
Skips unnecessary clones. |
.github/workflows/claude-evaluation.yml |
Skips unnecessary clones. |
.github/actions/setup-bc-container-repo/action.yml |
Adds the skip-repo input. |
Onat Buyukakkus (onbuyuka)
pushed a commit
that referenced
this pull request
Jul 29, 2026
Address @haoranpb's review by leaning the harness toward "fail loud, less machinery": - Drop the gold-withholding contamination guard (working-tree strip + .git relocation). The agent runs under repo_path, so the app-level guard is unnecessary complexity; real isolation is a runtime concern. - Replace the "unscorable" result state with fail-loud semantics: a gold query that does not compile/run now raises (turning the run red) instead of being silently excluded, since it is a harness/dataset bug that must be fixed. An empty agent output is still tracked as a build failure. Removed the `scorable` flag and its summary/bceval-export exclusions. - Bump environment_setup_version 26.0 -> 29.0 across the dataset. - Simplify the data-query prompt to the minimal harness contract (write one query object to query.al, no API properties) and drop the AL tutorial bullets. - Remove the now-obsolete gold-withholding unit tests. Threads 1-3 (skip-repo as a category property, optional repo field) are handled generically by #761; this PR will take those up on rebase after it merges. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
Magnus Hartvig Grønbech (Groenbech96)
approved these changes
Jul 30, 2026
Sun Haoran (haoranpb)
pushed a commit
that referenced
this pull request
Jul 30, 2026
Address @haoranpb's review by leaning the harness toward "fail loud, less machinery": - Drop the gold-withholding contamination guard (working-tree strip + .git relocation). The agent runs under repo_path, so the app-level guard is unnecessary complexity; real isolation is a runtime concern. - Replace the "unscorable" result state with fail-loud semantics: a gold query that does not compile/run now raises (turning the run red) instead of being silently excluded, since it is a harness/dataset bug that must be fixed. An empty agent output is still tracked as a build failure. Removed the `scorable` flag and its summary/bceval-export exclusions. - Bump environment_setup_version 26.0 -> 29.0 across the dataset. - Simplify the data-query prompt to the minimal harness contract (write one query object to query.al, no API properties) and drop the AL tutorial bullets. - Remove the now-obsolete gold-withholding unit tests. Threads 1-3 (skip-repo as a category property, optional repo field) are handled generically by #761; this PR will take those up on rebase after it merges. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
Sun Haoran (haoranpb)
added a commit
that referenced
this pull request
Jul 30, 2026
Marko Aleksandric (AleksandricMarko)
added a commit
that referenced
this pull request
Jul 31, 2026
…ding Applies Haoran's review feedback for the extensibility-request categories: - Rename to explicit `extensibility-request-implement` / `extensibility-request-triage` across category ids, enum members, classes, dataset files and module files (#1/#2/#4). - Slim both prompt templates to reflect real-world usage; category-specific rules now live in the ai-ext-fix / argus-triage skills (#3). - Merge duplicate `bcbench.results` import in commands/evaluate.py (#5). - Uptake #761: both ext entries subclass `RepoGroundedEntry` (#6/#7). - Triage `patch` defaults to None; stripped from the dataset (#8). - Constrain triage label fields with a `ManagedLabel` Literal to catch dataset typos (#9). - Move the ext dataset classes into dataset/extensibility_request.py, like code review (#10). - Drop `fetch_commit_if_missing` from both ext pipelines to match the other pipelines (#11). - Keep ext-implement judge-only for now; container build/publish noted as future work (#12). - Grade triage with the NL2AL LMChecklist instead of the code-review-style hybrid: `ExtRequestTriageResult` removed in favour of `JudgeBasedEvaluationResult`, expected_* fields replaced by an `expected` checklist, evaluators `lm_checklist`, core score `test_passed`; 17 triage entries migrated (#13/#14/#15/#16). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inspired by #740 , BC-Bench should support categories that do NOT need a repository.
The PR mostly consists of 2 parts:
A tricky issue had to address: the folders to store custom instructions, skills were repo-based, had to create an abstract function for non-repo-based categories.