Provide the ability to create non repo based datasets

.github/actions/setup-bc-container-repo/action.yml

@@ -21,6 +21,10 @@ inputs:
     description: Skip BC container setup (only clone repository)
     required: false
     default: "false"
+  skip-repo:
+    description: Skip cloning the dataset repository (workspace is scaffolded by the pipeline)
+    required: false
+    default: "false"
 
 outputs:
   repo_path:
@@ -57,6 +61,7 @@ runs:
       shell: pwsh
 
     - name: Azure Login with OIDC for cloning internal repository
+      if: inputs.skip-repo != 'true'
       uses: azure/login@v3
       with:
         client-id: ${{ inputs.azure-client-id }}
@@ -87,8 +92,10 @@ runs:
         Write-Output "::add-mask::$env:GITHUB_TOKEN"
 
         # Get Azure DevOps access token via OIDC
-        $env:ADO_TOKEN = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query accessToken -o tsv
-        Write-Output "::add-mask::$env:ADO_TOKEN"
+        if ('${{ inputs.skip-repo }}' -ne 'true') {
+          $env:ADO_TOKEN = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query accessToken -o tsv
+          Write-Output "::add-mask::$env:ADO_TOKEN"
+        }
 
-        .\scripts\Setup-ContainerAndRepository.ps1 -InstanceId "${{ inputs.instance-id }}" -Category "${{ inputs.category }}" ${{ inputs.skip-container == 'true' && '-SkipContainer' || '' }}
+        .\scripts\Setup-ContainerAndRepository.ps1 -InstanceId "${{ inputs.instance-id }}" -Category "${{ inputs.category }}" ${{ inputs.skip-container == 'true' && '-SkipContainer' || '' }} ${{ inputs.skip-repo == 'true' && '-SkipRepo' || '' }}
       shell: pwsh

.github/workflows/claude-evaluation.yml

@@ -104,6 +104,7 @@ jobs:
           azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
           github-token: ${{ secrets.GITHUB_TOKEN }}
           skip-container: ${{ needs.get-entries.outputs.requires-container != 'true' }}
+          skip-repo: ${{ needs.get-entries.outputs.requires-repo != 'true' }}
 
       - name: Setup Python with UV
         uses: ./.github/actions/setup-python-uv

.github/workflows/copilot-evaluation.yml

@@ -111,6 +111,7 @@ jobs:
           azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
           github-token: ${{ secrets.GITHUB_TOKEN }}
           skip-container: ${{ needs.get-entries.outputs.requires-container != 'true' }}
+          skip-repo: ${{ needs.get-entries.outputs.requires-repo != 'true' }}
 
       - name: Setup Python with UV
         uses: ./.github/actions/setup-python-uv

.github/workflows/get-entries.yml

@@ -30,6 +30,9 @@ on:
       requires-container:
         description: Whether this category needs a BC container ("true"/"false")
         value: ${{ jobs.get-entries.outputs.requires-container }}
+      requires-repo:
+        description: Whether this category needs the dataset repository cloned ("true"/"false")
+        value: ${{ jobs.get-entries.outputs.requires-repo }}
 
 jobs:
   get-entries:
@@ -38,6 +41,7 @@ jobs:
       entries: ${{ steps.get-entries.outputs.entries }}
       runner: ${{ steps.runtime-config.outputs.runner }}
       requires-container: ${{ steps.runtime-config.outputs.requires-container }}
+      requires-repo: ${{ steps.runtime-config.outputs.requires-repo }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v5

CATEGORIES.md

@@ -18,7 +18,7 @@ Start with `EvaluationCategory` in [src/bcbench/types.py](src/bcbench/types.py).
 - `summary_class` / `aggregate_class` — the aggregate views used by result summaries and leaderboards.
 - `pipeline` — the category-specific setup, agent run, and evaluation behavior.
 - `evaluators` / `core_score` — the bc-eval evaluator list and headline score, emitted to workflows by [src/bcbench/commands/category.py](src/bcbench/commands/category.py).
-- `requires_container` / `runner` — whether the category needs a BC container and which runner evaluates it.
+- `requires_container` / `requires_repo` / `runner` — whether the category needs a BC container, whether it needs the dataset repository cloned, and which runner evaluates it.
 - Prompt template — the category-specific prompt in [src/bcbench/agent/shared/config.yaml](src/bcbench/agent/shared/config.yaml), loaded by [src/bcbench/agent/shared/prompt.py](src/bcbench/agent/shared/prompt.py).
 
 Keep dataset entry classes and result classes focused on typed data. Put category-specific behavior in the pipeline.

EXPERIMENT.md

@@ -20,17 +20,17 @@ All configurations live in [`config.yaml`](src/bcbench/agent/shared/config.yaml)
 
 | Setting | Default | What it does |
 |---|---|---|
-| `instructions.enabled` | `false` | Copy the **entire** `instructions/<owner>-<repo>/` folder (instructions + skills + agents) into the target repo before running the agent |
-| `skills.enabled` | `false` | Copy **only** `instructions/<owner>-<repo>/skills/` |
-| `agents.enabled` and `agents.name` | `false` | Copy **only** `instructions/<owner>-<repo>/agents/` and pass `--agent=<name>` to the CLI |
+| `instructions.enabled` | `false` | Copy the **entire** `instructions/<profile>/` folder (instructions + skills + agents) into the target repo before running the agent |
+| `skills.enabled` | `false` | Copy **only** `instructions/<profile>/skills/` |
+| `agents.enabled` and `agents.name` | `false` | Copy **only** `instructions/<profile>/agents/` and pass `--agent=<name>` to the CLI |
 | `mcp.servers` | _(none)_ | List of MCP servers to register |
 | `plugins` | _(all disabled)_ | List of agent plugins to load for the run — one entry per plugin, local or cloned from GitHub at a revision, passed to the CLI via `--plugin-dir` |
 
 Note: `instructions.enabled: true` is a superset — you don't also need to enable `skills` or `agents` to get them. Use `skills`/`agents` when you want to isolate the effect of just that piece.
 
 ### Custom instructions / skills / custom agents
 
-Files live under `src/bcbench/agent/shared/instructions/<owner>-<repo>/`. The folder name mirrors the dataset's repo path with `/` replaced by `-` (e.g. `microsoft/BCApps` -> `microsoft-BCApps`).
+Files live under `src/bcbench/agent/shared/instructions/<profile>/`, where `<profile>` is the dataset entry's `customization_profile`. Repo-grounded categories derive it from the repo path with `/` replaced by `-` (e.g. `microsoft/BCApps` -> `microsoft-BCApps`), which reproduces the customization a developer would already have checked in. Categories that scaffold their own workspace and have no repo (e.g. `nl2al`) name their own folder and place it alongside the repo-keyed ones.
 
 The files checked in today are **placeholders**. Replace them with whatever you want to test — your own AGENTS.md, your own skills, your own agent definitions — then toggle the corresponding flag in `config.yaml`.
 
@@ -76,7 +76,7 @@ Loading a plugin makes its capabilities **available** — it does not guarantee
 - **MCP servers / hooks are non-discretionary.** An MCP server's tools and a plugin's hooks are loaded every run and exercised automatically (a `SessionStart` hook can even inject context). Nothing extra is needed to test these.
 - **Skills are discretionary.** The agent *sees* the loaded skills (they appear in the model's available-skills list, verified — including task-relevant ones like `systematic-debugging` for a bug-fix), but only invokes one when it judges it worthwhile. On a well-specified task (bug-fix, code-review) it typically just does the work directly and invokes nothing. So to test a **skill** plugin you must *encourage* usage.
 
-To encourage a skill, use the **custom instructions** lever (`instructions` toggle → the repo's `AGENTS.md`): even a light nudge flips skill usage on. Append a subtle nudge like the one below to the target repo's `AGENTS.md` (under `src/bcbench/agent/shared/instructions/<owner>-<repo>/`) and set `instructions.enabled: true`:
+To encourage a skill, use the **custom instructions** lever (`instructions` toggle → the repo's `AGENTS.md`): even a light nudge flips skill usage on. Append a subtle nudge like the one below to the target repo's `AGENTS.md` (under `src/bcbench/agent/shared/instructions/<profile>/`) and set `instructions.enabled: true`:
 
 ```md
 ## Using your skills