Add counterfactual dataset - #615

Draft

Jiawen Sun (Jiawen-CS) wants to merge 57 commits into

mainfrom

thesis/counterfactual-dataset

Commits on Apr 14, 2026

Add counterfactual dataset (255 entries + problem statements)
Jiawen Sun
committed
5169543
Add counterfactual experiment notebooks (aligned with main, deps stubbed)
Jiawen Sun
committed
c1b37e5
Add counterfactual-evaluation category
Jiawen Sun
committed
7e581c8
Add analysis package: family models, aggregator, metrics, annotation
Jiawen Sun
committed
6bc0ea2

Commits on Apr 16, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
67a9e97
Split counterfactual-evaluation into per-variant categories (cf-1..cf-4)
Jiawen Sun
committed
ea1584a
Add cf-1..cf-4 to workflow category dropdowns
Jiawen Sun
committed
9f036f2
Auto-detect counterfactual dataset path from InstanceId
Jiawen Sun
committed
48785c1

Commits on Apr 18, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
c28de64
Fix load base intances problem
Jiawen Sun
committed
fc6f4e3
fix matrix issue
Jiawen Sun
committed
7059b6f
fix container problems
Jiawen Sun
committed
d005880
remove low quality dataset
Jiawen Sun
committed
fdf88ea
change validation action for dataset
Jiawen Sun
committed
51cba47
fix patch failed issue
Jiawen Sun
committed
48e5ed5

Commits on Apr 20, 2026

update cf3 and cf4 datasets
Jiawen Sun
committed
37a48cf
remove images
Jiawen Sun
committed
9118019
Merge branch 'main' into thesis/counterfactual-dataset
Jiawen Sun (Jiawen-CS)
authored
459f0ad
fix cf1 and cf2
Jiawen Sun
committed
a704657
Merge branch 'thesis/counterfactual-dataset' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
a526f91
impovement for cf3
Jiawen Sun
committed
d96eb2b
update dataset
Jiawen Sun
committed
61a6569
./
Jiawen Sun
committed
0e3bb7c

Commits on Apr 21, 2026

Update cf2
Jiawen Sun
committed
28279c1
./cf2
Jiawen Sun
committed
cb62056
update dataset
Jiawen Sun
committed
05ad4b3

Commits on Apr 24, 2026

fix: resolve CF COMPILE failures from run 24733495613

Jiawen Sun
and
Copilot
committed
2e26c36
Fix 3 failing CF-3 counterfactual entries

Jiawen Sun
and
Copilot
committed
503998f
fix(dataset): redesign 4 TEST_3b CF entries with real semantic mutations

Jiawen Sun
and
Copilot
committed
0b4048f
update cf2 issue
Jiawen Sun
committed
c7c1244
Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
79a7eff
fix(dataset): auto-fix 19 CF PATCH hunk headers (run 24890066865)

Jiawen Sun
and
Copilot
committed
ff320f5
Fix 14 CF test failures (11 TEST_3a + 3 TEST_3b)

Jiawen Sun
and
Copilot
committed
fe3d767
Fix 7 failing CF-3 counterfactual entries

Jiawen Sun
and
Copilot
committed
ab8716f
fix(dataset): fix 22 CF-1 entries with COMPILE/TEST/PATCH failures

Jiawen Sun
and
Copilot
committed
da8f504

Commits on Apr 25, 2026

fix(dataset): combined cf-1 fixes + auto-fix 6 cf-2 PATCH hunk headers

Jiawen Sun
and
Copilot
committed
14c0f40
fix(cf-2): fix 14 COMPILE failures in counterfactual dataset

Jiawen Sun
and
Copilot
committed
b8c9b8c
fix(cf-2): fix 13 TEST_3a failures in counterfactual dataset

Jiawen Sun
and
Copilot
committed
c1d868e
fix(dataset): convert 6 cf-2 entries with list-typed patch/test_patch to strings

Jiawen Sun
and
Copilot
committed
e4305cd

Commits on Apr 26, 2026

fix(dataset): fix 10 CF entries with corrupt/mismatched patches

Jiawen Sun
and
Copilot
committed
496394b

Commits on Apr 27, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
86bf0bc

Commits on Apr 28, 2026

fix(dataset): remove 82 failed CF entries and renumber variants

Jiawen Sun
and
Copilot
committed
59e7e00
Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
40cbd60

Commits on May 1, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
7788e43
remove problem_statement_override
Jiawen Sun
committed
2dab94d

Commits on May 3, 2026

combine cfs as one
Jiawen Sun
committed
974bdc9
remove dataset
Jiawen Sun
committed
b7235e3
change readme.md
Jiawen Sun
committed
d83f67a

Commits on May 26, 2026

Merge branch 'main' of https://github.com/microsoft/BC-Bench into thesis/counterfactual-dataset
Jiawen Sun
committed
3baa38a

Commits on May 28, 2026

experment tests for cf
Jiawen Sun
committed
3bcd28d

Commits on Jun 20, 2026

update experments
Jiawen Sun
committed
b688070
prepare for exp4
Jiawen Sun
committed
55e72cd

Commits on Jun 21, 2026

Merge branch 'main' into thesis/counterfactual-dataset

Jiawen Sun
and
Copilot
committed
428c48b
Fix copilot-evaluation matrix skip after main merge

Jiawen Sun
and
Copilot
committed
e01b63b
Fix cf container setup: resolve dataset path per-instance

Jiawen Sun
and
Copilot
committed
52dd2c5
Fix cf container name: strip __cf suffix

Jiawen Sun
and
Copilot
committed
e2bdc06
Skip bceval upload and leaderboard for cf in summarize-results

Jiawen Sun
and
Copilot
committed
5108a9a