Extensibility requests triage + implementation - #762
Merged
Marko Aleksandric (AleksandricMarko) merged 16 commits intoAug 10, 2026
Conversation
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
July 29, 2026 20:20
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds benchmark support for implementing and triaging Business Central extensibility requests.
Changes:
- Adds
ext-implementandext-triagedatasets, pipelines, results, and category wiring. - Adds extensibility agent skills, triage phases, rules, and prompts.
- Updates tests, documentation, workflows, and agent configuration.
Reviewed changes
Copilot reviewed 60 out of 60 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_type_exhaustiveness.py |
Covers new entry types |
tests/test_agent_skills.py |
Uses explicit UTF-8 |
tests/conftest.py |
Adds extensibility fixtures |
src/bcbench/types.py |
Registers new categories |
src/bcbench/results/extriage.py |
Adds triage results |
src/bcbench/results/__init__.py |
Exports triage result |
src/bcbench/evaluate/extriage.py |
Implements triage pipeline |
src/bcbench/evaluate/extimplement.py |
Implements request pipeline |
src/bcbench/evaluate/__init__.py |
Exports new pipelines |
src/bcbench/dataset/dataset_entry.py |
Adds entry schemas |
src/bcbench/dataset/__init__.py |
Exports entry schemas |
src/bcbench/commands/evaluate.py |
Adds mock scenarios |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/templates/result-contract.md |
Defines triage output |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md |
Defines triage skill |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/shared-rules.md |
Defines shared rules |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/7-finalize.md |
Defines finalization |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/6-team-assignment.md |
Defines team assignment |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md |
Defines code analysis |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/4-requirements.md |
Defines requirements checks |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/3-classify.md |
Defines classification |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/2b-bcapps-check.md |
Defines BCApps check |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/2-eligibility.md |
Defines eligibility |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/1-preprocess.md |
Defines preprocessing |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/0-startup.md |
Defines startup checks |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/orchestrator.md |
Orchestrates triage phases |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/team-configuration/team_namespace_mapping.yaml |
Maps namespaces to teams |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/request_for_external_requirements.yaml |
Adds external requirements |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/general_requirements.yaml |
Adds general requirements |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/extensibility_enhancement_requirements.yaml |
Adds enhancement requirements |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml |
Adds event requirements |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml |
Adds enum requirements |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/comment-templates/comment_templates.yaml |
Adds response templates |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/request_for_external_implementation.yaml |
Adds accessibility rules |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/general_blockers.yaml |
Adds general blockers |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_warnings.yaml |
Adds event warnings |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_warnings.yaml |
Adds IsHandled warnings |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_implementation.yaml |
Adds IsHandled guidance |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_blockers.yaml |
Adds IsHandled blockers |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_alternative_suggestions.yaml |
Adds IsHandled alternatives |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_implementation.yaml |
Adds event guidance |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_alternative_suggestions.yaml |
Adds event alternatives |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/compatibility.md |
Maps host capabilities |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/ai-ext-fix/SKILL.md |
Adds implementation skill |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/ai-ext-fix/guidelines.md |
Adds AL event guidelines |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/argus-triage.agent.md |
Adds triage agent |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/argus-codebase-analysis.agent.md |
Adds analysis sub-agent |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/ai-ext-fix.agent.md |
Adds implementation agent |
src/bcbench/agent/shared/config.yaml |
Adds category prompts |
src/bcbench/agent/copilot/agent.py |
Expands logging |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30377/README.md |
Adds report request |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30362/README.md |
Adds Service-Post request |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30361/README.md |
Adds archive request |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30346/README.md |
Adds accessibility request |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30336/README.md |
Adds routing-line request |
dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30223/README.md |
Adds Purchase Line request |
dataset/extriage.jsonl |
Adds triage benchmark entries |
CATEGORIES.md |
Documents new categories |
.github/workflows/copilot-evaluation.yml |
Exposes implementation category |
.github/workflows/claude-evaluation.yml |
Exposes implementation category |
Comments suppressed due to low confidence (3)
src/bcbench/evaluate/extriage.py:145
- These fields come from unconstrained agent JSON, but the code assumes
labels_to_setislist[str]and coerces arbitrary comment/state values withstr(). For example,labels_to_set: 1raisesTypeErrorin_normalize_labelsand aborts the entry. Validate the fullFinal_Outputcontract before grading and save a failed result for invalid field types.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:103 flexible_validationandvalidation_notesare over-indented relative to the requirement item, making this YAML invalid rather than attaching the relaxed policy todata_sensitivity_review.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:118flexible_validationandvalidation_notesare over-indented relative to the requirement item, making this YAML invalid rather than attaching the relaxed policy tomulti_extension_interaction.
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
July 29, 2026 20:28
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 60 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (8)
src/bcbench/agent/shared/config.yaml:84
- This prompt requires
argus-triage, but the same agent config hasskills.enabled: false, sosetup_agent_skills()never installs that skill into the target checkout. Both Copilot and Claude will therefore be asked to run a skill that is unavailable. Enable/install the skill for this category, or make the prompt self-contained.
Task: Run the `argus-triage` skill end to end on the extensibility request below and produce its
triage decision. Follow the skill's phases (preprocess, eligibility, classify, requirements,
src/bcbench/agent/shared/config.yaml:70
- The NAV checkout and every new problem statement use
App/Layers/..., notsrc/Layers/.... This instruction prevents agents from recognizing W1 files and propagating changes to localization counterparts.
- W1-first layering: when the file to change exists under `src/Layers/W1/...`, change W1 first, then apply
the same change to every same-named counterpart file in the other country/region layers. If the file
exists only in a specific country layer, edit that layer directly.
src/bcbench/dataset/dataset_entry.py:224
- The offline task omits metadata required by the mandated eligibility phases: current issue state/type, author,
updatedAt, and dated comment authorship. In particular, entries already labeledmissing-infocannot evaluate the “bot last commenter” or 30-day rules, so their expected open/closed decision is not derivable from the prompt. Model these fields and render them here, or remove those checks in offline mode.
src/bcbench/evaluate/extriage.py:108 - Malformed judge output currently escapes as
JSONDecodeError/AttributeErrorand fails the whole evaluation, while string values such as"false"are truthy and can incorrectly pass. Convert unreadable or schema-invalid verdicts toLLMJudgeErrorand require an actual boolean.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:36 - Not every rule stored in a
blockersfile means auto-reject. For example,potential_security_riskhasaction: include_human,entity_not_foundhasaction: request_clarification, andishandled_unsafe_code_blockhasaction: suggest_alternative. Mapping every match toauto-rejectcloses requests that should instead be routed for human review or clarification.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:30 - Phase 3 emits subtype
extend_existing_enum, but this requirements file names the sectionextend_enum. The requirements for target enum and compatibility therefore cannot be selected for requests that extend an existing enum.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:88 - These keys are over-indented beneath the scalar
validation_hints, making this YAML invalid; the same defect is repeated fordata_sensitivity_reviewandmulti_extension_interactionbelow. Any YAML-aware reader will fail to load the event requirements, blocking Phase 4 for event requests.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/4-requirements.md:18 - The requirements data includes
action: auto_reject(for examplereject_ishandled_only_for_locktable), but this dispatcher recognizes onlyreject. Such a matched rule falls through and becomesmissing-inforather than closing the rejected request.
Sun Haoran (haoranpb)
left a comment
Sun Haoran (haoranpb)
left a comment
Collaborator
There was a problem hiding this comment.
Great work, it is close. Left some comments about styling => let's spell the name out.
Regarding the evaluation method, take a look at how nl2al is using LMChecklist, sounds like a good fit for your senarios
…ensibility/implementation # Conflicts: # src/bcbench/dataset/__init__.py # src/bcbench/dataset/dataset_entry.py
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
July 31, 2026 08:25
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 60 changed files in this pull request and generated 2 comments.
Suppressed comments (8)
src/bcbench/agent/shared/config.yaml:86
- The triage prompt requires
argus-triage, but this configuration still hasskills.enabled: false, andsetup_agent_skills()only copies profile skills when that flag is true. The evaluated NAV checkout therefore has no such skill or knowledge files, so the agent cannot execute the workflow this prompt requires. Enable/install the skill for these categories (ideally category-selectively) before advertising it in the prompt.
Task: Run the `argus-triage` skill end to end on the extensibility request below and produce its
triage decision. Follow the skill's phases (preprocess, eligibility, classify, requirements,
codebase analysis, team assignment, finalize) and its knowledge base to decide the managed labels,
the advisory comment, and whether the request should stay open or be closed.
src/bcbench/agent/shared/config.yaml:70
- The NAV dataset and every new problem statement use
App/Layers/W1/..., notsrc/Layers/W1/.... With this path, an agent following the benchmark prompt will miss W1 and skip required country-layer propagation. UseApp/Layers/W1/...here.
- W1-first layering: when the file to change exists under `src/Layers/W1/...`, change W1 first, then apply
the same change to every same-named counterpart file in the other country/region layers. If the file
exists only in a specific country layer, edit that layer directly.
src/bcbench/evaluate/extriage.py:108
- A malformed judge artifact currently escapes as
JSONDecodeError(orAttributeErrorwhen the JSON is not an object), whileevaluate()only catchesLLMJudgeError. Since this file is LLM-generated, one invalid response crashes the entire evaluation instead of recordingcomment_ok=False. Validate and wrap parse/type failures asLLMJudgeError.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:36 - The Phase 5 algorithm maps every rule in the
blockersbatch toauto-reject, but that batch contains other required outcomes:ishandled_unsafe_code_blockusessuggest_alternative,potential_security_riskusesinclude_human, andentity_not_foundusesrequest_clarification. Those cases will therefore be closed as rejections instead of becoming missing-info/manual-review outcomes. Dispatch on each matched rule'sactionrather than its file category.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:32 - Phase 3 emits the subtype
extend_existing_enum, and the orchestrator uses that same name when loading subtype rules, but this requirements file definesextend_enum. Existing-enum requests therefore do not match their subtype-specific requirements. Rename this key to the canonical subtype.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:184 - Phase 4 only recognizes requirement actions
rejectandagent-not-processable;auto_rejectis never handled there. As a result, a request solely trying to bypassLockTablecan miss the required rejection path. Use the Phase 4 action vocabulary (reject) for this requirement.
src/bcbench/evaluate/extriage.py:126 - This new evaluator has no focused tests, although the analogous judge-based pipeline is covered in
tests/test_nl2al_pipeline.py. Add tests for missing/unparsable output, wrapped versus flatFinal_Output, invalid field types, judge failure, and pass/fail aggregation so contract regressions do not crash evaluation jobs.
src/bcbench/evaluate/extimplement.py:57 - The new judge-based pipeline is untested, while
NL2ALPipeline.evaluate()has dedicated empty/non-empty-diff coverage intests/test_nl2al_pipeline.py. Add equivalent tests forExtImplementPipeline.evaluate()and its workspace setup, especially that only AL edits are captured and empty output persists the expected result.
…ding Applies Haoran's review feedback for the extensibility-request categories: - Rename to explicit `extensibility-request-implement` / `extensibility-request-triage` across category ids, enum members, classes, dataset files and module files (#1/#2/#4). - Slim both prompt templates to reflect real-world usage; category-specific rules now live in the ai-ext-fix / argus-triage skills (#3). - Merge duplicate `bcbench.results` import in commands/evaluate.py (#5). - Uptake #761: both ext entries subclass `RepoGroundedEntry` (#6/#7). - Triage `patch` defaults to None; stripped from the dataset (#8). - Constrain triage label fields with a `ManagedLabel` Literal to catch dataset typos (#9). - Move the ext dataset classes into dataset/extensibility_request.py, like code review (#10). - Drop `fetch_commit_if_missing` from both ext pipelines to match the other pipelines (#11). - Keep ext-implement judge-only for now; container build/publish noted as future work (#12). - Grade triage with the NL2AL LMChecklist instead of the code-review-style hybrid: `ExtRequestTriageResult` removed in favour of `JudgeBasedEvaluationResult`, expected_* fields replaced by an `expected` checklist, evaluators `lm_checklist`, core score `test_passed`; 17 triage entries migrated (#13/#14/#15/#16). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
July 31, 2026 10:00
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 59 out of 59 changed files in this pull request and generated 1 comment.
Suppressed comments (13)
src/bcbench/agent/shared/config.yaml:75
- The triage prompt depends on “the skill's
Final_Outputobject,” but the default config still has bothskills.enabled: falseandagents.enabled: false, and the runners read that config directly. Consequently, normal workflow runs never copyargus-triageor select its agent, so the model is asked to follow a JSON contract and triage procedure that are unavailable. Make skill/agent selection category-aware, or make this prompt self-contained.
- Write your final decision as valid JSON to `triage_result.json` in {{repo_path}}, containing exactly the skill's `Final_Output` object. Do not apply it through any host tool.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/general_requirements.yaml:112
- This key is indented deeper than its sibling
rationale; the subsequent dedent makes the newly added knowledge file invalid YAML, so a YAML loader cannot consume the general requirements. Align it with the otherquality_standardskeys.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:88 - These mapping keys are over-indented beneath the inline
validation_hintsvalue, which is invalid YAML. Align them with the other fields of this requirement so the IsHandled requirements file can be loaded.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:103 - These mapping keys are over-indented beneath the inline
validation_hintsvalue, making this YAML invalid. Align them with the other fields of the requirement.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:118 - These mapping keys are over-indented beneath the inline
validation_hintsvalue, making this YAML invalid. Align them with the other fields of the requirement.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:30 - The classifier and orchestrator call this subtype
extend_existing_enum, but the requirements file names itextend_enum. A subtype lookup based on the emittedSUBTYPEtherefore misses these mandatory requirements. Use the same identifier throughout the workflow.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md:31 - These hard-coded repository assumptions do not match this category's checkout. Entries target
microsoftInternal/NAVand their AL projects are underApp/Layers/..., whereas the skill checksmicrosoft/BCAppsTestand./src; the startup source check will therefore fail before triage. Derive repository and code roots from the evaluation context/project paths.
Everything is local in **`microsoft/BCAppsTest`**:
- The **issue** is in this repo → operated on with `gh`.
- The **AL source** is checked out at **`CODE_ROOT = ./src`** → Phase 5 reads local files
(no remote GitHub API, no codebase token).
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6
- Every other subtype rule identifies this target as
event-request/ishandled; this hyphenated value does not match the classifier's type/subtype contract, so action filtering can skip the only rule in this file. Use the established subtype identifier.
applies_to: ["event-request-ishandled"]
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:33
- This blanket mapping ignores each rule's declared action. For example,
potential_security_riskusesinclude_humanand explicitly requiresagent-not-processable, whileentity_not_foundusesrequest_clarification; both are in the blocker batch and will be converted toauto-reject/closed here. Dispatch matched blockers according to theiractioninstead of closing every match.
src/bcbench/dataset/extensibility_request.py:48 - The triage skill classifies and emits
extensibility-enhancementas a managed request-type label, but this public input type rejects that label incurrent_labels. A previously triaged enhancement entry therefore cannot be loaded. Keep the schema aligned with the skill's managed-label set.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:184 - Phase 4 only recognizes
action: rejectandaction: agent-not-processable;auto_rejectis not handled there. A request whose IsHandled block only skipsLockTablewill therefore fall through to the generic missing-info path instead of being closed as this rule requires. Use the action value defined by the requirements workflow.
src/bcbench/evaluate/ext_request_implement.py:59 - The new pipeline's raw-diff and empty-diff result paths are untested, while analogous judge-based pipeline behavior is covered in
tests/test_nl2al_pipeline.py. Add tests that mockstage_and_get_difffor both outcomes and verify the savedJudgeBasedEvaluationResult; otherwise regressions in this category can pass the exhaustiveness test unnoticed.
src/bcbench/evaluate/ext_request_triage.py:47 - No test exercises the triage artifact contract introduced here. Add coverage for a missing file, a whitespace-only file, and a non-empty UTF-8 JSON file, asserting the empty/raw result saved in each case; existing judge-based pipelines have dedicated tests, so the current exhaustiveness fixture does not cover this behavior.
Rename the copied argus-triage skill/agents to extensibility-request-triage (and ai-ext-fix -> extensibility-request-implement) and rework the triage skill to run offline for evals: request comes from prompt text, source under caller-supplied CODE_ROOT, decision written to triage_result.json instead of applied via gh. Drop Argus branding, ISSUE_NUMBER, BCAppsTest and ./src hard-coding, and author/timestamp eligibility checks. Fix knowledge/rule bugs found in review: invalid YAML indentation in general_requirements.yaml and event_request_requirements.yaml; align enum subtype extend_enum -> extend_existing_enum; correct IsHandled applicability id to event-request/ishandled; add missing approved_extensibility_enhancement comment template; dispatch Phase 5 blockers by each rule's own action instead of blanket auto-reject. Update types.py comment and ext_request_triage.py docstring to drop stale names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
July 31, 2026 10:43
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.
Suppressed comments (3)
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/extensibility-request-triage/phases/4-requirements.md:14
- The requirement files embed subtype requirements under
subtypes(for example,event_request_requirements.yamlcontains bothishandledandregular), and no separate subtype files exist. Telling the agent to evaluate every requirement in the type file makes regular events fail IsHandled-only checks and makes one enum subtype fail the other subtype's checks. Select only common requirements plus the active subtype.
src/bcbench/dataset/extensibility_request.py:51 extensibility-enhancementis a managed request-type label inshared-rules.md:10-14and is emitted by the classifier, but it is absent from this literal. Any future triage entry already carrying that managed label will fail Pydantic validation before evaluation starts.
tests/test_type_exhaustiveness.py:45- This signature exceeds the repository's 200-character Ruff line length and will be reformatted by the required
ruff-formathook. Apply the formatter (or split the parameters) so the formatting check does not leave this file modified.
Sun Haoran (haoranpb)
left a comment
Sun Haoran (haoranpb)
left a comment
Collaborator
There was a problem hiding this comment.
Great progress, you should be able to trigger some runs and how/if things works.
Start with test run (with few entries) and cheaper models.
With LMChecklist, you should be able to view results in Kuisto, link shared in private chat.
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 3, 2026 07:49
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 52 changed files in this pull request and generated 6 comments.
Suppressed comments (7)
src/bcbench/agent/shared/config.yaml:75
- The triage prompt depends on the added skill/custom agent, but normal evaluation runs never install or select either one:
config.yamlstill has instructions, skills, and agents disabled, and both new workflow paths callbcbench evaluatewithout an override. Consequently the NAV checkout receives none of the step files or rule YAMLs, so the model only sees this short prompt and cannot execute the triage policy being benchmarked. Please make agent/skill setup category-aware and selectextensibility-request-triagefor this category.
- Write your final decision as valid JSON to `triage_result.json` in {{repo_path}}, containing exactly the skill's `Final_Output` object. Do not apply it through any host tool.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:103
- These keys are over-indented relative to the
data_sensitivity_reviewmapping, so the YAML cannot be parsed and this policy is not attached to the requirement.
flexible_validation: true
validation_notes: |
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:118
- These keys are over-indented relative to the
multi_extension_interactionmapping, making the YAML invalid and preventing the relaxed validation policy from loading.
flexible_validation: true
validation_notes: |
dataset/extensibility_request_triage.jsonl:1
- Every entry's critical checklist asks for an
issue_statefield, but the emitted artifact contract explicitly requiresrequest_stateand forbidsissue_state. Since LMChecklist receives the raw JSON, these assertions test a field that compliant agents never emit and can incorrectly fail the state criterion. Rename these checklist references torequest_statethroughout this dataset.
{"metadata": {}, "repo": "microsoftInternal/NAV", "instance_id": "microsoftInternal__NAV-Ext_Request_Triage-29447", "base_commit": "06e095c485e474431177e2170c0e22182bfdcc28", "created_at": "2025-06-01", "environment_setup_version": "27.0", "project_paths": ["App/Layers/W1/BaseApp"], "title": "[Event Request] Codeunit 5880 \"Phys. Invt. Order-Finish\"", "description": "### Why do you need this change?\n\nHi,\ni need an event in the Codeunit 5880 \"Phys. Invt. Order-Finish\" in the procedure \"CreateOrderTrackingBufferLines\" to modify the TempInvtOrderTrackingBuffer record. Each pass through the final loop finalizes one tracking buffer line together with its own DocNo and LineNo, and I need to read and adjust those values on the exact line being processed right before it is persisted \u2014 a single hook placed before or after the whole loop would no longer expose the individual line, its document/line context, or the computed quantities I have to react to.\n\n### Describe the request\n\nAdd a new event in the procedure \"CreateOrderTrackingBufferLines\", before modifying the record TempInvtOrderTrackingBuffer.\n\n procedure CreateOrderTrackingBufferLines(DocNo: Code[20]; LineNo: Integer)\n var\n ExpInvtOrderTracking: Record \"Exp. Invt. Order Tracking\";\n ItemTrackingSetup: Record \"Item Tracking Setup\";\n begin\n PhysInvtRecordLine.Reset();\n PhysInvtRecordLine.SetCurrentKey(\"Order No.\", \"Order Line No.\");\n PhysInvtRecordLine.SetRange(\"Order No.\", DocNo);\n PhysInvtRecordLine.SetRange(\"Order Line No.\", LineNo);\n PhysInvtRecordLine.SetFilter(\"Quantity (Base)\", '<>%1', 0);\n if PhysInvtRecordLine.Find('-') then\n repeat\n ItemTrackingSetup.\"Serial No.\" := PhysInvtRecordLine.\"Serial No.\";\n ItemTrackingSetup.\"Lot No.\" := PhysInvtRecordLine.\"Lot No.\";\n ItemTrackingSetup.\"Package No.\" := PhysInvtRecordLine.\"Package No.\";\n UpdateBufferRecordedQty(ItemTrackingSetup, PhysInvtRecordLine.\"Quantity (Base)\", LineNo);\n OnCreateOrderTrackingBufferLinesFromPhysInvtRecordLine(TempInvtOrderTrackingBuffer, PhysInvtRecordLine);\n until PhysInvtRecordLine.Next() = 0;\n\n ExpInvtOrderTracking.Reset();\n ExpInvtOrderTracking.SetRange(\"Order No\", DocNo);\n ExpInvtOrderTracking.SetRange(\"Order Line No.\", LineNo);\n if ExpInvtOrderTracking.Find('-') then\n repeat\n ItemTrackingSetup.\"Serial No.\" := ExpInvtOrderTracking.\"Serial No.\";\n ItemTrackingSetup.\"Lot No.\" := ExpInvtOrderTracking.\"Lot No.\";\n ItemTrackingSetup.\"Package No.\" := ExpInvtOrderTracking.\"Package No.\";\n UpdateBufferExpectedQty(ItemTrackingSetup, ExpInvtOrderTracking.\"Quantity (Base)\", LineNo);\n OnCreateOrderTrackingBufferLinesFromExpInvtOrderTracking(TempInvtOrderTrackingBuffer, ExpInvtOrderTracking);\n until ExpInvtOrderTracking.Next() = 0;\n\n TempInvtOrderTrackingBuffer.Reset();\n if TempInvtOrderTrackingBuffer.Find('-') then\n repeat\n TempInvtOrderTrackingBuffer.\"Qty. To Transfer\" :=\n TempInvtOrderTrackingBuffer.\"Qty. Recorded (Base)\" - TempInvtOrderTrackingBuffer.\"Qty. Expected (Base)\";\n TempInvtOrderTrackingBuffer.\"Outstanding Quantity\" := TempInvtOrderTrackingBuffer.\"Qty. To Transfer\";\n TempInvtOrderTrackingBuffer.Open := TempInvtOrderTrackingBuffer.\"Outstanding Quantity\" <> 0;\n\n // Start Event\n\n OnCreateOrderTrackingBufferLinesOnBeforeModifyTempInvtOrderTracking(TempInvtOrderTrackingBuffer, DocNo, LineNo);\n\n // End Event\n\n TempInvtOrderTrackingBuffer.Modify();\n until TempInvtOrderTrackingBuffer.Next() = 0;\n end;\n\n**New Event**\n```\n[IntegrationEvent(false, false)]\n local procedure OnCreateOrderTrackingBufferLinesOnBeforeModifyTempInvtOrderTracking(var TempInvtOrderTracking: Record \"Invt. Order Tracking\" temporary; DocNo: Code[20]; LineNo: Integer)\n begin\n end;\n```", "comments": "", "current_labels": [], "expected": [{"text": "The triage decision sets labels_to_set to exactly these managed labels and no others: SCM, event-request.", "level": "critical"}, {"text": "The triage decision sets issue_state to 'open'.", "level": "critical"}, {"text": "The advisory comment posted to the requester reaches the same triage conclusion and provides guidance consistent with the following reference comment:\n\n✅ Analysis complete - approved for implementation\n\n**Implementation:**\n\nAdd event publisher in procedure `CreateOrderTrackingBufferLines` before `TempInvtOrderTrackingBuffer.Modify()`:\n\n```al\nTempInvtOrderTrackingBuffer.Reset();\nif TempInvtOrderTrackingBuffer.Find('-') then\n repeat\n TempInvtOrderTrackingBuffer.\"Qty. To Transfer\" :=\n TempInvtOrderTrackingBuffer.\"Qty. Recorded (Base)\" - TempInvtOrderTrackingBuffer.\"Qty. Expected (Base)\";\n TempInvtOrderTrackingBuffer.\"Outstanding Quantity\" := TempInvtOrderTrackingBuffer.\"Qty. To Transfer\";\n TempInvtOrderTrackingBuffer.Open := TempInvtOrderTrackingBuffer.\"Outstanding Quantity\" <> 0;\n OnCreateOrderTrackingBufferLinesOnBeforeModifyTempInvtOrderTracking(TempInvtOrderTrackingBuffer, DocNo, LineNo);\n TempInvtOrderTrackingBuffer.Modify();\n until TempInvtOrderTrackingBuffer.Next() = 0;\n```\n\nAdd event publisher procedure at the end of the codeunit (before the closing brace):\n\n```al\n[IntegrationEvent(false, false)]\nlocal procedure OnCreateOrderTrackingBufferLinesOnBeforeModifyTempInvtOrderTracking(var TempInvtOrderTracking: Record \"Invt. Order Tracking\" temporary; DocNo: Code[20]; LineNo: Integer)\nbegin\nend;\n```", "level": "expected"}]}
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step5-codebase-analysis.md:52
- Not every rule in the
blockersbatch is an auto-rejection. For example,potential_security_riskhasaction: include_human, whileentity_not_foundrequests clarification; this instruction closes both instead of producingagent-not-processableormissing-info. Dispatch matched blockers by their declaredaction.
src/bcbench/dataset/extensibility_request.py:51 - The classifier supports
extensibility-enhancement, and the success path emits[TEAM_LABEL, TYPE], but that type is absent from the managed-label contract. As a result, entries already carrying this label cannot validate and the documented output label is not representable byManagedLabel. Addextensibility-enhancementto this literal (or remove the unsupported classification).
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step4-requirements-check.md:16 - The requirements data uses
action: auto_reject(for examplereject_ishandled_only_for_locktable), but this dispatcher only recognizesreject. Such a matched requirement falls through to the ordinary failure path and becomesmissing-infoinstead of closing the request. Handleauto_rejectas a rejection here as well.
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 3, 2026 12:53
View session
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 70 out of 71 changed files in this pull request and generated no new comments.
Suppressed comments (10)
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:88
- These keys are over-indented beneath the scalar
validation_hintsvalue, which makes this requirements file invalid YAML. Dedent them to the same level as the other fields ofperformance_considerations.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:103 - These keys are over-indented beneath
validation_hints, so YAML parsing fails before the triage agent can load the requirements. Dedent them to the requirement-item level.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/general_requirements.yaml:113 - This indentation still makes the YAML invalid:
reject_images_without_text_equivalentestablishes one child indentation, thenrationalededents to a different level, whileacceptance_criteriais swallowed by the block scalar. Any YAML loader will fail or lose these rules. Align allquality_standardschildren consistently.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:118 - These keys are also over-indented beneath
validation_hints, leaving the requirements file invalid YAML. Dedent them to the requirement-item level.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:30 - The classifier emits
extend_existing_enum, but this subtree is namedextend_enum; subtype-specific requirements for extending an enum therefore cannot be selected consistently. Use the classifier's identifier here.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - This applicability value differs from the
event-request/ishandledidentifier used by the classifier and every other IsHandled rule. Applicability filtering will skip this alternative-suggestion rule.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step5-codebase-analysis.md:55 - Not every rule in the blockers file is an auto-rejection. For example,
potential_security_riskhasaction: include_human, whileentity_not_foundhasaction: request_clarification; this instruction closes both requests instead of producingagent-not-processableormissing-info. Dispatch using each matched rule'saction.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:184 - This rejection action is nested inside
quality_standardsand usesauto_reject, but Step 4 only checks a requirement-levelaction: reject. A request that asks only to bypassLockTablecan therefore be treated as ordinary missing information instead of being closed. Move the action to the requirement level and use the recognized value.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-implement.agent.md:27 - Agent assets are installed under
.githubfor Copilot but under.claudefor Claude (AgentType.get_target_dir, types.py:172-177). This hard-coded path makes the implementation agent unable to load its mandatory skill in Claude experiments. Resolve the skill from the active agent root instead.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-triage.agent.md:26 - The shared setup installs this tree beneath
.githubfor Copilot and.claudefor Claude (types.py:172-177). Hard-coding.githubhere—and again in the step files—means the triage agent cannot find any procedure or knowledge YAML in Claude experiments. MakeTRIAGE_ROOTdepend on the active agent root throughout the triage instructions.
Setup-ContainerAndRepository.ps1 resolves the dataset path via Get-BCBenchDatasetPath in a param default, which runs even with -SkipContainer/ -SkipRepo. Its ValidateSet only accepted bug-fix/test-generation/code-review/ nl2al, so selecting extensibility-request-implement or -triage in the evaluation workflows failed at setup before the (judge-based, container-less) run could start. Map both categories to their dataset files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 4, 2026 08:57
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 72 changed files in this pull request and generated 3 comments.
Suppressed comments (14)
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:88
- These keys are over-indented beneath the inline
validation_hintsvalue, which makes this YAML invalid instead of attaching the relaxed policy toperformance_considerations. Dedent them to the other requirement fields.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:103 - These keys are over-indented beneath the inline
validation_hintsvalue, making the YAML invalid and preventing the relaxed data-sensitivity policy from loading. Dedent them to the other requirement fields.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:30 - The classifier emits
extend_existing_enum, but this requirements tree names the subtypeextend_enum. Requests that add values to an existing enum therefore cannot select these subtype requirements. Use the classifier's identifier consistently.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - This applicability value differs from the
event-request/ishandledidentifier emitted and used by every other IsHandled rule. Filtering byTYPE/SUBTYPEskips this alternative-suggestion rule entirely.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step5-codebase-analysis.md:52 - Treating every rule in the
blockersbatch as auto-reject ignores each rule's declared action. For example,potential_security_riskisinclude_humanandentity_not_foundisrequest_clarification; this instruction closes those requests instead of routing them to human review or missing-info. Dispatch onactionbefore stopping.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/comment-templates/comment_templates.yaml:107 extensibility-enhancementis a supported classifier result, and Step 7 selectsapproved_<TYPE>for every feasible request, but the approval templates stop at enum requests. A feasible enhancement therefore has no template for its required advisory comment. Add anapproved_extensibility_enhancementtemplate or remove that supported type.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/general_requirements.yaml:113- This indentation makes the requirements file invalid YAML:
reject_images_without_text_equivalentandrationaleare at different mapping depths, andacceptance_criteriais currently part of the rationale block scalar rather than a structured key. Any YAML loader will fail before triage can evaluate these requirements.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:184 - Step 4 only recognizes requirement-level
action: rejectoraction: agent-not-processable, but this action is nested underquality_standardsand uses the unrecognized valueauto_reject. A request whose sole purpose is skippingLockTabletherefore will not take the mandated rejection path. Move it to the requirement level and use the recognized rejection action.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-implement.agent.md:27 - This path only exists for Copilot runs. The same shared agent is installed under
.claude/agentsfor Claude, while its skill is installed under.claude/skills, so the Claude workflow cannot load the mandatory procedure and stops before editing. Resolve the installed agent root or provide agent-specific paths.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-triage.agent.md:26 - This hard-coded root only exists for Copilot. For Claude, setup copies the same profile under
.claude, so the custom agent cannot find any step or knowledge file; the step files also repeat the.github/instructionsroot. Resolve the installed agent root consistently for both runners.
.github/workflows/copilot-evaluation.yml:35 - Selecting either new category here still runs
bcbench evaluatewith the shared defaults, where bothskills.enabledandagents.enabledare false. The workflow passes no category-specific override, so the implementation run lacks its skill and the triage run lacks the step/rule/template files that define its result contract. Wire these choices to a configuration that enables the corresponding skill/custom agent before exposing them.
- "extensibility-request-implement"
- "extensibility-request-triage"
.github/workflows/claude-evaluation.yml:28
- Selecting either new category here still runs
bcbench evaluatewith the shared defaults, where bothskills.enabledandagents.enabledare false. The workflow passes no category-specific override, so the implementation run lacks its skill and the triage run lacks the step/rule/template files that define its result contract. Wire these choices to a configuration that enables the corresponding skill/custom agent before exposing them.
- "extensibility-request-implement"
- "extensibility-request-triage"
src/bcbench/evaluate/ext_request_implement.py:42
- This new judge-based pipeline has no tests for its category-specific empty-diff and raw-patch persistence paths. The analogous NL2AL behavior is covered in
tests/test_nl2al_pipeline.py:25-52; add equivalent coverage so changes tostage_and_get_diffhandling do not silently turn empty agent runs into job failures or lose valid output.
src/bcbench/evaluate/ext_request_triage.py:48 - There is no coverage for the new artifact contract: missing/blank
triage_result.jsonmust persist an empty result, while a non-empty file must be preserved verbatim. Similar judge-based pipeline behavior is tested intests/test_nl2al_pipeline.py:25-52; add corresponding triage tests to protect these branches.
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 4, 2026 09:16
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 72 changed files in this pull request and generated 1 comment.
Suppressed comments (12)
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-triage.agent.md:26
- This path only exists for Copilot.
setup_instructions_from_config()installs the same tree under.claudefor Claude runs, so the Claude triage agent cannot find any step or knowledge files and stops before producing its result. ResolveTRIAGE_ROOTfrom the agent-specific installation directory.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-implement.agent.md:28 - The implementation skill is installed at
.claude/skills/...for Claude, not.github/skills/.... Consequently every Claude implementation run following this mandatory first step fails to load the authoritative procedure. Make the path agent-specific or locate the installed skill by name.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step5-codebase-analysis.md:53 - Not every rule in
general_blockers.yamlis an auto-rejection: for example,potential_security_riskusesinclude_human, whileentity_not_foundusesrequest_clarification. Collapsing all blocker-file matches to auto-reject closes requests that should instead be marked agent-not-processable or missing-info. Dispatch using each rule'saction.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/general_requirements.yaml:115 - This mapping is malformed YAML:
reject_images_without_text_equivalentstarts the child mapping at one indentation level, butrationaleis then dedented to a different, still-nested level;acceptance_criteriais also currently part of therationalescalar. The triage agent cannot load the general requirements file. Align these keys withinquality_standards.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:88 - These keys are indented beneath the scalar
validation_hintsvalue rather than alongside the other requirement properties, which makes this requirements file invalid YAML. Dedent them to the same level asmandatoryandvalidation_hints.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:103 - These keys are over-indented beneath
validation_hints, making the YAML invalid and preventing the triage agent from loading any event requirements. Dedent them to the requirement-property level.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:118 - These keys are again over-indented beneath the preceding scalar. This third malformed block must also be dedented for the event requirements YAML to parse.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:30 - The classifier emits
extend_existing_enum, and Step 5 uses that same subtype identifier, but this requirements tree names itextend_enum. Requests that add values to an existing enum therefore never select these subtype-specific requirements.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - All other IsHandled rules use the
event-request/ishandledapplicability identifier emitted by the classifier/orchestrator. This different identifier causes this alternative-suggestion rule to be skipped entirely.
src/bcbench/dataset/extensibility_request.py:39 extensibility-enhancementis a supported triage type and successful requests emitTYPEas a managed label, but it is missing from thisManagedLabelunion. Dataset entries that already carry that managed label fail validation even though the workflow itself supports the type.
src/bcbench/evaluate/ext_request_implement.py:45- The new judge-based pipeline has no tests for either result path, while the analogous NL2AL pipeline tests empty output, raw output, and exception propagation in
tests/test_nl2al_pipeline.py:25-52. Add tests that verify an empty diff and a generated patch each persist the expected result.
src/bcbench/evaluate/ext_request_triage.py:51 - There are no tests for the new artifact-reading behavior. Add coverage for a missing file, a blank file, and a non-empty
triage_result.json, asserting that each branch persists the corresponding empty/raw judge result; similar judge-based pipeline coverage exists intests/test_nl2al_pipeline.py:25-52.
…t label Two triage input-requirement files failed to parse and were silently dropped at runtime: - event_request_requirements.yaml: flexible_validation/validation_notes were nested inside the validation_hints scalar (3 blocks). - general_requirements.yaml: quality_standards children were inconsistently indented, swallowing acceptance_criteria into the rationale literal and breaking required_format at the code fence. Also add "extensibility-enhancement" to ManagedLabel so the schema matches the classifier (step3/step5 + extensibility_enhancement_requirements.yaml). Addresses PR review comments (invalid YAML, missing managed label). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 4, 2026 10:33
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 72 changed files in this pull request and generated no new comments.
Suppressed comments (7)
src/bcbench/agent/shared/config.yaml:121
- This shared config is loaded directly by both runners for every category, and
setup_custom_agent()always returns this single name; both runners then append--agent=extensibility-request-implement. As a result, triage runs (and existing bug-fix/test-generation/code-review runs) execute the implementation agent instead of their category-appropriate behavior. Select the custom agent byEvaluationCategory(triage vs implement) and leave it disabled for unrelated categories rather than setting one global agent name.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:30 - The classifier emits
SUBTYPE: "extend_existing_enum", while this requirements tree usesextend_enum. Subtype lookup therefore skips thetarget_enumandcompatibility_checkrequirements for requests that add values to an existing enum. Use the classifier's identifier consistently.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - Every other IsHandled rule uses the applicability identifier
event-request/ishandled, and Step 5 represents the type/subtype with that slash form. This different identifier causes the regular-event alternative rule to be filtered out for IsHandled requests.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-implement.agent.md:28 - This agent is also installed and invoked by the Claude workflow, where
setup_agent_skills()places the skill under.claude/skills, not.github/skills. The mandatory first step therefore opens a nonexistent path and cannot load the implementation procedure on Claude runs. Resolve the installed skill path for both agent types.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/extensibility-request-triage.agent.md:26 - The Claude runner installs this profile under
.claude, but this agent and the step files fixTRIAGE_ROOTto.github/instructions/.... Once the triage agent is selected, Claude runs cannot find Step 0 or any knowledge YAML. MakeTRIAGE_ROOTagent-specific (.github/...for Copilot and.claude/...for Claude) throughout the triage instructions.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step5-codebase-analysis.md:52 - The loaded
general_blockers.yamldoes not make every match an auto-rejection:potential_security_riskhasaction: include_human, andentity_not_foundhasaction: request_clarification. This instruction converts both into closed/rejected requests, contradicting the rule contract. Dispatch blocker outcomes from each rule'sactioninstead of the containing filename.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/step7-labels-comments.md:30 - Offline
GH_REQUESTdeliberately contains no author identity, and the workflow state listed here also has no author, but several selected templates require{author_name}(for examplemissing_info_requirementsandrejected_request). Those paths can only emit an unresolved placeholder or invent a username. Either include author metadata in the entry/task or remove author-addressed placeholders from all offline templates.
…ensibility/implementation
Copilot started reviewing on behalf of
Marko Aleksandric (AleksandricMarko)
August 5, 2026 12:03
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 72 changed files in this pull request and generated no new comments.
Suppressed comments (5)
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/extensibility-request-implement/SKILL.md:69
- Ignoring all but the first extension point makes valid benchmark tasks incomplete. For example, the newly added 30223 entry requests three events in one issue and 30173 requests two; both are expected to implement every requested change. Treat multiple extension points belonging to the single supplied request as one task, and only reject multiple unrelated request threads.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - All other IsHandled rules and the classifier identify this subtype as
event-request/ishandled; the hyphenated value will not match that subtype, so this alternative-suggestion rule can be skipped. Use the shared identifier.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:184 - Step 4 only stops and closes a request for
action: reject(step4-requirements-check.md:15-16).auto_rejectis not a recognized requirements action, so a request whose sole purpose is skippingLockTablecan continue instead of being rejected as this rule requires.
src/bcbench/agent/shared/config.yaml:121 - This global agent name is passed unconditionally for every Copilot and Claude evaluation:
setup_custom_agent()does not receive the category and the runners append its return value as--agent. Consequently triage runs the implementation agent instead of the new triage agent, while existing categories (and profiles that do not contain this agent) are also forced to use a nonexistent or inappropriate agent. Make custom-agent selection category-aware and leave it disabled for categories without a configured agent.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:30 - The classifier emits subtype
extend_existing_enum(step3-request-types.md:13), so this differently named key is never selected and its mandatory target/compatibility requirements can be skipped. Use the classifier's exact subtype identifier.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 71 out of 72 changed files in this pull request and generated 2 comments.
Suppressed comments (6)
src/bcbench/agent/shared/config.yaml:121
setup_custom_agent()is category-agnostic, so this value is passed as--agentfor every Copilot and Claude run. That forces bug-fix, test-generation, code-review, and even extensibility-request-triage to use the implementation agent; in particular, triage will follow the.al-editing agent instead of writingtriage_result.json. Select the agent fromEvaluationCategory(implement vs triage vs the existing category behavior) rather than configuring one global name.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/enum_request_requirements.yaml:31- Step 3 emits the subtype name
extend_existing_enum, while this requirements tree calls itextend_enum. An agent matching requirements byTYPE/SUBTYPEwill not associate thesetarget_enumandcompatibility_checkrequirements with existing-enum requests. Rename this key to the emitted subtype value.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/input-requirements/event_request_requirements.yaml:184 - Step 4 only treats
action: reject(oragent-not-processable) as a terminal requirement failure. Thisauto_rejectvalue is not part of that contract, so an IsHandled request used only to skipLockTablecan fall through as missing information instead of being closed as intended. Use the recognized rejection action.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/instructions/extensibility-request-triage/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6 - The subtype identifier is inconsistent with every other IsHandled rule, which uses
event-request/ishandled. If applicability metadata is honored, this alternative rule is skipped for the exact subtype it targets. Use the canonical identifier.
src/bcbench/evaluate/ext_request_implement.py:42 - This new evaluation path has no tests for either branch: an empty diff must save
create_empty_output, while a generated patch must be preserved as raw judge output. The analogous judge-based pipeline is covered intests/test_nl2al_pipeline.py; add equivalent tests so result serialization and empty-output handling cannot regress.
src/bcbench/evaluate/ext_request_triage.py:48 - The artifact-reading behavior is untested: missing files, whitespace-only files, and valid raw JSON take different result paths. Add pipeline tests (similar to
tests/test_nl2al_pipeline.py) that assert each case saves the expectedJudgeBasedEvaluationResult, especially because a missingtriage_result.jsonis the primary failure mode for this category.
Sun Haoran (haoranpb)
left a comment
Sun Haoran (haoranpb)
left a comment
Collaborator
There was a problem hiding this comment.
Great work! Didn't look through the dataset, but the rest looks good to merge.
See the individual comments for some small things.
You probably want to look into expanding the dataset, both size and diversity if relevant.
| "--disable-builtin-mcps", | ||
| f"--model={model}", | ||
| "--log-level=debug", | ||
| "--log-level=all", |
Collaborator
There was a problem hiding this comment.
I think log level all and debug are equivalent
Collaborator
Author
There was a problem hiding this comment.
Good catch — reverted back to --log-level=debug\ (the original) in bffe360.
Revert config.yaml instructions/skills/agents toggles to disabled and agents.name to ALTest (the minimal/production-like defaults). Agent- specific enablement for a run belongs in a separate per-experiment PR, not the committed baseline. The two ext-category prompt templates remain. Also restore --log-level=debug in the Copilot runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31
Marko Aleksandric (AleksandricMarko)
marked this pull request as ready for review
August 10, 2026 10:43
Marko Aleksandric (AleksandricMarko)
enabled auto-merge (squash)
August 10, 2026 10:43
Sun Haoran (haoranpb)
approved these changes
Aug 10, 2026
Marko Aleksandric (AleksandricMarko)
deleted the
extensibility/implementation
branch
August 10, 2026 12:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR adds support for evaluating agents on Business Central (AL) extensibility requests, introducing two new BC-Bench evaluation categories:
Both categories are judge-based (scored downstream by the LMChecklist judge, like NL2AL) and follow the existing category-based structure, so they slot into the current dataset, runner, and evaluation pipeline alongside bug-fix and test-generation.
Included:
Fixes AB#613841