Add per-article BCQuality coverage for code-review dataset (AB#646153) - #776
Open
Wenjie Fan (gggdttt) wants to merge 2 commits into
Open
Add per-article BCQuality coverage for code-review dataset (AB#646153)#776Wenjie Fan (gggdttt) wants to merge 2 commits into
Wenjie Fan (gggdttt) wants to merge 2 commits into
Conversation
Annotate code-review gold entries with the BCQuality knowledge article(s) they exercise and report per-article coverage. - ReviewComment.article and metadata.articles (both optional, backward-compatible) carry the <domain>/<slug> association; FP-guard entries (expected_comments=[]) use the entry-level list. - New dataset/coverage.py aggregates declared articles and, given a BCQuality checkout (--bcquality-root / BCQUALITY_ROOT), flags zero-coverage and unknown slugs; degrades to declared-only when absent. - 'bcbench dataset coverage' renders per-domain coverage with a JSON --github-output. - Backfill the security domain (18 entries, 10 articles) as a worked example; 671 passed / 1 skipped.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-article BCQuality coverage tracking for code-review datasets.
Changes:
- Extends dataset schemas with optional article annotations.
- Adds inventory comparison and coverage reporting.
- Adds a CLI command, tests, and security-domain annotations.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
dataset/codereview.jsonl |
Backfills security article annotations. |
src/bcbench/commands/dataset.py |
Adds the coverage CLI command. |
src/bcbench/dataset/__init__.py |
Exports coverage APIs. |
src/bcbench/dataset/codereview.py |
Adds comment-level article support. |
src/bcbench/dataset/coverage.py |
Implements coverage aggregation and inventory comparison. |
src/bcbench/dataset/dataset_entry.py |
Adds entry-level article metadata. |
tests/test_codereview_coverage.py |
Tests coverage logic and inventory handling. |
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-004", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/AppConstants.Codeunit.al b/src/AppConstants.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/AppConstants.Codeunit.al\n@@ -0,0 +1,30 @@\n+codeunit 50103 \"App Constants\"\n+{\n+ Access = Internal;\n+\n+ var\n+ ApiVersionTok: Label 'v2.0', Locked = true;\n+ DefaultCurrencyTok: Label 'USD', Locked = true;\n+ DateFormatTok: Label 'yyyy-MM-dd', Locked = true;\n+ AppIdTok: Label 'BC-INVENTORY-APP', Locked = true;\n+\n+ procedure GetApiVersion(): Text\n+ begin\n+ exit(ApiVersionTok);\n+ end;\n+\n+ procedure GetDefaultCurrency(): Code[10]\n+ begin\n+ exit(DefaultCurrencyTok);\n+ end;\n+\n+ procedure GetDateFormat(): Text\n+ begin\n+ exit(DateFormatTok);\n+ end;\n+\n+ procedure GetAppId(): Text\n+ begin\n+ exit(AppIdTok);\n+ end;\n+}\ndiff --git a/src/QueryStringSecretClient.Codeunit.al b/src/QueryStringSecretClient.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/QueryStringSecretClient.Codeunit.al\n@@ -0,0 +1,15 @@\n+codeunit 50304 \"Query String Secret Client\"\n+{\n+ procedure FetchAccount(ApiKey: Text)\n+ var\n+ HttpClient: HttpClient;\n+ HttpResponseMessage: HttpResponseMessage;\n+ begin\n+ HttpClient.Get(this.BuildAccountUrl(ApiKey), HttpResponseMessage);\n+ end;\n+\n+ local procedure BuildAccountUrl(ApiKey: Text): Text\n+ begin\n+ exit('https://api.contoso.com/accounts?api_key=' + ApiKey);\n+ end;\n+}\n", "expected_comments": [{"file": "src/QueryStringSecretClient.Codeunit.al", "line_start": 13, "line_end": 13, "severity": "high", "domain": "security", "body": "API key is placed in the URL query string. Use an Authorization header, or SetSecretRequestUri() if a secret URI is unavoidable."}], "category": "code-review", "description": "Clean codeunit with configuration constants that are not secrets: API version, currency codes, labels, and format strings", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-005", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/ValidatedImportConfig.Table.al b/src/ValidatedImportConfig.Table.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ValidatedImportConfig.Table.al\n@@ -0,0 +1,34 @@\n+table 50104 \"Validated Import Config\"\n+{\n+ Caption = 'Validated Import Configuration';\n+ DataClassification = CustomerContent;\n+\n+ fields\n+ {\n+ field(1; \"Code\"; Code[20])\n+ {\n+ Caption = 'Code';\n+ NotBlank = true;\n+ }\n+ field(2; \"Source Table ID\"; Integer)\n+ {\n+ Caption = 'Source Table';\n+ TableRelation = AllObjWithCaption.\"Object ID\" where(\"Object Type\" = const(Table));\n+ ValidateTableRelation = true;\n+ }\n+ field(3; \"Max Records\"; Integer)\n+ {\n+ Caption = 'Maximum Records';\n+ MinValue = 1;\n+ MaxValue = 10000;\n+ }\n+ }\n+\n+ keys\n+ {\n+ key(PK; \"Code\")\n+ {\n+ Clustered = true;\n+ }\n+ }\n+}\ndiff --git a/src/BroadFinanceAccess.PermissionSet.al b/src/BroadFinanceAccess.PermissionSet.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/BroadFinanceAccess.PermissionSet.al\n@@ -0,0 +1,15 @@\n+permissionset 50305 \"Broad Finance Access\"\n+{\n+ Assignable = true;\n+ Caption = 'Broad Finance Access', Locked = true;\n+ Permissions =\n+ tabledata * = RIMD,\n+ table * = X,\n+ tabledata Customer = R,\n+ tabledata Vendor = R,\n+ tabledata Item = R,\n+ tabledata \"Sales Header\" = R,\n+ tabledata \"Sales Line\" = R,\n+ codeunit \"Release Sales Document\" = X,\n+ codeunit \"Sales-Post\" = X;\n+}\n", "expected_comments": [{"file": "src/BroadFinanceAccess.PermissionSet.al", "line_start": 6, "line_end": 6, "severity": "critical", "domain": "security", "body": "Permission set grants RIMD on all table data. Replace the wildcard with the minimum specific tabledata permissions required."}, {"file": "src/BroadFinanceAccess.PermissionSet.al", "line_start": 7, "line_end": 7, "severity": "high", "domain": "security", "body": "Permission set grants execute permission on all tables. Grant execute only on the specific objects this role requires."}], "category": "code-review", "description": "Clean table with proper input validation: ValidateTableRelation, OnValidate triggers, MinValue/MaxValue, Editable=false on system fields", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-006", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/InventoryReader.PermissionSet.al b/src/InventoryReader.PermissionSet.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/InventoryReader.PermissionSet.al\n@@ -0,0 +1,11 @@\n+permissionset 50106 \"Inventory Reader\"\n+{\n+ Caption = 'Inventory Reader';\n+ Assignable = true;\n+\n+ Permissions =\n+ tabledata Item = r,\n+ tabledata \"Item Ledger Entry\" = r,\n+ tabledata \"Item Category\" = r,\n+ codeunit \"Inventory Lookup\" = X;\n+}\ndiff --git a/src/InventoryLookup.Codeunit.al b/src/InventoryLookup.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/InventoryLookup.Codeunit.al\n@@ -0,0 +1,15 @@\n+codeunit 50108 \"Inventory Lookup\"\n+{\n+ Access = Internal;\n+ Permissions = tabledata Item = r;\n+\n+ procedure GetItemDescription(ItemNo: Code[20]): Text[100]\n+ var\n+ Item: Record Item;\n+ begin\n+ Item.SetLoadFields(Description);\n+ if Item.Get(ItemNo) then\n+ exit(Item.Description);\n+ exit('');\n+ end;\n+}\ndiff --git a/src/ExcessiveInherentAccess.Codeunit.al b/src/ExcessiveInherentAccess.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ExcessiveInherentAccess.Codeunit.al\n@@ -0,0 +1,17 @@\n+codeunit 50306 \"Excessive Inherent Access\"\n+{\n+ procedure LookupCustomerName(CustomerNo: Code[20]): Text\n+ begin\n+ exit(this.GetCustomerName(CustomerNo));\n+ end;\n+\n+ [InherentPermissions(PermissionObjectType::TableData, Database::Customer, 'RIMD')]\n+ local procedure GetCustomerName(CustomerNo: Code[20]): Text\n+ var\n+ Customer: Record Customer;\n+ begin\n+ if Customer.Get(CustomerNo) then\n+ exit(Customer.Name);\n+ exit('');\n+ end;\n+}\n", "expected_comments": [{"file": "src/ExcessiveInherentAccess.Codeunit.al", "line_start": 8, "line_end": 8, "severity": "high", "domain": "security", "body": "InherentPermissions grants RIMD tabledata access even though this procedure only reads Customer. Reduce the permission to the minimal read access required."}], "category": "code-review", "description": "Clean permission sets with least-privilege access: read-only for readers, read-insert for editors, no RIMD grants", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-001", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/SecureAPIManager.Codeunit.al b/src/SecureAPIManager.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/SecureAPIManager.Codeunit.al\n@@ -0,0 +1,46 @@\n+codeunit 50100 \"Secure API Manager\"\n+{\n+ Access = Internal;\n+\n+ var\n+ KeyNotConfiguredErr: Label 'API key is not configured for %1.', Comment = '%1 = configuration code';\n+ RequestFailedErr: Label 'The API request failed. Check the configuration.', Comment = 'Shown when an outbound API call fails.';\n+ EndpointTok: Label 'https://api.businesscentral.dynamics.com/v2.0/data', Locked = true;\n+ BearerTok: Label 'Bearer %1', Locked = true;\n+ StorageKeyTok: Label 'ApiKey_%1', Locked = true;\n+\n+ [NonDebuggable]\n+ procedure StoreKey(ConfigCode: Code[20]; KeyValue: SecretText)\n+ begin\n+ IsolatedStorage.SetEncrypted(GetStorageKey(ConfigCode), KeyValue, DataScope::Module);\n+ end;\n+\n+ [NonDebuggable]\n+ procedure GetKey(ConfigCode: Code[20]) Result: SecretText\n+ begin\n+ if not IsolatedStorage.Contains(GetStorageKey(ConfigCode), DataScope::Module) then\n+ Error(KeyNotConfiguredErr, ConfigCode);\n+ IsolatedStorage.Get(GetStorageKey(ConfigCode), DataScope::Module, Result);\n+ end;\n+\n+ procedure CallEndpoint(ConfigCode: Code[20])\n+ var\n+ Client: HttpClient;\n+ Headers: HttpHeaders;\n+ Response: HttpResponseMessage;\n+ AuthHeader: SecretText;\n+ begin\n+ AuthHeader := SecretStrSubstNo(BearerTok, GetKey(ConfigCode));\n+ Headers := Client.DefaultRequestHeaders();\n+ Headers.Add('Authorization', AuthHeader);\n+ if not Client.Get(EndpointTok, Response) then\n+ Error(RequestFailedErr);\n+ if not Response.IsSuccessStatusCode() then\n+ Error(RequestFailedErr);\n+ end;\n+\n+ local procedure GetStorageKey(ConfigCode: Code[20]): Text[50]\n+ begin\n+ exit(CopyStr(StrSubstNo(StorageKeyTok, ConfigCode), 1, 50));\n+ end;\n+}\ndiff --git a/src/HardcodedSecretClient.Codeunit.al b/src/HardcodedSecretClient.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/HardcodedSecretClient.Codeunit.al\n@@ -0,0 +1,18 @@\n+codeunit 50301 \"Hardcoded Secret Client\"\n+{\n+ procedure CallApi()\n+ var\n+ HttpClient: HttpClient;\n+ HttpHeaders: HttpHeaders;\n+ HttpResponseMessage: HttpResponseMessage;\n+ begin\n+ HttpHeaders := HttpClient.DefaultRequestHeaders();\n+ HttpHeaders.Add('X-Api-Key', this.GetApiKey());\n+ HttpClient.Get('https://api.contoso.com/data', HttpResponseMessage);\n+ end;\n+\n+ local procedure GetApiKey(): Text\n+ begin\n+ exit('sk-1234567890abcdef');\n+ end;\n+}\n", "expected_comments": [{"file": "src/HardcodedSecretClient.Codeunit.al", "line_start": 16, "line_end": 16, "severity": "critical", "domain": "security", "body": "Hardcoded API key in source code. Retrieve secrets from encrypted isolated storage or another secure store instead.", "article": "security/isolatedstorage-setencrypted-for-sensitive-values"}], "category": "code-review", "description": "Clean codeunit using SecretText, NonDebuggable, IsolatedStorage.SetEncrypted, and HTTPS enforcement with no security issues", "expect_findings": true, "source": "vsoadmin"} |
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-009", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/OutlookAddinDeployer.Codeunit.al b/src/OutlookAddinDeployer.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/OutlookAddinDeployer.Codeunit.al\n@@ -0,0 +1,37 @@\n+namespace Microsoft.Integration.Outlook;\n+\n+codeunit 50104 \"Outlook Addin Deployer\"\n+{\n+ Access = Internal;\n+\n+ var\n+ EndpointTok: Label 'https://outlook.office365.com/api/v2.0/addins/deploy', Locked = true;\n+ StatusErr: Label 'Deployment failed (HTTP %1): %2', Comment = '%1 is the HTTP status code, %2 is the raw response body.';\n+ ConnectErr: Label 'Failed to connect to the deployment service: %1', Comment = '%1 is the underlying error text.';\n+\n+ procedure DeployAddin(ManifestPath: Text)\n+ var\n+ Client: HttpClient;\n+ Content: HttpContent;\n+ Response: HttpResponseMessage;\n+ Headers: HttpHeaders;\n+ Payload: Text;\n+ ResponseText: Text;\n+ begin\n+ Payload := '{\"manifest\":\"' + ManifestPath + '\"}';\n+ Content.WriteFrom(Payload);\n+ Content.GetHeaders(Headers);\n+ Headers.Add('Authorization', 'Bearer ' + GetAccessToken());\n+ if Client.Post(EndpointTok, Content, Response) then begin\n+ Response.Content.ReadAs(ResponseText);\n+ if not Response.IsSuccessStatusCode() then\n+ Error(StatusErr, Response.HttpStatusCode(), ResponseText);\n+ end else\n+ Error(ConnectErr, GetLastErrorText());\n+ end;\n+\n+ local procedure GetAccessToken(): Text\n+ begin\n+ exit('dummy_access_token_for_testing');\n+ end;\n+}\n", "expected_comments": [{"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 21, "line_end": 21, "domain": "security", "severity": "medium", "body": "The manifest path is concatenated directly into a JSON payload, allowing JSON injection. Build the payload with a JsonObject so values are escaped."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 28, "line_end": 28, "domain": "security", "severity": "medium", "body": "The error surfaces the raw HTTP status code and full response body to the user, leaking internal service details. Log the details and show a generic message."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 30, "line_end": 30, "domain": "security", "severity": "medium", "body": "GetLastErrorText() is shown to the user, exposing internal system details. Log the raw error and present a sanitized message."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 33, "line_end": 33, "domain": "security", "severity": "high", "body": "The access token is returned as plain Text instead of SecretText, exposing it in memory and to the debugger. Return and handle it as SecretText.", "article": "security/secrettext-for-credentials"}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 35, "line_end": 35, "domain": "security", "severity": "high", "body": "A hardcoded access token is embedded in source code. Retrieve the token from a secure store or OAuth flow instead of hardcoding it.", "article": "security/isolatedstorage-setencrypted-for-sensitive-values"}], "category": "code-review", "description": "True positive security findings: error_exposure (verified line numbers)", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-011", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/ElecVATSubmission.Codeunit.al b/src/ElecVATSubmission.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ElecVATSubmission.Codeunit.al\n@@ -0,0 +1,24 @@\n+namespace Microsoft.Finance.VAT;\n+\n+codeunit 13610 \"Elec VAT Submission\"\n+{\n+ Access = Internal;\n+\n+ procedure SubmitReturn(AuthorityUrl: Text): Boolean\n+ var\n+ Client: HttpClient;\n+ Content: HttpContent;\n+ Response: HttpResponseMessage;\n+ begin\n+ Content.WriteFrom('{}');\n+ exit(Client.Post(AuthorityUrl, Content, Response));\n+ end;\n+\n+ procedure CheckHealth(ServiceUrl: Text): Boolean\n+ var\n+ Client: HttpClient;\n+ Response: HttpResponseMessage;\n+ begin\n+ exit(Client.Get(ServiceUrl, Response));\n+ end;\n+}\n", "expected_comments": [{"file": "src/ElecVATSubmission.Codeunit.al", "line_start": 14, "line_end": 14, "domain": "security", "severity": "high", "body": "A caller-supplied URL is passed to HttpClient.Post without host or scheme validation, allowing SSRF requests to internal or attacker-chosen endpoints.", "article": "security/validate-user-configurable-urls"}, {"file": "src/ElecVATSubmission.Codeunit.al", "line_start": 22, "line_end": 22, "domain": "security", "severity": "high", "body": "A caller-supplied URL is passed to HttpClient.Get without host or scheme validation, allowing SSRF requests to internal or attacker-chosen endpoints.", "article": "security/validate-user-configurable-urls"}], "category": "code-review", "description": "True positive security findings: input_validation (trimmed to core input validation cases)", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-012", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/HttpAuthenticationBasic.Codeunit.al b/src/HttpAuthenticationBasic.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/HttpAuthenticationBasic.Codeunit.al\n@@ -0,0 +1,37 @@\n+codeunit 2359 \"Http Authentication Basic\"\n+{\n+ Access = Public;\n+ InherentEntitlements = X;\n+ InherentPermissions = X;\n+\n+ var\n+ Credential: SecretText;\n+ UsernameDomainTok: Label '%1\\%2', Comment = '%1 = domain, %2 = user name', Locked = true;\n+\n+ [NonDebuggable]\n+ procedure Initialize(Username: SecretText; Domain: Text; Password: SecretText)\n+ begin\n+ Credential := SecretStrSubstNo('%1:%2', QualifyUser(Username, Domain), Password);\n+ end;\n+\n+ procedure GetAuthorizationHeader() Header: SecretText\n+ begin\n+ Header := ToBase64(Credential);\n+ end;\n+\n+ [NonDebuggable]\n+ local procedure QualifyUser(Username: SecretText; Domain: Text): SecretText\n+ begin\n+ if Domain = '' then\n+ exit(Username);\n+ exit(SecretStrSubstNo(UsernameDomainTok, Domain, Username));\n+ end;\n+\n+ local procedure ToBase64(Value: SecretText) Base64Value: SecretText\n+ var\n+ Convert: DotNet Convert;\n+ Encoding: DotNet Encoding;\n+ begin\n+ Base64Value := Convert.ToBase64String(Encoding.UTF8().GetBytes(Value.Unwrap()));\n+ end;\n+}\n", "expected_comments": [{"file": "src/HttpAuthenticationBasic.Codeunit.al", "line_start": 30, "line_end": 30, "domain": "security", "severity": "medium", "body": "ToBase64 transforms SecretText credential material and calls Unwrap() without [NonDebuggable], so the plaintext credential is visible in the debugger.", "article": "security/nondebuggable-required-when-unwrapping-secrettext"}], "category": "code-review", "description": "True positive security findings: procedures handling passwords or SecretText values without [NonDebuggable]", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-013", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/ExpenseAgentAdmin.PermissionSet.al b/src/ExpenseAgentAdmin.PermissionSet.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ExpenseAgentAdmin.PermissionSet.al\n@@ -0,0 +1,15 @@\n+// ------------------------------------------------------------------------------------------------\n+// Copyright (c) Microsoft Corporation. All rights reserved.\n+// Licensed under the MIT License. See License.txt in the project root for license information.\n+// ------------------------------------------------------------------------------------------------\n+namespace Microsoft.Agents.Expense;\n+\n+permissionset 50700 \"Expense Agent Admin\"\n+{\n+ Assignable = true;\n+ Caption = 'Expense Agent Administration';\n+\n+ Permissions =\n+ tabledata \"Agent Creation Control\" = RIMD,\n+ tabledata \"Expense Report Rule Violation\" = IMD;\n+}\ndiff --git a/src/ExpenseAgentConsumption.Table.al b/src/ExpenseAgentConsumption.Table.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ExpenseAgentConsumption.Table.al\n@@ -0,0 +1,51 @@\n+// ------------------------------------------------------------------------------------------------\n+// Copyright (c) Microsoft Corporation. All rights reserved.\n+// Licensed under the MIT License. See License.txt in the project root for license information.\n+// ------------------------------------------------------------------------------------------------\n+namespace Microsoft.Agents.Expense;\n+\n+table 50600 \"Expense Agent Consumption\"\n+{\n+ Caption = 'Expense Agent Consumption';\n+ DataClassification = CustomerContent;\n+ InherentEntitlements = RIX;\n+ InherentPermissions = RIX;\n+\n+ fields\n+ {\n+ field(1; \"Entry No.\"; Integer)\n+ {\n+ Caption = 'Entry No.';\n+ DataClassification = SystemMetadata;\n+ AutoIncrement = true;\n+ }\n+ field(10; Amount; Decimal)\n+ {\n+ Caption = 'Amount';\n+ DataClassification = CustomerContent;\n+ }\n+ field(20; \"User Security ID\"; Guid)\n+ {\n+ Caption = 'User Security ID';\n+ DataClassification = EndUserPseudonymousIdentifiers;\n+ }\n+ }\n+\n+ keys\n+ {\n+ key(PK; \"Entry No.\")\n+ {\n+ Clustered = true;\n+ }\n+ }\n+\n+ procedure LogConsumption(CallerSecurityId: Guid; ConsumptionAmount: Decimal)\n+ var\n+ ConsumptionEntry: Record \"Expense Agent Consumption\";\n+ begin\n+ ConsumptionEntry.Init();\n+ ConsumptionEntry.\"User Security ID\" := CallerSecurityId;\n+ ConsumptionEntry.Amount := ConsumptionAmount;\n+ ConsumptionEntry.Insert();\n+ end;\n+}\n", "expected_comments": [{"file": "src/ExpenseAgentConsumption.Table.al", "line_start": 11, "line_end": 11, "domain": "security", "severity": "medium", "body": "InherentEntitlements and InherentPermissions of RIX grant read/insert/execute to every user regardless of assigned permission sets. Remove the inherent grants and control access explicitly.", "article": "security/inherent-permissions-minimal-grant"}, {"file": "src/ExpenseAgentConsumption.Table.al", "line_start": 42, "line_end": 42, "domain": "security", "severity": "medium", "body": "The procedure accepts an arbitrary UserSecurityId, letting a caller log consumption against any user's identity. Derive the user from UserSecurityId() instead of trusting the parameter."}, {"file": "src/ExpenseAgentAdmin.PermissionSet.al", "line_start": 13, "line_end": 13, "domain": "security", "severity": "medium", "body": "RIMD on Agent Creation Control lets assigned users delete creation-control records, removing a security guardrail. Grant only the permissions actually required.", "article": "security/inherent-permissions-minimal-grant"}, {"file": "src/ExpenseAgentAdmin.PermissionSet.al", "line_start": 14, "line_end": 14, "domain": "security", "severity": "medium", "body": "IMD on Expense Report Rule Violation lets users delete recorded policy violations, enabling them to hide their own violations. Remove delete and modify access.", "article": "security/inherent-permissions-minimal-grant"}], "category": "code-review", "description": "True positive security findings: permission (trimmed to 5 representative findings)", "expect_findings": true, "source": "vsoadmin"} |
Comment on lines
+61
to
+64
| # BCQuality knowledge article this finding derives from, as `<domain>/<slug>` | ||
| # (e.g. `security/hardcoded-secret`). Optional and backward-compatible; drives | ||
| # per-article coverage tracking. Older entries leave it unset (counted as unannotated). | ||
| article: Annotated[str, Field(pattern=r"^[a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*$")] | None = None |
| write_step_outputs({github_output: entry.environment_setup_version}) | ||
|
|
||
|
|
||
| @dataset_app.command("coverage") |
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-015", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/ExternalIntegrationMgt.Codeunit.al b/src/ExternalIntegrationMgt.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ExternalIntegrationMgt.Codeunit.al\n@@ -0,0 +1,24 @@\n+namespace Microsoft.Integration.Partner;\n+\n+codeunit 50205 \"External Integration Mgt.\"\n+{\n+ Access = Internal;\n+\n+ procedure PostToPartner(EndpointUrl: Text): Boolean\n+ var\n+ Client: HttpClient;\n+ Content: HttpContent;\n+ Response: HttpResponseMessage;\n+ begin\n+ Content.WriteFrom('{}');\n+ exit(Client.Post(EndpointUrl, Content, Response));\n+ end;\n+\n+ procedure GetFromProvider(EndpointUrl: Text): Boolean\n+ var\n+ Client: HttpClient;\n+ Response: HttpResponseMessage;\n+ begin\n+ exit(Client.Get(EndpointUrl, Response));\n+ end;\n+}\n", "expected_comments": [{"file": "src/ExternalIntegrationMgt.Codeunit.al", "line_start": 14, "line_end": 14, "domain": "security", "severity": "high", "body": "A caller-supplied URL is passed to HttpClient.Post without host or scheme validation, allowing SSRF requests to internal or attacker-chosen endpoints."}, {"file": "src/ExternalIntegrationMgt.Codeunit.al", "line_start": 22, "line_end": 22, "domain": "security", "severity": "high", "body": "A caller-supplied URL is passed to HttpClient.Get without host or scheme validation, allowing SSRF requests to internal or attacker-chosen endpoints."}], "category": "code-review", "description": "True positive security findings: URLs from table fields used in HTTP requests without validation (SSRF risk). Three procedures use user-configurable URLs directly, while two procedures correctly validate using Uri.AreURIsHaveSameHost and Uri.IsValidURIPattern.", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-016", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/ExpenseHtmlNotifier.Codeunit.al b/src/ExpenseHtmlNotifier.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/ExpenseHtmlNotifier.Codeunit.al\n@@ -0,0 +1,12 @@\n+codeunit 50900 \"Expense Html Notifier\"\n+{\n+ Access = Internal;\n+\n+ var\n+ BodyTemplateTok: Label '<p>Dear %1,</p><p>%2</p>', Locked = true;\n+\n+ internal procedure BuildNotificationBody(EmployeeName: Text; Description: Text): Text\n+ begin\n+ exit(StrSubstNo(BodyTemplateTok, EmployeeName, Description));\n+ end;\n+}\n", "expected_comments": [{"file": "src/ExpenseHtmlNotifier.Codeunit.al", "line_start": 10, "line_end": 10, "domain": "security", "severity": "high", "body": "User-supplied EmployeeName and Description are substituted into the HTML body without encoding, enabling stored or reflected XSS. HTML-encode the values before embedding them."}], "category": "code-review", "description": "True positive security findings: xss (user-supplied data embedded in HTML without encoding)", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-008", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/PartnerConfigKeyManager.Codeunit.al b/src/PartnerConfigKeyManager.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/PartnerConfigKeyManager.Codeunit.al\n@@ -0,0 +1,9 @@\n+codeunit 50100 \"Partner Config Key Manager\"\n+{\n+ Access = Internal;\n+\n+ internal procedure StoreApiKey(KeyName: Text; ApiKey: Text)\n+ begin\n+ IsolatedStorage.SetEncrypted(KeyName, ApiKey, DataScope::Module);\n+ end;\n+}\n", "expected_comments": [{"file": "src/PartnerConfigKeyManager.Codeunit.al", "line_start": 5, "line_end": 5, "domain": "security", "severity": "high", "body": "The API key is accepted as a plain Text parameter instead of SecretText, so the secret is exposed in memory and to anyone inspecting the call stack or debugger.", "article": "security/secrettext-for-credentials"}], "category": "code-review", "description": "True positive security findings: encryption (trimmed to 5 representative findings)", "expect_findings": true, "source": "vsoadmin"} | ||
| {"repo": "microsoft/BCApps", "instance_id": "synthetic__security-009", "base_commit": "70fd0246a0a4dbc72cb183ca719106722c03be4d", "created_at": "2026-05-15T00:00:00Z", "environment_setup_version": "27.0", "project_paths": [], "metadata": {"area": "security"}, "patch": "diff --git a/src/OutlookAddinDeployer.Codeunit.al b/src/OutlookAddinDeployer.Codeunit.al\nnew file mode 100644\n--- /dev/null\n+++ b/src/OutlookAddinDeployer.Codeunit.al\n@@ -0,0 +1,37 @@\n+namespace Microsoft.Integration.Outlook;\n+\n+codeunit 50104 \"Outlook Addin Deployer\"\n+{\n+ Access = Internal;\n+\n+ var\n+ EndpointTok: Label 'https://outlook.office365.com/api/v2.0/addins/deploy', Locked = true;\n+ StatusErr: Label 'Deployment failed (HTTP %1): %2', Comment = '%1 is the HTTP status code, %2 is the raw response body.';\n+ ConnectErr: Label 'Failed to connect to the deployment service: %1', Comment = '%1 is the underlying error text.';\n+\n+ procedure DeployAddin(ManifestPath: Text)\n+ var\n+ Client: HttpClient;\n+ Content: HttpContent;\n+ Response: HttpResponseMessage;\n+ Headers: HttpHeaders;\n+ Payload: Text;\n+ ResponseText: Text;\n+ begin\n+ Payload := '{\"manifest\":\"' + ManifestPath + '\"}';\n+ Content.WriteFrom(Payload);\n+ Content.GetHeaders(Headers);\n+ Headers.Add('Authorization', 'Bearer ' + GetAccessToken());\n+ if Client.Post(EndpointTok, Content, Response) then begin\n+ Response.Content.ReadAs(ResponseText);\n+ if not Response.IsSuccessStatusCode() then\n+ Error(StatusErr, Response.HttpStatusCode(), ResponseText);\n+ end else\n+ Error(ConnectErr, GetLastErrorText());\n+ end;\n+\n+ local procedure GetAccessToken(): Text\n+ begin\n+ exit('dummy_access_token_for_testing');\n+ end;\n+}\n", "expected_comments": [{"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 21, "line_end": 21, "domain": "security", "severity": "medium", "body": "The manifest path is concatenated directly into a JSON payload, allowing JSON injection. Build the payload with a JsonObject so values are escaped."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 28, "line_end": 28, "domain": "security", "severity": "medium", "body": "The error surfaces the raw HTTP status code and full response body to the user, leaking internal service details. Log the details and show a generic message."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 30, "line_end": 30, "domain": "security", "severity": "medium", "body": "GetLastErrorText() is shown to the user, exposing internal system details. Log the raw error and present a sanitized message."}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 33, "line_end": 33, "domain": "security", "severity": "high", "body": "The access token is returned as plain Text instead of SecretText, exposing it in memory and to the debugger. Return and handle it as SecretText.", "article": "security/secrettext-for-credentials"}, {"file": "src/OutlookAddinDeployer.Codeunit.al", "line_start": 35, "line_end": 35, "domain": "security", "severity": "high", "body": "A hardcoded access token is embedded in source code. Retrieve the token from a secure store or OAuth flow instead of hardcoding it.", "article": "security/isolatedstorage-setencrypted-for-sensitive-values"}], "category": "code-review", "description": "True positive security findings: error_exposure (verified line numbers)", "expect_findings": true, "source": "vsoadmin"} |
…erage # Conflicts: # src/bcbench/dataset/__init__.py
Wenjie Fan (gggdttt)
marked this pull request as ready for review
August 11, 2026 13:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds per-article BCQuality coverage tracking to the
code-reviewcategory so we can see, per BCQuality knowledge article, which articles have gold coverage and which have none. Follow-up from the 2026-08-06 PR Review Agent sync (AB#646153).Why
Today gold entries are classified only by coarse
domain/area(7 values). We cannot tell which specific BCQuality article an entry exercises, nor which articles have zero gold coverage. This blocks Wael's self-improving-agent pipeline from targeting coverage gaps.Changes
ReviewComment.article— the<domain>/<slug>BCQuality article a finding derives from (e.g.security/secrettext-for-credentials).metadata.articles— entry-level list for false-positive-guard entries (expected_comments = []) that test an article by omission and so have no per-commentarticle.dataset/coverage.py— pure logic: aggregate declared articles, enumerate the BCQuality article inventory from a checkout, compute covered / zero-coverage / unknown-slug / unannotated.bcbench dataset coverage— renders a per-domain coverage table. Inventory comes from--bcquality-rootor$BCQUALITY_ROOT(reusing the engine'sBCQUALITY_ROOTconvention); when absent it degrades to declared-only (zero-coverage undetermined). Supports--github-output(JSON) for CI.Example output
Validation
ruff format/checkclean,tyclean (the one remainingtydiagnostic is a pre-existingunused-ignoreinredteam.pyfrom main, untouched here),pytest717 passed / 2 skipped (+11 new coverage tests). Dataset still loads all 110 entries; the JSONL diff touches only the security lines (original per-line separator style preserved).bcbench dataset coverage --bcquality-root <checkout>verified end-to-end.Notes
main(includes Add neutral ignored_comments bucket to code-review scoring #768ignored_commentsand Collect code-review entries per reviewed commit (bug 643990) #770 per-commit collection); only conflict was thedataset/__all__export list.ignored_commentswork (Add neutral ignored_comments bucket to code-review scoring #768): FP-guard gold cases can be classified by the article whose false positive they guard.AB#646153