feat: denylist high-risk corpus categories in validation (ABE-2)

[HofniGartner]

Summary

Implements mitigation ABE-2 from the ai-agent-risk resync run: block High-risk corpora from eval submissions.

• Add a denylist (not allowlist) of prohibited corpus categories: hr_personal, customer_support_identifiable, credentials_secrets, financial_payment, sensitive_slack_dms, legal_privileged
• Fail validate when metadata.yaml corpus_categories or supportive/ YAML/JSON declares denylisted tags
• README note + unit tests

Risk assessment

Run: 2026-08-09T060900Z_must-should-resync

Mitigation: ABE-2 — Do not feed High-category corpora into evals

Test plan

• pytest tests/test_corpus_denylist.py

Made with Cursor

[GuyZivRH]

PR #68 Consolidated Review — Corpus category denylist (ABE-2)

PR: #68
Author: HofniGartner
Branch: HofniGartner:feat/abe-2-corpus-denylist → main (cross-fork)
HEAD: ce216f6
Size: +211 / −0 · 5 files · 1 commit
CI at consolidation: none reported (fork PR)
Merge state: MERGEABLE, BEHIND main (missing #60 Konflux 2b8485c)
Consolidated: 2026-08-11 · prefix twoct_
Sources folded in: opmam_, mzkpw_, ujrnt_, ykprt_, xkdfg_

---

Final verdict

Request changes.

Consensus likes the ABE-2 shape (denylist, validate-time, supportive scan, tests, README). Two reviews would Approve now; three ask for changes or hold for CI. The decisive technical issue is metadata categories are not normalized before denylist match — hr-personal / HR-Personal bypass while supportive YAML/JSON correctly normalize. That is a trivial hole in a security-policy control and must be fixed before merge. Also rebase for CI and strip Cursor co-author per repo convention.

---

What shipped

File	Role
abevalflow/corpus_denylist.py	Denylist set + normalize/scan/validate helpers
abevalflow/schemas.py	Optional SubmissionMetadata.corpus_categories
scripts/validate.py	Hook when metadata parses
tests/test_corpus_denylist.py	6 tests (all pass locally)
README.md	Operator-facing ABE-2 note

Denied tags: hr_personal, customer_support_identifiable, credentials_secrets, financial_payment, sensitive_slack_dms, legal_privileged.

Enforcement: metadata.yaml corpus_categories + top-level corpus_categories / corpus_category in supportive/**/*.{yaml,yml,json}.

---

Verdict tally

Review	Verdict	Notes
opmam_	Request changes	Metadata normalize bypass (confirmed); Cursor; rebase
ykprt_	Request changes	Hold for CI; missed normalize bug; said Cursor OK
mzkpw_	Approve w/ minor	Supportive-only e2e test; incorrectly assumed case/spacing always normalized
ujrnt_	Approve w/ nits	Also flags metadata normalize + list-of-dicts scan + Cursor
xkdfg_	Approve	Missed normalize asymmetry; treated parse-skip as pure strength

---

Consensus strengths

1. Denylist over allowlist — unknown tags allowed; only known High-risk block.
2. Clean module boundary — corpus_denylist.py + thin validate.py hook.
3. Supportive scan — recursive YAML/JSON; scalar + list; corpus_category singular.
4. Clear ABE-2 error — lists blocked tags; points at public fixtures.
5. Runs across engines — not harbor-only; whenever metadata parses.
6. Tests + README — 6 passed locally; schema field documents the six categories.

---

Must fix before merge

1. Normalize metadata corpus_categories before match

validate_corpus_categories appends raw declared then runs find_denylisted (exact match). Supportive path goes through collect_categories → normalize_category. Result:

Declared value	Blocked?
hr_personal	yes
hr-personal	no
HR-Personal	no

Ask: collect_categories(declared) (or equivalent) before match; add regression test that corpus_categories: [hr-personal] fails validate_submission. Optional: Pydantic field_validator to normalize on load.

Raised by opmam_, ujrnt_. Contradicts mzkpw_ / ykprt_ security notes that assumed full-path normalization.

2. Strip AI co-author / “Made with Cursor”

Commit has Co-authored-by: Cursor; PR body has “Made with Cursor”. Repo convention: no AI author/co-author. (ykprt_ treated this as acceptable — minority; fold as must for hygiene.)

3. Rebase on main + get CI signal

Branch is 1 behind (#60). Cross-fork shows empty checks. Rebase/push so Actions run (or document full local suite on tip). Hard gate for ykprt_; agreed by all others as should/must.

---

Should fix

#	Issue	Sources	Ask
1	Supportive-only e2e test	mzkpw_	validate_submission with clean metadata + denylisted tag only in supportive/
2	Fail-open parse skips	opmam_, mzkpw_	Invalid YAML/JSON continues — prefer error/warn for corpus-tag files, or document fail-open
3	List / multi-doc YAML	ujrnt_, opmam_	Top-level list-of-dicts and multi-doc ignored — handle or document
4	Skip when metadata is None	mzkpw_, ujrnt_	Supportive scan skipped if metadata fails parse — consider unconditional supportive scan
5	CRLF on new modules	opmam_	Convert corpus_denylist.py / tests to LF
6	Scope honesty	opmam_, mzkpw_	Tag declaration only — not content/PII scanning; note in README
7	Normalization unit test	mzkpw_	Assert HR Personal → hr_personal
8	Docs cross-ref	mzkpw_	Mention ABE-2 in Docs/certification_and_checks.md

---

Nice to have / out of scope

• Schema-level denylist validator for clearer field errors (opmam_).
• Scan tags if teams misuse it for sensitivity (opmam_).
• Externalize denylist when ai-agent-risk taxonomy evolves (mzkpw_).
• Undeclared sensitive content in CSV/PDF — not ABE-2 (mzkpw_, xkdfg_ overstated “deep” scan; only top-level keys).
• Internal GitLab risk-report URL may be inaccessible outside Red Hat (ykprt_).

---

Minority / corrected claims

Claim	Review	Consolidation
“Case/spacing variants — Normalized — good”	mzkpw_	Only supportive path; metadata path is the must-fix
“Normalization … ensure submitters know canonical form”	ykprt_	Incomplete — metadata bypass exists today
Cursor co-author “acceptable”	ykprt_	Reject — strip per repo convention
“Ready to merge” / Approved	xkdfg_	Overturned by normalize bug + no CI
Silent parse skip as pure strength	xkdfg_	Strength for crash-safety; weakness for fail-closed policy

---

Suggested GitHub action

1. Request changes on must-fix #1 (normalize metadata + test) and #2 (Cursor trailer).
2. Ask rebase onto main for CI.
3. After fixes + green CI (or explicit tip suite): Approve. Supportive-only e2e + list-of-dicts can land in the same fixup or a fast follow.

---

Merge bar

• Metadata categories normalized before denylist match + hyphen/case regression test
• No AI co-author / “Made with Cursor” on commit/PR
• Rebased on main
• CI green or equivalent local proof on tip
• Prefer: supportive-only validate_submission test
• Prefer: document or harden parse-skip / nested / list structures
• Prefer: LF on new Python files