Skip to content

Support multiple reviewer/area chair roles per submission#3009

Open
melisabok wants to merge 18 commits intomasterfrom
fix/parametrize-reviewers-roles
Open

Support multiple reviewer/area chair roles per submission#3009
melisabok wants to merge 18 commits intomasterfrom
fix/parametrize-reviewers-roles

Conversation

@melisabok
Copy link
Copy Markdown
Member

@melisabok melisabok commented Apr 9, 2026

Summary

Lets a venue define more than one reviewer role (and more than one area chair role), with a choice between a shared per-submission group or separate per-role per-submission groups. Single-role venues behave identically to before.

Motivation

Some venues need to run distinct reviewer committees for different kinds of submissions or different evaluation criteria — for example Expert_Reviewers vs Technical_Reviewers, or a main AC track alongside a technical AC track. The workflow already accepted alternate names in reviewer_roles / area_chair_roles at the top-level group, but the per-submission plumbing collapsed everything back to a single primary role: one Submission1/Reviewers group, one Official_Review invitation, one assignment deployment, one Submitted subgroup. That made it impossible to keep the committees separate once submissions came in — assignments, review forms, and "submitted" tracking all landed on the wrong group, and notifications fanned out to the umbrella instead of the reviewer's own role.

This branch treats the multi-role case as a first-class configuration. Two new Venue Request fields — reviewer_group_layout and area_chair_group_layout — let the PC pick shared (existing behavior) or per_role (each role gets its own per-submission group). The role name is then threaded through group creation, invitation setup, matching, assignment deployment, and the review process function, so per-role groups, per-role review forms, and per-role notifications all work end-to-end.

Key changes

  • New Venue / ReviewStage attributes submission_reviewer_roles and submission_area_chair_roles drive how per-submission groups and review invitations are fanned out.
  • get_reviewers_id / get_area_chairs_id / get_anon_reviewer_id take an optional name= so callers can target a specific role.
  • Group builder, invitation builder, matching, and assignment deployment iterate over the configured roles instead of using a single primary name.
  • Assignment invitations carry a submission_committee_name; matching deployment reads it instead of inferring the target group name from the committee id.
  • The review process function resolves the reviewer's role from the anon signature prefix and scopes the Submitted subgroup and reviewer-email recipients to that role.
  • New reviewer_roles / reviewer_group_layout / area_chair_roles / area_chair_group_layout fields on the Conference_Review_Workflow request form.

Test plan

  • pytest tests/test_two_submission_committee_roles.py (new end-to-end with two reviewer roles and two AC roles under per_role).
  • pytest tests/test_reviewers_only.py (single-role regression).
  • pytest tests/test_acs_and_reviewers.py (reviewer + AC regression).
  • pytest tests/test_matching_v2.py tests/test_icml_conference_with_templates.py tests/test_tasks.py tests/test_change_venue_email.py.

@melisabok melisabok changed the title Fix/parametrize reviewers roles Support multiple reviewer/area chair roles per submission Apr 10, 2026
@melisabok melisabok marked this pull request as ready for review April 23, 2026 20:11
Copilot AI review requested due to automatic review settings April 23, 2026 20:11
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class support for venues that have multiple per-submission reviewer and/or area-chair roles (e.g., Expert_Reviewers + Technical_Reviewers), ensuring per-role paper groups, assignment deployment, review invitations, and notifications target the correct role-specific groups while preserving single-role behavior.

Changes:

  • Introduces submission_reviewer_roles / submission_area_chair_roles and updates venue/group/invitation/stage logic to generate per-role per-paper groups and role-specific review invitation wiring.
  • Updates matching + assignment deployment to resolve the per-paper committee group name from the deployed Assignment invitation (submission_committee_name) instead of parsing group ids.
  • Refactors recruitment processes to derive Invited/Declined/Recruitment invitation ids from committee_id (removing per-role domain content keys), and adds an end-to-end test covering a two-role workflow.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/test_two_submission_committee_roles.py New end-to-end test covering two reviewer roles + two AC roles under per_role layout.
tests/test_tasks.py Updates assertions to reflect derived recruitment group ids.
tests/test_reviewers_only.py Removes assertions for domain content keys that are no longer stored.
tests/test_matching_v2.py Sets submission_*_roles explicitly for matching tests.
tests/test_icml_conference_with_templates.py Wires ReviewStage with submission_reviewer_roles for template workflow regression coverage.
tests/test_change_venue_email.py Updates assertions to reflect derived recruitment group ids.
tests/test_acs_and_reviewers.py Removes assertions for domain content keys that are no longer stored.
openreview/workflows/workflows.py Adds request-form fields for multi-role reviewer/AC configuration and group layout selection.
openreview/workflows/workflow_process/conference_review_workflow_deployment.py Sets default ReviewStage to target the primary submission reviewer role.
openreview/workflows/workflow_process/committee_invited_group_template_process.py Stops persisting invited-message invitation ids on the domain.
openreview/workflows/workflow_process/committee_group_template_process.py Stops persisting per-role committee id/invited/declined/recruitment ids on the domain.
openreview/workflows/process/deploy_assignments_process.py Forces materialization of per-role paper groups from proposed assignment edges before status updates.
openreview/workflows/process/committee_recruitment_response_process.py Derives declined + invited-message ids directly from committee_id.
openreview/workflows/process/committee_recruitment_request_reminder_process.py Derives invited/declined/recruitment/message ids directly from committee_id.
openreview/workflows/process/committee_recruitment_request_process.py Derives invited group + recruitment/message ids directly from committee_id.
openreview/workflows/process/committee_recruitment_request_edit_reminder_process.py Derives invited/declined/recruitment/message ids directly from committee_id.
openreview/venue/venue.py Adds submission role lists, name overrides for committee id helpers, and per-role creation of submission-group + assignment/matching setup.
openreview/venue/process/review_process.py Tracks reviewer roles per invitation; updates Submitted subgroup creation + reviewer notification targeting.
openreview/venue/matching.py Uses submission_committee_name from deployed assignment invitation to deploy/undeploy into correct per-paper role groups.
openreview/venue/invitation.py Stores reviewer_roles on review invitations; role-specific per-paper invitees/readers; persists submission_committee_name on Assignment invitations.
openreview/venue/helpers.py Loads submission_*_roles from the domain group content.
openreview/venue/group.py Threads role overrides into per-paper group invitations; persists submission role lists; creates umbrella groups for multi-role venues.
openreview/stages/venue_stages.py Adds submission_reviewer_roles to ReviewStage; iterates roles for readers/signatures; allows identity-reader overrides per role.
openreview/conference/helpers.py Keeps legacy Conference path single-role by explicitly setting submission_reviewer_roles.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread openreview/venue/group.py
Comment on lines 109 to 113
readers.append(self.venue.get_senior_area_chairs_id(number))
readers.append(self.venue.get_area_chairs_id(number))
readers.append(self.venue.get_area_chairs_id(number, name=name))
if openreview.stages.IdentityReaders.REVIEWERS_ASSIGNED in self.venue.area_chair_identity_readers:
readers.append(self.venue.get_reviewers_id(number))
return readers
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In per_role layouts, self.venue.get_reviewers_id(number) can point to a per-submission group that is never created (e.g. .../Submission1/Reviewers when submissions use .../Submission1/Expert_Reviewers + .../Submission1/Technical_Reviewers). This means area-chair paper-group readers may reference a non-existent group and won't actually grant access to the assigned reviewers. Consider deriving the correct per-submission reviewer group(s) from name (e.g., map AC role -> corresponding submission_reviewer_roles entry) or include all venue.submission_reviewer_roles for that paper when REVIEWERS_ASSIGNED is enabled.

Copilot uses AI. Check for mistakes.
Comment thread openreview/venue/process/review_process.py
Comment thread openreview/workflows/process/deploy_assignments_process.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants