Medical Reasoning Evaluation

Medical Reasoning Evaluation is a medical framework for evaluating language models across 10 standardized medical datasets and scoring strategies, with the Verifiers and Verl RL frameworks. The framework provides standardized dataset adapters, answer extraction heuristics, and multiple scoring strategies to measure medical reasoning capabilities.

Features

• 10 medical datasets - MedQA, MedMCQA, PubMedQA, MedBullets, MetaMedQA, MMLU-Pro Health, MedXpertQA, HealthBench, MedCaseReasoning, PubHealthBench.
• Dual evaluation frameworks - Verifiers (environment-based) and Verl (async with Groq rollouts).
• Five scoring strategies - Multiple-choice Accuracy, LLM-as-judge, Rubric-based Multi-criteria, Semantic Equivalence, and Hybrid Routing.
• Multi-strategy answer extraction - XML tags, boxed LaTeX, anchored phrases, last-token fallback with negation detection.
• Resilient API integration - Exponential backoff retry logic with configurable limits for rate-limited APIs.
• Streaming and in-memory modes - Stream large datasets lazily or load them fully for random access.

Evaluation Methods

• Multiple-Choice Accuracy: Letter/option extraction with normalization and negation-aware matching.
• LLM-as-Judge: Model-based correctness checks for open-ended responses where exact matching is insufficient.
• Rubric-based Multi-Criteria Scoring: Criterion-level judging with normalized totals for structured evaluation.
• Semantic Equivalence: Normalized matching utilities for free-form answers with valid phrasing variation.
• Hybrid Routing: Per-example routing between MCQ scoring and judge-based scoring for mixed-task datasets.

Datasets

Dataset	HuggingFace	Task Type	Evaluation Method	Why this strategy
MedQA	GBaker/MedQA-USMLE-4-options	MCQ (A-D)	MCQ accuracy	Canonical single-answer MCQ benchmark
MedMCQA	lighteval/med_mcqa	MCQ (A-D)	MCQ accuracy	Structured options with deterministic key
PubMedQA	openlifescienceai/pubmedqa	MCQ (Yes/No/Maybe)	MCQ accuracy	Canonical 500-example test split with fixed labels
MedBullets	mkieffer/MedBullets	MCQ (A-D or A-E)	MCQ accuracy	Option-based exam format
MetaMedQA	maximegmd/MetaMedQA	MCQ (A-E)	MCQ accuracy	Option-based QA with deterministic answer
MMLU-Pro Health	TIGER-Lab/MMLU-Pro	MCQ (A-J)	MCQ accuracy	Broad option space, still closed-form grading
MedXpertQA	TsinghuaC3I/MedXpertQA	MCQ (expert-level)	MCQ accuracy	Expert MCQ format with fixed answer key
HealthBench	neuralleap/healthbench-*	Rubric-based	Multi-criteria rubric scoring	Requires criterion-level quality judgment
MedCaseReasoning	zou-lab/MedCaseReasoning	Open-ended diagnosis	LLM-as-judge	Open-form diagnoses need semantic judging
PubHealthBench	Joshua-Harris/PubHealthBench	Mixed (MCQ + freeform)	Hybrid (MCQ + judge)	Mixed examples require dynamic scoring route

Installation

The project uses uv for dependency management. First, ensure uv is installed:

# Install uv (if not already installed)
pip install uv

Then install the project dependencies:

# Install dependencies
uv sync

# Activate the virtual environment
source .venv/bin/activate

Configuration

Create a .env file in the repository root:

GROQ_API_KEY=your-groq-key

Quick Start

1) Load a dataset adapter

from med_reason_evals import MedQADataset

dataset = MedQADataset(split="test", streaming=False)
verifiers_dataset = dataset.get_verifiers_dataset()
verl_dataset = dataset.get_verl_dataset()

2) Run evaluation with Verifiers

evaluate() is a coroutine — await it, or wrap it in asyncio.run() as below.

import asyncio
import os
from openai import AsyncOpenAI
from med_reason_evals.verifiers import MedQAEvaluator

client = AsyncOpenAI(
    api_key=os.environ["GROQ_API_KEY"],
    base_url="https://api.groq.com/openai/v1",
)

evaluator = MedQAEvaluator(use_think=True, answer_format="xml")
results = asyncio.run(
    evaluator.evaluate(
        client=client,
        model="openai/gpt-oss-120b",
        num_examples=100,
    )
)

# GenerateOutputs is a TypedDict — access it by key, not attribute.
# Per-example parallel arrays: results["reward"][i] pairs with results["prompt"][i]
rewards = results["reward"]
print(sum(rewards) / len(rewards))

3) Run evaluation with Verl

import asyncio
from med_reason_evals.verl import MedQAEvaluator

evaluator = MedQAEvaluator(split="test")
results = asyncio.run(evaluator.evaluate(num_examples=100))

Documentation

• Package overview: src/med_reason_evals/
• Dataset adapters: src/med_reason_evals/data/
• Shared utilities: src/med_reason_evals/utils/
• Verifiers-based evaluation: src/med_reason_evals/verifiers/
• Verifiers utilities: src/med_reason_evals/verifiers/utils/
• Verifiers rewards: src/med_reason_evals/verifiers/rewards/
• Verl-based evaluation: src/med_reason_evals/verl/
• Verl rewards: src/med_reason_evals/verl/rewards/

Contributing

Please see CONTRIBUTING.md for contribution guidelines.

License

This project is licensed under the MIT License. See LICENSE for details.