When Identity Skews Debate

Official code for the ACL 2026 Oral Paper:

When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning
Hyeong Kyu Choi, Xiaojin Zhu, and Sharon Li
arXiv:2510.07517v5

Overview: Identity Bias Coefficient (IBC)

For agent i with peer j, the paper defines:

Conformity_i = P[ y_{i,t}   = y_{j,t-1}  |  y_{i,t-1} ≠ y_{j,t-1} ]
Obstinacy_i  = P[ y_{i,t}   = y_{i,t-1}  |  y_{i,t-1} ≠ y_{j,t-1} ]
Δ_i          = Conformity_i − Obstinacy_i
IBC_i        = Δ_i^vanilla  − Δ_i^anonymized

debate/metrics.py aggregates these across all (sample, agent) pairs in the homogeneous single-peer ring topology.

---

Setup

1. Clone and create the conda environment

git clone https://github.com/deeplearning-wisc/MAD-identity-bias.git
cd MAD-identity-bias

conda env create -f environment.yml
conda activate identitybias

2. Hugging Face access token

Several checkpoints (e.g. Llama 3.1, gpt-oss-20b) are gated. Put a token with read access in a one-line file at the repo root:

echo "hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" > token

main.py and the model wrappers will read this file automatically.

3. (Optional) Configure cache directories

By default, Hugging Face caches models and datasets under ~/.cache/. To redirect them to a larger drive, export before running:

export HF_HOME=/path/to/hf-cache
export HF_DATASETS_CACHE=/path/to/datasets-cache

The scripts in scripts/ additionally accept a --model_dir / --data_dir argument forwarded to main.py.

---

Run Experiments

Smoke test — one experiment

A short end-to-end check that the pipeline is wired up correctly:

python main.py \
  --model qwen2.5-7b --data pro_medicine \
  --data_size 5 --num_agents 5 --debate_rounds 1 \
  --modes vanilla anonymized

python analyze.py      # writes out/results/summary.json
python dashboard.py    # writes out/dashboard/index.md + delta_reduction.png

Full sweep

Each scripts/run-*.sh runs one model across the four datasets and both modes (vanilla + anonymized). Run them sequentially or across separate GPUs:

bash scripts/run-qwen.sh       # qwen2.5-7b
bash scripts/run-llama.sh      # llama3.1-8b
bash scripts/run-mistral.sh    # mistral0.3
bash scripts/run-qwen32.sh     # qwen2.5-32b (needs ≥ 2 GPUs)
bash scripts/run-gptoss.sh     # gpt-oss-20b

Useful environment overrides recognized by the scripts:

Variable	Default	Purpose
DATA_SIZE_OVERRIDE	0 (paper-default sizes)	Cap each dataset for a fast smoke run
DATASETS	all four	Whitespace-separated subset to run
NUM_AGENTS	5	Number of agents in the debate
ROUNDS	1	Debate rounds (set 2+ to plot per-round IBC)
PYTHON	python	Interpreter to use

After all sweeps finish:

python analyze.py     # aggregates out/history/*.jsonl → out/results/summary.json
python dashboard.py   # writes out/dashboard/index.md + delta_reduction.png

Heterogeneous-agent setting

Add --multi_persona to main.py to swap homogeneous agents for the heterogeneous persona pool defined in model/model_utils.py.

python main.py --model qwen2.5-7b --data pro_medicine --multi_persona \
  --num_agents 5 --debate_rounds 1 --modes vanilla anonymized

Outputs

• out/history/<run>.jsonl — one line per sample, recording every agent's raw response, extracted answer, and per-round correctness for both Vanilla and Anonymized debates. Re-runs append to this file; delete the file (or rename) to start fresh.
• out/results/<run>.json — recomputed metrics for one experiment.
• out/results/summary.json — table-friendly aggregate keyed by (model, dataset, persona setting). Contains every metric, including the ones the dashboard intentionally omits.
• out/dashboard/index.md — markdown dashboard with run inventory and IBC values.
• out/dashboard/charts/delta_reduction.png — the single figure embedded by the dashboard.

---

Citation

@inproceedings{choi2026identity,
  title     = {When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning},
  author    = {Choi, Hyeong Kyu and Zhu, Xiaojin and Li, Sharon},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics},
  year      = {2026},
}