README.md

LLM-as-Meta-Reviewer Assistants'

Folder Organization

• Annotation Files
  • Contains human annotation for micro and macro evaluation.
• LLM Responses
  • This folder contains responses generated by three LLMs (i.e., PalM2, GPT-3.5, and Llama2) for four prompt levels. Here, LLM1, LLM2, and LLM3 represents PalM2, GPT-3.5, and Llama2. The responses are in the columns res_prompt1,res_prompt2, res_prompt3, and res_prompt4.
• GPT Ratings
  • Contains the ratings given by GPT-4o on the LLM-generated responses. The human and GPT ratings are merged together in the files. GPT_rating_LLMx and Macro_GPT_score contains micro and macro evaluation ratings of both human and GPT-4o respectively.
  • corr_average_likert_ceil file contains the correlations between human and GPT-4o in micro-evaluation for each LLM.
• Meta-Review Data
  • The curated dataset used for the meta-review generation. Review1, Review2, and Review3 columns have the peer reviews and Meta_Review column contain the meta-reviews written by humans.

Citation

If you find our work useful for your research then cite using this BibTeX:

@article{hossain2024llms,
  title={LLMs as Meta-Reviewers' Assistants: A Case Study},
  author={Hossain, Eftekhar and Sinha, Sanjeev Kumar and Bansal, Naman and Knipper, Alex and Sarkar, Souvika and Salvador, John and Mahajan, Yash and Guttikonda, Sri and Akter, Mousumi and Mahadi Hassan, Md and others},
  journal={arXiv e-prints},
  pages={arXiv--2402},
  year={2024}
}