Error in reproducing the results

[lezhang7]

Thanks for the great work. Could you explain how to evaluate on BEIR to reproduce the results?

I tried

bash launch_job.sh jhu-clsp/rank1-7b NFCorpus default 1
but got

Output directory: results/jhu-clsp/rank1-7b/NFCorpus_None
WARNING:mteb.models.overview:Failed to extract metadata from model: 'rank1' object has no attribute 'model_card_data'. Upgrading to sentence-transformers v3.0.0 or above is recommended.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Selected tasks  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Retrieval
    - NFCorpus, s2p
    - ```

[orionw]

Hi @lezhang7!

That looks like the message is cut off. That is just a warning message, there should be the output and the evaluation after that. Without going too deep, it looks like you're running it correctly

Let me know if there was another error message or a discrepancy in the score.

[lezhang7]

Hi @orionw . I appreciate your quick response. There's indeed a output file at rank1/results/jhu-clsp/rank1-7b/NFCorpus_None/no_model_name_available/no_revision_available/NFCorpus.json

but the result is null

{
  "dataset_revision": "ec0fa4fe99da2ff19ca1214b7966684033a58814",
  "task_name": "NFCorpus",
  "mteb_version": "1.38.9",
  "scores": {},
  "evaluation_time": 0.08560919761657715,
  "kg_co2_emissions": null
}

[lezhang7]

Hi @orionw ， I am trying to catch up for EMNLP, and would like to incorporate this method as a strong baseline. Is it possible to help me solve the issues? Or do you have the results for the method on plain BM25 top100 results without GPT-4 rewriting the queries? Appreciate your reply

[orionw]

Hi @lezhang7, I don't have that result ready.

MTEB is designed to skip any run that already has a file. If you want to re-run it you'll need to delete that file. Then you can run it again and it will attempt to evaluate.

I'm not sure what would cause an empty run, but if you run it with no run file saved, it should give you a more informative message. I haven't seen this problem myself.

Feel free to post what that new log says. The other thing I would look into is checking your environment and making sure it matches. My MTEB version I forked has some differences with the regular version so you should be using that version, etc.