success rate of libero-spatial is lower than the paper

[atticus77777]

Install conda completely according to the readme, and evaluate using the default configuration（I use a6000，torch==2.2.0 cuda==11.8 transformers==4.40.1 ，10 tasks ，each task 50 episodes ）but libero-object sr is 98.8，higher than paper.

[AOLIAO12312]

Thank you very much for your excellent open-source work.

However, we also encountered the same discrepancy between our reproduction results and the performance reported in the official paper. To provide specific reference data, we conducted 50 evaluation episodes for each task on an RTX 4090 GPU using the default random seed (seed = 7). Our results are as follows:

• libero-spatial: 93.2%
• libero-object: 98.4%
• libero-goal: 97.2%
• libero-long: 94.2%
• Avg: 95.8%

As shown above, our empirical results—particularly on the libero-spatial benchmark—are noticeably lower than the figures presented (97.6%)in the paper.

Could you please confirm if this variance falls within the expected range of performance fluctuation? We would greatly appreciate any insights or suggestions regarding experimental settings or potential factors that might affect reproducibility, to help us further investigate.

[du-owen]

@AOLIAO12312 @atticus77777
I had the same issue (max. 94% accuracy on spatial), what worked for me was installing the requirements again with pip install -e . and specifically using pip install mujoco==3.3.2, after this I reached the baseline of 99% on spatial