How can I reproduce experimental results?

Question

How can I reproduce experimental results?

hyoonsoo opened this issue 2 years ago · 6 comments

Hello, I am very impressed with the CoPO project. Thank you for sharing a great paper and code.
I wanted to see the trained multi-agent, so I visualized it using the weight stored in copo_code/copo/best_checkpoint/ and copo_code/vis.py. file. (without any modifications)
However, unlike the paper, I was able to render agents with lower performance(lower succeess rate).
How should I modify the code to see the higher performance of agents like your paper?
I look forward to your reply. Thank you.

Answer 1 · 2022-09-08T04:49:54.000Z

This is due to the changes of the environments. MetaDrive multiagent benchmark became much harder! I will update models and exp results soon! Best regards! Peng Zhenghao (彭正皓)

…

在 2022年9月8日，11:32，IMO ***@***.***> 写道： Closed #25 as completed. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

Answer 2 · 2022-10-22T23:11:43.000Z

Hi Yoonsoo,

Finally, I finished benchmarking the results of various MARL algorithms in MetaDrive MARL environments. Please kindly refer to this page:

https://github.com/metadriverse/metadrive-benchmark/tree/main/MARL

And I also upload latest trained models so you can run it to visualize the behaviors! This time I don’t find any performance discrepancy in the latest models (which proved that the performance discrepancy is due to the update of environment)!

https://github.com/decisionforce/CoPO#visualization

Thanks!

Answer 3 · 2022-11-21T06:42:40.000Z

I've been waiting for a new update!!
Thank you for your kind reply.

Answer 4 · 2022-11-21T18:58:19.000Z

As I updated last month, the result is in https://github.com/metadriverse/metadrive-benchmark/tree/main/MARL

Do you have any question?

Answer 5 · 2023-03-21T22:47:33.000Z

Hi Zhenghao,
I trained the intersection using torch copo (train_copo.py) and tried to evaluate the performance using copo_code/new_vis.py.
However, it gives unpickled = pickle.loads(data) TypeError: an integer is required (got type bytes).
I used the file that was stored in TEST/CoPOTraininger_Multi.../checkpoint_000440/algorithm_state.pkl. Did I use the wrong pkl file?

Answer 6 · 2023-03-21T23:26:34.000Z

Hi Zhenghao,
I trained the intersection using torch copo (train_copo.py) and tried to evaluate the performance using copo_code/new_vis.py.
However, it gives unpickled = pickle.loads(data) TypeError: an integer is required (got type bytes).
I used the file that was stored in TEST/CoPOTraininger_Multi.../checkpoint_000440/algorithm_state.pkl. Did I use the wrong pkl file?

Hi @XilunZhangRobo

Let's discuss this in a new issue.