[Re] Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress
AugustinChrtn opened this issue · 50 comments
Original article: Lopes, M., Lang, T., Toussaint, M., & Oudeyer, P. Y. (2012). Exploration in model-based reinforcement learning by empirically estimating learning progress. Advances in neural information processing systems, 25.
PDF URL: https://github.com/AugustinChrtn/Reproduction/blob/master/article.pdf
Metadata URL: https://github.com/AugustinChrtn/Reproduction/blob/master/metadata.yaml
Code URL: https://github.com/AugustinChrtn/Reproduction/
Scientific domain: Computational Neuroscience
Programming language: Python
Suggested editor:
Thank you for your submission. An editor will be soon assigned. @benoit-girard Can you handle this submission (either editing it or assigning an editor)?
Thank you for your answer! @benoit-girard is my PhD supervisor and cannot edit this submission due to conflict of interest.
Oh sorry, I didn't check the authors... @oliviaguest Can you handle this submission (either editing it or assigning an editor)?
Yes, I can, @rougier .
Thank you @oliviaguest!
👋 Hi @hhihn @kkhetarpal @ghost-nn-machine @MA-Ramirez can any/all of you review this for ReScience C? 😊 Please let me know as soon as possible if you can take this on, I would appreciate that of course, but also be aware I might be harder to reach than usual till next week. Thank you!
Dear editors, Dear @rougier, Dear @oliviaguest,
Any news from the reviewing process?
I wish you all a great summer!
Best regards
mehdi
Hi @MehdiKhamassi I guess August is not the best month to find reviewers. Let's hope we'll find them early September.
@oliviaguest Any progress on that?
Sorry indeed. It's so slow-going. As you see, nobody even replied above; I'll keep trying. If you have any suggestions, please give them to us. 😊
Dear all,
thank you very much for the invitation to review the article.
Nonetheless, this time I won’t be able to review it due to time constraints.
All the best
@oliviaguest You can try to ask all the reviewers at once to check if someone willing to review.
dear @ReScience/reviewers can anybody take this on? 😊
I would be willing to review this submission. But do note that I might be largely unreachable between Oct 20 to end-November. I can have my initial review submitted prior to this period.
[Edit on Sep 26]
As the upcoming weeks are going to be busy, I would no longer be available for review during this period. I can revisit this in December if needed.
I would be happy to review this. However, we should have a "compute requirements" section as well in the submission process to better understand what resources reviewers might need.
@oliviaguest I think you have two potential reviewers.
I am sorry for taking so long to check this, I have been unwell. Apologies, to all above.
@HaoZeke and @appukuttan-shailesh thank you for offering to review this. Please let me know if you need any help. Please see here for some information: https://rescience.github.io/edit/
Hi @oliviaguest,
As I had updated on my post above, I am currently away and won't be able to take this up prior to December. Also, once back I would be shifting to a new city and therefore might be slow on proceeding with this. I would therefore recommend appointing another reviewer to speed things along. If in case nobody volunteers by December, I will be willing to take it up on my return.
@appukuttan-shailesh thank you for the heads up.
@oliviaguest @appukuttan-shailesh @HaoZeke Happy new year all and gentle reminder for the review. If we can target end of January that would be cool since we'll soon transition to a new website.
@rougier, @oliviaguest : I am afraid I won't be able to attend to this in the short term. As mentioned previously I have just moved to take up a new position, and therefore a bit occupied at the moment. It would be best to appoint another reviewer for a faster evaluation.
@appukuttan-shailesh ok
@oliviaguest can you find another reviewer?
Dear editors, Dear @rougier, Dear @oliviaguest,
Any news from the reviewing process?
I wish you all a nice day!
Best regards
mehdi
@HaoZeke Are tou still availabe to review? If so you can start (don't wait for secodn reviewer I mean). If you could do it in two weeks that would wonderful.
@oliviaguest Can you find another reviewer or should be broadcast to all reviewers ?
@rougier can reviewers be tagged like this @ReScience/reviewers? If so, that's great. Please if anyone has the time and capacity, please let us know. 😊
Another solution would be to ask one of ReScience published author if he/she is willing to review.
I can review this work, should be able to return the review by the end of March.
In this submission, the authors attempt to reproduce the work of Lopes et al (Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress). They discuss the challenges, discrepancies and potential solutions for resolving the identified issues. While the authors of the original paper have not released their source code, they have assisted the authors of this paper and have clarified some aspect of the work. This paper does not fit into my area of expertise, I have attempted to read, understand, and follow the instructions of the authors.
I started with forking and cloning the source code released by submission 73. List of dependencies used by authors are rather short, which reduces the chance of discrepancy management issues (dependency hell). Containerizing the environment to run everything inside a docker container was straightforward. The dockerfile, the steps I took to reproduce the results, and the output of the scripts ran are added to the fork inside a directory called artifact.
The repository released by authors is sufficiently documented and properly structured. It took approximately two hours to run all the scripts on a modern workstation. While re-running the scripts provided by authors was easy, sifting through the results to find the corresponding figure was more convoluted than it should be. To put it simply, the code generates too many figures and naming schema is not clear enough to identify the matching figure in the paper. This is not a major issue, but they authors could refactor the naming schema to include the figure number, if a figure is included in their paper.
The authors extensively discuss the their findings and the challenges they have faced. In particular, Table 2 provides a great overview of all the challenges, the resolutions, and the results. Although they did not manage to successfully reproduce all, their investigation is thorough and the clarifications provided by the original authors are helpful. Overall, I believe this submission complements the original paper.
Question for the authors:
Can you provide some information on how to match the figures generated by the code to the figures in the paper?
@oliviaguest I can do the review, give me a few days (and a reminder if I'm late)
@AugustinChrtn I've cloned and installed your dependencies but I get problem running main. I'll open an issue on your repo.
Dear @mo-arvan, thank you for your detailled feedback, the docker and your question.
I modified the structure of the repository and clarified how to match each plot to a Figure of the article. I separated the different metrics (or agents for parameter fitting) in different folders. In addition, the plot names the code generates now have the number of the figure they correspond to in their names. I also added a clearer output for the computational time and for the figures that the code is generating online. Thank you for pointing this out and I hope that these changes will make the results of the code easier to reuse and understand. Please let me know if I should change anything else in the code or in the article!
Dear @rougier, I answered to your issue and I hope that you can run main now. Please let me know if you cannot.
Thanks, seems to be working flawlessly now (and good point with the estimated time)
I've read the article and I would like to congratulate for this work. The paper is really really clear and explain both the different difficulties, how you manage to solve problems and it represents a fair amount of work. I've got only a small minor problem during testing but it has been fixed since then. I've only some some minor points:
-
Your requirements file is bit strict in terms of version for the different library. On the one hand this ensures reproducibility (which is good) but in the other hand you need to install a specific python version which I found a bit annoying. This does not need correction and you precise specification is ok. Just to let you know.
-
You need to add a license to your code because no license means no right for anyone to do anything with your code (see https://choosealicense.com/). We have no recommendation for ReScience as long as it is an open license.
The "major" point concerns the title/conclusion. The semantic is [Re] for replication, [¬Re] for failed replication and [~Re] for approximate replication which I think correspond to your case since you did not manage to replicate all the results in spite of a lot of efforts you put in this work. The idea with the title is to quickly warn the reader that the work is not totally reproducible. If you disagree with the change [Re] to [~Re], we can discuss it.
One this is done, I think the paper can be published. I'll let @oliviaguest decides on the next step.
Dear @rougier, thank you very much for your feedback. We did spend a lot of time and effort to replicate this paper and we're very happy that you liked it. I answer to all of your points below:
Minor points:
- I agree that the requirement file is rather strict. It was to make sure that people would be able to generate the exact same plots and results as the ones we present.
- Thank you for pointing it out. I added an MIT license.
Major point: I do agree that the paper is ~Re rather than Re. I changed the metadata (and the title).
We would like to thank again the two reviewers for their feedback and we are ready to proceed depending on @oliviaguest 's instructions.
Thanks. Let's wait for Monday for @oliviaguest to react, else I'll publish the paper.
Dear @rougier, @oliviaguest,
I am ready to proceed depending on your instructions.
Sorry for delay. @oliviaguest can we publish?
@AugustinChrtn In the meantime, you can start filling the metadata with all the information from this review.
@rougier Thank you for the instructions. I updated the metadata.yaml file !
Dear Editors, Dear @rougier , Dear @oliviaguest ,
This is my yearly message asking if there is any news from the reviewing process? :-)
I wish you all a great summer!
Best regards
mehdi
Sorry for the loooong delay. I'll publish the paper today. @AugustinChrtn Do you have a link to the latex files, it'll make my life easier.
Sure! I downloaded the source files from overleaf. Please let me know if you need something else!
Augustin
[
Rescience Chartouny.zip
](url)
Thanks. Can you check https://sandbox.zenodo.org/records/105346 if everything looks right? This is a sandbox version, not the final publication.
I checked the sandbox version and everything looks right to me! Thank you!
Ok, so lt's try to to publish the finale version.
It's online https://zenodo.org/records/13627804 !!! Congratulations!
Thank you very much @rougier!