IDSIA/sacred

Best practice examples for using MongoDB with sacred.

Closed this issue · 4 comments

yuvval commented

Hi,

Since MongoDB is the recommended observer for sacred, are there best practice examples for using MongoDB with sacred?

It will be really helpful for newbies to Mongo.

For example:
(1) How to aggregate experiments results and filter them according to some configurations.
(2) How to delete experiments according to some configuration filter
(3) How to get the file of an artifact (e.g. retrieve a model checkpoint)
(4) How to delete specific outcomes (artifacts) from a set of experiment
(5) How to reproduce an experiment stored in MongoDB or at least extract the code for that experiment.

Point (3) is useful when we save the outcomes (e.g. models / predictions) during development, but would like to free disk space when this information is no longer needed.

Thanks!

yuvalatzmon commented

OP here:

I prepared a toy (MNIST) project, for demonstrating the usage of SACRED. It contains a jupyter notebook with demo code according to questions (1)-(4) above:
https://github.com/yuvalatzmon/SACRED_HYPEROPT_Example/blob/master/mongo_queries.ipynb

  • 👍1
  • ❤️2
JarnoRFB commented

I am also working on a library called incense for getting data back from the mongoDB into python. It is in a pretty early stage, so feedback is very welcome.

  • 🎉3
Qwlouse commented

My workflow was usually to gather results into a pandas dataframe in a jupyter notebook, and do the analysis / plotting from there. I had a few helper functions in a separate project for that. But overall I would say that this is still an unsolved problem, and I am looking forward to see where incense is going.

stale commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.