Question - is there a way to export sacred results off-computer?

[FranzEricSchneider]

I've been running a series of sacred experiments on Computer A for a month or so now, and I had two questions about saving the sacred results off of the computer.

1. Computer A is having some strange issues, so I'd like to do a fresh ubuntu install. Is it possible to upload the MongoDB sacred results somewhere and then download them again after the install? Maybe as a zip of the database or something?
2. These experiments are taking a while, and it would be nice to parallelize and run on a few different machines. However, I'm not sure how I would get the sacred results from Computer A and Computer B together in one place. Similar to the first question, is it possible to zip/upload/capture the results from Computer B and then download/add those results to the Computer A database so they could be viewed and compared?

I'm pretty unfamiliar with databases and MongoDB so maybe this is a straight database question, thanks for any suggestions.

[FranzEricSchneider]

I found this resource about zipping a database and tried that: https://dev.to/lucis/setting-up-a-mongodb-backup-on-google-drive-3bbe

mongodump -d DB_NAME --archive=DB_save_8-5-2022.gz --gzip

Then tried using mongorestore on another computer:

mongorestore --gzip --archive=DB_save_8-5-2022.gz

When I run mongosh I can see the restored database, I think:

test> show dbs
admin    40.00 KiB
config  108.00 KiB
local    72.00 KiB
DB_NAME  10.64 GiB
sacred   40.00 KiB

However, when I try to view the database in omniboard like I used to it is unfortunately entirely blank.

omniboard localhost:27017:DB_NAME
(node:28895) [MONGODB DRIVER] Warning: Current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Omniboard is listening on port 9000!
Connection to sacred at mongodb://localhost:27017 established successfully!

Any tips welcome, I'm also still unclear how one would go merging experiments run on different computers.

[FranzEricSchneider]

I had a dumb typo,

omniboard localhost:27017:DB_NAME

Should instead have been

omniboard  -m   localhost:27017:DB_NAME

So the process above worked for zipping a database, restoring it on another computer, and reconnecting to it. If anyone has thoughts on combining experiment results from Computer A and B as mentioned in the original question that would still be valuable!

[thequilo]

Hi @FranzEricSchneider, great that you found a solution! Other than that, you could use the FileStorageObserver, which stores the logs in JSON files that can easily be copied between machines. It doesn't support omniboard or other related tools, though.

[FranzEricSchneider]

Hi @thequilo , do you know if it is possible to convert the data stored from a Mongo observer into this file storage form? That might be one way to combine experiments from multiple computers, which I'm still struggling with.

[thequilo]

There is currently no easy way to do this. You can loop through the data stored in your mongo db manually and pass them to a file storage observer, similar to #815 but the other way around.

[FranzEricSchneider]

Alright, I will close this issue. If anyone does come up with a way to combine sacred runs from multiple computers I would be very interested, even in my first usage of sacred it's been necessary to run tests on multiple computers because the tests take a long time to run. My only thought at the moment is to go through the existing database on one of the computers and alter all the ids to be something like (13 -> 1013, 24 -> 1024) to prevent ID clashes, then do the database zip/unzip mentioned above.

[FranzEricSchneider]

Okay, in order to zip sacred tests from multiple computers I needed to make sure none of the IDs overlapped. In order to deal with overlaps I went to computer B and changed the IDs in the mongo database like so:

$ mongosh
> # Use the relevant db name
> use mmseg
> # Repeat for all necessary IDs
> runX=db.runs.findOne({_id:X});
> runX._id=100X;
> db.runs.insertOne(runX);
> db.runs.deleteOne({_id:X});
> db.metrics.updateMany({run_id:X}, {$set:{"run_id":100X}});

Then I zipped the database on computer B, moved it to computer A, restored the database, and was left with all of the results on computer A.