Postprocess memory and thread usage report

[CedricMidoux]

Hi Silas,

Is there a simple way to get a report on memory usage, thread, computation time, ... of different rules?
These metrics could be used for optimization for future analysis.
There is something like that in snakemake but I'm not too satisfied with it.

Thanks.

[SilasK]

You mean there is the snakemake --report which gathers much of the data. You are not satisfied which the output?
Do you think if you would have access to the raw data used for the report you could do something better?

For rules, that I estimated to be critical I made benchmark files that save the memory, io usage and time. This could be an other way to check it.

[CedricMidoux]

I use snakemake --report (note : I had to install pygraphviz, can we add it to atlasenv.yml ?)

atlas run all --report snakemake-report.html

We can see the rulesgraph, it's great (but not very readable).
However statistics are only based on "runtime" (although the runtime considers the waiting time not the effective running time) and "creation date".
Beyond time, cpu and threads usage can be useful.
I don't know how to represent this easily. maybe with an areaplot or a waffle-chart ?
I'll see what I can do with benchmark files ...

[SilasK]

Ok, then the benchmark files should have the actual running time, and mem usage. But I'm not shur if threads is there.