Refactor main methods for scripts to run without snakemake
fhg-isi opened this issue · 1 comments
I tried to run
https://github.com/PyPSA/pypsa-eur/blob/master/scripts/prepare_sector_network.py
as python script to debug and learn about it.
I got an error due to non-existing config file
test/config.overnight.yaml
After changing it to
config/test/config.overnight.yaml
the command snakemake = mock_snakemake(..)
worked.
Edit
Current version does not include the non-existing config file but has it out-commented
# configfiles="test/config.overnight.yaml",
I suggest to :
a) Check all main functions of the scripts if they work without "snakemake" in globals().
(Maybe automatically run them in a github workflow if possible.)
Edit: There is already a CI workflow to run test.sh which calls the rule solve_elec_networks
. Currently the pipeline fails.
Maybe extend workflow with rule solve_sector_networks
?
b) Update or remove paths to config files for calls to mock_snakemake.
c) Extract main content starting after line if __name__ == "__main__"
to an extra function def main(..):
and move it to the top of the file.
d) Improve handling of missing input files to make it easier for beginners. Here are some ideas:
- Warning/Hint about missing input: Include checks for missing input files, for example (might not work; just to illustrate):
output_path = snakemake.output[0]
network_path = snakemake.input.network
if not os.path.exists(network_path):
message = f'Network file {network_path} does not exist yet. ' +\
'Snakemake command to create it: ' + \
f'snakemake --cores 1 {network_path}' +\
'Snakemake command to run full skript and auto-generate required inputs' +\
f'snakemake --cores 1 {output_path }'
raise KeyError(message)
- Adapt mock_snakemake to automatically try to generate input files, if they do not exist yet.
e) Consider to provide a consistent set of pre-existing input files for a "small" example, that allows
to run each of the rule immediately with default settings for macksnakemake... so that users do not
need to generate all the inputs beforehand. That would allow for a quicker entry into debugging/learning.
Furhtermore, if there is a bug in a previous step, it would not block the whole chain.
Related:
Please chang from bug to enhancement. (First I thought I found a bug but used an older version. Later I was not able to convert the ticket type from bug to feature request/enhancement.)