pypsa-meets-earth/pypsa-earth

Empty GEGIS paths in build_demand_profiles

Closed this issue · 9 comments

energyLS commented

Checklist

  • I am using the current main branch or the latest release. Please indicate.
  • I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

In the rule build_demand_profiles the GEGIS paths are empty, hence the rule fails. Not sure why.
Details: I am running the country Morocco MA.
Reproduce by run solve_all_networks, GADM clustering.

Error Message

INFO:snakemake.logging:[Tue Sep 12 16:33:27 2023]
rule build_demand_profiles:
    input: networks/base.nc, resources/bus_regions/regions_onshore.geojson, resources/shapes/gadm_shapes.geojson
    output: resources/demand_profiles.csv
    log: logs/build_demand_profiles.log
    jobid: 18
    benchmark: benchmarks/build_demand_profiles
    reason: Missing output files: resources/demand_profiles.csv; Input files updated by another job: networks/base.nc, resources/bus_regions/regions_onshore.geojson
    resources: tmpdir=/tmp, mem_mb=3000, mem_mib=2862
INFO:snakemake.logging:rule build_demand_profiles:
    input: networks/base.nc, resources/bus_regions/regions_onshore.geojson, resources/shapes/gadm_shapes.geojson
    output: resources/demand_profiles.csv
    log: logs/build_demand_profiles.log
    jobid: 18
    benchmark: benchmarks/build_demand_profiles
    reason: Missing output files: resources/demand_profiles.csv; Input files updated by another job: networks/base.nc, resources/bus_regions/regions_onshore.geojson
    resources: tmpdir=/tmp, mem_mb=3000, mem_mib=2862

INFO:snakemake.logging:
/home/energyls/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
INFO:pypsa.io:Imported network base.nc has buses, lines, transformers
Traceback (most recent call last):
  File "/mnt/c/Users/scl38887/Documents/git/aldehyde/workflow/subworkflows/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpn6hqqqgy.build_demand_profiles.py", line 229, in <module>
    build_demand_profiles(
  File "/mnt/c/Users/scl38887/Documents/git/aldehyde/workflow/subworkflows/pypsa-earth-sec/pypsa-earth/.snakemake/scripts/tmpn6hqqqgy.build_demand_profiles.py", line 155, in build_demand_profiles
    gegis_load = xr.open_mfdataset(load_paths, combine="nested")
  File "/home/energyls/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/xarray/backends/api.py", line 1003, in open_mfdataset
    raise OSError("no files to open")
OSError: no files to open
ekatef commented

Thanks for reporting @energyLS!

Could you please specify how to reproduce the issue? Which country are you working with?

Apart of that, a standard question (sorry for that!) when have you updated your local repo? There has been a fix for a demand data issue a couple of weeks ago. Not sure if it's similar to one you experience now.

energyLS commented

Hey @ekatef, sorry for the incomplete error message, I have added some details :) Yes I am running on the latest main (cloned yesterday). I think this is not related to the fix you are proposing. I will run some more tests and then provide some more details :)

  • 🚀1
ekatef commented

@energyLS Thanks for the details. I can reproduce the issue :) Interestingly, it appears for "MA", but doesn't for "NG". Great catch!

ekatef commented

The reason is a missed code for Morocco in the configs/osm_config.yaml. Added a fix in #864.

@energyLS thanks a lot for reporting!

energyLS commented

The reason is a missed code for Morocco in the configs/osm_config.yaml. Added a fix in #864.

@energyLS thanks a lot for reporting!

Thank you for the quick fix! :)

davide-f commented

Thanks for reporting, and I'm happy to keep this issue open until we verify the dictionary is complete of all countries by continent.
Who is interested in investigating this issue? :)

ekatef commented

@davide-f there are investigation results attached as a comment to #864. The current config_osm excludes some island states. Not sure if it is intended, as the model is now functional everywhere on the Earth, right?

davide-f commented

@davide-f there are investigation results attached as a comment to #864. The current config_osm excludes some island states. Not sure if it is intended, as the model is now functional everywhere on the Earth, right?

I think we where crosscommenting xD (see comment in the PR) [I went first through the issues then the PRs].
So, I think we should cover at least all UN countries even though they are small islands. (I'm happy to help)
In the py function they were functional.
Beyond the UN countries, I feel we could go when the need arises.

  • 🚀1
ekatef commented

@davide-f there are investigation results attached as a comment to #864. The current config_osm excludes some island states. Not sure if it is intended, as the model is now functional everywhere on the Earth, right?

I think we where crosscommenting xD (see comment in the PR) [I went first through the issues then the PRs]. So, I think we should cover at least all UN countries even though they are small islands. (I'm happy to help) In the py function they were functional. Beyond the UN countries, I feel we could go when the need arises.

No problem with cross-commenting and thanks a lot for looking into this! 😄
Agree with the overall strategy and happy to implement the check.

If you don't mind, I'll continue on #864 as it's a bit more instrumental ;)