buildDomainGRASS.py

Question

buildDomainGRASS.py

Closed this issue 6 years ago · 17 comments

Using the g.extensions command: I can get extensions with GRASS 7.2.2 but not with 7.3, the version recommended in the Readme.
There is an additional settings.ini file in the Domain builder folder I think it's an old format. Do I need to do anything with this or can it be eliminated? If so it should be outlined in the readme.
When I run buildDomainGRASS.py it gives an error on line 14 importing readsettings:

Traceback (most recent call last):
File "D:\GSFLOW\Pre_processor_codes\Shullcas_8_GSFLOW-
GRASS\domain_builder\buildDomainGRASS.py", line 14, in

from readSettings import Settings
ImportError: No module named readSettings
(Wed Nov 29 16:43:18 2017) Command finished (0 sec)

Maybe my current directory is wrong because I seem to have problems locating the correct files here and in the next issue.
To temporarily work around I drop a copy of readSsettings.py into the domain builder folder.

Looks like the name 'Settings_AW.ini' and the Linux \ slashes are hard coded on line 11 of buildDomainGRASS.py.
When I try and run this script from GRASS is cannot find Settings_AW.ini...even when I change the names? Again maybe it is an issue which what is supposed to be the current directory? Error:

D:\GSFLOW\Pre_processor_codes\Shullcas_8_GSFLOW-GRASS\domain_builder\buildDomainGRASS.py
Error opening or parsing input file: ../Run/settings_AW.ini
Using specified input file: ../Run/settings_AW.ini
(Wed Nov 29 16:29:22 2017) Command finished (0 sec)

I temporarily work around by giving a full path for my settings file but then I mess other stuff up and get more errors with the settings stuff.

Answer 1 · 2017-11-30T06:02:30.000Z

Hi Lauren,

Could you try with GRASS 7.4? It is a little tricky to find for Windows, but the GRASS team is transitioning to this and 7.5, and from your comment, seems to have discontinued extensions for 7.3. Here is a link: https://wingrass.fsv.cvut.cz/grass74/x86_64/. Updating the README.md will go on my to-do list, but I won't have much time before AGU.
Just deleted it!
This line should add this file to the path: sys.path.append(os.path.join('..', 'Run')). We should check why it doesn't work. Your temporary workaround should be OK.
That's an OOPS! It was to give a default settings file. Deleting the line should work OK. I will try it in the morning.

Answer 2 · 2017-12-01T19:37:08.000Z

Also just a reminder to self or others that for #4 however you change how the name of the settings file is specified, it should be reflected in the README

Answer 3 · 2017-12-01T20:55:56.000Z

Updates:

I used GRASS GIS 7.4 and confirmed that this worked. Could you follow the above link and check?
This worked on Windows with GRASS 7.4 and the current repository version for me. @Lauren-Somers Could you try?
Fixed now. This line was not needed and has been removed.

Re: your comment this past hour, the settings file, settings_template.ini, is in the README. (Ignore my now-deleted line.) I have also started to clean up our directories and make sure that we don't have extraneous input/run files sitting around. @gcng most of these are from your testing; could you have a look? I don't want to delete anything of yours by mistake...

Answer 4 · 2017-12-01T23:53:24.000Z

1/3: I will try with GRASS 7.4 next chance I get.

4/Re my comment: When running buildDomainGRASS.py from GRASS are you supposed to also specify the name of the settings file? If so I think it would be helpful to include that in the readme. At the moment it just says to run by entering something like "python ~/models/GSFLOW-GRASS/domain_builder/buildDomainGRASS.py.sh".

Answer 5 · 2017-12-04T18:08:59.000Z

Downloaded GRASS 7.4. Strangely having same issue as 7.3. I also get the same error message for both the extensions you made and the previously existing extensions but I don't think I did anything differently than with 7.2.

(Mon Dec 04 13:06:16 2017)
g.extension v.gsflow.export
WARNING: Extension <v.gsflow.export> already installed. Re-installing...
Downloading precompiled GRASS Addons <v.gsflow.export>...
ERROR: Extension <v.gsflow.export> not found
(Mon Dec 04 13:06:17 2017) Command finished (1 sec)

Answer 6 · 2017-12-04T19:11:51.000Z

That is weird. I can try again on Windows. Could you uninstall 7.2 and 7.3 and reinstall 7.4 just to be extra sure that you are using it and not having any routines contaminated by other versions of the software?

Answer 7 · 2017-12-05T17:05:38.000Z

I re-downloaded GRASS 7.4 and it can now get the extension (did not end up having to delete other versions). When I run buildDomain.py it runs for a while and then gets stuck in the same place as when I was trying it with 7.2:

File "C:\Users\lsomers\AppData\Roaming\GRASS7\addons/scripts/v.gsflow.segments.py", line 319, in
GRASS\domain_builder\buildDomainGRASS.py", line 115

I also tried GRASS 7.5 which is now listed as the development version available on the GRASS website. For Grass 7.5, it is able to get the extensions but I get a different error in running the build domain scripts earlier on.

Answer 8 · 2017-12-05T17:12:54.000Z

I'm assuming it was this full error, so the real problem being on Line 272?

Traceback (most recent call last):
  File "C:\Users\lsomers\AppData\Roaming\GRASS7\addons/scrip
ts/v.gsflow.segments.py", line 319, in <module>
    main()
  File "C:\Users\lsomers\AppData\Roaming\GRASS7\addons/scrip
ts/v.gsflow.segments.py", line 272, in main
    nseg = np.arange(1, len(cats)+1)
TypeError: len() of unsized object
Traceback (most recent call last):
  File "D:\GSFLOW\Pre_processor_codes\Shullcas_8_GSFLOW-
GRASS\domain_builder\buildDomainGRASS.py", line 115, in
<module>
    overwrite=True)
  File "C:\Program Files\GRASS GIS
7.2.2\etc\python\grass\pygrass\modules\interface\module.py",
line 586, in __call__
    return self.run()
  File "C:\Program Files\GRASS GIS
7.2.2\etc\python\grass\pygrass\modules\interface\module.py",
line 716, in run
    module=self.name, errors=stderr)
grass.exceptions.CalledModuleError: Module run
v.gsflow.segments v.gsflow.segments input=streams_inbasin
icalc=1 cdpth=0.4 fdpth=0.42 awdth=4.0 bwdth=0.23 iupseg=0,0
flow=0,0 runoff=0,0 etsw=0,0 pptsw=0,0 roughch=0.035
roughbk=0.06 width1=5 width2=5 output=segments --o ended
with error
Process ended with non-zero return code 1. See errors in the
(error) output.
(Fri Dec 01 13:59:59 2017) Command finished (42 sec)

The declared problem is that "cats" is unsized. Therefore, I think that the problem is happening somewhere earlier in the script, or perhaps somewhere before it.

@Lauren-Somers have you tried reinstalling all of the extensions, using the Windows batch file? There have been changes made to many of them, and I want to make sure that you are using the most up-to-date versions of everything before I try to troubleshoot.

Answer 9 · 2017-12-05T17:30:36.000Z

Yes, that's the error message I get and yes, I just tried reinstalling all the extensions (by copying and pasting from the readme but I think it should be the same) and that's what I get. I also just ran on GRASS 7.5 to the same point as an edit to previous message. Which version of GRASS would you recommend I use or should it matter as long as I can get extensions? It seems my options are 7.2, 7.4 and 7.5. I will delete all other version of GRASS incase there are interactions between them.

Answer 10 · 2017-12-05T17:39:17.000Z

Glad we have the same versions! I am going to double-check that there are no changes that I made on the Windows machine that I forgot to transfer to GitHub.

GRASS 7.4 or 7.5 should work. I have successfully done this on Windows with 7.4.

This error occurs because a previous process has failed -- and therefore you cannot query these category values. I think that the best ways to proceed are (in no particular order):

Run v.db.select on your streams map. This is copied to create segments. Perhaps an earlier issue with building the streams vector prevented their category values from being copied.
Toss your whole GRASS GIS folder on Google Drive so I can look at it.
Run the earlier steps in the analysis one at a time and watch for warning messages where GRASS may have the initial issue that leads to this one
Create a new GRASS location and re-run from the beginning (perhaps something isn't being overwritten properly)? If this works, then there is a bug in the code.

Answer 11 · 2017-12-05T19:36:16.000Z

Okay, sounds good. I think you're right about an earlier issue because some of the grass data looks weird/empty. I've dropped everything including grassdata in this google drive folder:

https://drive.google.com/open?id=1NHkMZn5xyAdAuHVjGLr6uq3vsKqFWKQL

I tried your first suggestion:
v.db.select streams
ERROR: Vector map not found

For the second suggestion I get a few different warnings as it runs. The most serious sounding is:
WARNING: Unable to rename null file 'D:\grassdata\Shullcas\PERMANENT.tmp/unknown\8632.1' to 'D:\grassdata\Shullcas\PERMANENT\cell_misc\accumulation_onmap\nullcmpr': File exists

I get this for six different files in a row. Again, not sure if the two directional slashes is part of the problem or not.

I'll try your final suggestion when I get a chance. Thanks!

Answer 12 · 2017-12-05T20:15:21.000Z

This overwrite issue is almost certainly the problem.

Unable to rename null file 'D:\grassdata\Shullcas\PERMANENT.tmp/unknown\8632.1' to 'D:\grassdata\Shullcas\PERMANENT\cell_misc\accumulation_onmap\nullcmpr

I do not know what this overwrite issue might be. My best suggestion would be to start from scratch with a new GRASS GIS location.

The slash thing is still weird, but I wonder if it is because GRASS was written originally for UNIX-like systems, so their messages may default to forwards slashes.

I won't have much time to address this prior to AGU -- especially since the machine with Windows is being used by students -- but will get to it as soon as I can.

Answer 13 · 2017-12-05T22:35:23.000Z

Okay, thanks for all your help! I know it's a super busy time of year.

I tried creating a new location with the same result. I just had one last thing and then I'll give it a rest for a while. Could you attach here the DEM geotif you are using for Shullcas? Can't remember if you modified it at all after I passed it over.

Answer 14 · 2018-10-04T10:42:28.000Z

Hi!
I get similar error messages and I haven´t yet managed to debug the error.
I work on Ubuntu 18.04.1 LTS, GRASS 7.4, GSFLOW 1.2.2

My error message is:

Traceback (most recent call last):
  File
"/home/mezcla/.grass7/addons/scripts/v.gsflow.segments",
line 512, in <module>
    main()
  File
"/home/mezcla/.grass7/addons/scripts/v.gsflow.segments",
line 380, in main
    nseg = np.arange(1, len(cats)+1)
TypeError: len() of unsized object
Traceback (most recent call last):
  File "buildDomainGRASS.py", line 135, in <module>
    overwrite=True)
  File "/usr/lib/grass74/etc/python/grass/pygrass/modules/in
terface/module.py", line 649, in __call__
    return self.run()
  File "/usr/lib/grass74/etc/python/grass/pygrass/modules/in
terface/module.py", line 770, in run
    self.wait()
  File "/usr/lib/grass74/etc/python/grass/pygrass/modules/in
terface/module.py", line 791, in wait
    module=self.name, errors=stderr)
grass.exceptions.CalledModuleError: Module run
v.gsflow.segments v.gsflow.segments input=streams_inbasin
icalc=1 cdpth=0.4 fdpth=0.42 awdth=4.0 bwdth=0.23 iupseg=0,0
flow=0,0 runoff=0,0 etsw=0,0 pptsw=0,0 roughch_value=0.035
roughch_raster=0.035 roughch_points=0.035
roughch_pt_col=0.035 roughbk_value=0.06 width1=5 width2=5
width_points=5 width_points_col=5 fp_width_value=0
output=segments --o ended with error
Process ended with non-zero return code 1. See errors in the
(error) output.
Using specified input file: /gsflow_path_simdir/myloi.ini
Importing DEM and generating hydrologic correction

For

v.db.select streams_all

I get 1864 registers (streams)

I would be really grateful for any suggestions.

ps. Attached dem, ini, temp_prec
myloi.zip

Answer 15 · 2018-10-19T07:23:11.000Z

Hi @aandrovitsanea and sorry for the delayed reply! I will download your files and test them over the weekend.

Answer 16 · 2018-10-19T07:25:03.000Z

Thanks! Looking forward to hearing from you.

Answer 17 · 2018-10-20T02:42:14.000Z

I can resolve this. The problem at least for @aandrovitsanea is that with just the basin clipped out, the algorithm thinks that there is offmap flow everywhere (because it doesn't actually see the basin boundary). I thought about how to fix this, but in fact, the DEM must be at least 2 MODFLOW grid cells larger than the model domain. This will stop the offmap flow / NULL cell error (pictured below) and will also allow the constant-head boundary condition at the outlet to be maintained. (Because of the NULL cells, the drainage network could not be built.) I will close this discussion and update the manual to include this information. Please send an email or open a new error if this issue repeats or something else comes up.