aria-tools/ARIA-tools

[BUG] Invalid shape when multiplying lake_masks and gdal

Closed this issue · 8 comments

aaryan-rampal commented

Describe the bug
When I try to run this command: ariaTSsetup.py -f ../products/\*.nc -b 29.8459\ 29.8506\ -98.2566\ -98.247 -nt all --mask Download, I run into a bug which prevents me from downloading the water masks.

To Reproduce
Steps to reproduce the behavior:

  1. Download the files using ariaDownload.py --track 107 -b "29.8459 29.8506 -98.2566 -98.247" --start 20210101.
  2. Command used: Feed the downloaded products to ariaTSsetup.py with the same bounding box and try to download the water masks
  3. Error Output:
/home/arampal/tools/miniforge/envs/ARIA-tools/bin/ariaTSsetup.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('ARIAtools==1.1.6')
ARIA-tools Version: 1.1.6
*****************************************************************
*** Time-series Preparation Function ***
*****************************************************************
Shapefile ./user_bbox.json created for input user bounds
Multi-core version
All (127) GUNW products meet spatial bbox criteria.
Group GUNW products into spatiotemporally continuous interferograms.
All (127) interferograms are spatially continuous.
User specified use of all 64 threads for gdal multiprocessing
Thread count specified for gdal multiprocessing = ALL_CPUS
Download/cropping DEM
Downloading glo_90 tiles: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:40<00:00,  3.37s/it]
Applied cutline to produce 3 arc-sec SRTM DEM: ./DEM/glo_90.dem
***Downloading water mask... ***
Traceback (most recent call last):
//
// removed a lot of vscode python debugger traceback calls
//
  File "/home/arampal/tools/miniforge/envs/ARIA-tools/bin/ariaTSsetup.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/arampal/ARIA-tools/tools/bin/ariaTSsetup.py", line 18, in <module>
    main(inps)
  File "/home/arampal/ARIA-tools/tools/ARIAtools/tsSetup.py", line 470, in main
    inps.mask = prep_mask(**mask_dict)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arampal/ARIA-tools/tools/ARIAtools/mask_util.py", line 107, in prep_mask
    mask_array = lake_masks*gdal.Open(maskfilename).ReadAsArray()
                 ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (7,13) (6,12)

Expected behavior
Not really sure, this is my first time using this program

Desktop (please complete the following information):

  • ARIA-tools git tag: [e.g. git show --summary]
commit 32018d59312f0267e077aba1e16de4f1c70a98d9 (HEAD -> dimension, dev)
Merge: 35c01cd 60ea380
Author: BB <bbuzz318@icloud.com>
Date:   Wed Sep 6 13:48:34 2023 -0700

    Merge pull request #381 from aria-tools/sss_latsortbug

    Sort pairs by latitude
  • OS
    Running on Ubuntu server over SSH
alexfore commented

Hi @aaryan-rampal -- I'm able to reproduce this issue. Forcing GDAL to be 3.8.3 seems to resolve this issue when the input products are V2. We will update the requirements.txt and environment.yml to specify this shortly

aaryan-rampal commented

my gdal is 3.8.4 yet it still doesn't work. What do you mean by input products are v2?

aaryan-rampal commented

I tried with the v2v3 branch, which gave this error

/home/arampal/tools/miniforge/envs/A/bin/ariaTSsetup.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('ARIAtools==1.1.6')
2024-04-22 13:04:41,911 - ARIAtools.util.dem - WARNING - Cannot proceed with VRT format, using ENVI format instead
Downloading glo_90 tiles: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:31<00:00,  2.61s/it]
Traceback (most recent call last):
  File "/home/arampal/tools/miniforge/envs/A/bin/ariaTSsetup.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/bin/ariaTSsetup.py", line 628, in <module>
    main()
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/bin/ariaTSsetup.py", line 483, in main
    maskfilename = ARIAtools.util.mask.prep_mask(**mask_dict)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/ARIAtools/util/mask.py", line 106, in prep_mask
    mask_array = lake_masks * osgeo.gdal.Open(maskfilename).ReadAsArray()
                 ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (16,26) (15,25)

The only notable difference is 2024-04-22 13:04:41,911 - ARIAtools.util.dem - WARNING - Cannot proceed with VRT format, using ENVI format instead. Everything else is the same. My pip show gdal is:

Version: 3.8.4
Summary: GDAL: Geospatial Data Abstraction Library
Home-page: http://www.gdal.org
Author: Frank Warmerdam
Author-email: warmerdam@pobox.com
License: MIT
Location: /home/arampal/tools/miniforge/envs/A/lib/python3.11/site-packages
Requires:
Required-by:
aaryan-rampal commented

If I run this without --mask Download, I get this issue:

/home/arampal/tools/miniforge/envs/A/bin/ariaTSsetup.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('ARIAtools==1.1.6')
2024-04-22 13:37:07,147 - ARIAtools.util.dem - WARNING - Cannot proceed with VRT format, using ENVI format instead
/home/arampal/tools/miniforge/envs/A/lib/python3.11/site-packages/dem_stitcher/stitcher.py:146: UserWarning: The directoryglo_90_tiles exists; We are writing new files to this directory
  warn(f"The directory{tile_dir} exists; " "We are writing new files to this directory", category=UserWarning)
Downloading glo_90 tiles: 100%|██████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:37<00:00,  3.16s/it]
/bin/sh: 1: parallel: not found
Traceback (most recent call last):
  File "/home/arampal/tools/miniforge/envs/A/bin/ariaTSsetup.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/bin/ariaTSsetup.py", line 628, in <module>
    main()
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/bin/ariaTSsetup.py", line 532, in main
    ARIAtools.util.vrt.dim_check(ref_arr_record, prod_arr_record)
  File "/home/arampal/ARIA-tools-v3v4/ARIA-tools/tools/ARIAtools/util/vrt.py", line 575, in dim_check
    raise Exception(
Exception: Inconsistent product dims between products ./unwrappedPhase/20220115_20220103 and None: check respective width (25, None) and height (15, None) and geotrans ((50.8924796429999, 0.000833332999999999, -0.0, 25.264989894, -0.0, -0.000833333), None)

I added code which implemented parallel processing for ARIA-tools, but it was written off of the dev branch. I noticed that the TODO in extractProduct.py export_products had been acted on in the new v2v3 branch, but it doesn't work as it gives the error above. I'd like to contribute my code if it helps since it works currently right off of the dev branch

alexfore commented

GDAL 3.8.4 introduced an inconsistency between the behavior of gdal.Rasterize and gdal.Warp which leads to the observed mismatch in array sizes. We are working on a improved water mask generation process to address that issue. This particular issue will manifest with GDAL 3.8.4 until we implement our improved water mask.

The branch you used also needs the gnu parallel utility installed but did not contain updated specs in requirements.txt / enviorment.yaml. When these changes are merged into dev they should be ready for use.

aaryan-rampal commented 
ARIA-tools Version: 1.1.6
*****************************************************************
*** Time-series Preparation Function ***
*****************************************************************
Shapefile ./user_bbox.json created for input user bounds
Multi-core version
All (45) GUNW products meet spatial bbox criteria.
Group GUNW products into spatiotemporally continuous interferograms.
All (45) interferograms are spatially continuous.
Thread count specified for gdal multiprocessing = 2
Download/cropping DEM
Downloading glo_90 tiles: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:30<00:00,  2.57s/it]
Applied cutline to produce 3 arc-sec SRTM DEM: ./DEM/glo_90.dem
***Downloading water mask... ***
Traceback (most recent call last):
  File "/home/arampal/tools/miniforge/envs/ARIA-tools/bin/ariaTSsetup.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/arampal/ARIA-tools/tools/bin/ariaTSsetup.py", line 18, in <module>
    main(inps)
  File "/home/arampal/ARIA-tools/tools/ARIAtools/tsSetup.py", line 470, in main
    inps.mask = prep_mask(**mask_dict)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arampal/ARIA-tools/tools/ARIAtools/mask_util.py", line 104, in prep_mask
    mask_array = lake_masks*gdal.Open(maskfilename).ReadAsArray()
                 ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (16,26) (15,25)

It still doesn't work when I force gdal to be 3.8.3

alexfore commented

I have tested this and it does work with GDAL 3.8.3. In any case this will be closed by work we are doing now. I will close this issue with that PR when we are ready. Secondly, your posted Exception is from a different case than was posted in this issue report, so I can't comment on that.

dbekaert commented

this has been fixed under the latest release. Now also available on conda