ssec/sift

Cannot do a Band Difference in 1.1.2

ScottLindstrom opened this issue · 8 comments

ScottLindstrom commented

I've loaded up GOES-16 Band 13/Band 15 CONUS sectors. Something like this:
OR_ABI-L1b-RadC-M6C13_G16_s20201341001116_e20201341003500_c20201341004030.nc

from iris / the cluster. They load and display in SIFT with no issue. I can make an RGB with them as well. But when I try to make an algebraic, SIFT does not cooperate. I get this trace-back:

 File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 229, in update_display
    self._get_shared_color_limits(shared_info, layer_info, this_prez)
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 148, in _get_shared_color_limits
    unit_info = self.document[this_prez.uuid][Info.UNIT_CONVERSION]
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 916, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 908, in __missing__
    raise KeyError(key)
KeyError: <Info.UNIT_CONVERSION: 'unit_conversion'>
DEBUG 17:02:45 layer_tree:setData:L690 setData 0
DEBUG 17:02:45 layer_tree:setData:L703 toggle layer visibility for row 0 to False
Traceback (most recent call last):
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 229, in update_display
    self._get_shared_color_limits(shared_info, layer_info, this_prez)
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 148, in _get_shared_color_limits
    unit_info = self.document[this_prez.uuid][Info.UNIT_CONVERSION]
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 916, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 908, in __missing__
    raise KeyError(key)
KeyError: <Info.UNIT_CONVERSION: 'unit_conversion'>

Other unexpected things that happen: It does create the layer, and it put the SWD (the name I give the difference field) at the top of the layer list. It only lists the two layers -- SWD and B13. B15 is somehow vanished until I unclick B13 at which point all 3 layers show up in the layer list window. If I unclick B13 and B15, I see a black field (the SWD). I can't do a colormap change on it -- also get the unit_conversion error above thrown.

The files I used are in ftp:/ftp.ssec.wisc.edu/pub/ssec/scottl/

Or, on the cluster: /arcdata/goes_restricted/grb/goes16/2020/2020_05_13_134/abi/L1b/RadC/

Two important notes:
(1) I am running this by tunneling into crusher from outside. I don't know how that affects what is going on.
(2) I'm not completely remembering exactly how SIFT was installed. I run 'source ./SIFT_1.1.2/bin/activate ' before running SIFT to get things going.

djhoese commented

Interestingly I get a different but similar error:

WARNING: Traceback (most recent call last):
  File "/home/davidh/repos/git/sift/uwsift/view/create_algebraic.py", line 147, in _create_algebraic
    self.doc.create_algebraic_composite(operations=operations, namespace=namespace, info=info)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 2101, in create_algebraic_composite
    dataset[Info.UNIT_CONVERSION] = units_conversion(dataset)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 156, in units_conversion
    punits = preferred_units(dsi)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 144, in preferred_units
    if dsi[Info.STANDARD_NAME] == 'toa_bidirectional_reflectance':
  File "/home/davidh/miniconda3/envs/sift_pyqt5_py37/lib/python3.7/collections/__init__.py", line 914, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/davidh/miniconda3/envs/sift_pyqt5_py37/lib/python3.7/collections/__init__.py", line 906, in __missing__
    raise KeyError(key)
KeyError: <Info.STANDARD_NAME: 'standard_name'>

And my SIFT application completely crashes. I think this calls for a 1.1.3 release. I'm running Satpy's master branch so that could explain the difference, but I'm still surprised.

djhoese commented

I think I found two issues, at least for the version of everything I'm running. Two things:

  1. There is a bug in the metadata guidebook stuff that isn't setting standard_name correctly. Once fixed I was able to create a difference layer. However, the limits were terribly wrong.
  2. Satpy's ABI readers are including metadata for valid_range but it is set to (0, 4095) which is the original in-file range. I'm made a PR for Satpy to fix this (pytroll/satpy#1213). Once merged this shouldn't be a problem. I'll have to double check AMI and AHI too.

Edit: Doesn't look like AMI or AHI have this valid_range in their metadata (it's a pretty CF standard kind of thing which neither of those instruments are).

djhoese commented

@ScottLindstrom Is there a chance that there is an error a little higher up in your terminal when you get this error?

ScottLindstrom commented

I didn't see one. All the debug in opening I cleared, and then tried the difference

katherinekolman commented

I tested creating an algebraic layer with the Satpy and SIFT master branches and it seems to be working (at least on my end). I used files from the ftp server and didn't get any error output.

Just to confirm, is this what it is supposed to look like?
s1

I can also do colormap changes:
s2

ScottLindstrom commented
ScottLindstrom commented
katherinekolman commented

This is what the image looks like when the range is changed to -10 to +10:
s3