NWCSAF GEO images black with numpy 2

Question

NWCSAF GEO images black with numpy 2

gerritholl opened this issue 5 months ago · 1 comments

Describe the bug

With numpy 2, images produced from NWCSAF GEO composites such as cloudtype and cloud_top_height appear black.

To Reproduce

from satpy import Scene
from satpy.utils import debug_on; debug_on()
import numpy as np
fns = ["S_NWC_CTTH_MSG4_MSG-N-VISIR_20230222T170000Z.nc", "S_NWC_CT_MSG4_MSG-N-VISIR_20230201T073000Z.nc"]
sc = Scene(filenames=fns, reader=["nwcsaf-geo"])
sc.load(["cloud_top_height", "cloudtype"])
ls = sc.resample("nqeuro3km", radius_of_influence=15000)
ls.save_datasets(writer="geotiff", keep_palette=True)

Expected behavior

I expected coloured palette images.

Actual results

[DEBUG: 2024-08-07 10:25:51 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml',)
[DEBUG: 2024-08-07 10:25:51 : satpy.readers.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/202302010730/S_NWC_CT_MSG4_MSG-N-VISIR_20230201T073000Z.nc', '/media/nas/x21308/scratch/NWCSAF/202302221700/S_NWC_CTTH_MSG4_MSG-N-VISIR_20230222T170000Z.nc']
[DEBUG: 2024-08-07 10:25:51 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2024-08-07 10:25:51 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/pyorbital/etc
[DEBUG: 2024-08-07 10:25:51 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2024-08-07 10:25:51 : satpy.readers.nwcsaf_nc] Reading ctth_alti.
[DEBUG: 2024-08-07 10:25:51 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2024-08-07 10:25:51 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2024-08-07 10:25:51 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2024-08-07 10:25:51 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ct.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ctth_status_flag.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ctth_alti_pal.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.yaml_reader] No coordinates found for DataID(name='ctth_alti_pal', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ct_status_flag.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ctth_conditions.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ct_quality.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ctth_quality.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ct_pal.
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.yaml_reader] No coordinates found for DataID(name='ct_pal', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:52 : satpy.readers.nwcsaf_nc] Reading ct_conditions.
[DEBUG: 2024-08-07 10:25:52 : satpy.scene] Unloading dataset: DataID(name='ctth_alti', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:52 : satpy.scene] Unloading dataset: DataID(name='ct', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:52 : satpy.scene] Resampling DataID(name='cloud_top_height', resolution=3000)
[INFO: 2024-08-07 10:25:53 : satpy.resample] Using default KDTree resampler
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling add-93c45dc6e66ec87f1121b9481a937450
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ctth_status_flag', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ctth_status_flag
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ctth_conditions', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ctth_conditions
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ctth_quality', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ctth_quality
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='cloudtype', resolution=3000)
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling add-4040886413dba3654a72aeaae5287ccb
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ct_status_flag', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ct_status_flag
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ct_conditions', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ct_conditions
[DEBUG: 2024-08-07 10:25:53 : satpy.scene] Resampling DataID(name='ct_quality', resolution=3000, modifiers=())
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2024-08-07 10:25:53 : satpy.resample] Resampling ct_quality
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Adding enhancement configuration from file: /home/gholl/Arbeit/checkouts-perforce/config/enhancements/generic.yaml
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Data for DataID(name='cloud_top_height', resolution=3000) will be enhanced with options:
        [{'name': 'palettize', 'method': <function palettize at 0x7f9ea9a10540>, 'kwargs': {'palettes': [{'dataset': 'ctth_alti_pal', 'color_scale': 255}]}}]
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f9eaaada690>
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Starting outermost env
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f9eaab40550> created
[DEBUG: 2024-08-07 10:25:53 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f9eaab40550>.
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f9eaaada690>
[DEBUG: 2024-08-07 10:25:53 : rasterio._io] Path: _UnparsedPath(path='cloud_top_height_20230222_170000.tif'), mode: w, driver: GTiff
[DEBUG: 2024-08-07 10:25:53 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f9eaaada690>
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f9eaab40550> options
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f9eaab40550>.
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exiting outermost env
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f9eaaada690>
[DEBUG: 2024-08-07 10:25:53 : satpy.writers] Data for DataID(name='cloudtype', resolution=3000) will be enhanced with options:
        [{'name': 'palettize', 'method': <function palettize at 0x7f9ea9a10540>, 'kwargs': {'palettes': [{'dataset': 'ct_pal', 'color_scale': 255}]}}]
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f9ea99fefc0>
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Starting outermost env
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f9ea99fcb50> created
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f9ea99fcb50>.
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f9ea99fefc0>
[DEBUG: 2024-08-07 10:25:53 : rasterio._io] Path: _UnparsedPath(path='cloudtype_20230201_073000.tif'), mode: w, driver: GTiff
[DEBUG: 2024-08-07 10:25:53 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f9ea99fefc0>
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f9ea99fcb50> options
[DEBUG: 2024-08-07 10:25:53 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f9ea99fcb50>.
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exiting outermost env
[DEBUG: 2024-08-07 10:25:53 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f9ea99fefc0>
[INFO: 2024-08-07 10:25:53 : satpy.writers] Computing and writing results...
/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/dask/core.py:127: RuntimeWarning: invalid value encountered in sin
  return func(*(_execute_task(a, cache) for a in args))
/data/gholl/mambaforge/envs/py312/lib/python3.12/site-packages/dask/core.py:127: RuntimeWarning: invalid value encountered in cos
  return func(*(_execute_task(a, cache) for a in args))
/data/gholl/checkouts/trollimage/trollimage/colormap.py:182: RuntimeWarning: All-NaN slice encountered
  outside_range_bin = max(np.nanmax(arr), values.max()) + 1

Screenshots

Environment Info:

OS: openSUSE Leap 15.3
Satpy version: main (v0.50.0-38-g6f475f199)
trollimage version: main (v1.24.0-7-gc665c4d)

Additional context

I found this when working on #2870. It is related to #2872 and pytroll/trollimage#175, but #2874 (in combination with pytroll/trollimage#176) solves the problem only for cloud type, not for cloud top height.

Answer 1 · 2024-08-07T08:52:12.000Z

Never mind, there was something wrong with the input file.