Unpin pinned dependency versions
Closed this issue · 3 comments
pont-us commented
Describe the bug
Currently, the following dependencies are pinned in environment.yml:
werkzeug <2.2 # >=2.2 slows down S3 tests (deps: moto->flask->werkzeug)xarray >=2022.6,<2023.09.0 # because of issue https://github.com/dcs4cop/xcube/issues/897
In the case of xarray, 61 unit tests fail if the current latest version (2024.2.0) is installed. Of course these pins can't be retained indefinitely, and both pinned packages are already significantly behind the current version. We should fix the incompatibilities and unpin them as soon as possible.
pont-us commented
The initial xarray pin was motivated by #897 ; with the latest (2024.2.0) xarray version, it has been joined by another 60 (!) unit test failures and errors. Fortunately many of them seem to have similar causes, so we probably won't need 61 separate fixes.
pont-us commented
Summary of failing tests with 2024.2.0:
FAILED test/cli/test_dump.py::DumpTest::test_dump_ds - AssertionError: '<xarray.Dataset> Size: 8MB\nDimensions: (lon: 360, ...
FAILED test/cli/test_level.py::LevelDataTest::test_all_defaults - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_invalid_inputs - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_nc - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_agg_method - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_agg_methods - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_link - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_num_levels_max - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_existing_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_tile_size_and_num_levels - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_defaults - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_user_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_delete_block_file - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_dry_run - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_no_dry_run - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/mldataset/test_fs.py::FsMultiLevelDatasetTest::test_io_nl_4_ts_256 - xcube.core.mldataset.fs.FsMultiLevelDatasetError: Failed to write dataset t...
FAILED test/core/mldataset/test_fs.py::FsMultiLevelDatasetTest::test_io_nl_4_ts_256_agg - xcube.core.mldataset.fs.FsMultiLevelDatasetError: Failed to write dataset t...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset /tmp/xcubeiv...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset '/tmp/xcubew...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset /tmp/xcubeu0...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset 'xcube/testi...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset 'xcube/testi...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/test_dump.py::DumpDatasetTest::test_dump_dataset - AssertionError: "<xarray.DataArray 'precipitation' (time: 5, lat: 180, lon:...
FAILED test/core/test_normalize.py::TestNormalize::test_normalize_zonal_lat_lon - ValueError: cannot re-index or align objects with conflicting indexes found...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_failures - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_list - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_list_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_tuple - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_tuple_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_true - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_true_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_append_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_find_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_insert_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_replace_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_update_corrupt_cube - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::ZarrStoreTest::test_local - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_chunked - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_dont_unchunk_if_unchunked - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_all - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_coord_var - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_coords_only - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_data_var - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_data_var_coords_only - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_invalid_path - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_invalid_vars - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/zarrstore/test_generic.py::CommonS3ZarrStoreTest::test_it - ValueError: When encoding chunked arrays of datetime values, both the units...
ERROR test/core/store/fs/test_registry.py::NewCubeDataTestMixin::test_open_packed - ValueError: When encoding chunked arrays of datetime values, both the units...
ERROR test/core/store/fs/test_registry.py::NewCubeDataTestMixin::test_open_unpacked - ValueError: When encoding chunked arrays of datetime values, both the units...
= 61 failed, 1356 passed, 39 skipped, 6357 warnings, 2 errors in 212.92s (0:03:32) =
pont-us commented
Just for reference, I'm attaching a script I wrote to investigate one of the errors covered by this issue. The script demonstrates the changed xcube behaviour when serializing a chunked datetime array for which the encoding specifies the units but not the (encoded) dtype.