ArgumentError: Unable to determine chunksize of non-range views.

Question

ArgumentError: Unable to determine chunksize of non-range views.

Closed this issue 6 months ago · 21 comments

Hello,

I am trying to see if I am doing something wrong here. This piece of code worked a couple of months ago and no longer works. I tried on multiple dataset/cube and I have the same error.

Thanks!

using YAXArrays
using Zarr
using DimensionalData
using Dates
path="gs://cmip6/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp585/r1i1p1f1/3hr/tas/gn/v20190710/"
store = zopen(path, consolidated=true)
ds = open_dataset(store)

ds.tas[Ti=Where(x-> Dates.monthday(x) != (2,29))]
ERROR: ArgumentError: Unable to determine chunksize of non-range views.
Stacktrace:
  [1] eachchunk_view(::DiskArrays.Chunked{DiskArrays.ChunkRead{…}}, vv::SubArray{Float32, 3, ZArray{…}, Tuple{…}, false})
    @ DiskArrays ~/.julia/packages/DiskArrays/6JA8Z/src/subarray.jl:29
  [2] eachchunk(a::DiskArrays.SubDiskArray{Float32, 3, ZArray{…}, Tuple{…}, false})
    @ DiskArrays ~/.julia/packages/DiskArrays/6JA8Z/src/subarray.jl:25
  [3] rebuild(A::YAXArray{…}, data::DiskArrays.SubDiskArray{…}, dims::Tuple{…}, refdims::Tuple{}, name::DimensionalData.NoName, metadata::Dict{…})
    @ YAXArrays.Cubes ~/.julia/packages/YAXArrays/b5XBB/src/Cubes/Cubes.jl:201
  [4] rebuild
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/array.jl:85 [inlined]
  [5] rebuildsliced
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/array.jl:100 [inlined]
  [6] rebuildsliced
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/array.jl:99 [inlined]
  [7] view
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/indexing.jl:125 [inlined]
  [8] _dim_view
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/indexing.jl:110 [inlined]
  [9] #view#110
    @ ~/.julia/packages/DimensionalData/GaADx/src/array/indexing.jl:81 [inlined]
 [10] getindex(::YAXArray{Float32, 3, ZArray{…}, Tuple{…}, Dict{…}}; kwargs::@Kwargs{Ti::Where{…}})
    @ YAXArrays.Cubes ~/.julia/packages/YAXArrays/b5XBB/src/Cubes/Cubes.jl:488
 [11] top-level scope
    @ REPL[33]:1
Some type information was truncated. Use `show(err)` to see complete types.

Answer 1 · 2024-08-07T16:53:34.000Z

What versions of the packages are you on?

Answer 2 · 2024-08-07T17:25:18.000Z

edit - See messages below this one for relevant information.

I am on latest versions for all packages (DimensionalData, YAXArrays, DiskArrays). I tried with older version, close to last spring 2024, but somehow, I still haven't found where it is breaking yet...

I will try to look closely at the date when the code worked (i.e. JuliaDataCubes/YAXArrays.jl#357)

Here's some configurations I tried so far:

# latest
  [0703355e] DimensionalData v0.27.6
  [3c3547ce] DiskArrays v0.4.4
  [c21b50f5] YAXArrays v0.5.10

⌅ [0703355e] DimensionalData v0.26.8
⌃ [3c3547ce] DiskArrays v0.4.3
⌃ [c21b50f5] YAXArrays v0.5.5

⌅ [0703355e] DimensionalData v0.26.8
  [3c3547ce] DiskArrays v0.4.4
⌃ [c21b50f5] YAXArrays v0.5.5

  [0703355e] DimensionalData v0.27.6
⌃ [3c3547ce] DiskArrays v0.4.2
⌃ [c21b50f5] YAXArrays v0.5.6

  [0703355e] DimensionalData v0.27.6
⌃ [3c3547ce] DiskArrays v0.4.2
⌃ [c21b50f5] YAXArrays v0.5.7

  [0703355e] DimensionalData v0.27.6
⌃ [3c3547ce] DiskArrays v0.4.2
  [c21b50f5] YAXArrays v0.5.10

  [0703355e] DimensionalData v0.27.6
⌃ [3c3547ce] DiskArrays v0.4.3
  [c21b50f5] YAXArrays v0.5.10

Answer 3 · 2024-08-07T17:30:50.000Z

I found versions where it works!

ds.tas[Ti=Where(x-> Dates.monthday(x) != (2,29))]
384×192×251120 YAXArray{Float32,3} with dimensions:
  Dim{:lon} Sampled{Float64} 0.0:0.9375:359.0625 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} Float64[-89.28422753251364, -88.35700351866494, …, 88.35700351866494, 89.28422753251364] ForwardOrdered Irregular Points,
  Ti Sampled{DateTime} DateTime[2015-01-01T03:00:00, …, 2101-01-01T00:00:00] ForwardOrdered Irregular Points
units: K
name: tas
Total size: 68.97 GB

Versions are:

⌅ [0703355e] DimensionalData v0.25.8
⌅ [3c3547ce] DiskArrays v0.3.23
⌃ [c21b50f5] YAXArrays v0.5.3

Answer 4 · 2024-08-07T17:45:03.000Z

Due to Zarr and DimensionalData requirements in the MWE, I am unable to install DiskArrays@v0.4.0 to see if it is the version the break the code or if this is due to YAXArrays going to v0.5.4

Answer 5 · 2024-08-07T17:53:07.000Z

Zarr was causing the req incompatibilities. I tried with a NetCDF file with the following versions and I get the error. So, either from DiskArrays going to v0.4.0 or YAXArrays going to v0.5.4

Env with error:

⌅ [0703355e] DimensionalData v0.26.8
⌃ [3c3547ce] DiskArrays v0.4.0
  [30363a11] NetCDF v0.12.0
⌃ [c21b50f5] YAXArrays v0.5.4

Answer 6 · 2024-08-07T22:59:30.000Z

Most likely DiskArrays 0.4, it's a major reworking of some indexing internals. @meggart will know

Answer 7 · 2024-08-08T13:18:31.000Z

This comes from changes to eachchunk I managed to reduce this to a DiskArrays problem on DiskArrays master:

using DiskArrays
using DiskArrays.TestTypes
julia> a = TestTypes.ChunkedDiskArray(rand(100,100),(10,10))
100×100 ChunkedDiskArray{Float64, 2, Matrix{Float64}}

Chunked: (
    [10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
    [10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
)


julia> v = view(a, [1,2,4],:);

julia> eachchunk(v)
ERROR: ArgumentError: Unable to determine chunksize of non-range views.
Stacktrace:
 [1] eachchunk_view(::DiskArrays.Chunked{…}, vv::SubArray{…})
   @ DiskArrays ~/.julia/dev/DiskArrays/src/subarray.jl:29
 [2] eachchunk(a::DiskArrays.SubDiskArray{Float64, 2, ChunkedDiskArray{…}, Tuple{…}, false})
   @ DiskArrays ~/.julia/dev/DiskArrays/src/subarray.jl:25
 [3] top-level scope
   @ REPL[17]:1
Some type information was truncated. Use `show(err)` to see complete types.

Answer 8 · 2024-08-08T13:24:35.000Z

I am not sure what is the reason we have this check there and whether we could try to forward this to the unchunked access of the data.

Answer 9 · 2024-08-08T13:28:49.000Z

This might already be solved in #181

Answer 10 · 2024-08-08T13:49:49.000Z

Great, will keep an eye on latest updates from DiskArrays.jl and report back when it is updated. Thanks!

Answer 11 · 2024-08-09T13:48:54.000Z

Is there a way I can test the #181 commit? I tried with the commit (add DiskArrays#5ef1432d4e925cf550c1cfdd3e083eca80db1fe9 and add DiskArrays#5ef1432), but didn't worked. Perhaps it is because the commit is from another repo? (I never tried to pin a package with a specific commit).

I'd like to test the commit to see if it is resolved. Thanks!

Answer 12 · 2024-08-09T13:52:55.000Z

Its in https://github.com/ConnectedSystems/DiskArrays.jl fix-index-issue branch

You probably have to manually clone it?

Answer 13 · 2024-08-09T14:30:37.000Z

Its in https://github.com/ConnectedSystems/DiskArrays.jl fix-index-issue branch

You probably have to manually clone it?

ah! And then do a dev? I'll try to so how hard it is (I guess I would need to dev YAXArrays too and change the [deps] section)

Answer 14 · 2024-08-09T14:36:21.000Z

no you can just add it and it will work:
] add https://github.com/ConnectedSystems/DiskArrays.jl#fix-index-issue

or of cource git clone and then dev

Answer 15 · 2024-08-09T16:11:04.000Z

It works! 😄

ds.tas[Ti=Where(x-> Dates.monthday(x) != (2,29))]
╭────────────────────────────────────╮
│ 384×192×251120 YAXArray{Float32,3} │
├────────────────────────────────────┴─────────────────────────────────────────────────────── dims ┐
  ↓ lon Sampled{Float64} 0.0:0.9375:359.0625 ForwardOrdered Regular Points,
  → lat Sampled{Float64} [-89.28422753251364, -88.35700351866494, …, 88.35700351866494, 89.28422753251364] ForwardOrdered Irregular Points,
  ↗ Ti  Sampled{DateTime} [2015-01-01T03:00:00, …, 2101-01-01T00:00:00] ForwardOrdered Irregular Points
├──────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 10 entries:
  "units"         => "K"
  "history"       => "2019-07-21T06:26:13Z altered by CMOR: Treated scalar dimension: 'height'. 201…
  "name"          => "tas"
  "cell_methods"  => "area: mean time: point"
  "cell_measures" => "area: areacella"
  "long_name"     => "Near-Surface Air Temperature"
  "coordinates"   => "height"
  "standard_name" => "air_temperature"
  "_FillValue"    => 1.0f20
  "comment"       => "near-surface (usually, 2 meter) air temperature"
├─────────────────────────────────────────────────────────────────────────────────────── file size ┤
  file size: 68.97 GB

  [0703355e] DimensionalData v0.27.6
  [3c3547ce] DiskArrays v0.4.5 `https://github.com/ConnectedSystems/DiskArrays.jl#fix-index-issue`
  [30363a11] NetCDF v0.12.0
  [c21b50f5] YAXArrays v0.5.10
  [0a941bbe] Zarr v0.9.4

@rafaqz Note that if I did dev DiskArrays after add ...ConnectedSystems..., the dev version was reverting back to 0.4.4. Only adding the version from the ConnectedSystem was sufficient to test the commit. Thanks again!

Answer 16 · 2024-08-09T20:09:12.000Z

@rafaqz I have a side-question for this working example (the MWE provided in this thread: JuliaDataCubes/YAXArrays.jl#357, reproduced below). I am trying to rebuild the array (i.e. updating the dimensions values), but I have trouble understanding how I can do it. Any pointer would be helpful!

using YAXArrays
using Zarr
using DimensionalData
using Dates
path="gs://cmip6/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp585/r1i1p1f1/3hr/tas/gn/v20190710/"
store = zopen(path, consolidated=true)
ds = open_dataset(store)

# Selecting only entry without Feb 29th.
ds_subset = ds.tas[Ti=Where(x-> Dates.monthday(x) != (2,29))]

# Taking and reinterpreting the timevector
date_vec = lookup(ds_subset, dimt)
# New time vector
datevec_noleap = CFTime.reinterpret.(DateTimeNoLeap, date_vec)

Then I'd like to rebuild the dimensions :Ti with datevec_noleap, for example, I can rebuild with

rebuild(:Ti, datevec_noleap)
newdim = Ti [DateTimeNoLeap(2001-01-01T00:00:00), …, DateTimeNoLeap(2010-12-31T00:00:00)]

However, I still haven't been able to assign this new rebuilded dim to ds_subset

Answer 17 · 2024-08-09T20:48:47.000Z

DD is mostly straight functional code... You have to rebuild constructed objects and assign them to a new variable.

In this case, set is a rebuild helper. So you can do new_array = set(some_dd_array, Ti => datevec_noleap) and set will figure out how to rebuild it correctly.

You can put pretty much any dimensions of lookup property after the => as there is no ambiguity as to what you mean to do.

Answer 18 · 2024-08-09T20:59:21.000Z

Nice, will give a try monday morning 😄

Thanks a lot!

Answer 19 · 2024-08-12T18:22:09.000Z

DD is mostly straight functional code... You have to rebuild constructed objects and assign them to a new variable.

In this case, set is a rebuild helper. So you can do new_array = set(some_dd_array, Ti => datevec_noleap) and set will figure out how to rebuild it correctly.

You can put pretty much any dimensions of lookup property after the => as there is no ambiguity as to what you mean to do.

It works nicely, thanks!

Answer 20 · 2024-08-12T20:08:06.000Z

Its in ConnectedSystems/DiskArrays.jl fix-index-issue branch

You probably have to manually clone it?

You can also checkout the PR branch locally from a normally dev'ed DiskArrays via this tutorial
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Answer 21 · 2024-10-22T12:44:44.000Z

Fixed by #181