Error with v2.6.0

Question

Error with v2.6.0

baumgold opened this issue a year ago · 9 comments

After upgrading to Arrow v2.6.0 I get the following error in one of my apps:

MethodError: Cannot `convert` an object of type Vector{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}} to an object of type SubArray{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}, 1, Arrow.Struct{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}, Tuple{Arrow.Primitive{UInt16, Vector{UInt16}}, Arrow.Primitive{Union{Missing, Int32}, Vector{Int32}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}, Arrow.Primitive{Int32, Vector{Int32}}, Arrow.Primitive{UInt32, Vector{UInt32}}, Arrow.Primitive{Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Vector{Arrow.Decimal{19, 9, Int128}}}, Arrow.Primitive{Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Vector{Arrow.Decimal{19, 9, Int128}}}}}, Tuple{UnitRange{Int64}}, true}

Closest candidates are:
  convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray
   @ LinearAlgebra ~/julia/julia-1.9.0/share/julia/stdlib/v1.9/LinearAlgebra/src/factorization.jl:59
  convert(::Type{T}, ::T) where T<:AbstractArray
   @ Base abstractarray.jl:16
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:64
  ...

Stacktrace:
  [1] push!
    @ ./array.jl:1060
  [2] #ToList#16
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:117
  [3] ToList
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:100 [inlined]
  [4] #arrowvector#18
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:214
  [5] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:211 [inlined]
  [6] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
  [7] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
  [8] #49
    @ ./none:0
  [9] iterate
    @ ./generator.jl:47 [inlined]
 [10] collect_to!
    @ ./array.jl:840
 [11] collect_to!
    @ ./array.jl:848
 [12] collect_to!
    @ ./array.jl:848
 [13] collect_to_with_first!
    @ ./array.jl:818
 [14] collect
    @ ./array.jl:792
 [15] _totuple
    @ ./tuple.jl:401 [inlined]
 [16] Tuple
    @ ./tuple.jl:369 [inlined]
 [17] #arrowvector#48
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:93
 [18] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:89 [inlined]
 [19] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
 [20] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
 [21] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:58 [inlined]
 [22] #toarrowvector#2
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:37
 [23] toarrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:34 [inlined]
 [24] #145
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:329
 [25] eachcolumn
    @ ~/.julia/packages/Tables/AcRIE/src/utils.jl:70
 [26] toarrowtable
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:326
 [27] macro expansion
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:187 [inlined]
 [28] macro expansion
    @ ./task.jl:476 [inlined]
 [29] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:177
 [30] #122
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:58
 [31] #open#409
    @ ./io.jl:395
 [32] open
    @ ./io.jl:392 [inlined]
 [33] #write#121
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:57 [inlined]
 [34] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:56 [inlined]

Answer 1 · 2023-06-07T16:47:41.000Z

you probably manually asserted (initialized) Vector{} of something at some point, see #452

Answer 2 · 2023-06-07T16:52:52.000Z

https://github.com/JuliaHEP/UnROOT.jl/actions/runs/5202197179/jobs/9383461173?pr=258#step:6:382

hmm, I hit this in a much more general case -- I convert to Arrow and read back, and it works. But some how it hits this thing during == which is worrisome, will provide a reproducer later.

Answer 3 · 2023-06-07T20:11:30.000Z

yeah ok this seems super broken now @quinnj

julia> vector_vector_int = [[[1]], [[1], [2]], [[1], [2], [3]], [[1], [2], [3], [4]], [[1], [2], [3], [4], [5]]]
5-element Vector{Vector{Vector{Int64}}}:
...

julia> df = DataFrame(; vector_vector_int)
5×1 DataFrame
 Row │ vector_vector_int
     │ Array…
─────┼───────────────────────────
   1 │ [[1]]
   2 │ [[1], [2]]
   3 │ [[1], [2], [3]]
   4 │ [[1], [2], [3], [4]]
   5 │ [[1], [2], [3], [4], [5]]

julia> Arrow.write(path, df)
"/tmp/jl_i5KrCRaB6b"

julia> DataFrame(Arrow.Table(path))
Error showing value of type DataFrame:
ERROR: MethodError: no method matching SubArray{Vector{Int64}, 1, Arrow.List{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}, Tuple{UnitRange{Int64}}, true}(::SubArray{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, 1, Arrow.List{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}, Tuple{UnitRange{Int64}}, true})

Closest candidates are:
  SubArray{T, N, P, I, L}(::Any, ::Any, ::Any, ::Any) where {T, N, P, I, L}
   @ Base subarray.jl:19

Answer 4 · 2023-06-07T21:03:14.000Z

It'd be nice to have a repro from @baumgold's original error, because it seems like a different thing than what @Moelf is showing later on.

Answer 5 · 2023-06-07T22:11:05.000Z

I have a relatively easy fix for @Moelf's case. I've been trying to figure out a way to reproduce @baumgold's error, but have been unsuccessful so far.

Answer 6 · 2023-06-07T23:01:34.000Z

I’ve been working to come up with a simple reproducible example. Been tied up with some other stuff today but hope to spend more time tonight and tomorrow. Will let you know as soon as I have something. Thanks, all.

Answer 7 · 2023-06-08T00:06:00.000Z

@quinnj - I think I managed to do it. See my latest comment in #456. I'm not quite sure how we should solve this. Any suggestions?

Answer 8 · 2023-06-08T14:02:51.000Z

Ok, I have a fix for @Moelf's issue here: #459. I think we also have a good fix for @baumgold's issue, so let's merge those 2 and get a new release kicked off. These errors are involving pretty complicated types, so it doesn't seem worth reverting/yanking anything IMO, let's just move forward with a patch fix.

Answer 9 · 2023-06-09T03:27:21.000Z

Fixed in #456