gher-uliege/DINCAE.jl

ERROR: LoadError: GPU compilation of MethodInstance

ctroupin opened this issue · 5 comments

ctroupin commented

Situation

I'm trying the run gridding of CPR data, which were previously working.

Environment

  • Version 1.9.3 (2023-08-24)
  • Package status:
Project CPRDINCAE v0.1.0
Status `~/Projects/CPR-DINCAE/Project.toml`
  [8f4d0f93] Conda v1.9.1
  [0d879ee6] DINCAE v2.0.2 `https://github.com/gher-uliege/DINCAE.jl#main`
  [864edb3b] DataStructures v0.18.15
  [8bb1440f] DelimitedFiles v1.9.1
  [587475ba] Flux v0.14.6
  [2fb1d81b] GeoArrays v0.8.3
  [61d90e0f] GeoJSON v0.7.2
  [bb4c363b] GridInterpolations v1.1.2
  [7073ff75] IJulia v1.24.2
⌃ [6a3955dd] ImageFiltering v0.7.6
  [a98d9a8b] Interpolations v0.14.7
  [85f8d34a] NCDatasets v0.12.17
  [647866c9] PolygonOps v0.1.2
  [438e738f] PyCall v1.96.1
  [d330b81b] PyPlot v2.11.2
⌃ [02a925ec] cuDNN v1.1.1
  [ade2ca70] Dates
  [9a3f8284] Random
  [10745b16] Statistics v1.9.0

Full error

┌ Warning: Performing scalar indexing on task Task (runnable) @0x00007f36b7bfc010.
│ Invocation of getindex resulted in scalar indexing of a GPU array.
│ This is typically caused by calling an iterating implementation of a method.
│ Such implementations *do not* execute on the GPU, but very slowly on the CPU,
│ and therefore are only permitted from the REPL for prototyping purposes.
│ If you did intend to index this array, annotate the caller with @allowscalar.
└ @ GPUArraysCore ~/.julia/packages/GPUArraysCore/uOYfN/src/GPUArraysCore.jl:106
ERROR: LoadError: GPU compilation of MethodInstance for (::GPUArrays.var"#broadcast_kernel#26")(::CUDA.CuKernelContext, ::CuDeviceVector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Float32, Float32}}, 1}, ::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{1}, Tuple{Base.OneTo{Int64}}, …}, ::Int64) failed
KernelError: passing and using non-bitstype argument

Argument 4 to your kernel function is of type Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{1}, Tuple{Base.OneTo{Int64}}, ChainRulesCore.var"#49#50", Tuple{Base.Broadcast.Extruded{CuDeviceVector{ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{Tuple{Float32, Float32}}, NamedTuple{(:elements,), Tuple{Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}}}}}, 1}, Tuple{Bool}, Tuple{Int64}}, Base.Broadcast.Extruded{Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}}, Tuple{Bool}, Tuple{Int64}}}}, which is not isbits:
  .args is of type Tuple{Base.Broadcast.Extruded{CuDeviceVector{ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{Tuple{Float32, Float32}}, NamedTuple{(:elements,), Tuple{Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}}}}}, 1}, Tuple{Bool}, Tuple{Int64}}, Base.Broadcast.Extruded{Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}}, Tuple{Bool}, Tuple{Int64}}} which is not isbits.
    .2 is of type Base.Broadcast.Extruded{Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}}, Tuple{Bool}, Tuple{Int64}} which is not isbits.
      .x is of type Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}} which is not isbits.


Stacktrace:
  [1] check_invocation(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/validation.jl:96
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:123 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:121
  [5] codegen
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:110 [inlined]
  [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:106
  [7] compile
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:98 [inlined]
  [8] #1037
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:104 [inlined]
  [9] JuliaContext(f::CUDA.var"#1037#1040"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:47
 [10] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:103
 [11] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:125
 [12] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:103
 [13] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:318 [inlined]
 [14] macro expansion
    @ ./lock.jl:267 [inlined]
 [15] cufunction(f::GPUArrays.var"#broadcast_kernel#26", tt::Type{Tuple{CUDA.CuKernelContext, CuDeviceVector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Float32, Float32}}, 1}, …}}; kwargs::Base.Pairs{Symbol, Union{}, …})
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:313
 [16] cufunction
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:310 [inlined]
 [17] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:104 [inlined]
 [18] #launch_heuristic#1080
    @ ~/.julia/packages/CUDA/35NC6/src/gpuarrays.jl:17 [inlined]
 [19] launch_heuristic
    @ ~/.julia/packages/CUDA/35NC6/src/gpuarrays.jl:15 [inlined]
 [20] _copyto!
    @ ~/.julia/packages/GPUArrays/5XhED/src/host/broadcast.jl:65 [inlined]
 [21] copyto!
    @ ~/.julia/packages/GPUArrays/5XhED/src/host/broadcast.jl:46 [inlined]
 [22] copy
    @ ~/.julia/packages/GPUArrays/5XhED/src/host/broadcast.jl:37 [inlined]
 [23] materialize(bc::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{1}, Nothing, …})
    @ Base.Broadcast ./broadcast.jl:873
 [24] map(f::Function, x::CuArray{ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{Tuple{Float32, Float32}}, NamedTuple{(:elements,), Tuple{Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}}}}}, 1, …}, xs::Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}})
    @ GPUArrays ~/.julia/packages/GPUArrays/5XhED/src/host/broadcast.jl:84
 [25] ProjectTo
    @ ~/.julia/packages/ChainRulesCore/0t04l/src/projection.jl:238 [inlined]
 [26] macro expansion
    @ ~/.julia/packages/ChainRulesCore/0t04l/src/projection.jl:343 [inlined]
 [27] _project_namedtuple(f::NamedTuple{(:pos, :x), Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{CuArray{ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{Tuple{Float32, Float32}}, NamedTuple{(:elements,), Tuple{Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}}}}}, 1, …}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}, x::NamedTuple{(:pos, :x), Tuple{Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}}, ChainRulesCore.NoTangent}})
    @ ChainRulesCore ~/.julia/packages/ChainRulesCore/0t04l/src/projection.jl:342
 [28] (::ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}, NamedTuple{(:pos, :x), Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{CuArray{ChainRulesCore.ProjectTo{ChainRulesCore.Tangent{Tuple{Float32, Float32}}, NamedTuple{(:elements,), Tuple{Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}}}}}, 1, …}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}})(dx::NamedTuple{(:pos, :x), Tuple{Vector{ChainRulesCore.Tangent{Tuple{Float32, Float32}, Tuple{Int64, Int64}}}, ChainRulesCore.NoTangent}})
    @ ChainRulesCore ~/.julia/packages/ChainRulesCore/0t04l/src/projection.jl:335
 [29] ProjectTo
    @ ~/.julia/packages/ChainRulesCore/0t04l/src/projection.jl:317 [inlined]
 [30] _project
    @ ~/.julia/packages/Zygote/4SSHS/src/compiler/chainrules.jl:189 [inlined]
 [31] (::Zygote.var"#back#302"{:pos, Zygote.Context{true}, …})(Δ::Vector{Tuple{Int64, Int64}})
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/lib/lib.jl:234
 [32] #2184#back
    @ ~/.julia/packages/ZygoteRules/OgCVT/src/adjoint.jl:71 [inlined]
 [33] Pullback
    @ ./none:0 [inlined]
 [34] (::Zygote.Pullback{Tuple{DINCAE.var"#121#124"{CuArray{Float64, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}, Int64}, Tuple{Zygote.ZBack{ChainRules.var"#hcat_pullback#1399"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, …}}, Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:pos, Zygote.Context{true}, …}}, …}})(Δ::CuArray{Float64, 2, …})
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [35] #680
    @ ~/.julia/packages/Zygote/4SSHS/src/lib/array.jl:215 [inlined]
 [36] (::Base.var"#4#5"{Zygote.var"#680#685"})(a::Tuple{Tuple{CuArray{Float64, 2, …}, Zygote.Pullback{Tuple{DINCAE.var"#121#124"{CuArray{Float64, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}, Int64}, Tuple{Zygote.ZBack{ChainRules.var"#hcat_pullback#1399"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, …}}, Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:pos, Zygote.Context{true}, …}}, …}}}, CuArray{Float64, 2, …}})
    @ Base ./generator.jl:36
 [37] iterate
    @ ./generator.jl:47 [inlined]
 [38] collect(itr::Base.Generator{Base.Iterators.Zip{Tuple{Vector{Tuple{CuArray{Float64, 2, …}, Zygote.Pullback{Tuple{DINCAE.var"#121#124"{CuArray{Float64, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}, Int64}, Tuple{Zygote.ZBack{ChainRules.var"#hcat_pullback#1399"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float64, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, …}}, Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:pos, Zygote.Context{true}, …}}, …}}}}, Vector{CuArray{Float64, 2, …}}}}, Base.var"#4#5"{Zygote.var"#680#685"}})
    @ Base ./array.jl:782
 [39] map
    @ ./abstractarray.jl:3385 [inlined]
 [40] (::Zygote.var"#map_back#682"{DINCAE.var"#121#124"{CuArray{Float64, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}, 1, …})(Δ::Vector{CuArray{Float64, 2, …}})
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/lib/array.jl:215
 [41] (::Zygote.var"#collect_pullback#720"{Zygote.var"#map_back#682"{DINCAE.var"#121#124"{CuArray{Float64, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}, 1, …}, Nothing})(ȳ::Vector{CuArray{Float64, 2, …}})
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/lib/array.jl:249
 [42] Pullback
    @ ~/.julia/packages/DINCAE/5Y5oF/src/points.jl:420 [inlined]
 [43] (::Zygote.Pullback{Tuple{DINCAE.var"##costfun#120", Int64, …}, Any})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [44] Pullback
    @ ~/.julia/packages/DINCAE/5Y5oF/src/points.jl:375 [inlined]
 [45] (::Zygote.Pullback{Tuple{typeof(Core.kwcall), NamedTuple{(:laplacian_penalty, :laplacian_error_penalty), Tuple{Int64, Int64}}, …}, Any})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [46] Pullback
    @ ~/.julia/packages/DINCAE/5Y5oF/src/model.jl:172 [inlined]
 [47] (::Zygote.Pullback{Tuple{DINCAE.var"#53#57"{Bool, Int64, …}, CuArray{Float64, 4, …}, …}, Tuple{Zygote.Pullback{Tuple{Type{NamedTuple{(:laplacian_penalty, :laplacian_error_penalty)}}, Tuple{Int64, Int64}}, Tuple{Zygote.Pullback{Tuple{Type{NamedTuple{(:laplacian_penalty, :laplacian_error_penalty), Tuple{Int64, Int64}}}, Tuple{Int64, Int64}}, Tuple{Zygote.var"#2224#back#315"{Zygote.Jnew{NamedTuple{(:laplacian_penalty, :laplacian_error_penalty), Tuple{Int64, Int64}}, Nothing, …}}}}}}, Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:truth_uncertain, Zygote.Context{true}, …}}, …}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [48] Pullback
    @ ~/.julia/packages/DINCAE/5Y5oF/src/model.jl:201 [inlined]
 [49] (::Zygote.Pullback{Tuple{typeof(DINCAE.loss_function), DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, …}, Any})(Δ::Float32)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [50] #291
    @ ~/.julia/packages/Zygote/4SSHS/src/lib/lib.jl:206 [inlined]
 [51] #2173#back
    @ ~/.julia/packages/ZygoteRules/OgCVT/src/adjoint.jl:71 [inlined]
 [52] Pullback
    @ ~/.julia/packages/DINCAE/5Y5oF/src/flux.jl:26 [inlined]
 [53] (::Zygote.Pullback{Tuple{DINCAE.var"#4#5"{DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, Tuple{CuArray{Float32, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}}}, Tuple{Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:samples, Zygote.Context{true}, …}}, Zygote.var"#2173#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing, Nothing}}, Zygote.Pullback{Tuple{typeof(DINCAE.loss_function), DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, …}, Any}}}, …}})(Δ::Float32)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface2.jl:0
 [54] (::Zygote.var"#122#123"{Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, Zygote.Pullback{Tuple{DINCAE.var"#4#5"{DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, Tuple{CuArray{Float32, 4, …}, Vector{NamedTuple{(:pos, :x), Tuple{CuArray{Tuple{Float32, Float32}, 1, …}, CuArray{Float32, 2, …}}}}}}}, Tuple{Zygote.var"#2184#back#303"{Zygote.var"#back#302"{:samples, Zygote.Context{true}, …}}, Zygote.var"#2173#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing, Nothing}}, Zygote.Pullback{Tuple{typeof(DINCAE.loss_function), DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, …}, Any}}}, …}}, …})(Δ::Float32)
    @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:419
 [55] train_epoch!(::Tuple{DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, …}, dl::DINCAE.PointCloud{CuArray{Float32}, Float32, …}, learning_rate::Float64; clip_grad::Float64)
    @ DINCAE ~/.julia/packages/DINCAE/5Y5oF/src/flux.jl:29
 [56] kwcall(::NamedTuple{(:clip_grad,), Tuple{Float64}}, ::typeof(DINCAE.train_epoch!), ::Tuple{DINCAE.StepModel{DINCAE.var"#52#56"{Float64}, DINCAE.var"#53#57"{Bool, Int64, …}}, Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, …}, dl::DINCAE.PointCloud{CuArray{Float32}, Float32, …}, learning_rate::Float64)
    @ DINCAE ~/.julia/packages/DINCAE/5Y5oF/src/flux.jl:20
 [57] macro expansion
    @ ~/.julia/packages/DINCAE/5Y5oF/src/points.jl:657 [inlined]
 [58] macro expansion
    @ ./timing.jl:273 [inlined]
 [59] reconstruct_points(T::Type, Atype::Type, filename::String, varname::String, grid::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, …}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, …}}, fnames_rec::Vector{String}; epochs::Int64, batch_size::Int64, truth_uncertain::Bool, enc_nfilter_internal::Vector{Int64}, skipconnections::UnitRange{Int64}, clip_grad::Float64, regularization_L1_beta::Int64, regularization_L2_beta::Int64, save_epochs::StepRange{Int64, Int64}, upsampling_method::Symbol, probability_skip_for_training::Float64, jitter_std_pos::Tuple{Float32, Float32}, ntime_win::Int64, learning_rate::Float64, learning_rate_decay_epoch::Float64, min_std_err::Float64, loss_weights_refine::Tuple{Float64}, auxdata_files::Vector{Any}, savesnapshot::Bool, laplacian_penalty::Int64, laplacian_error_penalty::Int64)
    @ DINCAE ~/.julia/packages/DINCAE/5Y5oF/src/points.jl:651
 [60] kwcall(::NamedTuple{(:epochs,), Tuple{Int64}}, ::typeof(DINCAE.reconstruct_points), T::Type, Atype::Type, filename::String, varname::String, grid::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, …}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, …}}, fnames_rec::Vector{String})
    @ DINCAE ~/.julia/packages/DINCAE/5Y5oF/src/points.jl:529
 [61] top-level scope
    @ ~/Projects/CPR-DINCAE/src/run_DINCAE_0.jl:58
 [62] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [63] top-level scope
    @ REPL[6]:1
in expression starting at /home/ctroupin/Projects/CPR-DINCAE/src/run_DINCAE_0.jl:58

How to reproduce?

run_DINCAE_0.jl (private repos)

Alexander-Barth commented
Alexander-Barth commented

The issue is fixed for my reproducer.
@ctroupin, does it also work for you?

ctroupin commented

Thanks, yes I confirm this is working now, even if I sometimes get a "segmentation fault", which I cannot reproduced as needed.

...
[ Info: gamma: 10.0
no refine
skip connections at level 5
skip connections at level 4
skip connections at level 3
skip connections at level 2
Segmentation fault (core dumped)

maybe be due to errors in the pre-compilation; another issue will be open is this error comes back.

Alexander-Barth commented

errors in the pre-compilation

strange... Did you restart julia after updating ?

ctroupin commented

yes yes, I've run it twice, first time I got segmentation fault, then restarting Julia I got this:

DINCAE [0d879ee6-e5ed-4b6f-b65c-f78122b59944]
│  ┌ Warning: Module CUDA with build ID fafbfcfd-70bd-43e1-000f-0f28f04f3eaf is missing from the cache.
│  │ This may mean CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1793
│  ┌ Error: Error during loading of extension NNlibCUDAExt of NNlib, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
...

but the code is finally running.