[BUG]: Spontaneous TaskFailedException
Closed this issue · 5 comments
gm89uk commented
What happened?
After running a search with a custom loss function overnight, I opened an excel sheet and the TaskFailedException happened. I'm running on 16GB ram, I'm not sure if that has anything to do with it, but I've been running the same code for months, with many more things open and this only happened with beta2.
Thank you again for your work with this.
Version
v1.0.0beta2
Operating System
Windows
Interface
Julia REPL
Relevant log output
ERROR: TaskFailedExceptionre Equation
Stacktrace: 7.582e-01 1.594e+01 y = 2.3557
[1] wait 7.182e-01 2.713e-02 y = -0.201 * a
@ .\task.jl:352 [inlined]-02 y = 0.23635 * (r - a)
[2] fetch 6.126e-01 6.060e-02 y = (0.61059 * (r - p)) - a
@ .\task.jl:372 [inlined]-02 y = r - ((p * 1.8532) + (a / 0.44432))
[3] _main_search_loop!(state::SymbolicRegression.SearchUtilsModule.SearchState{…}, datasets::Vector{…}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{…}, options::Options{…})
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:829
[4] _equation_search(datasets::Vector{Dataset{…}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multithreading, 1, true}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{…}, DynamicExpressions.OperatorEnumModule.OperatorEnum, DynamicExpressions.NodeModule.Node, Expression, @NamedTuple{}, MutationWeights, true, true, nothing, Nothing, 5}, saved_state::Nothing)
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:533
[5] equation_search(datasets::Vector{…}; options::Options{…}, saved_state::Nothing, runtime_options::Nothing, runtime_options_kws::@Kwargs{…})
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:52338)))) * 1.2304
[6] equation_search 1.932e-03 y = ((0.15476 * ((l + (d * 1.2641)) + g)) + (r - ((a - -13.138) * ((p * 0.13204) - -2.3591)))) * 1.2316
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:504 [inlined]707) * a)) - p) * 1.2366) - p)
[7] equation_search(X::Matrix{…}, y::Matrix{…}; niterations::Int64, weights::Vector{…}, options::Options{…}, variable_names::Vector{…}, display_variable_names::Vector{…}, y_variable_names::Nothing, parallelism::Symbol, numprocs::Nothing, procs::Nothing, addprocs_function::Nothing, heap_size_hint_in_bytes::Nothing, runtests::Bool, saved_state::Nothing, return_state::Bool, run_id::Nothing, loss_type::Type{…}, verbosity::Int64, progress::Nothing, X_units::Nothing, y_units::Nothing, extra::@NamedTuple{}, v_dim_out::Val{…}, multithreaded::Nothing, varMap::Nothing)- p) / 0.78218
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:475- (((p * 0.12872) - -2.3832) * (a - -6.1259)))) / 0.76823
[8] #equation_search#21065e-04 y = (r + (((0.13712 * ((((g * 0.76823) - (p * (r / 1.5913))) + l) + (d * 1.2775))) - p) - (((p * 0.12872) - -2.3832) * (a - -6.1228)))) / 0.76823
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:497 [inlined]p * 0.50005)))) * 0.13297)) - ((a - -6.1258) * ((p * 0.13199) - -2.4217))) / 0.77853
[9] _update(m::SRRegressor{…}, verbosity::Int64, old_fitresult::Nothing, old_cache::Nothing, X::@NamedTuple{…}, y::Vector{…}, w::Vector{…}, options::Options{…}, classes::Nothing)) / 0.77853
@ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:191 0.13464) + (r - (((0.13255 * p) - -2.4211) * (a - -6.1259)))) - p) / 0.77253
[10] update(m::SRRegressor{DynamicQuantities.SymbolicDimensions{…}, DataType}, verbosity::Int64, old_fitresult::Nothing, old_cache::Nothing, X::@NamedTuple{a::SubArray{…}, r::SubArray{…}, p::SubArray{…}, d::SubArray{…}, l::SubArray{…}, g::SubArray{…}}, y::Vector{Float32}, w::Vector{Float32})
@ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:142(r + -2.3217) * (r - d)) * -0.063971) * g) + p) * 0.64795)))) + l) * -0.13559))) - p) / 0.79963
[11] fit(m::SRRegressor{DynamicQuantities.SymbolicDimensions{…}, DataType}, verbosity::Int64, X::@NamedTuple{a::SubArray{…}, r::SubArray{…}, p::SubArray{…}, d::SubArray{…}, l::SubArray{…}, g::SubArray{…}}, y::Vector{Float32}, w::Vector{Float32})
@ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:136(p + 0.65992)) + -2.3217) * g) * (-0.063971 * (r - d))) + (p * 0.63568)) * r))) + l) * -0.13559))) - p) / 0.79963
[12] fit_only!(mach::Machine{SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, true}; rows::Nothing, verbosity::Int64, force::Bool, composite::Nothing)
@ MLJBase C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:69280831 * r) * (-1.6799 + (r - (d * d)))) * g) + ((-0.29098 * d) + p)) * (r * 0.63569))))) * 0.13196) - ((a - -6.1258) * ((0.13254 * p) - -2.4217))) + r) - p) / 0.79963) / 0.9875
[13] fit_only!84e-01 4.239e-04 y = ((r - p) + ((((l + ((1.3129 * d) + g)) - (((((r * ((d * d) - (-1.9856 + r))) * 0.091608) * g) + ((d * (-0.094242 * (d * d))) + p)) * (r * 0.63569))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
@ C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:617 [inlined](((r * ((d * d) - (-1.9856 + r))) * 0.091608) * g) + ((d * (-0.094242 * (a + (d * d)))) + p)) * (r * 0.63569)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
[14] #fit!#63679e-01 2.768e-04 y = ((r - p) + (((l + (((1.3129 * d) + g) - ((((((r + 0.63569) * ((d * d) - (-1.9856 + r))) * (0.091608 * g)) + ((d * -0.094242) * (a + (d * d)))) + p) * (0.63569 * r)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
@ C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:789 [inlined]((((r + 0.63569) * ((d * d) - (-1.9856 + r))) * (0.091608 * g)) + ((d * -0.094242) * (a + (d * d)))) + (p / 1.0506)) * (0.63569 * r)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
[15] fit!(mach::Machine{SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, true})───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ MLJBase C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:786
[16] top-level scope
@ REPL[55]:1
nested task error: TaskFailedException
Stacktrace:
[1] wait
@ .\task.jl:352 [inlined]
[2] fetch
@ .\task.jl:372 [inlined]
[3] (::SymbolicRegression.var"#60#65"{SymbolicRegression.SearchUtilsModule.SearchState{…}, Int64, Int64})()
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:932
nested task error: InterruptException:
Stacktrace:
[1] Array
@ .\boot.jl:477 [inlined]
[2] vect
@ .\array.jl:163 [inlined]
[3] convert
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MutationWeights.jl:125 [inlined]
[4] sample_mutation(w::MutationWeights)
@ SymbolicRegression.CoreModule.MutationWeightsModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MutationWeights.jl:135
[5] next_generation(dataset::Dataset{…}, member::PopMember{…}, temperature::Float64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics, options::Options{…}; tmp_recorder::Dict{…})
@ SymbolicRegression.MutateModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\Mutate.jl:182
[6] next_generation
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\Mutate.jl:154 [inlined]
[7] reg_evol_cycle(dataset::Dataset{…}, pop::Population{…}, temperature::Float64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics, options::Options{…}, record::Dict{…})
@ SymbolicRegression.RegularizedEvolutionModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\RegularizedEvolution.jl:36
[8] s_r_cycle(dataset::Dataset{…}, pop::Population{…}, ncycles::Int64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics; verbosity::Int64, options::Options{…}, record::Dict{…})
@ SymbolicRegression.SingleIterationModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SingleIteration.jl:47
[9] s_r_cycle
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SingleIteration.jl:19 [inlined]
[10] _dispatch_s_r_cycle(in_pop::Population{…}, dataset::Dataset{…}, options::Options{…}; pop::Int64, out::Int64, iteration::Int64, verbosity::Int64, cur_maxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics)
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:1066
[11] _dispatch_s_r_cycle
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:1049 [inlined]
[12] macro expansion
@ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:916 [inlined]
[13] (::SymbolicRegression.var"#59#64"{…})()
@ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SearchUtils.jl:258
Some type information was truncated. Use `show(err)` to see complete types.Extra Info
No response
MilesCranmer commented
Do you mean that the search was still running, and opening the .csv file, which was being written to by PySR simultaneously, crashed things?
I think Excel has some kind of autosave feature so perhaps a concurrent write messed with Julia? Not sure. The easiest solution is to copy the file, and then open the copied version in excel.
But the error message you have seems unrelated - that’s the (harmless) message you see if you just Ctrl-C while it is running the search.
gm89uk commented
Hi Miles,
Sorry I should have clarified, the Excel worksheet opened was not the csv search, just a very large one that takes a lot of memory; I'm aware opening the HoF CSV during the search will crash it.
Sorry I cannot provide more insight to it! I will let you know if I can repro it tomorrow.
MilesCranmer commented
I'm aware opening the HoF CSV during the search will crash it.
I think it's actually generally safe to do this! Was just thinking about possible explanations and that was a possibility. But in general usage I don't think this would cause issues.
The error itself seems to be fairly benign though, this is what happens if the process is told to exit by hitting <ctrl-c>. But perhaps it was told to exit by some other mechanism, such as the computer going to sleep or something? I'm not sure. The InterruptException you see (I think) is just Julia exiting because something told it to stop
gm89uk commented
It's quite plausible I pressed ctrl + c while on the REPL by mistake. Thank you again for looking into it
MilesCranmer commented
That must be it! Let me know if this sort of thing comes up again though without input