jump-dev/NLopt.jl

Failure in callback under Julia-1.5.0-DEV

Closed this issue · 5 comments

dmbates commented

NLopt is the optimizer used in the MixedModels package. Our tests have started failing under Julia-1.5.0-DEV and under
julia-nightly in a callback to evaluate the objective function.

A simple example is transcript2.txt
I am using the MixedModels#master there but I think the same will occur under the latest release.

palday commented

I suspect the problem originates from this change in 1.5:

Immutable structs (including tuples) that contain references can now be allocated on the stack, and allocated inline within arrays and other structs (#33886). This significantly reduces the number of heap allocations in some workloads. Code that requires assumptions about object layout and addresses (usually for interoperability with C or other languages) might need to be updated; for example any object that needs a stable address should be a mutable struct.

NEWS page for 1.5
JuliaLang/julia#33886

If we change the algorithm MixedModels uses, we can sometimes avoid the problem:

using MixedModels, NLopt
using MixedModels: dataset

m1 = LinearMixedModel( @formula(yield ~ 1 + (1|batch)), dataset(:dyestuff))
m1.optsum.optimizer = :LN_NELDERMEAD
fit!(m1)

Nelder-Mead mostly works, as does COBYLA, but BOBYQA never does. I suspect that small differences in the way the memory is being allocated leads to being a small amount off in the relevant pointer and so you get the adjacent Core.Compiler.UseRef instead of the relevant NLopt.Callback_Data.

palday commented

See also #142.

Using NLopt#master solves this issue. Perhaps a release would be in order so that compatibility is maintained for Julia 1.5? I see the last one was in August 2018.....

dmbates commented

Thanks for looking into this @palday

stevengj commented

Yes, a new release would be a good idea. I also want to switch over to the JLL artifacts system for building the library (JuliaPackaging/Yggdrasil#1028 and JuliaRegistries/General#14686)…