instantiate command results in type error, preventing dependency installation

Question

instantiate command results in type error, preventing dependency installation

Closed this issue 3 years ago · 10 comments

Hello!

Pushing into Julia via this repo, thanks for putting it together!

I'm running Julia 1.5.3, and following the installation instructions in repo README.md, I am hitting a TypeError when executing the instantiate command. Stacktrace, here:

(@v1.5) pkg> activate .
 Activating environment at `~/repos/StatisticalRethinkingTuring.jl/Project.toml`

(StatisticalRethinkingTuring) pkg> instantiate
 Installing known registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
ERROR: TypeError: in typeassert, expected VersionNumber, got a value of type Pkg.Types.VersionSpec
Stacktrace:
 [1] load_urls(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:510
 [2] #download_source#54 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:686 [inlined]
 [3] download_source at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:685 [inlined]
 [4] instantiate(::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Pkg.BinaryPlatforms.MacOS, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:875
 [5] instantiate(::Pkg.Types.Context) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:795
 [6] #instantiate#169 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:791 [inlined]
 [7] instantiate() at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:791
 [8] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:401
 [9] do_cmd(::REPL.LineEditREPL, ::String; do_rethrow::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:382
 [10] do_cmd at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [11] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:546
 [12] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [13] invokelatest at ./essentials.jl:709 [inlined]
 [14] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/LineEdit.jl:2355
 [15] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:1144
 [16] (::REPL.var"#38#42"{REPL.LineEditREPL,REPL.REPLBackendRef})() at ./task.jl:356

Searching, I found this discourse discussion on the same exception, and mentions that there is possibly incompatibility in Mainfest.toml across different versions of Julia. I've inspected the Manifest.toml in the repo, and don't see any reference to a specific Julia version, and being unfamiliar with Julia, don't know where else to go. Any pointers would be helpful.

Thanks much in advance!

Answer 1 · 2021-02-28T00:01:49.000Z

Hi,

I recommend deleting the manifest file and trying again. If you can activate the environment, you might have to update with ]update.

Was the Manifest.toml file in the repo when you added it? Generally, it is added to gitignore to prevent these types of conflicts.

Edit: I can verify that Manifest.toml is in the repo. Rob, is there a reason that you are not including it in your gitignore file?

Answer 2 · 2021-02-28T00:15:55.000Z

Thanks for jumping in Chris. I can certainly remove the Manifest.toml files from the repos. Nicholaus ( @nehalecky ) can you delete the Manifest.toml as Chris suggest? My understanding was that including the Manifest.toml file is the basis for reproducible projects.

I wonder what happens if you skip the instantiate step and what triggers this behavior. Only a few folks seem to run into it, could it be the base configuration? My base configuration is very minimal (Pkg, Pluto, DrWatson). I would love to get to the bottom of this.

I've also seen the not backward compatible discussion and didn't quite understand it.

Answer 3 · 2021-02-28T00:42:32.000Z

Good point. Including Manifest.toml should ensure that all of the dependencies will have the same versions. I believe I encountered the same issue in my PR for ModelComparisons. That might provide some insight. I can also try to reproduce the problem on my system tomorrow if that would help.

Answer 4 · 2021-02-28T00:50:34.000Z

That would definitely help. If @nehalecky reports it works after deleting the Manifest,toml, at least we have a simple work around.

Answer 5 · 2021-02-28T01:08:43.000Z

@itsdfish and @goedman, thanks so much for the quick replies!

@itsdfish, your workaround worked! To summarize steps:

delete Manifest.toml in repo root
]update
activate .
instantiate

Then I was able to launch Pluto as described. Generally, makes sense about the reproducibility desired, however, it's probably beneficial make this caveat explicit in the README.md, just in case someone else encounters it!

Happy to close this, unless you'd like a PR that captures the above. Let me know, and thanks again!

Answer 6 · 2021-02-28T01:39:24.000Z

Thanks very much for the update. I’ll update the READMEs for all projects. Thanks again for sticking with it!

Answer 7 · 2021-02-28T09:59:05.000Z

According to the documentation, instantiate should work:

help?> Pkg.instantiate
  Pkg.instantiate(; verbose = false)

  If a Manifest.toml file exists in the active project, download all the packages declared in that manifest. Otherwise, resolve a set of feasible packages from the Project.toml files and install them. verbose = true prints the build output to stdout/stderr instead of redirecting
  to the build.log file. If no Project.toml exist in the current active project, create one with all the dependencies in the manifest and instantiate the resulting project.

However, I receive a different error when following your directions:

(base) dfish@dfish-MS-7850:~/Desktop/StatisticalRethinkingTuring.jl$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> pwd()
"/home/dfish/Desktop/StatisticalRethinkingTuring.jl"

(@v1.5) pkg> activate .
 Activating environment at `~/Desktop/StatisticalRethinkingTuring.jl/Project.toml`

(StatisticalRethinkingTuring) pkg> instantiate
ERROR: AssertionError: sourcepath !== nothing
Stacktrace:
 [1] is_package_downloaded at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:1670 [inlined]
 [2] #12 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:113 [inlined]
 [3] _all(::Pkg.Operations.var"#12#13"{Pkg.Types.Context}, ::Array{Pkg.Types.PackageSpec,1}, ::Colon) at ./reduce.jl:828
 [4] #all#638 at ./reducedim.jl:735 [inlined]
 [5] all at ./reducedim.jl:735 [inlined]
 [6] is_instantiated at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:113 [inlined]
 [7] instantiate(::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Pkg.BinaryPlatforms.Linux, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:828
 [8] instantiate(::Pkg.Types.Context) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:795
 [9] #instantiate#169 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:791 [inlined]
 [10] instantiate() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:791
 [11] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:401
 [12] do_cmd(::REPL.LineEditREPL, ::String; do_rethrow::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:382
 [13] do_cmd at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [14] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:546
 [15] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [16] invokelatest at ./essentials.jl:709 [inlined]
 [17] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/LineEdit.jl:2355
 [18] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:1144
 [19] (::REPL.var"#38#42"{REPL.LineEditREPL,REPL.REPLBackendRef})() at ./task.jl:356

(StatisticalRethinkingTuring) pkg>

If I delete Manifest.toml, the simplest sequence of steps is:

activate .
instantiate

You might want to note that instantiate is only required for initial set up. activate . is sufficient for future use. Subsequent calls to instantiate will trigger recompilation. So I think that is the easiest workaround, but it is not clear to me why instantiate does not work as intended.

Answer 8 · 2021-02-28T10:23:28.000Z

"Only a few folks seem to run into it, could it be the base configuration? My base configuration is very minimal (Pkg, Pluto, DrWatson). I would love to get to the bottom of this."

My understanding is that the global and project specific environments should be independent of each other. The only time the global environment can become a factor is if your scripts call packages that are not listed in your Project.toml.

Answer 9 · 2021-02-28T16:40:29.000Z

Thanks Chris. True, be it mostly by complaining about incompatible versions :-)

Also experimented a bit more and this might indeed have to do with the backwards incompatibility issue. Not completely surprising as I always develop on nightly. For now I've added a suggestion to delete Manifest.toml if the initial steps fail in all 3 projects (StatisticalRethinkingDHMC is not yet on GitHub, but will be one day!).

Until now I've never seen an incompatibility with other packages when upgrading the Manifest.toml (and I tend to do that pretty regularly).

But I'm certainly struggling with a solid workflow with non-registered projects and registered packages when the projects trigger functionality in a package (in this case StatisticalRethinking.jl) using Requires.jl. And as a consequence progress with both StatisticalRethinkingTuring and StatisticalRethinkingDHMC is way behind schedule!

Anyway, this has been a very useful issue for me. Thanks a lot. Shall we close the issue or just leave it open for a few months to help other folks jumping in?

Answer 10 · 2021-02-28T16:52:53.000Z

Before pushing the latest version of StatisticalRethinking.jl with e.g. trankplot(), compare() and plot_models() based on PSIS and WAIC as used in chapters 7-9 I'll have to wait until the renamed StatsModelComparisons.jl has been registered (probably next Tuesday, 3 days waiting period). Probably on Wednesday I'll push the projects.

I think part of the workflow with projects is very consistent tagging of the repos.

Apologies Nicholaus, this is all probably way more info then you asked for, but it helps me to write it down!