Add all dependencies of a package?

[notinaboat]

I have a cluster of embedded nodes that use a local registry for private packages.
Updating the "General" registry takes a very long time because network latency is high and because DEPOT_PATH is mounted by NFS.

I would like to copy all the package I use from "General" into the local registry.
It looks like PR #14 will enable this but I would still have to figure out all list of dependencies.

Maybe merge could accept an include_dependencies option?

LocalRegistry.merge(target_path, source_path, include = ["A", "B"], include_dependencies=true).

After this I could drop the "General" registry on the deployed nodes and efficiently deploy all my packages and their dependencies from the local registry.

[GunnarFarneback]

Updating the "General" registry takes a very long time because network latency is high and because DEPOT_PATH is mounted by NFS.

You could make your own snapshot of General and assign a new name and uuid. Pkg.UPDATED_REGISTRY_THIS_SESSION could possibly be of some use.

I would still have to figure out all list of dependencies.

They are all listed in Manifest.toml in your environment.

Maybe merge could accept an include_dependencies option?

LocalRegistry.merge(target_path, source_path, include = ["A", "B"], include_dependencies=true).

This is not well-defined since it depends on which versions you have of the packages and how the dependencies are resolved. Theoretically you could do something that recursively computed all dependencies for all possible versions but it's much easier and precise to let Pkg resolve everything and just read Manifest.toml.

[notinaboat]

You could make your own snapshot of General

In my setup I really need a subset of General. The time taken to create 25k+ files over NFS and embedded WiFi is prohibitive.

Pkg.UPDATED_REGISTRY_THIS_SESSION could possibly be of some use

Yes, I already use JULIA_PKG_OFFLINE = true (except when I need to update something).

Using Manifest.toml for dependencies makes sense.
I think if a include_dependencies=true option was added it would make sense to just use whatever the current dependencies are per Manifest.toml.

LocalRegistry.merge would have to be run again periodically anyway to bring in new versions. The local registry would end up containing all the historic versions and combinations of dependencies, which is what you want.

[GunnarFarneback]

Julia 1.7 will by default use a non-unpacked registry and thus no longer create 25k+ files, which should significantly reduce the value of this functionality.

[notinaboat]

Thanks @GunnarFarneback. The Julia 1.7 change does sound like it will address my current needs.
I can imagine a situation where I would still want the ability to create a registry that includes all the dependencies. But the scope is a little different...
If I was deploying a finished system to a customer I would want to ensure that they could easily rebuild everything 5, 10, 15 years later. I would want:

• A local clone of the main julia compiler repo and all its dependencies.
• A local clone of the repos for all the project dependencies.
• A local registry pointing to the local clones of dependencies.
• A script that can build an old version of the compiler and the project to allow bugs to be fixed long after the original repos have disappeared from the internet.
  ... but that is beyond the scope of this issue.

[musm]

@notinaboat did you find a solution to your problem? I'm looking at a similar system and was wondering if you ever managed to get something working.