Automate the NuGet package build
Closed this issue · 16 comments
The libsodium NuGet package is currently created from several downloads and binaries built by local Docker containers. That's much better than the solution before, but not as frictionless as it could be. It would be great if package building could be fully automated.
I've experimented a bit and converted the packaging scripts to run on CircleCI. There's now a repository at https://github.com/ektrah/libsodium-build that fully automatedly downloads the libsodium sources and Windows binaries from libsodium.org, compiles binaries for Linux and macOS, and generates a NuGet package ready for upload to nuget.org. It also runs some tests on the NuGet package to make sure it works.
I've also made changes to add support for Alpine Linux (as requested here) and CentOS 6 (as requested here).
What do you think about (a) replacing the current packaging process with that repository and (b) publishing an updated libsodium 1.0.16.1 NuGet package to make the changes available?
This is really cool!
But now that Travis supports Windows, isn't it something we could do on Travis? That would be more convenient than using both platforms.
How are you creating the Windows binaries so far? If that isn't automated yet, then moving this to Travis or AppVeyor might be a good idea. My repository still depends a lot on Docker, though, so I don't know if that runs on Travis. It's currently using 10+ different Docker images for building and testing.
I manually test on a real Windows system, and eventually use the builds/msvc/build/buildall.bat script to produce the binaries.
Which is ok. I trust this process more than CIs for releases.
But if the Nuget packages require way more work, delegating it to a CI may be a big timer saver.
Maintaining and troubleshooting the packaging scripts so they run everywhere as intended requires a lot more work than just getting them to run on a CI, so I think - for the NuGet packages - building those on a CI would be a big win. I could try to get them running on Travis sometime if necessary, but if building the Windows binaries manually is OK then I would go with CircleCI now since everything is fully working there right now.
The release cycle is pretty slow, so building these packages manually is fine. But that can be very useful for libhydrogen.
Does nuget make sense now that vcpkg is available?
@OlafvdSpek The NuGet package is intended for .NET Core wrappers of libsodium, not C++ users.
@jedisct1 Then please go ahead and publish a libsodium-1.0.16.1 NuGet package with my two changes.
@jedisct1 By the way, do you see any problem if libsodium is compiled to use memcpy@GLIBC_2.2.5 instead of memcpy@GLIBC_2.14?
It shouldn't make a difference, but glibc 2.2.5 was released 17 years ago, and glibc 2.14 8 years ago.
Why use so incredibly old versions?
.NET Core 2.2 is supported on RHEL 6, which apparently comes with glibc 2.12. The latest version of memcpy available in glibc 2.12 is memcpy@GLIBC_2.2.5. (The latest version of memcpy available in current glibc is memcpy@GLIBC_2.14.)
This also fixes #769, even though .NET Core 2.2 is not supported on CentOS 6.
RHEL 6 was released 9 years ago. Damn, that's old.
Anyway, you are the .NET expert, do what you think is the right thing to do!
What's a good way forward?
The requirements of NuGet.org for package metadata have changed since the packaging script were written, and it would be really good to have pre-built binaries included for Alpine Linux. I'm happy to update the scripts, but maintaining the packaging environment locally is really tedious.
Hi,
I really appreciate the work you are doing on this, especially since I don't know much about building and using NuGet packages.
We can definitely switch to automated builds, but I have some really basic questions:
- Your scripts use CircleCI. CircleCI is great, but MacOS support is only only available in paid plans. Could Travis be used for this? Or is there another way the MacOS binaries could be built or retrieved?
- What determines the set of Linux distributions? Does the current set mean that the NuGet packages cannot be used on, e.g. Arch Linux ?
- Where are the Windows binaries downloaded or built from? I don't see any reference to Windows in the CircleCI script.
- Every time a change is made to the
stablebranch, and it affects the generated code, what am I supposed to do so that new NuGet packages are built? - Prior to that, how can I check that a forthcoming change to
stablewill not break any of the environments the NuGet packages are built on? How can I do the same thing before a release? - Do these packages still need to be manually uploaded?
Thanks for all your work on libsodium!
Your questions seem to be hinting at something larger than just automating the NuGet package.
Ideally, on every commit to master or stable, a single CI would compile the code on Linux, macOS, and Windows, run the tests, package the .tar.gz, .zip and .nupkg file, and check that the .nupkg file runs on all supported platforms and distributions. Making a release would then just be a matter of uploading the files for a selected commit to libsodium.org and nuget.org.
The approach so far for packaging the .nupkg file was to disturb the libsodium release process as little as possible: After every release of a new libsodium version, the released .tar.gz and .zip files are downloaded from libsodium.org and used to create the .nupkg file that can be uploaded to nuget.org. This is, of course, not ideal, because if checking the .nupkg file reveals a problem, the new libsodium version is already released.
The first approach should be doable, if you want. That is, automating all of the libsodium release process, not just the NuGet package. It's a bit challenging to find a single CI that does this, because of the huge variety of platforms and distributions involved. It might be possible to use Azure Pipelines, which seems to offer unlimited build minutes for open source on Linux, macOS and Windows. But I have no experience with that.
What I've got running now is the post-hoc approach using CircleCI. I'll answer your questions based on this. If you're interested in the first approach, I could try to help you figure that out (if I can find time for it).
Your scripts use CircleCI. CircleCI is great, but MacOS support is only only available in paid plans. Could Travis be used for this? Or is there another way the MacOS binaries could be built or retrieved?
As said above, finding a single CI that does everything is a bit challenging. As you remember, we started some years ago with building the Linux binaries on openSUSE build service, then switched to local Docker containers, and can now build the macOS and Linux binaries on CircleCI. So things are improving over time.
CircleCI offers a free macOS plan to open source projects on request.
When I used Travis the last time, it ran on something like Ubuntu 12.04 and the macOS builds took forever to start. I've switched all my projects over to CircleCI since then and never looked back.
Azure Pipelines might be an option.
What determines the set of Linux distributions? Does the current set mean that the NuGet packages cannot be used on, e.g. Arch Linux ?
Since the libsodium NuGet package is intended to be used with .NET Core, it should ideally run (and be checked to run) on every platform supported by .NET Core. You can find the list of supported platforms and distributions for current versions of .NET Core here: 2.1, 2.2, 3.0-preview.
Microsoft updates these lists based on what they offer support for. If a Linux distribution is not listed, it doesn't mean that .NET Core doesn't run on it. Just that it hasn't been tested and might not work.
My approach for NSec is the same: I'm testing it on as many distributions supported by .NET Core as possible and list those in the documentation. It very likely runs on other distributions as well, but since I don't test on those, I don't know until someone tries it.
In general, making the same pre-compiled libsodium binaries work on a large range of Linux distributions is remarkable simple. I think the only dependency is the C standard library, so that libsodium needs to be compiled only for glibc and musl libc to cover all Linux distributions currently supported by .NET Core. (The only caveat being that some of these distributions have a really old glibc.)
Where are the Windows binaries downloaded or built from? I don't see any reference to Windows in the CircleCI script.
The Windows binaries are downloaded from libsodium.org. It's a bit hidden in the prepare.py file, which generates a Makefile that does the download.
Every time a change is made to the stable branch, and it affects the generated code, what am I supposed to do so that new NuGet packages are built?
Right now, the CircleCI script downloads specific .tar.gz and .zip files from libsodium.org and builds the NuGet package entirely from that. When new files are released, the file names need to be updated in the script repository. This triggers CircleCI to build a new NuGet package.
Prior to that, how can I check that a forthcoming change to stable will not break any of the environments the NuGet packages are built on? How can I do the same thing before a release?
With the post-hoc approach, you cannot really.
Do these packages still need to be manually uploaded?
Right now, yes. The CircleCI script builds the NuGet package and stores it on CircleCI. From there, it can be manually downloaded and uploaded to nuget.org. If you want, this step could be fully automated as well, though. (Or semi-automated. CircleCI supports delaying the last step until it's manually approved.)
@ektrah Any chance you could update the nuget package with 1.0.18? Much appreciated!
I've experimented a bit with Azure Pipelines but can't seem to get an Alpine container to run...