Precompiled binaries for easier installation?

Question

Precompiled binaries for easier installation?

ssolari opened this issue 8 years ago · 12 comments

Our specific use case is that we are using lz4framed inside Docker containers. The issue is that base linux in docker used for python often doesn't have the needed gcc components. We use the miniconda3 docker base image and of the dozens of python packages we use (many of which require compilation), lz4framed is the only one that needs to have gcc installed into the container.

We can obviously apt-get install gcc, but having lz4framed packaged up into binaries would be a great benefit to its users like us. I don't know what the compile dependencies are and if they impact wheels for pip, but if so conda can be used to build binaries so this isn't an issue and the package just installs anywhere easily.

Answer 1 · 2017-02-13T17:14:45.000Z

Unfortunately we're a little low on resources (time) at the moment, so I'm not in a position to offer a quick solution right now in terms of binaries per-release - sorry.

Are you thinking of binaries attached to PyPI or GitHub releases? (I'm aware Travis CI could automate this, at least for Linux binaries.) Does the Miniconda project itself supply the binaries for the packages which you say require compilation (but don't need it within docker?) or do all of them happen to have binary wheels already? Is it that the others have a pure-Python equivalent so can skip compilation maybe?
(One thing against having binary wheels supplied by the module itself is that we'd have to provide binaries for at least 2.7, 3.4, 3.5 & 3.6 and possibly in 32 & 64 bit flavours.)

As for dependencies, other than Python development headers, gcc & C development headers, it should have no other dependencies (e.g. python-dev or python3-dev packages should be enough apart from gcc itself). If another C-extension for Python builds successfully, so should this. (Feel free to post build failure output if that is not the case.)

Answer 2 · 2017-02-13T17:40:27.000Z

@ssolari: I notice there seems to be an (official?) mirror of PyPI which (Ana)conda might be able to use (although I cannot see py-lz4framed in the list). Maybe it's worth contacting the Anaconda team about how they update their PyPI "mirror" list.

Answer 3 · 2017-02-13T18:31:53.000Z

The resource issue is always a challenge. I am far from an expert, but I think the easiest way to go would be to create a conda recipe and host it on conda-forge. They are automatically integrating with Travis to make life easier and even compile binaries to all 3 major platforms (osx, linux, windows), which means the package is more widely available. https://conda-forge.github.io/#contribute.

This whole process can be fully automated and there are lots of examples of existing python packages that could be used to get something up and running fairly quickly. This should solve the 'maintaining multiple' problem. You would just need to maintain a single recipe that gets built across platforms etc, which is maybe manageable on low resources and also means that if you release a new version it could be packaged and available almost immediately.

Answer 4 · 2017-02-13T18:36:31.000Z

I was just looking at the example recipe and basically the single recipe will automatically build deploy against python 2.7, 3.4 and above for all three OS systems.

https://github.com/conda-forge/staged-recipes/blob/master/recipes/example/meta.yaml

Answer 5 · 2017-02-13T23:38:47.000Z

Am waiting to hear back how to make it run unit tests (which are currently not being found):
conda-forge/staged-recipes#2435

(Might not have time to finish this up until next week.)

Answer 6 · 2017-02-14T11:54:03.000Z

I'm sorry but I don't have any time to figure out why VS (Windows) builds don't like the source code and for adding VS-specific defines (it also does not seem to recognise C99 standard flag).
As for the Travis-CI builds - I think there's an unrelated problem building another recipe (doc8).
The docker build works but I don't know how to make it run the unit tests.

So even if I skipped the Windows builds it'd still not work fully right now. You're welcome to try to make it work but maybe it's easier to build a local binary distribution yourself and then use it within the docker image to avoid needing gcc etc. (Am not an expert on that but presumably that's a pretty standard thing.)

Answer 7 · 2017-02-14T16:04:27.000Z

Thanks for the effort. I think dropping Windows would be fine as the other lz4 package is linux/Mac only in Anaconda too, but I get that these things take time to figure out and debug.

As far as installation and docker etc. The goal was to simply be able to run:
conda install -c conda-forge py-lz4framed (on Mac and Linux only)

As I said. We do have a workaround by simply installing all needed gcc components into the docker image before we install py-lz4framed, but it adds 150MB to the image and takes a lot more time in the build.
Our use case is that we are creating microservices that need to be interoperable and py-lz4framed is needed to communicate with javascript. A lot of people doing microservices these days are moving to Docker containers and passing compressed lz4 data has benefits. I just think that making it easier to get py-lz4framed installed will increase its use and adoption, which I think is a good thing. I completely understand time constraints to do this though and thanks for the very quick effort!

Answer 8 · 2017-02-14T18:13:38.000Z

I'll leave this issue open for when I have a bit more time to investigate this (or someone else beats me to it).

Answer 9 · 2017-02-15T08:42:01.000Z

Thought I'd have one last stab at it - excluded Windows build & made tests work. It's up to the conda-forge team now in terms of how long this will take:
conda-forge/staged-recipes#2442

Answer 10 · 2017-02-15T09:05:56.000Z

Potential fix for WIndows bulids in future - prevent VERSION define being interpreted without quotes by:

Specifying version without quotes in setup.py:
'-DVERSION=%s' % VERSION
Quote via expansion in py-lz4framed.c:

#define QUOTE(str) #str
#define EXPAND_AND_QUOTE(str) QUOTE(str)

// ...

PyModule_AddStringConstant(module, "__version__", EXPAND_AND_QUOTE(VERSION))

Answer 11 · 2017-02-21T08:40:56.000Z

@ssolari - the recipe should now be available. Would you mind checking whether it works in your target setup?

Answer 12 · 2017-02-22T00:09:22.000Z

Works absolutely perfectly. Easy as

conda install -c conda-forge py-lz4framed

We actually use the conda env command with an environment.yml file in our Docker setup, but now py-lz4framed installs no problem and we don't have to install the gcc within the Docker image so that saves a good amount of space and build time.

thanks for making this happen. This issue is closed from my standpoint.