Docker build times (and errors)
Closed this issue · 8 comments
Hi there
JSONObject version 0.9.4
First of all thanks for this package. I really prefer having my custom DTOs over dicts and magic strings, and JSONObject was a godsend :)
I just created a small hello world application using JSONObject with a Docker file, but I'm running into issues with the deployment. The app contains the usual suspects (Flask, SQL Alchemy, Marshmallow, ...), and the build goes quickly until it hits JSONObject which takes ages in comparison to build.
Also, at the end, the build fails with the following message:
reading manifest file 'jsonobject.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'jsonobject.egg-info/SOURCES.txt'
Copying jsonobject.egg-info to build/bdist.linux-x86_64/wheel/jsonobject-0.9.4-py3.7.egg-info
running install_scripts
error: [Errno 2] No such file or directory: 'LICENSE'
----------------------------------------
Failed building wheel for jsonobject
Any idea how to work around this?
I can reproduce that it takes two and a half minutes to install using python3:
$ python3 -m venv py3env && py3env/bin/pip3 install --upgrade pip && time py3env/bin/pip3 install jsonobject
Collecting pip
Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-18.1
Collecting jsonobject
Using cached https://files.pythonhosted.org/packages/fa/14/16c90b1d4cd7135095d3ea4c3995881e22e48eb6b33904b6500b3de0aeb1/jsonobject-0.9.4.tar.gz
Collecting six (from jsonobject)
Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, jsonobject
Running setup.py install for jsonobject ... done
Successfully installed jsonobject-0.9.4 six-1.11.0
real 2m26.957s
user 2m10.004s
sys 0m6.519s
I haven't yet been able to reproduce the error locally.
Can you tell me more about what's being run and how it's being installed? If you're able to generate a minimal command like the one I have above and also use time to show how long it takes, that would be very helpful for further investigation.
Glad to hear you've enjoyed working with jsonobject so far!
I can reproduce the error: [Errno 2] No such file or directory: 'LICENSE' but the library still installs successfully:
➜ /tmp virtualenv _venv -p python3.6
Running virtualenv with interpreter /Users/vadan/anaconda3/bin/python3.6
Using base prefix '/Users/vadan/anaconda3'
New python executable in /private/tmp/_venv/bin/python3.6
Also creating executable in /private/tmp/_venv/bin/python
Installing setuptools, pip, wheel...done.
➜ /tmp source _venv/bin/activate
(_venv) ➜ /tmp pip install jsonobject
Looking in indexes: https://testpypi.python.org/simple, https://pypi.python.org/simple
Collecting jsonobject
Downloading https://files.pythonhosted.org/packages/fa/14/16c90b1d4cd7135095d3ea4c3995881e22e48eb6b33904b6500b3de0aeb1/jsonobject-0.9.4.tar.gz (371kB)
100% |████████████████████████████████| 378kB 6.5MB/s
Collecting six (from jsonobject)
Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: jsonobject
Running setup.py bdist_wheel for jsonobject ... error
Complete output from command /private/tmp/_venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/fv/ygdmdnss4y7cqcyv2_v7k2tc0000gp/T/pip-install-ffl3vlgu/jsonobject/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/fv/ygdmdnss4y7cqcyv2_v7k2tc0000gp/T/pip-wheel-znrsyvz1 --python-tag cp36:
You are running without Cython installed. It is highly recommended to run
pip install cython==0.27.3
before you continue
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.6
creating build/lib.macosx-10.7-x86_64-3.6/jsonobject
copying jsonobject/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/jsonobject
copying jsonobject/exceptions.py -> build/lib.macosx-10.7-x86_64-3.6/jsonobject
running build_ext
building 'jsonobject.api' extension
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/jsonobject
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/api.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/api.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/api.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/api.cpython-36m-darwin.so
building 'jsonobject.base' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/base.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/base.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/base.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/base.cpython-36m-darwin.so
building 'jsonobject.base_properties' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/base_properties.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/base_properties.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/base_properties.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/base_properties.cpython-36m-darwin.so
building 'jsonobject.containers' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/containers.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/containers.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/containers.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/containers.cpython-36m-darwin.so
building 'jsonobject.properties' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/properties.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/properties.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/properties.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/properties.cpython-36m-darwin.so
building 'jsonobject.utils' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include -arch x86_64 -I/Users/vadan/anaconda3/include/python3.6m -c jsonobject/utils.c -o build/temp.macosx-10.7-x86_64-3.6/jsonobject/utils.o
gcc -bundle -undefined dynamic_lookup -L/Users/vadan/anaconda3/lib -arch x86_64 -L/Users/vadan/anaconda3/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/jsonobject/utils.o -o build/lib.macosx-10.7-x86_64-3.6/jsonobject/utils.cpython-36m-darwin.so
installing to build/bdist.macosx-10.7-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.7-x86_64
creating build/bdist.macosx-10.7-x86_64/wheel
creating build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/api.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/containers.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/__init__.py -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/base.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/base_properties.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/exceptions.py -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/properties.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
copying build/lib.macosx-10.7-x86_64-3.6/jsonobject/utils.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/wheel/jsonobject
running install_egg_info
running egg_info
writing jsonobject.egg-info/PKG-INFO
writing dependency_links to jsonobject.egg-info/dependency_links.txt
writing requirements to jsonobject.egg-info/requires.txt
writing top-level names to jsonobject.egg-info/top_level.txt
reading manifest file 'jsonobject.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'jsonobject.egg-info/SOURCES.txt'
Copying jsonobject.egg-info to build/bdist.macosx-10.7-x86_64/wheel/jsonobject-0.9.4-py3.6.egg-info
running install_scripts
error: [Errno 2] No such file or directory: 'LICENSE'
----------------------------------------
Failed building wheel for jsonobject
Running setup.py clean for jsonobject
Failed to build jsonobject
Installing collected packages: six, jsonobject
Running setup.py install for jsonobject ... done
Successfully installed jsonobject-0.9.4 six-1.11.0
(_venv) ➜ /tmp pip list
Package Version
---------- -------
jsonobject 0.9.4
pip 18.1
setuptools 40.4.3
six 1.11.0
wheel 0.32.1
The install took less than 30secs to complete. Notice the message You are running without Cython installed. It is highly recommended to run pip install cython==0.27.3
I believe the fix is to simply add the LICENSE file name to the MANIFEST.in file. #153
@phsumi I've released @victoradan's changes as version 0.9.5. Could you try again and let me know how that affects the issues you were seeing?
My guess is that as @victoradan pointed out the extra time comes from the presence of Cython, which triggers setup to recompile the pyx files. While compilation can differ between versions and platforms and recompiling can sometimes be necessary, you can often get away with not compiling (by not having Cython present), and if this works for you it may also reduce install time.
While compilation can differ between versions and platforms and recompiling can sometimes be necessary, you can often get away with not compiling (by not having Cython present), and if this works for you it may also reduce install time.
I believe that the translation to C with Cython should only be different if you are using a different Cython version or python2 vs python3 to run Cython. It may be possible to conditionally skip running the code translation step as we provide some default C files.We'd be happy to accept a PR that implements that.
IIRC the compilation (C -> object code) is required on every install as there isn't an ability to include object code in wheels for linux. I may be incorrect about that, or something may have changed. If you can outline a process where we can provide the final compiled form on pypi, we'd be happy to change our processes to provide that
Hi all
Thanks for all the super helpful feedback. You guys rock :)
- Upgrading to 0.9.5 fixed the issue, the error is gone
- Building the Docker image (based on python:3.6.6-alpine3.7) without Cython worked, and build times are much better. I have a déja-vu that I ran into issues without Cython on my Win machine, but for now, I'll just not worry about that. JsonObject is still by far the slowest component though, not sure what can be done there. But we can live with it :)
Thanks again!
Philipp