Compilation error when installing python requirements, succeeds without explicit csvkit version

Question

Compilation error when installing python requirements, succeeds without explicit csvkit version

gabegm opened this issue 4 years ago · 0 comments

gabegm commented 4 years ago

Context

OS: Ubuntu 16.04 x64

Steps to Reproduce

Follow the steps from the official docs here untill the python_requirements.py step
Run $ sudo pip install --no-use-wheel -r python_requirements.txt, dependency building should fail
Install each dependency individually, i.e. $ sudo pip install LIBRARY==VERSION
$ sudo pip install csvkit==0.9.0 should fail
Remove the version from the csvkit requirement, i.e. csvkit
Rerun $ sudo pip install --no-use-wheel -r python_requirements.txt, this should now build successfully

Current Result

python_requirements.txt failure

$ sudo pip install --no-use-wheel -r python_requirements.txt
DEPRECATION: --no-use-wheel is deprecated and will be removed in the future.  Please use --no-binary :all: instead.
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): chardet==2.3.0 in /usr/local/lib/python2.7/dist-packages (from -r python_requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): argparse==1.2.1 in /usr/lib/python2.7 (from -r python_requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): brewery==0.6 in /usr/local/lib/python2.7/dist-packages (from -r python_requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): redis==2.4.9 in /usr/local/lib/python2.7/dist-packages (from -r python_requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): python-varnish==0.1.2 in /usr/local/lib/python2.7/dist-packages (from -r python_requirements.txt (line 5))
Collecting gdal==2.2.2 (from -r python_requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/11/cd/edef955a6ef6cb4f2113647d7465b34663e074451b66919e734466ba5197/GDAL-2.2.2.tar.gz (475kB)
    100% |████████████████████████████████| 481kB 2.5MB/s 
Collecting csvkit==0.9.0 (from -r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/37/39/e8a2bd3fe4c132fb0a09f7e8382341c4aeda2144506518465eccb4ffa655/csvkit-0.9.0.tar.gz
Collecting openpyxl==2.1.3 (from -r python_requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/86/7a/bba821c554b5fbde20aaa70cf5ae8c284883f38162deee7852e068db6246/openpyxl-2.1.3.tar.gz (96kB)
    100% |████████████████████████████████| 102kB 10.7MB/s 
Collecting xlrd>=0.7.1 (from csvkit==0.9.0->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/a6/b3/19a2540d21dea5f908304375bd43f5ed7a4c28a370dc9122c565423e6b44/xlrd-2.0.1.tar.gz (100kB)
    100% |████████████████████████████████| 102kB 10.7MB/s 
Collecting sqlalchemy>=0.6.6 (from csvkit==0.9.0->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/29/25/1fbcbf47928731ff46cbf3eaaaacbdcc042dd85f919580055f30a0dbdb0a/SQLAlchemy-1.4.9.tar.gz (7.5MB)
    11% |███▊                            | 880kB 45.4MB/s eta 0:00:01debug2: channel 0: window 998845 sent adjust 49731
    76% |████████████████████████▋       | 5.8MB 43.1MB/s eta 0:00:01debug2: channel 0: window 999409 sent adjust 49167
    100% |████████████████████████████████| 7.5MB 168kB/s 
  Running setup.py (path:/tmp/pip-build-56Rm7O/sqlalchemy/setup.py) egg_info for package sqlalchemy produced metadata for project name unknown. Fix your #egg=sqlalchemy fragments.
Collecting six>=1.6.1 (from csvkit==0.9.0->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9ba89a21c9743a0c59fbd0205c7f9426/six-1.15.0.tar.gz
Collecting python-dateutil==2.2 (from csvkit==0.9.0->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/75/c5/85d027471fa665f8c8b8eb0b925f9d84b4eee745a257b16de4957de99e81/python-dateutil-2.2.tar.gz (259kB)
    100% |████████████████████████████████| 266kB 4.7MB/s 
Collecting dbf==0.94.003 (from csvkit==0.9.0->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/7c/ef/79873517838910e8f1bd5610ab3de380a3886675103072727266eec91bbc/dbf-0.94.003.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 10.5MB/s 
Collecting jdcal (from openpyxl==2.1.3->-r python_requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/7b/b0/fa20fce23e9c3b55b640e629cb5edf32a85e6af3cf7af599940eb0c753fe/jdcal-1.4.1.tar.gz
Installing collected packages: gdal, xlrd, unknown, jdcal, openpyxl, six, python-dateutil, dbf, csvkit
  Running setup.py install for gdal ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-56Rm7O/gdal/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-KGJm8q-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying gdal.py -> build/lib.linux-x86_64-2.7
    copying ogr.py -> build/lib.linux-x86_64-2.7
    copying osr.py -> build/lib.linux-x86_64-2.7
    copying gdalconst.py -> build/lib.linux-x86_64-2.7
    copying gnm.py -> build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/gdal.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/osr.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/__init__.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/gnm.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/gdalconst.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/gdal_array.py -> build/lib.linux-x86_64-2.7/osgeo
    copying osgeo/ogr.py -> build/lib.linux-x86_64-2.7/osgeo
    running build_ext
    building 'osgeo._gdal' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/extensions
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/usr/include/python2.7 -I. -I/usr/include -c extensions/gdal_wrap.cpp -o build/temp.linux-x86_64-2.7/extensions/gdal_wrap.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    extensions/gdal_wrap.cpp:3168:22: fatal error: cpl_port.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-56Rm7O/gdal/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-KGJm8q-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-56Rm7O/gdal/

csvkit failure

$ pip install csvkit==0.9.0
Collecting csvkit==0.9.0
  Downloading https://files.pythonhosted.org/packages/37/39/e8a2bd3fe4c132fb0a09f7e8382341c4aeda2144506518465eccb4ffa655/csvkit-0.9.0.tar.gz
Collecting xlrd>=0.7.1 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/a6/0c/c2a72d51fe56e08a08acc85d13013558a2d793028ae7385448a6ccdfae64/xlrd-2.0.1-py2.py3-none-any.whl (96kB)
    100% |████████████████████████████████| 102kB 8.8MB/s 
Collecting sqlalchemy>=0.6.6 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/e0/24/75f1f1464457e9c01b138145636997020133566282df710a41a3cfdcec76/SQLAlchemy-1.4.9-cp27-cp27mu-manylinux1_x86_64.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 840kB/s 
Collecting openpyxl>=2.0.3 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/39/08/595298c9b7ced75e7d23be3e7596459980d63bc35112ca765ceccafbe9a4/openpyxl-3.0.7-py2.py3-none-any.whl (243kB)
    100% |████████████████████████████████| 245kB 5.3MB/s 
Collecting six>=1.6.1 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting python-dateutil==2.2 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/75/c5/85d027471fa665f8c8b8eb0b925f9d84b4eee745a257b16de4957de99e81/python-dateutil-2.2.tar.gz (259kB)
    100% |████████████████████████████████| 266kB 4.8MB/s 
Collecting dbf==0.94.003 (from csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/7c/ef/79873517838910e8f1bd5610ab3de380a3886675103072727266eec91bbc/dbf-0.94.003.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 10.3MB/s 
Collecting importlib-metadata; python_version < "3.8" (from sqlalchemy>=0.6.6->csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/9a/5a/5bee793e43f6499b7e28de7c0333f49752590826b08a456f3874f555f4f5/importlib_metadata-4.0.0.tar.gz (41kB)
    100% |████████████████████████████████| 51kB 11.5MB/s 
  Running setup.py (path:/tmp/pip-build-Co1hk3/importlib-metadata/setup.py) egg_info for package importlib-metadata produced metadata for project name unknown. Fix your #egg=importlib-metadata fragments.
Collecting et-xmlfile (from openpyxl>=2.0.3->csvkit==0.9.0)
  Downloading https://files.pythonhosted.org/packages/22/28/a99c42aea746e18382ad9fb36f64c1c1f04216f41797f2f0fa567da11388/et_xmlfile-1.0.1.tar.gz
Building wheels for collected packages: csvkit, python-dateutil, dbf, unknown, et-xmlfile, unknown
  Running setup.py bdist_wheel for csvkit ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/b1/e6/5e/12a42a3bc016cf19674eec242ff08479bcd84377270bb7f43d
  Running setup.py bdist_wheel for python-dateutil ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/97/40/49/e02356db91efa7a5da28ec2e5c61f0fb2c0aa21bbd5da95805
  Running setup.py bdist_wheel for dbf ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/fa/fb/ec/4da45ba6853875f5adc64cb3cc0fccc799f739ed4e7584a082
  Running setup.py bdist_wheel for unknown ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/d4/24/c8/3783bc6cc33cace61d33e1b6a76659800cdf3d2bf2bc7a1c42
  Running setup.py bdist_wheel for et-xmlfile ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/2a/77/35/0da0965a057698121fc7d8c5a7a9955cdbfb3cc4e2423cad39
  Running setup.py bdist_wheel for unknown ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-Co1hk3/unknown/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpkSZCFhpip-wheel- --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-build-Co1hk3/unknown/setup.py'
  
  ----------------------------------------
  Failed building wheel for unknown
  Running setup.py clean for unknown
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-Co1hk3/unknown/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-build-Co1hk3/unknown/setup.py'
  
  ----------------------------------------
  Failed cleaning build dir for unknown
Successfully built csvkit python-dateutil dbf unknown et-xmlfile
Failed to build unknown
Installing collected packages: xlrd, unknown, sqlalchemy, et-xmlfile, openpyxl, six, python-dateutil, dbf, csvkit
Successfully installed csvkit-0.9.0 dbf-0.94.3 et-xmlfile-1.0.1 openpyxl-3.0.7 python-dateutil-2.2 six-1.15.0 sqlalchemy-1.4.9 unknown-0.0.0 xlrd-2.0.1
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Expected result

Successful compilation of dependencies

$ sudo pip install --no-use-wheel -r python_requirements.txt
DEPRECATION: --no-use-wheel is deprecated and will be removed in the future.  Please use --no-binary :all: instead.
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): chardet==2.3.0 in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): argparse==1.2.1 in /usr/lib/python2.7 (from -r python_requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): brewery==0.6 in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): redis==2.4.9 in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): python-varnish==0.1.2 in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): gdal==2.2.2 in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): csvkit in /home/ubuntu/.local/lib/python2.7/site-packages (from -r python_requirements.txt (line 7))
Collecting openpyxl==2.1.3 (from -r python_requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/86/7a/bba821c554b5fbde20aaa70cf5ae8c284883f38162deee7852e068db6246/openpyxl-2.1.3.tar.gz (96kB)
    100% |████████████████████████████████| 102kB 7.9MB/s 
Requirement already satisfied (use --upgrade to upgrade): xlrd>=0.7.1 in /home/ubuntu/.local/lib/python2.7/site-packages (from csvkit->-r python_requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil==2.2 in /home/ubuntu/.local/lib/python2.7/site-packages (from csvkit->-r python_requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): six>=1.6.1 in /home/ubuntu/.local/lib/python2.7/site-packages (from csvkit->-r python_requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): sqlalchemy>=0.6.6 in /home/ubuntu/.local/lib/python2.7/site-packages (from csvkit->-r python_requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): dbf==0.94.003 in /home/ubuntu/.local/lib/python2.7/site-packages (from csvkit->-r python_requirements.txt (line 7))
Collecting jdcal (from openpyxl==2.1.3->-r python_requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/7b/b0/fa20fce23e9c3b55b640e629cb5edf32a85e6af3cf7af599940eb0c753fe/jdcal-1.4.1.tar.gz
Collecting importlib-metadata; python_version < "3.8" (from sqlalchemy>=0.6.6->csvkit->-r python_requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/9a/5a/5bee793e43f6499b7e28de7c0333f49752590826b08a456f3874f555f4f5/importlib_metadata-4.0.0.tar.gz (41kB)
    100% |████████████████████████████████| 51kB 11.8MB/s 
  Running setup.py (path:/tmp/pip-build-YG4Uph/importlib-metadata/setup.py) egg_info for package importlib-metadata produced metadata for project name unknown. Fix your #egg=importlib-metadata fragments.
  Requirement already satisfied (use --upgrade to upgrade): unknown from https://files.pythonhosted.org/packages/9a/5a/5bee793e43f6499b7e28de7c0333f49752590826b08a456f3874f555f4f5/importlib_metadata-4.0.0.tar.gz#sha256=2e881981c9748d7282b374b68e759c87745c25427b67ecf0cc67fb6637a1bff9 in /home/ubuntu/.local/lib/python2.7/site-packages (from sqlalchemy>=0.6.6->csvkit->-r python_requirements.txt (line 7))
Installing collected packages: jdcal, openpyxl
  Running setup.py install for jdcal ... done
  Found existing installation: openpyxl 3.0.7
    Uninstalling openpyxl-3.0.7:
      Successfully uninstalled openpyxl-3.0.7
  Running setup.py install for openpyxl ... done
Successfully installed jdcal-1.4.1 openpyxl-2.1.3
You are using pip version 8.1.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.