make all failed with latest release (2023-02-06) (with "shapely")

Question

make all failed with latest release (2023-02-06) (with "shapely")

Closed this issue 2 years ago · 17 comments

Hi,
I get the following error, when I try to build ocrd_all in a new venv:

Successfully built ocrd sparklines future
Installing collected packages: wrapt, python-magic, psutil, opencv-python-headless, MarkupSafe, itsdangerous, future, Werkzeug, sparklines, memory-profiler, Jinja2, Deprecated, Flask, ocrd
Successfully installed Deprecated-1.2.0 Flask-2.2.3 Jinja2-3.1.2 MarkupSafe-2.1.2 Werkzeug-2.2.3 future-0.18.3 itsdangerous-2.1.2 memory-profiler-0.61.0 ocrd-2.45.1 opencv-python-headless-4.7.0.72 psutil-5.9.4 python-magic-0.4.27 sparklines-0.4.2 wrapt-1.15.0
sem -q --will-cite --fg --id ocrd_all_pipocrd-3.7 pip install --no-binary shapely --force-reinstall shapely
Collecting shapely
  Downloading shapely-2.0.1.tar.gz (275 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 275.5/275.5 kB 5.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting numpy>=1.14
  Using cached numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Building wheels for collected packages: shapely
  Building wheel for shapely (pyproject.toml): started
  Building wheel for shapely (pyproject.toml): finished with status 'error'
Failed to build shapely
make[1]: Leaving directory '/home/ocrdadmin/ocrd_all/core'

--> please clarify ...

Answer 1 · 2023-03-15T12:04:14.000Z

I started make all now in CI to see whether that issue affects different Python / Ubuntu versions, see log output.

Answer 2 · 2023-03-15T12:36:46.000Z

Good idea: I forgot to mentioned, that I use Python 3.7 (as I was in the impression, that this is the highest version, which I can use ...).
Of course, I am willing to use 3.8 (default for Ubuntu 20) or 3.10 (default for Ubuntu 22).
--> just tell me, what to do ...

Answer 3 · 2023-03-15T13:04:39.000Z

Just another hint: I do NOT have a GPU.

Answer 4 · 2023-03-15T13:06:58.000Z

I started make all now in CI to see whether that issue affects different Python / Ubuntu versions, see log output.

Looking at Ubuntu 22 with Python 3.7 (which I my case). I can see a different error there:
tensorflow-gpu 1.15.5 has requirement numpy<1.19.0,>=1.16.0, but you have numpy 1.21.6.
BUT, as I do not have a GPU, maybe this is even an additional error.

Answer 5 · 2023-03-15T13:09:05.000Z

CI fails with Python 3.7 in the final check, but the build passes. So that's not the issue which was reported above.

Answer 6 · 2023-03-15T13:13:22.000Z

@stefanCCS, please try pip install shapely==2.0.1 in your Python venv. That's the command which fails, and you have to find the reason for that failure.

Answer 7 · 2023-03-15T13:39:55.000Z

(ocrd-3.7) ocrdadmin@ocrd-02:~/ocrd_all$ pip install shapely==2.0.1
Requirement already satisfied: shapely==2.0.1 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (2.0.1)
Requirement already satisfied: numpy>=1.14 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (from shapely==2.0.1) (1.21.6)

Answer 8 · 2023-03-15T13:41:45.000Z

Here is more information from the original error message (I forgot to redirect stderr to my logfile):

Collecting numpy>=1.14
  Using cached numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Building wheels for collected packages: shapely
  Building wheel for shapely (pyproject.toml): started
  Building wheel for shapely (pyproject.toml): finished with status 'error'
Failed to build shapely
make[1]: *** [Makefile:74: install] Error 1
make[1]: Leaving directory '/home/ocrdadmin/ocrd_all/core'
make: *** [Makefile:198: /home/ocrdadmin/ocrd-3.7/bin/ocrd] Error 2
make: *** Deleting file '/home/ocrdadmin/ocrd-3.7/bin/ocrd'

Answer 9 · 2023-03-15T13:46:00.000Z

Could you please run pip with --verbose and show what's failing exactly? (Could be enough to just pass PIP_OPTIONS=--verbose ...)

Answer 10 · 2023-03-15T13:48:59.000Z

(ocrd-3.7) ocrdadmin@ocrd-02:~/ocrd_all$ pip --verbose install shapely==2.0.1
Using pip 23.0.1 from /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages/pip (python 3.7)
Requirement already satisfied: shapely==2.0.1 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (2.0.1)
Requirement already satisfied: numpy>=1.14 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (from shapely==2.0.1) (1.21.6)

Answer 11 · 2023-03-15T13:55:00.000Z

Then uninstall shapely before running the installation (or use a new venv).

Answer 12 · 2023-03-15T14:04:19.000Z

Then uninstall shapely before running the installation (or use a new venv).

I have started the whole process within a new venv.

Typically I perform an udpate like this:

Answer 13 · 2023-03-15T15:48:04.000Z

Now, I have created a new venv with Python 3.10, and I get this:

      copying shapely/_geos.pxd -> build/lib.linux-x86_64-cpython-310/shapely
      UPDATING build/lib.linux-x86_64-cpython-310/shapely/_version.py
      set build/lib.linux-x86_64-cpython-310/shapely/_version.py to '2.0.1'
      running build_ext
      building 'shapely.lib' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/src
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/ocrdadmin/ocrd-3.10/include -I/usr/include/python3.10 -I/tmp/pip-build-env-w3bax7o2/overlay/lib/python3.10/site-packages/numpy/core/include -c src/c_api.c -o build/temp.linux-x86_64-cpython-310/src/c_api.o
      In file included from src/c_api.h:22,
                       from src/c_api.c:15:
      src/geos.h:15:10: fatal error: geos_c.h: No such file or directory
         15 | #include <geos_c.h>
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for shapely
ERROR: Could not build wheels for shapely, which is required to install pyproject.toml-based projects
make[1]: *** [Makefile:74: install] Error 1
make[1]: Leaving directory '/home/ocrdadmin/ocrd_all/core'
make: *** [Makefile:198: /home/ocrdadmin/ocrd-3.10/bin/ocrd] Error 2
make: *** Deleting file '/home/ocrdadmin/ocrd-3.10/bin/ocrd'

Answer 14 · 2023-03-15T16:13:52.000Z

Thanks @stefanCCS – that helps.

Strange though. This comes from libgeos-dev , but we do have that in core's deps-ubuntu:
https://github.com/OCR-D/core/blob/432d852bfd6be7b171042422ebbb451fc0f64c72/Makefile#L62

Answer 15 · 2023-03-15T16:21:13.000Z

Er, you did run sudo make deps-ubuntu (after having recursively checked out everything) though?

Answer 16 · 2023-03-15T17:11:06.000Z

@stefanCCS, as @bertsky has explained it is necessary to run sudo make deps-ubuntu or install any required dependency manually (which is very challenging). Otherwise such build errors like the one caused by the missing geos_c.h which is part of package libgeos-dev are more or less normal.

Answer 17 · 2023-03-16T14:42:35.000Z

Oops, sorry - totally forgot about this (sudo make deps-ubuntu).
(and I also forgot to put this on my notes, which I will do right now)

Made a rebuild for both Python 3.7 and 3.10 and both have worked fine (at least ocrd --version work fine).
So, I will close this issue now - again, many thanks for your support.