undefined GDAL symbols in ogr/gdal_GRASS.so

Question

undefined GDAL symbols in ogr/gdal_GRASS.so

okanisis opened this issue 2 years ago · 13 comments

GDAL-GRASS 1.0.0 (from release tarball)
GDAL 3.5.0 (from git release branch)
GRASS 8.2.0 (from release tarball)
QGIS 3.24.3 (from git release branch)

gdalinfo --formats outputs:

ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegister_GRASS
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: RegisterOGRGRASS

QGIS outputs:

ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegister_GRASS
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: RegisterOGRGRASS
ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegister_GRASS
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: RegisterOGRGRASS

GRASS outputs:

ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegister_GRASS
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/gdalplugins/ogr_GRASS.so: undefined symbol: RegisterOGRGRASS

Answer 1 · 2022-06-04T10:15:07.000Z

To make this a reproducible issue, please answer these questions:

which operating system?
how did you configure the GDAL, GRASS GIS, and GDAL-GRASS driver packages?
may there be a leftover /etc/ld.so.conf.d/grass.conf on your system?
what is the output of echo $LD_LIBRARY_PATH?
is the content of /usr/lib/gdalplugins/ on your system related to your recent built?
at least on Fedora, the GDAL plugins directory would be /usr/lib64/gdalplugins, does it also exist and, if yes, what does it contain?

Answer 2 · 2022-06-04T15:47:21.000Z

To make this a reproducible issue, please answer these questions:

which operating system?

Arch Linux

% uname -a
Linux cryptsus 5.18.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 30 May 2022 17:53:11 +0000 x86_64 GNU/Linux

how did you configure the GDAL, GRASS GIS, and GDAL-GRASS driver packages?

GDAL

  cmake -B build -S gdal \
      -DCMAKE_BUILD_TYPE='None' \
      -DCMAKE_INSTALL_PREFIX='/usr' \
      -DBUILD_PYTHON_BINDINGS='ON' \
      -Wno-dev

GRASS GIS

  ./configure \
    --prefix=/opt/grass \
    --with-freetype-includes=/usr/include/freetype2 \
    --with-wxwidgets \
    --with-readline \
    --with-pthread \
    --with-netcdf \
    --with-nls \
    --with-geos \
    --with-postgres \
    --with-bzlib \
    --with-zstd \
    --with-opencl \
    --with-openmp

GDAL-GRASS

  ./configure \
    --prefix=${pkgdir}/usr \
    --with-grass=/opt/grass \
    --with-autoload="${pkgdir}/usr/lib/gdalplugins" \
    --with-postgres-includes=$(pg_config --includedir)

may there be a leftover /etc/ld.so.conf.d/grass.conf on your system?

There's one from the GRASS GIS install.

% cat /etc/ld.so.conf.d/grass.conf 
/opt/grass/lib

what is the output of echo $LD_LIBRARY_PATH?

From the system, it's empty.

% printenv LD_LIBRARY_PATH

is the content of /usr/lib/gdalplugins/ on your system related to your recent built?

It looks like it, the dates match the compile & install time.

% ls -lah /usr/lib/gdalplugins/         
total 356K
drwxr-xr-x   2 root root 4.0K Jun  3 15:03 .
drwxr-xr-x 216 root root 252K Jun  4 08:34 ..
-rw-r--r--   1 root root 2.7K Jun  3 12:39 drivers.ini
-rwxr-xr-x   1 root root  38K Jun  3 12:53 gdal_GRASS.so
-rwxr-xr-x   1 root root  50K Jun  3 12:53 ogr_GRASS.so

at least on Fedora, the GDAL plugins directory would be /usr/lib64/gdalplugins, does it also exist and, if yes, what does it contain?

/usr/lib64 is a symlink to /usr/lib and contains the same structure.

% ls /usr/lib64/gdalplugins -lah
total 356K
drwxr-xr-x   2 root root 4.0K Jun  3 15:03 .
drwxr-xr-x 216 root root 252K Jun  4 08:34 ..
-rw-r--r--   1 root root 2.7K Jun  3 12:39 drivers.ini
-rwxr-xr-x   1 root root  38K Jun  3 12:53 gdal_GRASS.so
-rwxr-xr-x   1 root root  50K Jun  3 12:53 ogr_GRASS.so

Answer 3 · 2022-06-04T16:29:21.000Z

The GDAL-GRASS configure contains ${pkgdir} which is unclear.

And what does this give?

ldd /usr/lib/gdalplugins/gdal_GRASS.so | grep gdal

Answer 4 · 2022-06-04T16:46:17.000Z

ldd /usr/lib/gdalplugins/gdal_GRASS.so | grep gdal

% ldd /usr/lib/gdalplugins/gdal_GRASS.so | grep gdal
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007fe30f400000)

The GDAL-GRASS configure contains ${pkgdir} which is unclear.

Apologies, they are built using makepkg and live inside a PKGBUILD file so there are variables abound in them. Tried to remove for clarity but missed one - should be ok to ignore from your end.

${pkgdir} is a reference to the install directory from inside the chroot before it's packaged up by the install() function inside the PKGBUILD.

Answer 5 · 2022-06-04T17:09:39.000Z

And this points to the same library?

ldd /usr/bin/gdal_translate | grep gdal

Just to avoid that you have GDAL twice on the system.

--

BTW: With Google I found a similar issue you have from 2017, Ubuntu related:
https://trac.osgeo.org/osgeolive/ticket/1933

I scanned through https://salsa.debian.org/debian-gis-team/gdal-grass/-/commits/master and found the related Debian "PIE" fix from 2017: https://salsa.debian.org/debian-gis-team/gdal-grass/-/commit/3ceb43613537f6cfbd8e4b36d388ca30af7bf076

(dropped in 2021, https://salsa.debian.org/debian-gis-team/gdal-grass/-/commit/9c9f5c1e32cc324116619485d5c36fc1ad90e6c9)

Perhaps worth an attempt? (I am just guessing, on my Fedora box I don't have such an issue and the Ubuntu test workflow used in this repo looks also rather simple.)

Answer 6 · 2022-06-04T17:29:05.000Z

And this points to the same library?

ldd /usr/bin/gdal_translate | grep gdal

Just to avoid that you have GDAL twice on the system.

ah, this looks like it

% ldd /usr/lib/gdalplugins/gdal_GRASS.so | grep gdal
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007fdedfa00000)
% ldd /usr/bin/gdal_translate | grep gdal
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007f0024000000)

Maybe it's remnant files left over from the configure system vs cmake in the latest gdal release? Looks like an upgrade of gdal 3.4 (configure) -> 3.5 (cmake) could of caused the library errors.

Wouldn't the gdal_translate executable be upgraded and replaced from the upgrade?

Answer 7 · 2022-06-04T18:55:31.000Z

ok, I did a clean build of gdal, grass then gdal-grass and here's the output of ldd on the freshly built packages:

% ldd /opt/grass/lib/libgrass_vector.so | grep gdal
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007fa2c5400000)

% ldd /usr/lib/gdalplugins/gdal_GRASS.so | grep gdal                         
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007faf9d600000)

% ldd /usr/bin/gdal_translate | grep gdal
        libgdal.so.31 => /usr/lib/libgdal.so.31 (0x00007fd16ba00000)

I never negated the PIE option on this build cycle as still looking into how best to do that with the links you shared.

Answer 8 · 2022-06-05T12:45:46.000Z

-fPIE enables PIE, -no-pie disables it.
-fpic enables PIC, -fno-pic disables it.

dpkg-buildflags uses a spec file for to disable it: -specs=/usr/share/dpkg/no-pie-compile.specs

$ cat /usr/share/dpkg/no-pie-compile.specs 
*self_spec:
+ %{!r:%{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fno-PIE}}}}}}

Answer 9 · 2022-06-13T18:55:48.000Z

FWIW, this also happens with libgdal-grass (1:1.0.0-1) in Debian unstable since the transition to GDAL 3.5.0:

$ gdalinfo --version
ERROR 1: /usr/lib/x86_64-linux-gnu/gdalplugins/ogr_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/x86_64-linux-gnu/gdalplugins/ogr_GRASS.so: undefined symbol: RegisterOGRGRASS
ERROR 1: /usr/lib/x86_64-linux-gnu/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegisterMe
ERROR 1: /usr/lib/x86_64-linux-gnu/gdalplugins/gdal_GRASS.so: undefined symbol: GDALRegister_GRASS
GDAL 3.5.0, released 2022/05/10

GDALRegisterMe is defined in port/cpl_vsil_crypt.cpp, but it's conditional to VSICRYPT_AUTOLOAD being defined which it's not.

RegisterOGRGRASS & GDALRegister_GRASS are defined in ogrgrassdriver.cpp and grass.cpp respectively.

These two should suffice based on the GDALDriverManager::AutoLoadDrivers() comment:

/************************************************************************/
/*                          AutoLoadDrivers()                           */
/************************************************************************/

/**
 * \brief Auto-load GDAL drivers from shared libraries.
 *
 * This function will automatically load drivers from shared libraries.  It
 * searches the "driver path" for .so (or .dll) files that start with the
 * prefix "gdal_X.so".  It then tries to load them and then tries to call a
 * function within them called GDALRegister_X() where the 'X' is the same as
 * the remainder of the shared library basename ('X' is case sensitive), or
 * failing that to call GDALRegisterMe().
 *
 * There are a few rules for the driver path.  If the GDAL_DRIVER_PATH
 * environment variable it set, it is taken to be a list of directories to
 * search separated by colons on UNIX, or semi-colons on Windows.  Otherwise
 * the /usr/local/lib/gdalplugins directory, and (if known) the
 * lib/gdalplugins subdirectory of the gdal home directory are searched on
 * UNIX and $(BINDIR)\\gdalplugins on Windows.
 *
 * Auto loading can be completely disabled by setting the GDAL_DRIVER_PATH
 * config option to "disable".
 */

Disabling PIE in gdal & libgdal-grass does not resolve the issue, neither does disabling as-needed linking in addition.

AutoLoadDrivers() uses CPLGetSymbol() which in turn uses dlopen() & dlsym() to obtain the address of the symbol, the latter of which fails.

It seems to be resolved by disabling mangling of the C++ symbols:

Index: libgdal-grass-1.0.0/grass.cpp
===================================================================
--- libgdal-grass-1.0.0.orig/grass.cpp
+++ libgdal-grass-1.0.0/grass.cpp
@@ -53,6 +53,8 @@ extern "C" {
 char *GPJ_grass_to_wkt( const struct Key_Value *,
                         const struct Key_Value *,
                         int, int );
+
+void GDALRegister_GRASS();
 }
 
 #define GRASS_MAX_COLORS 100000  // what is the right value
Index: libgdal-grass-1.0.0/ogrgrassdriver.cpp
===================================================================
--- libgdal-grass-1.0.0.orig/ogrgrassdriver.cpp
+++ libgdal-grass-1.0.0/ogrgrassdriver.cpp
@@ -30,6 +30,10 @@
 #include "cpl_conv.h"
 #include "cpl_string.h"
 
+extern "C" {
+    void RegisterOGRGRASS();
+}
+
 CPL_CVSID("$Id$")
 
 /************************************************************************/

$ gdalinfo --version
GDAL 3.5.0, released 2022/05/10

Answer 10 · 2022-06-13T19:38:01.000Z

It seems to be resolved by disabling mangling of the C++ symbols:

I confirm this diagnosis. Those function used to be defined as C symbols in gdal_frmts.h / ogrsf_frmts.h when the driver was in-tree.

Answer 11 · 2022-06-13T19:51:45.000Z

Hopefully fixed, thanks for the PR.

Answer 12 · 2022-06-14T05:08:24.000Z

Apologies, they are built using makepkg and live inside a PKGBUILD file so there are variables abound in them. Tried to remove for clarity but missed one - should be ok to ignore from your end.

${pkgdir} is a reference to the install directory from inside the chroot before it's packaged up by the install() function inside the PKGBUILD.

The Debian package used a similar hack, that was until I got fed up with it and added a patch to support DESTDIR.

You may want to add this patch to your package for Arch until 1.0.1 is released: 1383ae1

Answer 13 · 2022-06-14T12:41:25.000Z

@okanisis

GDAL-GRASS GIS driver 1.0.1 has now been released including the bugfixes:
https://github.com/OSGeo/gdal-grass/releases/tag/1.0.1