Remove ldd to allow for bundling cross-compiled applications

[probonopd]

(Based on the tread on qt-devel)

Remove ldd to allow for bundling cross-compiled applications. ldd not only provides the name of the libraries to be loaded, but also the paths to the locations where they are loaded from. We have to recreate this manually:

implement the transitiveness by running it on each library.

First, use the environment variable $LD_LIBRARY_PATH. The default search path
is known to qmake (variable $QMAKE_DEFAULT_LIBDIRS) and you can then search the
paths from (etc/ld.so.conf & /etc/ld.so.conf.d/*.

• If DT_RUNPATH is present, check $LD_LIBRARY_PATH first and ignore DT_RPATH
• if only DT_RPATH is present, check it before $LD_LIBRARY_PATH

If we're cross-compiling, we should have access to the path to the
sysroot, so we can find ld.so.conf there, e.g., /opt/poky/sysroots/x86_64-pokysdk-linux/etc/ld.so.conf.

[larpon]

I think I've got the basics down now. But I need help blending/merging it in with your code base. I'd suggest some housekeeping to the code base soon - I'm having a hard time finding my way around in both main.cpp and shared.cpp (getting too long for me to keep track of everything) :)

You can checkout a "proof of concept" for dependency resolving here. It currently only resolves for the mandatory binary from the commandline argument - I need some help to use it on resolved QML plugins and QML imports etc. (due to above reasons).

You basically call resolveDependencies with an executable or shared object as an argument. And in return you get a QMap structure (QMap<QString,ExecutableInfo>) where all the keys (QString) are absolute paths to each dependency. In the ExecutableInfo class structure you can, among other things, find the full absolute canonical path (e.g. symlinks are resolved) to the dependency. The dependants string list is only for the purpose of checking that the resolving works as expected.

The order of resolving paths is done as described in man ld.so - which is pretty much the same as described in the snippet from the mailing list.

There are still some quirks that needs to be worked out around the resolving of special contents of RPATH and RUNPATH variables ($ORIGIN $LIB $PLATFORM)

The blacklist should probably also be used to speed up resolving

[probonopd]

You can checkout a "proof of concept" for dependency resolving here

Thanks for working on this. Very cool. I will check it out!

I'd suggest some housekeeping to the code base soon - I'm having a hard time finding my way around in both main.cpp and shared.cpp (getting too long for me to keep track of everything) :)

I have tried to stay close to the macdeployqt code wherever possible, so that we can benefit from improvements made there more easily (should help in maintenance).

I need some help to use it on resolved QML plugins and QML imports etc. (due to above reasons).

Set -verbosity=3 and run linuxdeployqt on a complex app, watch its output. Should give you a clue about what it does, and how. For testing it might be the easiest to really make a ldd-replacement binary with the exact same output as the real ldd, which we would just use as a drop-in replacement for the real ldd, and compare the output.

[larpon]

I've tried implementing some code in the LddInfo findDependencyInfo(const QString &binaryPath) function - simply replacing it's ldd output and parsing with my recursive functions using objdump - and that's slowing down the complete executing to the almost unbearable. It does finish though - but it still bundles the wrong libraries (system libs in my case)? I've pulled in your commits that uses the correct qmake found via a custom $PATH but there's something wrong elsewhere that needs to be fixed first I think

Also there's still problems with plugins in this bit which seemingly doesn't use the info read from PATH - I've tried setting it to deploymentInfo.qtPath = qtPathToBeBundled but that gave some problems with a relative path somewhere else :)

And on a side note; I don't think this function is used anymore?

[probonopd]

I haven't written this from scratch but based it on linuxdeployqt so it's basically a matter of understanding what they did and trying to re-create something similar for Linux.

And on a side note; I don't think this function is used anymore?

...which might well be a bug...

[larpon]

Yeah. As I've read the code they find the library dependencies (non-recursively) of each "major part" (the executable(s), the plugins, the QML libraries for imported QML ...). Reading them non-recursively is apparently enough to do on the mac. I don't know if it's actually necessary to find them recursively on linux - I thought it would be a good idea since the AppImages need more system libraries included to run than what the mac app bundles does (or so it seems). But reading them recursively requires a different flow which would probably be significant enough to move away from the basics of the macdeployqt tool.

Anyway it's all about finding a bunch of files on the filesystem - how hard can it be? 🔨 ⚙️

[probonopd]

If ldd is so bad I wonder why no one has written a better (but syntax compatible) one in C, without the drawbacks mentioned.

[probonopd]

Maybe code from http://www.purinchu.net/software/elflibviewer.php could be used? It uses readelf from GNU binutils under the hood.

[larpon]

If I recall correctly objdump gives dependencies for cross-compiled executables as well doesn't it?

[probonopd]

lddtree may work too. It can fall back to objdump and readelf if scanelf is not found.
https://github.com/ncopa/lddtree

[probonopd]

I wonder whether we could use LIEF to replace ldd (like we might be able to replace patchelf as suggested by @lkollar in NixOS/patchelf#174 (comment)).