Calculated output filenames

Question

Calculated output filenames

typeless opened this issue 2 years ago · 7 comments

I want to collect all generated shared libs and create symlinks as per the Linux convention such as libfoo.so.1 and libfoo.so given libfoo.so.1.0.0.

For example,

shlibs-y += libfoo.so.1.0.0
shlibs-y += libbar.so.1.2.3
shlibs-y += libbaz.so.3.4.5

: foreach $(shlibs-y) |> ln -s -T %b %o |> ????

I can get the SONAME of the shared libs by objdump -p %f | awk '/SONAME/{print $2}'.
But how can I provide it to the outputs of the foreach rule?

Do I need to write it in Lua (if possible at all)?

Answer 1 · 2022-12-22T08:08:10.000Z

What I attempted:

--local inputs = tup.glob('$(shlibs-y)')
local inputs = tup.glob('$(ROOT)/src/libfoo/lib*')
local outputs
local command = 'ln -s -T %b %o'

print('table:\n')
for i, v in ipairs(inputs) do
	print('\t', i, v)
end
print('end table\n')

local k, v
for k, v in ipairs(inputs) do
	table.insert(outputs, v..'.xxx')
end

tup.foreach_rule(inputs, command, outputs)

It looks like tup.glob doesn't expand Tup variables like $(ROOT) and $(shlibs-y), where they are defined in a non-Lua file that includes this Lua file.

Answer 2 · 2022-12-26T13:50:34.000Z

Eventually, the way shown below works for me:

#!/bin/sh -e

for input in $@; do
        basefn=$(basename $input)
        # soname=$($OBJDUMP -p $D/$basefn | awk '/SONAME/{print $2}')
        # tup error: Unable to read from generated file '[build] sysroot/usr/lib/libxyz.so.1.0.0'. Your build configuration must be comprised of files you wrote yourself.
        soname=$(echo $basefn | sed -n 's;lib\([^.]\+\)\.so\(\.[0-9]\+\)\(\.[0-9]\+\)\(\.[0-9]\+\);lib\1.so\2;p')
        soname_nover=$(echo $basefn | sed -n 's;lib\([^.]\+\)\.so\(\.[0-9]\+\)\(\.[0-9]\+\)\(\.[0-9]\+\);lib\1.so;p')
        echo ": $input |> ln -s -T %b %o |> $soname"
        echo ": $input |> ln -s -T %b %o |> $soname_nover"
done

And in the Tupfile

run D=$(TUP_VARIANT_OUTPUTDIR) $(ROOT)/scripts/symlink-libs.sh $(libs-y)

The arguably more accurate $OBJDUMP way doesn't work though. But it makes sense.
I expect Tup should have all dependency data ready before running the generation commands. So, it cannot change the dependency graph 'on-the-fly' while the content of a generated file changed.

Answer 3 · 2022-12-26T22:23:22.000Z

I want to collect all generated shared libs and create symlinks as per the Linux convention such as libfoo.so.1 and libfoo.so given libfoo.so.1.0.0.

For example,
shlibs-y += libfoo.so.1.0.0
shlibs-y += libbar.so.1.2.3
shlibs-y += libbaz.so.3.4.5

: foreach $(shlibs-y) |> ln -s -T %b %o |> ????
I can get the SONAME of the shared libs by objdump -p %f | awk '/SONAME/{print $2}'. But how can I provide it to the outputs of the foreach rule?

Do I need to write it in Lua (if possible at all)?

Tup used to have a "reverse rule", with <| delimiters instead of |>, which would cause the foreach to loop over outputs for things like this. It was confusing and complicated though, and when Lua support was added, this feature was removed. So you are correct, using Lua would be the best way to do this in tup.

What I attempted:
--local inputs = tup.glob('$(shlibs-y)')
local inputs = tup.glob('$(ROOT)/src/libfoo/lib*')
local outputs
local command = 'ln -s -T %b %o'

print('table:\n')
for i, v in ipairs(inputs) do
	print('\t', i, v)
end
print('end table\n')

local k, v
for k, v in ipairs(inputs) do
	table.insert(outputs, v..'.xxx')
end

tup.foreach_rule(inputs, command, outputs)
It looks like tup.glob doesn't expand Tup variables like $(ROOT) and $(shlibs-y), where they are defined in a non-Lua file that includes this Lua file.

Tup should probably support the $(VAR) convention everywhere in the Lua parser, but I think that would only work in the command strings right now. You can always use Lua string functions instead, so tup.glob(ROOT .. '/src/libfoo/lib*') should work. Here's an example that I think does what you want, but you will probably need to still tweak it some:

-- Get the list of ROOT/src/libfoo/libfoo.so.1.2.3 files
local libs = tup.glob(ROOT .. '/src/libfoo/lib*')
local command = 'ln -s -T %b %o'

local k, v
for k, v in ipairs(libs) do
        -- Get just the filename without the path (libname will be libfoo.so.1.2.3)
        local libname = tup.file(v)

        -- Strip the last two numbers to get libfoo.so.1
        local soname = libname:gsub('%.%d+%.%d+$', '')

        -- Strip the last remaining number to get libfoo.so
        local soname_nover = soname:gsub('%.%d$', '')

        -- Create symlinks
        tup.rule(v, command, soname)
        tup.rule(v, command, soname_nover)
end

Eventually, the way shown below works for me:

#!/bin/sh -e

for input in $@; do
        basefn=$(basename $input)
        # soname=$($OBJDUMP -p $D/$basefn | awk '/SONAME/{print $2}')
        # tup error: Unable to read from generated file '[build] sysroot/usr/lib/libxyz.so.1.0.0'. Your build configuration must be comprised of files you wrote yourself.
        soname=$(echo $basefn | sed -n 's;lib\([^.]\+\)\.so\(\.[0-9]\+\)\(\.[0-9]\+\)\(\.[0-9]\+\);lib\1.so\2;p')
        soname_nover=$(echo $basefn | sed -n 's;lib\([^.]\+\)\.so\(\.[0-9]\+\)\(\.[0-9]\+\)\(\.[0-9]\+\);lib\1.so;p')
        echo ": $input |> ln -s -T %b %o |> $soname"
        echo ": $input |> ln -s -T %b %o |> $soname_nover"
done

And in the Tupfile

run D=$(TUP_VARIANT_OUTPUTDIR) $(ROOT)/scripts/symlink-libs.sh $(libs-y)

The arguably more accurate $OBJDUMP way doesn't work though. But it makes sense. I expect Tup should have all dependency data ready before running the generation commands. So, it cannot change the dependency graph 'on-the-fly' while the content of a generated file changed.

Yeah, unfortunately reading from a generated file like this to determine what the output filename should be won't work in Tup. The parsing is all done before any commands are run, so the library won't exist when the DAG is formed. Presumably you already have the soname info somewhere else when you generate the shared library though?

I think trying again with the Lua version would be a good path forward rather than using the run-script. I was hoping to remove run-scripts at some point in the future, but only if the Lua parser can easily handle the cases where people find they are needed (like this one!)

Answer 4 · 2022-12-27T03:03:54.000Z

Not that I am a regular user of the run scripts (I just found it for this problem). But I feel that the ability to use Tup as a language-agnostic, generic build engine simply by outputting a sequence of :-statements is quite neat. It reminds me of the dot language of Graphviz. Everyone can easily use any language. even Bash scripts, to draw a nice diagram that you would otherwise be very hard to do. Maybe there is a better way than run to enable uses like that though. A potential drawback is that it would probably risk fragmenting the community.

By the way, you have given many helpful comments about how to use Tup. But they are scattered all over places like different GitHub issues and threads in the user group. I think a wiki or somewhere to collect all those 'recipes' would be nice. Some tricks like using printf and xargs to overcome the difficulty of adapting various CLI peculiarities or how to run cppcheck with Tup could take me hours to come up with or find the answer on the internet. It would probably also help discover 'idiomatic usage' among users. Just my 2 cents.

Answer 5 · 2022-12-27T04:05:32.000Z

It turned out that I still have some problems to overcome.

The run-script can only be used with the Fuse server. But I use the ldpreload version to minimize the dependency for easier integration with the Yocto BSP. And Yocto doesn't seem to support host-side fuse natively, unfortunately.
I hope my colleagues don't have to be exposed to the build configuration other than listing the source files and some compiler/linker flags if possible. The Lua example you demonstrated above would help in this case if it can be included by a Tupfile. I have to find a way to pass the build configurations/sources lists into that Lua script.

Answer 6 · 2022-12-27T11:50:06.000Z

~~I am going to test a forked Tup that has libfuse3 built-in like PCRE and SQLite. Hope thats enough to have the run-script working.~~
Edit: I've made libfuse3 statically linked with the Tup executable, but it still requires fusermount3 to function. This gets even more complicated when running in containers (docker & podman). So, this route looks not quite viable either.

Maybe I am missing something. Using Lua scripts seems to mean that I have to fully buy into it rather than using it as needed and won't be able to reuse the build configurations defined in my existing Tupfiles (or I have to duplicate the configurations for Lua scripts). Considering that, I would try to use the run scripts first.

Looks like I cannot escape from Lua without facing the headache of FUSE + containers + Yocto 🤣 .

Answer 7 · 2022-12-28T03:17:56.000Z

While looking into the code to add the ability to get variables from Tupfile for my forked Tup, I found there is already a Lua function named tup_get_var (maybe it's not intended to be used by the users?), which is exactly what I need. Using tup_get_var and the Lua code you demonstrated, it WORKS and the run script can be entirely replaced. Thank you.