build fails after kafel update

[jvvv]

The kafel/src/Makefile now includes LDFLAGS.

Firstly, nsjail build fails in kafel if LDFLAGS contain any '-Wl,' CC linker options. I got around that by patching kafel/src/Makefile so that LD is changed to CC; I could also patch the LDFLAGS, but my distribution frowns on editing LDFLAGS in its package builds (adding to them is fine, but actually editting the contents not so much).

sed -i -e 's/(LD)/(CC)/' nsjail/kafel/src/Makefile

Next, the build fails in kafel when CC/LD chokes on the library flags in LDFLAGS. I got around that by patching nsjail/Makefile so that LDFLAGS contains the linker options and added LDLIBS for -l_library_ and -L_library_path_ portions.

diff --git a/Makefile b/Makefile
index 4c4417c..a406a6b 100644
--- a/Makefile
+++ b/Makefile
@@ -33,7 +33,8 @@ COMMON_FLAGS += -O2 -c \
 
 CXXFLAGS += $(USER_DEFINES) $(COMMON_FLAGS) $(shell pkg-config --cflags protobuf) \
 	-std=c++14 -fno-exceptions -Wno-unused -Wno-unused-parameter
-LDFLAGS += -pie -Wl,-z,noexecstack -lpthread $(shell pkg-config --libs protobuf)
+LDFLAGS += -pie -Wl,-z,noexecstack
+LDLIBS += -lpthread $(shell pkg-config --libs protobuf)
 
 BIN = nsjail
 LIBS = kafel/libkafel.a
@@ -51,7 +52,7 @@ endif
 NL3_EXISTS := $(shell pkg-config --exists libnl-route-3.0 && echo yes)
 ifeq ($(NL3_EXISTS), yes)
 	CXXFLAGS += $(shell pkg-config --cflags libnl-route-3.0)
-	LDFLAGS += $(shell pkg-config --libs libnl-route-3.0)
+	LDLIBS += $(shell pkg-config --libs libnl-route-3.0)
 endif
 
 .PHONY: all clean depend indent
@@ -67,7 +68,7 @@ ifneq ($(NL3_EXISTS), yes)
 	$(warning "You probably miss libnl3(-dev)/libnl-route-3(-dev) libraries")
 	$(warning "============================================================")
 endif
-	$(CXX) -o $(BIN) $(OBJS) $(LIBS) $(LDFLAGS)
+	$(CXX) -o $(BIN) $(OBJS) $(LIBS) $(LDFLAGS) $(LDLIBS)
 
 .PHONY: kafel_init
 kafel_init:

I can submit pull requests, but perhaps a different approach is preferred.

[jvvv]

I have found that building kafel with a separate make command as such:
LD="$CC" make -C kafel
before building nsjail fixes things for me.

Edit:
Feel free to close this since I think my work around proves I should have dug a little harder for a solution on my end.

[happyCoder92]

Hi! I verified sed/LD/CC/ fixes the issue with flags. Could you elaborate on "CC/LD chokes on the library flags in LDFLAGS" as I could not reproduce it after switching to CC.

[jvvv]

This is the error I get when the nsjail Makefile attempts to build the kafel submodule:

gcc -Wl,--as-needed,-O1,--sort-common -pie -Wl,-z,noexecstack -lpthread -lprotobuf  -lnl-route-3 -lnl-3  -r kafel.o context.o codegen.o expression.o includes.o parser_types.o policy.o range_rules.o syscall.o lexer.o parser.o syscalls/amd64_syscalls.o syscalls/i386_syscalls.o syscalls/aarch64_syscalls.o syscalls/mipso32_syscalls.o syscalls/mips64_syscalls.o syscalls/riscv64_syscalls.o syscalls/arm_syscalls.o -o libkafel_r.o
/usr/lib/gcc/x86_64-alpine-linux-musl/12.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lnl-route-3: No such file or directory
/usr/lib/gcc/x86_64-alpine-linux-musl/12.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lnl-3: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:59: ../libkafel.a] Error 1
make[2]: Leaving directory '/home/john/aports/testing/nsjail/src/nsjail-3.2/kafel/src'
make[1]: *** [build/Makefile.mk:41: src] Error 2
make[1]: Leaving directory '/home/john/aports/testing/nsjail/src/nsjail-3.2/kafel'
make: *** [Makefile:80: kafel/libkafel.a] Error 2
make: *** Waiting for unfinished jobs....

It is a bit odd, since I am certain that libnl-route-3 and libnl-3 are installed in gcc/ld's searchdirs. Otherwise, the nsjail link would fail, which it does not when kafel is built before starting nsjail build. Since I could not find a reason for ld failing to find those libraries and I could not find another way to prevent nsjail's LDFLAGS from leaking into the kafel build, I first thought to separate the LDLIBS portion (the -llib portions); hence the patch above. Unfortunately, the patch does nothing to resolve why ld is 'choking' on those libraries. I think it strange that the kafel link has no errors for the other two libraries (pthread and protobuf). All mentioned libraries are installed in /usr/lib.

I won't deny that it could be something on my end, but if so I would think I would have issues building other software.

[jvvv]

I just realized why the link is failing. My libnl3 install only has the shared libraries and the failure is for the static link. This proves the linker error is on my end. On the same token, there is no need to link kafel against nsjail's library dependencies. So I still think that nsjail's LDFLAGS should not include nsjail dependency libraries, but rather put them in the standard LDLIBS make variable.

[jvvv]

For the Alpine Linux package, I have gotten around this by building kafel separately before building nsjail with this:

LD="$CC" make -C kafel
make

Since this is working fine, I can close this issue, if that is preferred. The binaries test out well on four of the five architectures that are supported by both kafel and Alpine Linux (x86_64, x86, aarch64, armv7). I am getting textrels in the final binary for riscv64, but they seem to be bogus and don't point to any actual code points; kafel is built with -fPIC and so are the nl-route-3 and protobuf libraries. I will add that architecture once I track down what is causing the textrels showing up in the riscv64 build.