[FLI-935] Support relative linking to FFI library from Go module

[markphelps]

Problem

We are packing the libfliptengine.(so|dylib) with the flipt-client-go module however applications that use the flipt-client-go SDK do not seem able to resolve the relative path(s) for the libfliptengine library.

We've tried many things, all documented starting here: #187 (comment)

For whatever reason all advice/tips on the internet around setting #cgo directives still do not seem to work for this situation, where we are trying to link a dynamic lib (libfliptengine) within a Go module (library) flipt-client-go, to be used by other applications (user code).

The current state of the #cgo directives is here:

flipt-client-sdks/flipt-client-go/evaluation.go

Lines 3 to 9 in 3f7831c

	/*
	#cgo darwin,arm64 LDFLAGS: -L./ext/darwin_arm64 -lfliptengine -Wl,-rpath -Wl,./ext/darwin_arm64
	#cgo linux,arm64 LDFLAGS: -L./ext/linux_arm64 -lfliptengine -Wl,-rpath -Wl,./ext/linux_arm64
	#cgo linux,amd64 LDFLAGS: -L./ext/linux_x86_64 -lfliptengine -Wl,-rpath -Wl,./ext/linux_x86_64
	#include <string.h>
	#include <stdlib.h>
	#include "./ext/flipt_engine.h"

The only way we have gotten the Go Client SDK to work is by specifying the LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH on Mac) with the exact path to the library inside the go module when running the user code (ie: DYLD_LIBRARY_PATH=~/go/pkg/mod/go.flipt.io/flipt-client@v0.4.7/ext/darwin_arm64 ./main).

Without setting this env var, we get:
Library not loaded: /Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib

The above path is the location of the library when it is built and packaged on GH Actions.

Running otool -l on the built dylib (Mac) shows this path as the first line:

ext/darwin_arm64 » otool -L libfliptengine.dylib
libfliptengine.remote.dylib:
	/Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1241.60.3)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.60.24)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1953.255.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

The current assumption is when Go is trying to link to this library at runtime (from the user app), that is is reading this line in the lib and expecting the lib itself to actually 'live' there.

What we are doing with the LD_LIBRARY_PATH env var is effectively overriding this to use the actual location of the lib.

Ideal Solution

Ideally, we would be able to ship the dynamic library with the Go SDK and users could import this library into their applications and not need to then set LD_LIBRARY_PATH or DYLD_LIBRARY_PATH.

_FLI-935

[sylwang]

Running in a Docker environment and setting the LD_LIBRARY_PATH worked for me. However, I couldn't get it running locally on my MacOS.

I could see that the libfliptengine.dylib file is in the same path as the one being used by $DYLD_LIBRARY_PATH.

sy.wang@F2L0GF3HWP flipt-client-sdks % ls /Users/sy.wang/go/pkg/mod/go.flipt.io/flipt-client@v0.4.7/ext/darwin_arm64
libfliptengine.d	libfliptengine.dylib	libfliptengine.rlib
sy.wang@F2L0GF3HWP flipt-client-sdks % echo $DYLD_LIBRARY_PATH                                                      
/Users/sy.wang/go/pkg/mod/go.flipt.io/flipt-client@v0.4.7/ext/darwin_arm64

but I got this error while running a program that uses the flipt client go SDK:

dyld[81146]: Library not loaded: /Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib
  Referenced from: <15C50FBC-0856-3961-8468-F1170D0798DE> /private/var/folders/l5/.../T/go-buildxxxxx/b245/clickhouse-writer.test
  Reason: tried: '/Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib' (no such file), '/Users/runner/work/flipt-client-sdks/flipt-client-sdks/target/aarch64-apple-darwin/release/deps/libfliptengine.dylib' (no such file)

I tried many different things but no luck : (. I wonder if there is a possibility that the .dylib file was somehow corrupted...

[markphelps]

@sylwang , @erka just found a fix for this so we can include the libs without you having to set the env var! I tested it on my local mac and he tested on both mac and linux. will get a new release of the SDK out now so you can try and see if it fixes the issue you were seeing ☝🏻

[erka]

@markphelps I have to say that multi-stage docker image may still need LD_LIBRARY_PATH. Probably it would be nice to try it out.

[markphelps]

@sylwang I updated https://github.com/flipt-io/fliptgosdk to the latest release of the Go SDK (0.5.0) which shows this fixed! 🌮

Thanks again @erka

[erka]

This is a minimal example for multi stage builds of Dockerfile

FROM golang:1.22.1-bookworm as build-env

WORKDIR /app
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .

# Build the Go app
RUN go build -o main .

## Find used library
RUN strings -n 54 main | sed -n -e '/\/go\/pkg\/mod\/go.flipt.io\/flipt-client/ { /\/ext\// p; }' > flipt-ext-path
RUN cp $(cat flipt-ext-path)/libfliptengine.so libfliptengine.so

FROM gcr.io/distroless/cc-debian12
COPY --from=build-env /app/main /app/main
COPY --from=build-env /app/libfliptengine.so /app/libfliptengine.so
ENV LD_LIBRARY_PATH=/app
CMD ["/app/main"]

[sylwang]

Sorry for the delayed response! I was able to run the go client SDKs locally on my machine (MacOS). Thank you both!