Luax compile on M1 detects arch incorrectly for Zig.
wmealing opened this issue ยท 30 comments
Tried doing an install on the M1 macs, which detect uname -m as an arm64 instead of the arch that zig uses download uses (aarch64).
Do you want me to submit a pull request with the fix ?
After overcoming this, I found that ZIG doesn't compile any further on this architecture due to LTO issues.
11 fails with the error:
(error: LTO is not yet supported with the Mach-O object format. More details: ziglang/zig#8680)
12-dev fails with the error
error: LTO requires using LLD
The offer for the PR is still open, I can't build luax at the moment due to the above issue.
I began looking at the build.lua however at the moment it is above my skills.
Thanks again.
I've no mac to test luax so it would be great if you can submit a pull request (I guess a quick fix in build_env.sh
is enough).
LTO is disabled for mac binaries but not for temporary tools (such as lua). So this works on Linux to cross compile for mac but not on mac. I can disable LTO for these tools (no effect on the luax binaries, only the execution of the tests using a plain lua interpreter may take a few more milliseconds (no difference on my Linux PC)).
I've disabled LTO (for lua
and lz4
only) on the dev
branch. Can you rebase your fix on the dev
branch?
Tried it on 11 and 12, getting similiar errors.
info: zig can provide libc for related target aarch64-macos.11-none
`[2/673] CC ext/c/lz4/lib/lz4.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o.d ext/c/lz4/lib/lz4.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
[3/673] CC ext/c/lz4/lib/lz4hc.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o.d ext/c/lz4/lib/lz4hc.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
[4/673] CC ext/c/lz4/lib/lz4file.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o.d ext/c/lz4/lib/lz4file.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
[5/673] CC ext/c/lz4/lib/lz4frame.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o.d ext/c/lz4/lib/lz4frame.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
[6/673] CC ext/c/lz4/lib/xxhash.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o.d ext/c/lz4/lib/xxhash.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
[7/673] CC ext/c/lz4/programs/bench.c
FAILED: .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o
. tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o.d ext/c/lz4/programs/bench.c -o .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o
error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu'
info: zig can provide libc for related target aarch64-macos.11-none
I'll try again (its late/early here) maybe i'll be able to understand a bit more when i'm more awake.
Ok got past those errors, the LIBC has to be set to none in the tools/build_env.sh for mac os.. now a new error...
FAILED: .build/tmp/lua_runtime_bundle.dat . tools/build_env.sh; PATH=.build/tmp:$PATH LUA_PATH="./?.lua;.build/tmp/?.lua;libluax/?.lua;libluax/F/?.lua;libluax/L/?.lua;libluax/complex/?.lua;libluax/crypt/?.lua;libluax/fs/?.lua;libluax/imath/?.lua;libluax/import/?.lua;libluax/linenoise/?.lua;libluax/lpeg/?.lua;libluax/lz4/?.lua;libluax/mathx/?.lua;libluax/package/?.lua;libluax/ps/?.lua;libluax/qmath/?.lua;libluax/sh/?.lua;libluax/socket/?.lua;libluax/sys/?.lua;libluax/term/?.lua;libluax/version/?.lua" .build/tmp/lua -l tools/rc4_runtime luax/bundle.lua -lib -ascii .build/tmp/luax_config.lua libluax/F/F.lua libluax/L/L.lua libluax/complex/complex.lua libluax/crypt/crypt.lua libluax/fs/fs.lua libluax/imath/imath.lua libluax/import/import.lua libluax/linenoise/linenoise.lua libluax/lpeg/lpeg.lua libluax/lz4/lz4.lua libluax/mathx/mathx.lua libluax/package/package_hook.lua libluax/ps/ps.lua libluax/qmath/qmath.lua libluax/sh/sh.lua libluax/sys/sys.lua libluax/term/term.lua ext/c/lpeg/re.lua ext/c/luasocket/ftp.lua ext/c/luasocket/headers.lua ext/c/luasocket/http.lua ext/c/luasocket/ltn12.lua ext/c/luasocket/mbox.lua ext/c/luasocket/mime.lua ext/c/luasocket/smtp.lua ext/c/luasocket/socket.lua ext/c/luasocket/tp.lua ext/c/luasocket/url.lua ext/lua/argparse/argparse.lua ext/lua/cbor/cbor.lua ext/lua/inspect/inspect.lua ext/lua/json/json.lua ext/lua/serpent/serpent.lua > .build/tmp/lua_runtime_bundle.dat error: .build/tmp/luax_config.lua: File not found [339/675] CC[aarch64-linux-musl] ext/c/lz4/lib/lz4hc.c ninja: build stopped: subcommand failed.
However the file .build/tmp/luax_config.lua does exist and has contents.. working on that now.
The stat
command may not work on mac as on linux. Could you please post the result of LC_ALL=C stat -L -c '%s;%Y;%X;%W;%F;%f' .build/tmp/luax_config.lua
? or just LC_ALL=C stat -L .build/tmp/luax_config.lua
?
The OSX stat doesn't support -c, I had to use 'gstat' which is coreutils port renamed to gstat.
$ LC_ALL=C gstat -L -c '%s;%Y;%X;%W;%F;%f' .build/tmp/luax_config.lua
404;1704697291;1704697301;1704697291;regular file;81a4
$ LC_ALL=C stat -L .build/tmp/luax_config.lua
16777232 22569945 -rw-r--r-- 1 wmealing staff 0 404 "Jan 8 17:01:41 2024" "Jan 8 17:01:31 2024" "Jan 8 17:01:31 2024" "Jan 8 17:01:31 2024" 4096 8 0 .build/tmp/luax_config.lua
For completion, the contents are:
$ cat .build/tmp/luax_config.lua
--@lib
local version = "2.25.2-2-geadbbdb"
return {
version = version,
date = "2024-01-08",
copyright = "LuaX "..version.." Copyright (C) 2021-2024 cdelord.fr/luax",
authors = "Christophe Delord",
magic_id = "LuaX",
targets = {"x86_64-linux-musl", "x86_64-linux-gnu", "aarch64-linux-musl", "aarch64-linux-gnu", "x86_64-windows-gnu", "x86_64-macos-none", "aarch64-macos-none"},
}
When i run the command by hand, and i know it exists: It still throws the error:
$ . tools/build_env.sh; PATH=.build/tmp:$PATH LUA_PATH="./?.lua;.build/tmp/?.lua;libluax/?.lua;libluax/F/?.lua;libluax/L/?.lua;libluax/complex/?.lua;libluax/crypt/?.lua;libluax/fs/?.lua;libluax/imath/?.lua;libluax/import/?.lua;libluax/linenoise/?.lua;libluax/lpeg/?.lua;libluax/lz4/?.lua;libluax/mathx/?.lua;libluax/package/?.lua;libluax/ps/?.lua;libluax/qmath/?.lua;libluax/sh/?.lua;libluax/socket/?.lua;libluax/sys/?.lua;libluax/term/?.lua;libluax/version/?.lua" .build/tmp/lua -l tools/rc4_runtime luax/bundle.lua -lib -ascii .build/tmp/luax_config.lua libluax/F/F.lua libluax/L/L.lua libluax/complex/complex.lua libluax/crypt/crypt.lua libluax/fs/fs.lua libluax/imath/imath.lua libluax/import/import.lua libluax/linenoise/linenoise.lua libluax/lpeg/lpeg.lua libluax/lz4/lz4.lua libluax/mathx/mathx.lua libluax/package/package_hook.lua libluax/ps/ps.lua libluax/qmath/qmath.lua libluax/sh/sh.lua libluax/sys/sys.lua libluax/term/term.lua ext/c/lpeg/re.lua ext/c/luasocket/ftp.lua ext/c/luasocket/headers.lua ext/c/luasocket/http.lua ext/c/luasocket/ltn12.lua ext/c/luasocket/mbox.lua ext/c/luasocket/mime.lua ext/c/luasocket/smtp.lua ext/c/luasocket/socket.lua ext/c/luasocket/tp.lua ext/c/luasocket/url.lua ext/lua/argparse/argparse.lua ext/lua/cbor/cbor.lua ext/lua/inspect/inspect.lua ext/lua/json/json.lua ext/lua/serpent/serpent.lua > .build/tmp/lua_runtime_bundle.dat
error: .build/tmp/luax_config.lua: File not found
This is the current tree :
https://github.com/wmealing/luax/tree/dev
ok, so fs.stat
shall use gstat
on mac instead of stat
.
Can you also confirm that uname -s
returns Darwin
and uname -m
returns arm64
? (build_env.sh
and sys.lua
must be updated)?
bash-3.2$ uname -s
Darwin
bash-3.2$ uname -m
arm64
I updated build_env.sh but not sys.lua
ok, so fs.stat shall use gstat on mac instead of stat.
I installed gstat via brew.. is that part of the problem ? stat that ships with OSX is the second command above
bash-3.2$ which stat
/usr/bin/stat
bash-3.2$ which gstat
/opt/homebrew/bin//gstat
no, the problem is that fs.lua
uses stat
with the unsupported option -c
. I've pushed a fix on the dev
branch. Could you please test it?
Needed an additional patch to tools/build_env.sh
See https://github.com/wmealing/luax/blob/dev/tools/build_env.sh#L39
The check for the LIBC. Otherwise the build fails with the comment in #1 (comment)
Now with the patch and the build:
bash-3.2$ ninja
[666/674] LD[aarch64-linux-musl] .build/tmp/run/luaxruntime-aarch64-linux-musl
In file included from .zig/0.11.0/lib/libc/musl/crt/rcrt1.c:3:
.zig/0.11.0/lib/libc/musl/crt/../ldso/dlstart.c:146:20: warning: a function declaration without a prototype is deprecated in all versions of C and is treated as a zero-parameter prototype in C2x, conflicting with a subsequent definition [-Wdeprecated-non-prototype]
GETFUNCSYM(&dls2, __dls2, base+dyn[DT_PLTGOT]);
^
.zig/0.11.0/lib/libc/musl/crt/rcrt1.c:11:13: note: conflicting prototype is here
hidden void __dls2(unsigned char *base, size_t *sp)
^
1 warning generated.
[670/674] LD[x86_64-windows-gnu] .build/tmp/run/luaxruntime-x86_64-windows-gnu.exe
.zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:167:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
s->seen_eof = 1;
^ ~
.zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:176:22: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
else s->seen_eof = 1;
^ ~
.zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:1628:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
ifp.is_string = 1;
^ ~
.zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:158:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
s->seen_eof = 1;
^ ~
.zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:167:22: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
else s->seen_eof = 1;
^ ~
.zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:1629:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
ifp.is_string = 1;
^ ~
3 warnings generated.
3 warnings generated.
[673/674] LUAX .build/bin/luax-pandoc
FAILED: .build/bin/luax-pandoc
.build/bin/luax -q -t pandoc -o .build/bin/luax-pandoc luax/luax.lua
.build/bin/luax: No such file or directory
No such file or directory
[674/674] LUAX .build/bin/luax-lua
FAILED: .build/bin/luax-lua
.build/bin/luax -q -t lua -o .build/bin/luax-lua luax/luax.lua
.build/bin/luax: No such file or directory
No such file or directory
ninja: build stopped: subcommand failed.
$ ls .build/bin/
luax luax-aarch64-macos-none luax-x86_64-macos-none
luax-aarch64-linux-gnu luax-x86_64-linux-gnu luax-x86_64-windows-gnu.exe
luax-aarch64-linux-musl luax-x86_64-linux-musl
./luax from here is :
$ file .build/bin/luax
.build/bin/luax: Mach-O 64-bit executable arm64
However it crashes on execution.
bash-3.2$ ./.build/bin/luax
./.build/bin/luax: No such file or directory
No such file or directory
Segmentation fault: 11
Thanks for taking the time on this, i know how hard this kind of debugging is.
Do you have the same problem with the executable I've cross-compiled from Linux?
https://cdelord.fr/hey/luax-aarch64-macos-none.tar.xz
Same problem:
bash-3.2$ ./aarch64-macos-none/bin/luax
./aarch64-macos-none/bin/luax: No such file or directory
No such file or directory
Segmentation fault: 11$ ./aarch64-macos-none/bin/luax-aarch64-macos-none
./aarch64-macos-none/bin/luax-aarch64-macos-none: No such file or directory
No such file or directory
Segmentation fault: 11
I had thought it might be some executable protection on OSX, but i have no evidence to back that up.
This seems to work though:
bash-3.2$ ./aarch64-macos-none/bin/luax-lua
print("hello") <-- typed this.
hello <- echoed this.
If luax
can not read itself, it won't work.
luax-lua
is a pure Lua implementation of some LuaX modules, run by lua
, not luax
.
So if luax
can not read itself, its design does not work on mac (I had tested linux and windows only).
Ah, that'll do it then, i need to go afk for a while. I will test any further suggestions you have overnight.
I'm afraid luax
won't work on macos without a deep redesign.
To validate the hypothesis, can you run this test?
readmyself.c
contains:
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("executable: %s\n", argv[0]);
char buf[1024*1024];
FILE *f = fopen(argv[0], "rb");
size_t n = fread(&buf, 1, sizeof(buf), f);
fclose(f);
printf("size : %zu\n", n);
}
Compiling and running this program prints this on Linux:
$ gcc readmyself.c -o readmyself && ./readmyself
executable: ./readmyself
size : 16880
(should also work with clang
)
Do you get a segmentation fault on macos?
I'm testing a simplified design for LuaX:
- only one binary for the current host (the default host is Linux, for other architectures the ninja file must be regenerated)
- programs "compiled" with luax require luax to be executed (thus the executable does not have to read itself)
This is in the luax3
branch.
e.g.:
$ git clone https://github.com/CDSoft/luax
$ cd luax
$ git checkout luax3
$ tools/bang # regenerate build.ninja for your OS, requires lua 5.4
$ ninja
$ .build/bin/luax
โ ~ gcc readmyself.c -o readmyself && ./readmyself
.executable: ./readmyself
size : 33735
Oddly no crash.
โ ~ clang readmyself.c -o readmyself && ./readmyself
executable: ./readmyself
size : 33687
No segmentation faults on either.
I followed your above steps in the luax3 branch.
โ luax git:(luax3) โ .build/bin/luax
_ __ __ | https://cdelord.fr/luax
| | _ _ __ _\ \/ / |
| | | | | |/ _` |\ / | Version 2.25.2-3-gd9c7b6d (2024-01-11)
| |__| |_| | (_| |/ \ | Powered by Lua 5.4
|_____\__,_|\__,_/_/\_\ |
| Macos aarch64 none
Thanks for the test on macos!
The luax3
branch has been merged to master
.
Was able to build master.
It looks as though not all of the LTO fixes were pulled into master branch. There was something funky going on.
When I would try to build :
โ luax git:(master) ninja
[36/144] CC lua/lapi.c
FAILED: .build/tmp/obj/lua/lapi.o
ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.11.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.11.0/zig-local-cache .zig/0.11.0/zig cc -c -flto=thin -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax -Wno-constant-logical-operand -DLUA_USE_LINUX -MD -MF .build/tmp/obj/lua/lapi.o.d lua/lapi.c -o .build/tmp/obj/lua/lapi.o
error: LTO is not yet supported with the Mach-O object format. More details: https://github.com/ziglang/zig/issues/8680
[45/144] CC ext/c/lz4/lib/lz4hc.c
ninja: build stopped: subcommand failed.
I traced (or think i did) this down to build.lua
diff --git a/build.lua b/build.lua
index 77cfd5a..b1138cd 100644
--- a/build.lua
+++ b/build.lua
@@ -237,7 +237,7 @@ local include_path = {
local lto_opt = case(compiler) {
zig = "-flto=thin",
gcc = "-flto",
- clang = "-flto=thin",
+ clang = "",
}
Now when i run
โ luax git:(master) ninja clean
[1/1] bang build.lua -o build.ninja
load build.lua
21 variables
35 rules
227 build statements
557 lines
43687 bytes
write build.ninja
[1/1] CLEAN .build
โ luax git:(master) โ ninja
[144/144] CP .build/bin/luax
The luax binary is built and works.
โ luax git:(master) โ .build/bin/luax
_ __ __ | https://cdelord.fr/luax
| | _ _ __ _\ \/ / |
| | | | | |/ _` |\ / | Version 3.0 (2024-01-11)
| |__| |_| | (_| |/ \ | Powered by Lua 5.4
|_____\__,_|\__,_/_/\_\ |
| Macos aarch64 none
>>
i realise the fix probably wont be in this location, because you likely want flto in clang on non osx systems.
It looks like you are running the ninja file for Linux (-DLUA_USE_LINUX
is on the command line).
Did you run tools/bang -- macos-aarch64
before ninja
? (I though I had updated the documentation but something has been lost, sorry)
tools/bang
should be enough to detect the OS.
I'll close this issue. Feel free to reopen it if necessary.
The master branch should now have everything to compile LuaX on MacOS.