Support for FreeBSD-arm64
sec opened this issue · 131 comments
Hi,
Not to spam #14537 (as it's already quite big) :)
As full native build of either v6 and v7 is possible under FreeBSD-amd64, I went ahead and tried to make the same process work for ARM64.
After getting configure to success, I've tried:
- Using Linux arm64 host to cross compile to FreeBSD (with custom rootfs created with libs, etc.)
As it was used before, to cross compile from Linux-amd64 to Freebsd-x64, this fail at very early stage with:
[ 1%] Built target System.Globalization.Native-Static
Scanning dependencies of target coreclrpal
[ 1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/dec/state.c.o
[ 1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
In file included from /root/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
/root/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
#include <type_traits>
^~~~~~~~~~~~~
[ 1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/enc/backward_references.c.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
1 error generated.
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_time.c.o
make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
Command I've used ROOTFS_DIR=/home/ubuntu/arm64 ./src/coreclr/build-runtime.sh -arm64 -debug clang14 -cross -os FreeBSD.
This does look like something missing in configure step, as those headers are present on rootfs_dir and host system:
root@instance-20220624-1221:/# find . -name type_traits
./home/ubuntu/arm64/usr/include/c++/v1/tr1/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/experimental/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/type_traits
./usr/include/c++/11/tr2/type_traits
./usr/include/c++/11/tr1/type_traits
./usr/include/c++/11/experimental/type_traits
./usr/include/c++/11/type_traits
./root/runtime/src/coreclr/inc/clr_std/type_traits
so I guess something during configure stage should be added to look in those directories?
- Using FreeBSD arm64 host directly to compile
Trying with both-crossand without, it fails with the same error:
[ 4%] Building C object libs-native/System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_evp_pkey_ecdh.c.o
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:450:1: error: unknown type name 'fpsimd_context'
fpsimd_context* GetNativeSigSimdContext(native_context_t *mc)
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:9: error: unknown type name 'fpsimd_context'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:47: error: unknown type name 'fpsimd_context'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:82: error: no member named '__reserved' in '__mcontext'
fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:458:30: error: use of undeclared identifier 'FPSIMD_MAGIC'
if(fp->head.magic == FPSIMD_MAGIC)
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:460:46: error: use of undeclared identifier 'fpsimd_context'
_ASSERTE(fp->head.size >= sizeof(fpsimd_context));
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:461:69: error: no member named '__reserved' in '__mcontext'
_ASSERTE(size + fp->head.size <= sizeof(mc->uc_mcontext.__reserved));
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/dbgmsg.h:362:35: note: expanded from macro '_ASSERTE'
#define _ASSERTE(expr) do { if (!(expr)) { ASSERT("Expression: " #expr "\n"); } } while(0)
^~~~
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:28: error: use of undeclared identifier 'fpsimd_context'
} while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:70: error: no member named '__reserved' in '__mcontext'
} while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:480:7: error: unknown type name 'fpsimd_context'
const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
^
10 errors generated.
[ 4%] Linking C static library libSystem.Native.a
--- pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o] Error code 1
make[2]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
*** [all] Error code 6
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
1 error
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
Command used was the same as with ad. 1. This looks more complicated, as it have something to do with libunwind being used? No idea how to get over that :)
Including patches I've come up with.
If anyone expierenced with cross build could comment on that, to even confirm it's possible in the current state to make that work or would it require far more work?
If someone is interested in helping/testing/making this work I can provide access to either FreeBSD or Linux ARM64 box that I use.
OSX/BSD builds use the target system's libunwind:
from src/coreclr/pal/src/CMakeLists.txt
if(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)
# On OSX and *BSD, we use the libunwind that's part of the OS
set(CLR_CMAKE_USE_SYSTEM_LIBUNWIND 1)
endif(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)
which makes it add only include_directories(SYSTEM $ENV{ROOTFS_DIR}/usr/local/include)
Yup, I think we just need to update the list of compatible context registers for freebsd-arm64 to make it work.
If that would only make sense to me I would maybe help more, still learing on the fly. I found #40149 and doing quick change:
diff --git a/src/coreclr/pal/src/include/pal/context.h b/src/coreclr/pal/src/include/pal/context.h
index fec5139fd5e..2cc4941d653 100644
--- a/src/coreclr/pal/src/include/pal/context.h
+++ b/src/coreclr/pal/src/include/pal/context.h
@@ -407,7 +407,7 @@ inline void *FPREG_Xstate_Ymmh(const ucontext_t *uc)
#if defined(HOST_ARM64)
-#ifndef TARGET_OSX
+#if !defined(TARGET_OSX) && !defined(TARGET_FREEBSD)
#define MCREG_X0(mc) ((mc).regs[0])
#define MCREG_X1(mc) ((mc).regs[1])
@@ -482,7 +482,7 @@ const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
}
-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX
#define MCREG_X0(mc) ((mc)->__ss.__x[0])
#define MCREG_X1(mc) ((mc)->__ss.__x[1])native compile fail with:
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:177:2: error: unsupported architecture
#error unsupported architecture
^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:183:5: error: use of undeclared identifier 'ASSIGN_UNWIND_REGS'
ASSIGN_UNWIND_REGS
^
2 errors generated.
--- pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o] Error code 1
Is this the context registers that needs to be updated (looked at the code and there's no ARM64 for any of the targets)?
@sec, what are the detected features (the HAVE_xxx) for the FreeBSD arm64 build? When you do a clean build, it lists them to the console. From those, I should be able to tell where to update the context registers extraction.
@janvorli There you go:
-- Performing Test COMPILER_SUPPORTS_F_STACK_PROTECTOR_STRONG - Success
-- Performing Test COMPILER_SUPPORTS_W_IMPLICIT_FALLTHROUGH - Success
-- Performing Test HAVE_LTO - Failed
-- Performing Test COMPILER_SUPPORTS_F_ALIGNED_NEW - Success
-- Performing Test HAVE_DIRENT_D_TYPE - Success
-- Performing Test C_SUPPORTS_WUNGUARDED_AVAILABILITY - Success
-- Performing Test HAVE_IN_PKTINFO - Failed
-- Performing Test HAVE_IP_MREQN - Success
-- Performing Test HAVE_STATFS_VFS - Failed
-- Performing Test HAVE_STATFS_MOUNT - Success
-- Performing Test HAVE_FLOCK64 - Failed
-- Performing Test HAVE_IFADDRS - Success
-- Performing Test HAVE_UTSNAME_DOMAINNAME - Failed
-- Performing Test HAVE_STAT_BIRTHTIME - Success
-- Performing Test HAVE_STAT_FLAGS - Success
-- Performing Test HAVE_STAT_TIMESPEC - Success
-- Performing Test HAVE_STAT_TIM - Success
-- Performing Test HAVE_STAT_NSEC - Success
-- Performing Test HAVE_DIRENT_NAME_LEN - Success
-- Performing Test HAVE_STATFS_FSTYPENAME - Success
-- Performing Test HAVE_STATVFS_FSTYPENAME - Failed
-- Performing Test HAVE_GNU_STRERROR_R - Failed
-- Performing Test HAVE_READDIR_R - Success
-- Performing Test KEVENT_HAS_VOID_UDATA - Success
-- Performing Test HAVE_FDS_BITS - Success
-- Performing Test HAVE_PRIVATE_FDS_BITS - Success
-- Performing Test HAVE_SENDFILE_4 - Failed
-- Performing Test HAVE_SENDFILE_6 - Failed
-- Performing Test HAVE_SENDFILE_7 - Success
-- Performing Test HAVE_GETNAMEINFO_SIGNED_FLAGS - Success
-- Performing Test HAVE_SHM_OPEN_THAT_WORKS_WELL_ENOUGH_WITH_MMAP - Success
-- Performing Test HAVE_CLOCK_MONOTONIC - Success
-- Performing Test HAVE_CLOCK_REALTIME - Success
-- Performing Test BIND_ADDRLEN_UNSIGNED - Success
-- Performing Test IPV6MR_INTERFACE_UNSIGNED - Success
-- Performing Test INOTIFY_RM_WATCH_WD_UNSIGNED - Failed
-- Performing Test HAVE_MKSTEMPS - Success
-- Performing Test HAVE_MKSTEMP - Success
-- Performing Test HAVE_NETINET_TCP_VAR_H - Success
-- Performing Test HAVE_NETINET_UDP_VAR_H - Success
-- Performing Test HAVE_NETINET_IP_VAR_H - Success
-- Performing Test HAVE_NETINET_ICMP_VAR_H - Success
-- Performing Test HAVE_TCP_H_TCPSTATE_ENUM - Failed
-- Performing Test HAVE_GETDOMAINNAME_SIZET - Failed
-- Performing Test HAVE_IN_EXCL_UNLINK - Success
-- Performing Test HAVE_TCP_H_TCP_KEEPALIVE - Failed
-- Performing Test HAVE_BUILTIN_MUL_OVERFLOW - Success
-- Performing Test HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS - Success
-- Performing Test HAVE_VM_FLAGS_SUPERPAGE_SIZE_ANY - Failed
-- Performing Test HAVE_MAP_HUGETLB - Failed
-- Performing Test HAVE_LTTNG_TRACEPOINT_H - Failed
-- Performing Test HAVE_CPUSET_T - Success
-- Performing Test HAVE_TM_GMTOFF - Success
-- Performing Test HAVE_GREGSET_T - Failed
-- Performing Test HAVE___GREGSET_T - Failed
-- Performing Test HAVE_FPSTATE_GLIBC_RESERVED1 - Failed
-- Performing Test HAVE_SYSINFO_WITH_MEM_UNIT - Failed
-- Performing Test HAVE_FPREGS_WITH_CW - Failed
-- Performing Test REALPATH_SUPPORTS_NONEXISTENT_FILES - Failed
-- Performing Test SSCANF_SUPPORT_ll - Success
-- Performing Test SSCANF_CANNOT_HANDLE_MISSING_EXPONENT - Failed
-- Performing Test HAVE_LARGE_SNPRINTF_SUPPORT - Success
-- Performing Test HAVE_BROKEN_FIFO_SELECT - Failed
-- Performing Test HAVE_BROKEN_FIFO_KEVENT - Failed
-- Performing Test HAVE_SCHED_GET_PRIORITY - Success
-- Performing Test HAVE_WORKING_GETTIMEOFDAY - Success
-- Performing Test HAVE_WORKING_CLOCK_GETTIME - Success
-- Performing Test HAVE_CLOCK_MONOTONIC_COARSE - Success
-- Performing Test HAVE_CLOCK_THREAD_CPUTIME - Success
-- Performing Test MMAP_ANON_IGNORES_PROTECTION - Failed
-- Performing Test ONE_SHARED_MAPPING_PER_FILEREGION_PER_PROCESS - Failed
-- Performing Test PTHREAD_CREATE_MODIFIES_ERRNO - Failed
-- Performing Test SEM_INIT_MODIFIES_ERRNO - Failed
-- Performing Test HAVE_PROCFS_CTL - Failed
-- Performing Test HAVE_PROCFS_MAPS - Failed
-- Performing Test HAVE_PROCFS_STAT - Failed
-- Performing Test HAVE_PROCFS_STATUS - Success
-- Performing Test HAVE_COMPATIBLE_ACOS - Success
-- Performing Test HAVE_COMPATIBLE_ASIN - Success
-- Performing Test HAVE_COMPATIBLE_POW - Success
-- Performing Test HAVE_VALID_NEGATIVE_INF_POW - Success
-- Performing Test HAVE_VALID_POSITIVE_INF_POW - Success
-- Performing Test HAVE_COMPATIBLE_ATAN2 - Success
-- Performing Test HAVE_COMPATIBLE_EXP - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGB0 - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGBNAN - Success
-- Performing Test HAVE_COMPATIBLE_LOG - Success
-- Performing Test HAVE_COMPATIBLE_LOG10 - Success
-- Performing Test UNGETC_NOT_RETURN_EOF - Failed
-- Performing Test HAS_POSIX_SEMAPHORES - Success
-- Performing Test GETPWUID_R_SETS_ERRNO - Failed
-- Performing Test FILE_OPS_CHECK_FERROR_OF_PREVIOUS_CALL - Failed
-- Performing Test UNWIND_CONTEXT_IS_UCONTEXT_T - Success
-- Performing Test HAVE_XSWDEV - Success
-- Performing Test HAVE_XSW_USAGE - Failed
-- Performing Test HAVE_PUBLIC_XSTATE_STRUCT - Failed
-- Performing Test HAVE_PR_SET_PTRACER - Failed
-- Performing Test HAVE_FULLY_FEATURED_PTHREAD_MUTEXES - Success
-- Performing Test HAVE_PTHREAD_THREADID_NP - Failed
-- Performing Test HAVE_PTHREAD_GETTHREADID_NP - Success
edit: also attached full log from build attempt.
build_log.txt
It seems to me that your last change was correct. But you also need to add #elif (defined(HOST_UNIX) && defined(HOST_ARM64)) before the #else with #error that fires. In that #elif, you'd add:
ASSIGN_REG(Pc) \
ASSIGN_REG(Sp) \
ASSIGN_REG(Fp) \
ASSIGN_REG(Lr) \
ASSIGN_REG(X19) \
ASSIGN_REG(X20) \
ASSIGN_REG(X21) \
ASSIGN_REG(X22) \
ASSIGN_REG(X23) \
ASSIGN_REG(X24) \
ASSIGN_REG(X25) \
ASSIGN_REG(X26) \
ASSIGN_REG(X27) \
ASSIGN_REG(X28)I hope I got it right.
@janvorli Tried that, it failed with bunch of errors for every register:
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:190:2: error: use of undeclared identifier 'MCREG_X21'
ASSIGN_UNWIND_REGS
^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:178:2: note: expanded from macro 'ASSIGN_UNWIND_REGS'
ASSIGN_REG(X21) \
^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:189:25: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(unwContext->uc_mcontext) = winContext->reg;
^
<scratch space>:95:1: note: expanded from here
MCREG_X21
^
Leaving this empty, will go on but then fail again with similar errors. If I'm looking at this correctly, I think it's missing some #def for HOST_ARM64 and TARGET_FREEBSD inside src/coreclr/pal/src/include/pal/context.h with those registers?
And also there's again error: unknown type name 'fpsimd_context' - which FreeBSD doesn't have, but if I add || defined(TARGET_FREEBSD) next to TARGET_OSX those errors seems to go away (don't know if that will work in the end though, as I going blind in here. Also the undefined registers errors are there, so my bet is that fixing those first would be real fix).
I wodner, if there's libunwind used (which state FreeBSD is supported under x64 and arm64) - shouldn't this be used (if I assume this code here is about that).
As I saw @sdmaclea did the OSX part of changes - maybe some hints? :)
Just to make sure, this is native build under ARM64 FreeBSD host.
Ah, I haven't noticed that your change around line 490 was like this:
-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSXThat way, you have eliminated both definitions of the MCREG_xxx for arm64. What does the ucontext_t struct look like on FreeBSD arm64? We'll either need to define the MCREG_xxx for the FreeBSD separately or reuse the one for Linux / OSX if it happens to be the same.
Ah, found it in FreeBSD sources. So it is different and we will need to have another #elif TARGET_FREEBSD there. It seems that the definitions should look like:
#define MCREG_X21(mc) (mc.mc_gpregs.gp_x[21])
#define MCREG_X22(mc) (mc.mc_gpregs.gp_x[22])
#define MCREG_X23(mc) (mc.mc_gpregs.gp_x[23])
#define MCREG_X24(mc) (mc.mc_gpregs.gp_x[24])
#define MCREG_X25(mc) (mc.mc_gpregs.gp_x[25])
#define MCREG_X26(mc) (mc.mc_gpregs.gp_x[26])
#define MCREG_X27(mc) (mc.mc_gpregs.gp_x[27])
#define MCREG_X28(mc) (mc.mc_gpregs.gp_x[28])
#define MCREG_Fp(mc) (mc.mc_gpregs.gp_fp)
#define MCREG_Lr(mc) (mc.mc_gpregs.gp_lr)
#define MCREG_Sp(mc) (mc.mc_gpregs.gp_sp)
#define MCREG_Pc(mc) (mc.mc_gpregs.gp_pc)As for the floating point access we will also need an extra code here. Take the following with a grain of salt, it might not be exactly right or might need extra change at another place too:
inline
struct fpregs* GetNativeSigSimdContext(native_context_t *mc)
{
return &(mc->mc_fpregs);
}
inline
const struct fpregs* GetConstNativeSigSimdContext(const native_context_t *mc)
{
return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
}@janvorli Thanks, I was looking at the same files and was about to post :)
Can't find Pc and Fp in ucontext in structs btw.
Managed to get past some errors, but now a lot of:
[ 6%] Built target eventpipe_objs
Consolidate compiler generated dependencies of target mdruntimerw_wks_obj
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: use of undeclared identifier 'BSDREG_Cpsr'
ASSIGN_ALL_REGS
^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
ASSIGN_CONTROL_REGS \
^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:137:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
ASSIGN_REG(Cpsr) \
^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:333:62: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(registers.uc_mcontext) = BSDREG_##reg(ptrace_registers);
^
<scratch space>:248:1: note: expanded from here
BSDREG_Cpsr
^
[ 8%] Built target ceefgen_obj
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: use of undeclared identifier 'BSDREG_Fp'
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
ASSIGN_CONTROL_REGS \
^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:138:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
ASSIGN_REG(Fp) \
^
...
I haven't saw any existing BSDREG_X for defined (even for OSX)
I'm starting to think does this road make sense. Shouldn't this reuse existing ARM64 code and/or use machine specific libunwind for the rest?
We cannot reuse the code, because the structure of the ucontext_t is different from all other platforms, using different names of data fields for the registers etc. That's why we are adding new ones.
Can you please share the log from the clean build after you've applied my previous fix? It is not clear to me how come we are compiling code using the BSDREG_xxx stuff when the HAVE_BSD_REGS_T doesn't seem to be defined.
build_log.txt
freebsd_arm64_patch.txt
@janvorli Here's build output and git diff of everything I have tweaked :)
edit: accoring to CMakeOutput.txt, BSD_REG check passed:
Run Build Command(s):/usr/local/bin/gmake -f Makefile cmTC_c2fef/fast && /usr/local/bin/gmake -f CMakeFiles/cmTC_c2fef.dir/build.make CMakeFiles/cmTC_c2fef.dir/build
gmake[1]: Entering directory '/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o
/usr/local/bin/clang14 -D_FILE_OFFSET_BITS=64 -I/usr/local/include -fPIE -MD -MT CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o -MF CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o.d -o CMakeFiles/cmTC_c2fef.dir/BSD_REGS _T.c.o -c /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CheckTypeSize/BSD_REGS_T.c
Linking C executable cmTC_c2fef
/usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c2fef.dir/link.txt --verbose=1
/usr/local/bin/clang14 -fuse-ld=lld -pie -Wl,--build-id=sha1 CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o -o cmTC_c2fef
gmake[1]: Leaving directory '/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CMakeTmp'
CMakeError.log
CMakeOutput.log
edit 2: added CMakeFiles output, maybe it will help also
Ah, I've forgotten that the check_type_size("struct reg" BSD_REGS_T) doesn't log anything. In that case, we need to add definitions for BSDREG_xxx stuff. The HAVE_BSD_REGS_T branch in the source assumes it is either x64 or x86. So in context.h, look for the #if HAVE_BSD_REGS_T and then #ifdef HOST_64BIT under that. Inside that #ifdef, we need to put the current stuff under #ifdef HOST_AMD64 and add #elif defined(HOST_ARM64) branch where we put the following:
#define BSDREG_X0(reg) BSD_REGS_STYLE(reg,X[0],x[0])
.
.
.
#define BSDREG_X29(reg) BSD_REGS_STYLE(reg,X[29],x[29])
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,Fp,fp)
#define BSDREG_Sp(reg) BSD_REGS_STYLE(reg,Sp,sp)
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
#define BSDREG_Lr(reg) BSD_REGS_STYLE(reg,Lr,lr)
#define BSDREG_Cpsr(reg) BSD_REGS_STYLE(reg,Spsr,spsr)
And then in src/coreclr/pal/src/configure.cmake, find elseif(CLR_CMAKE_TARGET_FREEBSD). In that branch, the set(BSD_REGS_STYLE "((reg).r_##rr)") has to be different for amd64 and arm64. For arm64, it needs to be set(BSD_REGS_STYLE "((reg).rr)"), because the naming convention of the registers is different there. So it should be:
if (CLR_CMAKE_HOST_ARCH_AMD64)
set(BSD_REGS_STYLE "((reg).r_##rr)")
elseif(CLR_CMAKE_HOST_ARCH_ARM64)
set(BSD_REGS_STYLE "((reg).rr)")
else()
message(FATAL_ERROR "Unknown FreeBSD architecture")
endif()Thanks, will try that - any idea what about missing MCREG_Pc and MCREG_Fp - can't see those inside ucontext struct (or the name is not close to that) ?
Ah, sorry, the Fp is X29, so it should be
#define MCREG_Fp(mc) (mc.mc_gpregs.gp_x[29])And the MCREG_Pc - hmm, that's strange. I can see that there is gp_lr and gp_elr. I think that maybe the gp_elr is in fact the Pc.
Also related to the x29 above, for the most recent change I've suggested, this is wrong:
#define BSDREG_X29(reg) BSD_REGS_STYLE(reg,X[29],x[29])
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,Fp,fp)It should be just
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,X[29],x[29])With everything so far:
freebsd_arm_patch.txt
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: no member named 'pc' in 'reg'
ASSIGN_ALL_REGS
^~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
ASSIGN_CONTROL_REGS \
^~~~~~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:141:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
ASSIGN_REG(Pc)
^~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:333:62: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(registers.uc_mcontext) = BSDREG_##reg(ptrace_registers);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<scratch space>:310:1: note: expanded from here
BSDREG_Pc
^
/home/frank/runtime/src/coreclr/pal/src/include/pal/context.h:1020:46: note: expanded from macro 'BSDREG_Pc'
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
~~~~~~~~~~~~~~~~~~~~~~^~~
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/pal/src/config.h:158:44: note: expanded from macro 'BSD_REGS_STYLE'
#define BSD_REGS_STYLE(reg, RR, rr) ((reg).rr)
~~~~~ ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:491:13: error: no member named 'pc' in 'reg'
ASSIGN_CONTROL_REGS
^~~~~~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:141:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
ASSIGN_REG(Pc)
^~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:483:25: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) BSDREG_##reg(ptrace_registers) = lpContext->reg;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<scratch space>:385:1: note: expanded from here
BSDREG_Pc
^
/home/frank/runtime/src/coreclr/pal/src/include/pal/context.h:1020:46: note: expanded from macro 'BSDREG_Pc'
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
~~~~~~~~~~~~~~~~~~~~~~^~~
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/pal/src/config.h:158:44: note: expanded from macro 'BSD_REGS_STYLE'
#define BSD_REGS_STYLE(reg, RR, rr) ((reg).rr)
~~~~~ ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:600:9: error: unknown type name 'fpsimd_context'
fpsimd_context* fp = GetNativeSigSimdContext(native);
^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:744:15: error: unknown type name 'fpsimd_context'
const fpsimd_context* fp = GetConstNativeSigSimdContext(native);
^
4 errors generated.
(Note: using Ubuntu 18.04 with FreeBSD AARCH64 crossroot)
Ok, sounds like again elr vs pc.
So can you please make this change?:
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Elr,elr)Last thing is the floating point context. In context.cpp at line 743, we'll need to add before the #else:
#elif TARGET_FREEBSD
const struct fpregs* fp = GetConstNativeSigSimdContext(native);
if (fp)
{
lpContext->Fpsr = fp->fp_sr;
lpContext->Fpcr = fp->fp_cr;
for (int i = 0; i < 32; i++)
{
lpContext->V[i] = *(NEON128*) &fp->q[i];
}
}Similarly at line 599, we need to add before the #else:
#elif TARGET_FREEBSD
struct fpregs* fp = GetNativeSigSimdContext(native);
if (fp)
{
fp->fp_sr = lpContext->Fpsr;
fp->fp_cr = lpContext->Fpcr;
for (int i = 0; i < 32; i++)
{
*(NEON128*) &fp->q[i] = lpContext->V[i];
}
}Including above code:
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:607:34: error: no member named 'q' in 'fpregs'
*(NEON128*) &fp->q[i] = lpContext->V[i];
~~ ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:762:52: error: no member named 'q' in 'fpregs'
lpContext->V[i] = *(NEON128*) &fp->q[i];
~~ ^
2 errors generated.
Not sure why its throwing that. usr/include/machine/ucontext.h on the crossrootfs has that member and seen (https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/arm64/include/ucontext.h#L45)
@janvorli Big thanks for help/pointers. I was able to compile coreclr part (there were some typos to fix):
Repo successfully built.
Product binaries are available at /usr/home/sec/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug
Also I've pushed those changes here main...sec:runtime:freebsd.arm64
Any way to test if all those register parts are working correctly (aka how to run the tests for the part that just got compiled) ?
@Thefrank I also took few changes from your patch regarding RID's - thanks :) Would you be able to try to cross compile under Linux? I've tried, but on my setup (Linux ARM64) it fails with:
Scanning dependencies of target coreclrpal
[ 1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
In file included from /home/ubuntu/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
/home/ubuntu/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
#include <type_traits>
^~~~~~~~~~~~~
[ 1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/home/ubuntu/runtime/src/native/external/brotli/enc/backward_references.c.o
[ 1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
1 error generated.
make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Also if you'll take my branch, those can also be needed - don't know if those are needed or it's just my setup:
diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index d5dfc13504b..478029410b7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -133,7 +133,11 @@ elseif(FREEBSD)
set(CMAKE_CXX_COMPILER_TARGET ${triple})
set(CMAKE_ASM_COMPILER_TARGET ${triple})
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
- set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ if(CLR_CMAKE_HOST_OS STREQUAL FreeBSD)
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ else()
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -stdlib=libc++ -Qunused-arguments")
+ endif()
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld")
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -fuse-ld=lld")
elseif(ILLUMOS)
diff --git a/eng/native/configuretools.cmake b/eng/native/configuretools.cmake
index 6697524c659..b44bcc3417c 100644
--- a/eng/native/configuretools.cmake
+++ b/eng/native/configuretools.cmake
@@ -54,7 +54,11 @@ if(NOT WIN32 AND NOT CLR_CMAKE_TARGET_BROWSER)
set(TOOLSET_PREFIX ${ANDROID_TOOLCHAIN_PREFIX})
elseif(CMAKE_CROSSCOMPILING AND NOT DEFINED CLR_CROSS_COMPONENTS_BUILD AND
CMAKE_SYSTEM_PROCESSOR MATCHES "^(armv8l|armv7l|armv6l|aarch64|arm|s390x|ppc64le)$")
- set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+ if(FREEBSD)
+ set(TOOLSET_PREFIX "llvm-")
+ else()
+ set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+ endif()
else()
set(TOOLSET_PREFIX "")
endif()What should be next steps to get full SDK - as under FreeBSD ARM64 I don't have bootstrapped one, I think I will move to AMD64 FreeBSD and try there to cross compile?
edit: @Thefrank for the error, it's fp_q and also sorry I wrongly tagged @wfurt :)
edit 2: cross build under FreeBSD is going well... does this mean I will have working SDK on FreeBSD-arm64? 🤞
@sec I was happy to help. As for testing whether these changes are correct, running coreclr tests is the best way. These code paths are exercised by exception handling.
On the target, you can build all coreclr tests (in checked flavor, which means mostly optimized code, but including asserts in PAL / runtime) using
src/tests/build.sh checked -priority1(Please note that you have to have the runtime clr+libs part built there before building the tests.)
And then run them using
src/tests/run.sh checkedHm, not that easy @janvorli :) While doing crosscompile under FreeBSD-amd64, using command
ROOTFS_DIR=/home/sec/arm64_rootfs eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false -subset Clr.Native+Host.Native it fails after soem time:
Repo successfully built.
Product binaries are available at /usr/home/sec/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug
Executing "/usr/home/sec/runtime/src/coreclr/build-runtime.sh" -arm64 -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -debug -ci -os FreeBSD -hostarch x64 -component crosscomponents
Commencing CoreCLR Repo build
__DistroRid: freebsd-arm64
Setting up directories for build
Checking prerequisites...
Commencing build of " crosscomponents " target in "CoreCLR component" for FreeBSD.arm64.Debug in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
Invoking "/usr/home/sec/runtime/eng/native/gen-buildsys.sh" "/usr/home/sec/runtime/src/coreclr" "/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64" x64 clang Debug "" -DCLR_CMAKE_TARGET_ARCH=arm64 -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCLR_CROSS_COMPONENTS_BUILD=1
/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64 /usr/home/sec/runtime/src/coreclr
Not searching for unused variables given on the command line.
-- The C compiler identification is Clang 10.0.1
-- The CXX compiler identification is Clang 10.0.1
...
[ 2%] Linking CXX static library libgcinfo_universal_arm64.a
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:34:
/home/sec/arm64_rootfs/usr/local/include/libunwind.h:25:11: fatal error: 'libunwind-x86_64.h' file not found
# include "libunwind-x86_64.h"
^~~~~~~~~~~~~~~~~~~~
[ 2%] Built target gcinfo_universal_arm64
[ 2%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/file/directory.cpp.o
1 error generated.
--- pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o] Error code 1
make[3]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
--- CMakeFiles/crosscomponents.dir/rule ---
*** [CMakeFiles/crosscomponents.dir/rule] Error code 6
make[1]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
1 error
make[1]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
*** [crosscomponents] Error code 2
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
1 error
/usr/home/sec/runtime/src/coreclr
What's going on in here? @Thefrank how's your Linux crossbuild?
The cross components that are failing to build are compiled for x64, so they should be built on the host without the rootfs usage. It seems like there is some problem causing that the rootfs is still being used.
So for my Ubuntu 18.04 AMD64 -> FreeBSD AARCH64 an old bug has presented itself:
[edit: ROOTFS_DIR=/home/frank/freebsdarm/ ./src/coreclr/build-runtime.sh -arm64 --clang11 -cross -os FreeBSD]
/home/frank/runtime/src/coreclr/debug/daccess/daccess.cpp:6996:38: error: invalid suffix 'x' on integer constant
*dacTableAddress = baseAddress + DAC_TABLE_RVA;
^
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/inc/dactablerva.h:1:24: note: expanded from macro 'DAC_TABLE_RVA'
#define DAC_TABLE_RVA 0x c06bb8
^
1 error generated.
Previously under net6, this could be worked around in eng/native/gen-buildsys.sh
diff --git a/eng/native/gen-buildsys.sh b/eng/native/gen-buildsys.sh
index bf04c26f2b1..7d8793dd92c 100755
--- a/eng/native/gen-buildsys.sh
+++ b/eng/native/gen-buildsys.sh
@@ -97,7 +97,7 @@ if [[ "$build_arch" == "wasm" ]]; then
cmake_command="emcmake $cmake_command"
fi
-cmake_args_to_cache="$scan_build\n$SCAN_BUILD_COMMAND\n$generator\n$__UnprocessedCMakeArgs"
+cmake_args_to_cache=$(printf "$scan_build${SCAN_BUILD_COMMAND:+ }${SCAN_BUILD_COMMAND} $generator $__UnprocessedCMakeArgs")
cmake_args_cache_file="$2/cmake_cmd_line.txt"
if [[ -z "$__ConfigureOnly" ]]; then
if [[ -e "$cmake_args_cache_file" ]]; thenThis whole codeblock was removed in d22acef so this hacky-workaround won't work.
The contents of the dactablerva.h is generated by this script:
https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/tools/gen-dactable-rva.sh
Sounds like this generates the extra spaces, so we can fix it there:
@janvorli About that crosscomponents compile issue... Looking at the command "/usr/home/sec/runtime/src/coreclr/build-runtime.sh" -arm64 -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -debug -ci -os FreeBSD -hostarch x64 -component crosscomponents - there's no -cross so it shouldn't use cross fs, but it's using, so looks like it's env thing - anyway - if I run the command directly it fail with something more harder to fix (hope not):
Consolidate compiler generated dependencies of target cee_wks_obj
[ 21%] Building ASM object vm/wks/CMakeFiles/cee_wks_core.dir/__/arm64/asmhelpers.S.o
[ 21%] Building CXX object vm/wks/CMakeFiles/cee_wks_obj.dir/__/codeman.cpp.o
[ 21%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/map/map.cpp.o
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:9:5: error: unknown use of instruction mnemonic without a size suffix
mov x0, lr
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:10:5: error: unknown use of instruction mnemonic without a size suffix
ret lr
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:15:5: error: unknown use of instruction mnemonic without a size suffix
mov x0, sp
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:16:5: error: unknown use of instruction mnemonic without a size suffix
ret lr
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:21:5: error: invalid instruction mnemonic 'mrs'
mrs x0, dczid_el0
^~~
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:22:5: error: unknown use of instruction mnemonic without a size suffix
and x0, x0, 31
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:23:5: error: unknown use of instruction mnemonic without a size suffix
ret lr
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:32:13: error: unknown token in expression
mov w1, #0
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:33:16: error: expected ']' in brackets expression
str w1, [x0, #0xc0]
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:35:16: error: expected ']' in brackets expression
str lr, [x0, #((0xc0 +8)+8)]
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:38:5: error: unknown use of instruction mnemonic without a size suffix
mov x1, sp
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:39:16: error: expected ']' in brackets expression
str x1, [x0, #(0xc0 +8)]
^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:42:17: error: unknown token in expression
add x1, x0, #0x0
... a lot more of those errors ...
edit: running ./src/coreclr/build-runtime.sh -arm64 -hostarch arm64 -component crosscomponents -cmakeargs -DCLR_CROSS_COMPONENTS_BUILD=1 on FreeBSD-arm64 fail with
Consolidate compiler generated dependencies of target coreclr
[ 94%] Linking CXX shared library libcoreclr.so
ld.lld: error: unable to find library -lSystem.Globalization.Native-Static
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
--- dlls/mscoree/coreclr/libcoreclr.so ---
*** [dlls/mscoree/coreclr/libcoreclr.so] Error code 1
Albo native libs compile fine under ARM64 host, but executing ./src/coreclr/build-runtime.sh fail with:
[ 94%] Linking CXX shared library libcoreclr.so
ld.lld: error: unable to find library -lSystem.Globalization.Native-Static
[ 94%] Building CXX object jit/CMakeFiles/clrjit_win_x86_arm64.dir/typeinfo.cpp.o
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
and
sec@amper:~/runtime % ls artifacts/bin/native/FreeBSD-arm64-Debug/
./ libSystem.IO.Compression.Native.so.dbg libSystem.Net.Security.Native.a
../ libSystem.IO.Ports.Native.a libSystem.Net.Security.Native.so*
libSystem.Globalization.Native.a libSystem.IO.Ports.Native.so* libSystem.Net.Security.Native.so.dbg
libSystem.Globalization.Native.so* libSystem.IO.Ports.Native.so.dbg libSystem.Security.Cryptography.Native.OpenSsl.a
libSystem.Globalization.Native.so.dbg libSystem.Native.a libSystem.Security.Cryptography.Native.OpenSsl.so*
libSystem.IO.Compression.Native.a libSystem.Native.so* libSystem.Security.Cryptography.Native.OpenSsl.so.dbg
libSystem.IO.Compression.Native.so* libSystem.Native.so.dbg
edit: trying to build those components under Linux is working fine - maybe this is something wrong with cmakefiles def's ? :/
Tried to use Linux for crosscompile
- RVA is caused I think when you have bash version <5.1 - dirty hack was to remove 016 from the format string in script
eng/common/cross/toolchain.cmakeneed a fix when setting FreeBSD triple x86_64 -> aarch64
but even then it fails, at mono stage, so that's the far most progress:
[31/351] Building C object mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o
FAILED: mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o
/usr/bin/clang-10 --target=aarch64-unknown-freebsd12 --sysroot=/home/linux/arm64/ -DHAVE_CONFIG_H -I/home/linux/runtime/artifacts/obj -I/home/linux/runtime/src/native -Imono/mini -Imono/mini/../.. -Imono/mini/../../mono/eglib -I/home/linux/runtime/src/mono/mono/mini/../.. -I/home/linux/runtime/src/mono/mono/mini/.. -I/home/linux/runtime/src/mono/mono/mini/../eglib -I/home/linux/runtime/src/mono/mono/mini/../sgen -I/home/linux/runtime/src/mono/mono/mini/../eventpipe -Imono/mini/eventpipe -I/home/linux/runtime/src/native/public/. -std=gnu99 -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -g -g -fPIC -fvisibility=hidden -ffp-contract=off -MD -MT mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o -MF mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o.d -o mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o -c /home/linux/runtime/src/mono/mono/component/debugger-agent.c
<inline asm>:1:6: error: invalid token in expression
int $03
^
<inline asm>:1:6: error: invalid operand
int $03
^
2 errors generated.
ninja: build stopped: subcommand failed.
/home/linux/runtime/src/mono/mono.proj(541,5): error MSB3073: The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.
##vso[task.logissue type=error;sourcepath=/home/linux/runtime/src/mono/mono.proj;linenumber=541;columnnumber=5;code=MSB3073;](NETCORE_ENGINEERING_TELEMETRY=Build) The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.
Build FAILED.
/home/linux/runtime/src/mono/mono.proj(541,5): error MSB3073: The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.
0 Warning(s)
1 Error(s)
Commands I used:
ROOTFS_DIR=/home/linux/arm64/ ./eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false -subset Clr.Native+Host.Native- compile fineROOTFS_DIR=/home/linux/arm64/ ./eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false /p:AppHostSourcePath=/home/linux/runtime/artifacts/obj/freebsd-arm64.Debug/apphost/standalone/apphost- failed
- RVA is caused I think when you have bash version <5.1 - dirty hack was to remove 016 from the format string in script
frank@buildbot:~/runtime$ bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
O.K. /sigh/
removing that deleting the old artifacts/obj/coreclr/FreeBSD.arm64.Debug/inc/dactablerva.h and rerunning finished!
the new file contained #define DAC_TABLE_RVA 0xc06bb8
Repo successfully built.
Product binaries are available at /home/frank/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug
It also makes it past the cross components but runs into the same issue for mono.
Which is interesting because...FreeBSD doesn't have pkg-config as a part of the base OS. It's in ports (as pkgconf). That path looks like it gets added from the mono.proj
TOP EDIT: almost forgot to give credit to @janvorli for all the help
It's built!
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:43:38.40
I just have no way to test it as...I don't have any ARM64 hardware sitting around. Posting patch + binaries shortly
EDIT1:
freebsd_arm_patch.txt
EDIT2: https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/V7.0-ARM64-EXPERIMENTAL
EDIT3: build command ROOTFS_DIR=/home/frank/freebsd13arm/ ./build.sh -a arm64 --clang11 -cross -os FreeBSD
EDIT4: Tomorrow ill look into ASPNET and Installer, both of which should be much easier than runtime
I was wrong about RVa issue - I have same version of bash and it format diffrently - but made a fix for that - sec@1a6d288
Also all the other fixes (with some not needed removed) are integrated into my branch - would it be feasible to make PR for that at this stage @wfurt ?
Thanks to last mono fix - I was also able to crosscompile Linux x64 -> FreeBSD arm64
Linux arm64 -> FreeBSD arm64 - also fine, with small patch (edit: looks it's no longer need, maybe becuase clang11 was used, but I will leave it here for future reference anyway):
diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index d5dfc13504b..478029410b7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -133,7 +133,11 @@ elseif(FREEBSD)
set(CMAKE_CXX_COMPILER_TARGET ${triple})
set(CMAKE_ASM_COMPILER_TARGET ${triple})
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
- set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ if(CLR_CMAKE_HOST_OS STREQUAL FreeBSD)
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ else()
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -stdlib=libc++ -Qunused-arguments")
+ endif()
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld")
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -fuse-ld=lld")
elseif(ILLUMOS)FreeBSD x64 -> FreeBSD arm64 - two issues:
- on first run when it try to build
crosscomponents- looks like crossrootfs is used - on manual re-run of the command used to build those - there are some errors in
/tmp/asmhelpers-8f0fb3.s(more above)
FreeBSD arm64:
./src/coreclr/build-runtime.sh- ok./src/native/libs/build-native.sh- ok- don't know if building mono is possible without working runtime first (found https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/cross-building.md#build-corefx-for-a-new-architecture but it not enough information to me)
NOW, testing runtime under FreeBSD arm64 host...
[sec@amper ~]$ ~/dotnet7/dotnet --info
Host:
Version: 7.0.0-dev
Architecture: arm64
Commit: N/A
.NET SDKs installed:
No SDKs were found.
.NET runtimes installed:
Microsoft.NETCore.App 7.0.0-dev [/usr/home/sec/dotnet7/shared/Microsoft.NETCore.App]
Other architectures found:
None
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
Tried to make simple console app, restore, build, publish, copy to arm64 host, adjust json to use 7.0.0-dev and
[sec@amper ~/runtime/console_app]$ ~/dotnet7/dotnet bin/Debug/net6.0/publish/console_app.dll
Failed to create CoreCLR, HRESULT: 0x80004005
Attaching COREHOST trace file - hope it's something trivial to fix :)
host.txt
yes, I think so @sec. main branch for 7.0 is still generally open (closing soon). I'm on the road so my capacity is small for next 2-3 weeks.
@sec if run in jail, does the jail have allow_mlock on? Does the jail/OS have a defined and writable TMP folder?
Not running in jail and yes, there's a writeable /tmp folder.
truss work under ARM64? Might give that a shot
@Thefrank tried that already, nothing special, but attaching sample log
truss_output.txt
attempt at full SDK. Still no way to test it:https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/7.0.100-preview.7.22353.99-freebsd-arm64
@Thefrank nice - tried SDK, same result (one thing I notice is no runtime info about FreeBSD host etc., but that's why it fail right?):
sec@amper:~/sdk7 % ./dotnet --info
Failed to create CoreCLR, HRESULT: 0x80004005
Host:
Version: 7.0.0-preview.6.22329.5
Architecture: arm64
Commit: c9140d621a
.NET SDKs installed:
7.0.100-preview.7.22353.99 [/usr/home/sec/sdk7/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 7.0.0-preview.7.22351.1 [/usr/home/sec/sdk7/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.0-preview.6.22329.5 [/usr/home/sec/sdk7/shared/Microsoft.NETCore.App]
Other architectures found:
None
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
Output from SDK7 under x64 host:
.NET SDK:
Version: 7.0.100-preview.5.22307.18
Commit: bd8b088037
Runtime Environment:
OS Name: FreeBSD
OS Version: 13
OS Platform: FreeBSD
RID: freebsd.13-x64
Base Path: /usr/home/sec/dotnet7/sdk/7.0.100-preview.5.22307.18/
global.json file:
Not found
Host:
Version: 7.0.0-preview.5.22301.12
Architecture: x64
Commit: 425fedc0fb
.NET SDKs installed:
7.0.100-preview.5.22307.18 [/usr/home/sec/dotnet7/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 7.0.0-preview.5.22303.8 [/usr/home/sec/dotnet7/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.0-preview.5.22301.12 [/usr/home/sec/dotnet7/shared/Microsoft.NETCore.App]
@Thefrank could you share the steps/any patches that you used to make this - maybe we could use this to automate x64 builds and publish them on https://github.com/dotnet/runtimelab ?
Does anyone have any expierence in troubleshoting root cuase of this CLR fail - maybe something from https://github.com/dotnet/diagnostics will be to good use?
@Thefrank nice - tried SDK, same result (one thing I notice is no runtime info about FreeBSD host etc., but that's why it fail right?):
From what was posted it does not appear to be making it very far. Now WHY that is happening is beyond me
@Thefrank could you share the steps/any patches that you used to make this - maybe we could use this to automate x64 builds and publish them on https://github.com/dotnet/runtimelab ?
Runtime builds from head with only a quick sed patch to make it use a prebuilt SDK and that is how I do nightlies+tests for FreeBSD-x64
Building a WHOLE SDK requires a bit more tweaks but is rather easy from TAG.
However, if you want to build from a random commit, well, that takes a bit more work. For that I usually work backwards; examine what BUILDIDs dotnet/installer wants for dotnet/runtime and dotnet/aspnetcore, reverse them into YYYYMMDD.# format and plug them into the builds first for runtime and second for aspnetcore.
EDIT: Does lldb provide any insights? Is it even making it to managed code?
@Thefrank I was rather asking, how to you build e. g. installer repo in cross mode, as I didn't saw any swithces just for that (and --runtime-id freebsd-x64 /p:OSName=freebsd prepare SDK with correct names, but Linux inside) - but that's offtopic :)
As for lldb - not an export on this one, but doesn't look like it's getting into managed code - tried to use corerun with bogus parameters and it's throwing the same error before even checking if given file exists.
edit: I may be wrong, but maybe it's connected with fact, that cross build from FreeBSD-x64 host fail on cross components stage with some assembler compiler error (but is fine when doing the same from Linux host)?
Had fun with lldb, learning at 2am is great :) The code looks like it's failing when loaded/loading System.Private.CoreLib and then going this route:
* thread #1, name = 'corerun', stop reason = breakpoint 7.1
frame #0: 0x0000000041544ce8 libcoreclr.so`Assembly::Init(this=0x0000000040c11650, pamTracker=0x0000ffffffffccf0, pLoaderAllocator=0x0000000000000000) at assembly.cpp:196:53
193 // manifest modules of dynamic assemblies are always transient
194 m_pModule = ReflectionModule::Create(this, pPEAssembly, pamTracker, REFEMIT_MANIFEST_MODULE_NAME);
195 else
-> 196 m_pModule = Module::Create(this, mdFileNil, pPEAssembly, pamTracker);
197
198 InterlockedIncrement((LONG*)&g_cAssemblies);
199
after some steps
Process 25614 stopped
* thread #1, name = 'corerun', stop reason = step over
frame #0: 0x000000004156461c libcoreclr.so`Module::Initialize(this=0x0000000042224000, pamTracker=0x0000ffffffffccf0, szName=0x0000000000000000) at ceeload.cpp:443:63
440
441 #ifdef FEATURE_READYTORUN
442 m_pNativeImage = NULL;
-> 443 if ((m_pReadyToRunInfo = ReadyToRunInfo::Initialize(this, pamTracker)) != NULL)
444 {
445 m_pNativeImage = m_pReadyToRunInfo->GetNativeImage();
446 if (m_pNativeImage != NULL)
(lldb)
BEGIN: coreclr_initialize failed - Error: 0x80004005
time for bed...
edit: I think final fail is here:
Process 26961 stopped
* thread #1, name = 'corerun', stop reason = step over
frame #0: 0x000000004172268c libcoreclr.so`ReadyToRunInfo::Initialize(pModule=0x0000000042254000, pamTracker=0x0000ffffffffccf0) at readytoruninfo.cpp:561:9
558 if (!pLayout->IsNativeMachineFormat())
559 {
560 // For CoreCLR, be strict about disallowing machine mismatches.
-> 561 COMPlusThrowHR(COR_E_BADIMAGEFORMAT);
562 }
Is this something wrong with cross compiled runtime/sdk ?
It's not libcoreclr.so fault for sure
sec@amper:~/runtime % file artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so
artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.1, BuildID[sha1]=3d30f6f528036a298d342a15af8b3b1e47af6a09, stripped
sec@amper:~/runtime % ldd artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so
artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so:
libthr.so.3 => /lib/libthr.so.3 (0x4195e000)
librt.so.1 => /usr/lib/librt.so.1 (0x419ba000)
libunwind-aarch64.so.8 => /usr/local/lib/libunwind-aarch64.so.8 (0x419ef000)
libunwind.so.8 => /usr/local/lib/libunwind.so.8 (0x41a51000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x41aae000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x41ba9000)
libm.so.5 => /lib/libm.so.5 (0x41bf5000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x41c63000)
libc.so.7 => /lib/libc.so.7 (0x401e5000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x41ccc000)
libz.so.6 => /lib/libz.so.6 (0x41d21000)
libmd.so.6 => /lib/libmd.so.6 (0x41d68000)
So it's this one that's faulty?
sec@amper:~/runtime % file artifacts/bin/coreclr/FreeBSD.arm64.Debug/System.Private.CoreLib.dll
artifacts/bin/coreclr/FreeBSD.arm64.Debug/System.Private.CoreLib.dll: PE32+ executable (DLL) (console) Mono/.Net assembly, for MS Windows
When doing cross build Linux -> FreeBSD arm64, is this ok on Linux host - shouldn't this be "freebsd-arm64" by any chance?
./artifacts/bin/coreclr/FreeBSD.arm64.Debug/crossgen2/linux-arm64/System.Private.CoreLib.dll
./artifacts/bin/coreclr/FreeBSD.arm64.Debug/crossgen2/linux-arm64/publish/System.Private.CoreLib.dll
Checking Microsoft.NETCore.App.Crossgen2.freebsd-arm64.7.0.0-dev.nupkg there are Linux binaries inside:
sec@amper:~/newb/tools % file crossgen2
crossgen2: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=725302cc1678f04d8935457644ebd9deeb09f8ab, stripped
sec@amper:~/newb/tools % file dotnet
dotnet: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=eb185bf4cb8984561b845c57ea43484c3ed2b40a, stripped
sec@amper:~/newb/tools % file corerun
corerun: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, BuildID[sha1]=da7462690d3094983f41df3114505547a61835c8, with debug_info, not stripped
@Thefrank @janvorli @jkotas Maybe this is silly, but managed to get runtime/SDK to work (at some point at least). What I did was
- took Crossgen2 package built for FreeBSD-x64 (maybe any will work)
- took
System.Private.CoreLib.dllfrom cross compiled arm64 runtime/SDK - run
tools/crossgen2 System.Private.CoreLib.dll --targetarch arm64 --out System.Private.CoreLib.dll.2on it - replaced the original with new one
And
sec@amper:~/sdk7 % ./dotnet --info
.NET SDK:
Version: 7.0.100-preview.7.22353.99
Commit: be72bcc21b
Runtime Environment:
OS Name: FreeBSD
OS Version: 13
OS Platform: FreeBSD
RID: freebsd.13-arm64
Base Path: /usr/home/sec/sdk7/sdk/7.0.100-preview.7.22353.99/
Host:
Version: 7.0.0-preview.6.22329.5
Architecture: arm64
Commit: c9140d621a
.NET SDKs installed:
7.0.100-preview.7.22353.99 [/usr/home/sec/sdk7/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 7.0.0-preview.7.22351.1 [/usr/home/sec/sdk7/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.0-preview.6.22329.5 [/usr/home/sec/sdk7/shared/Microsoft.NETCore.App]
Run simple console app published from Windows host:
sec@amper:~/dotnet7 % ./dotnet ~/runtime/console_app/bin/Debug/net6.0/publish/console_app.dll
Hello, World!
But:
sec@amper:~/temp % ~/sdk7/dotnet new console
Segmentation fault (core dumped)
More info:
sec@amper:~/temp % lldb14 ~/sdk7/dotnet new console
(lldb) target create "/home/sec/sdk7/dotnet"
Current executable set to '/home/sec/sdk7/dotnet' (aarch64).
(lldb) settings set -- target.run-args "new" "console"
(lldb) r
Process 27236 launched: '/home/sec/sdk7/dotnet' (aarch64)
Process 27236 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x0000000041859f6c libunwind.so.8`___lldb_unnamed_symbol150 + 32
libunwind.so.8`___lldb_unnamed_symbol150:
-> 0x41859f6c <+32>: ldr x8, [x1]
0x41859f70 <+36>: mov w0, wzr
0x41859f74 <+40>: str x8, [x2]
0x41859f78 <+44>: ldr x19, [sp, #0x20]
(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
* frame #0: 0x0000000041859f6c libunwind.so.8`___lldb_unnamed_symbol150 + 32
frame #1: 0x000000004185af70 libunwind.so.8`_ULaarch64_step + 568
frame #2: 0x000000004178e8ec libcoreclr.so`___lldb_unnamed_symbol14794 + 256
frame #3: 0x00000000415b29ec libcoreclr.so`___lldb_unnamed_symbol7901 + 256
frame #4: 0x00000000413f75a0 libcoreclr.so`___lldb_unnamed_symbol2153 + 136
frame #5: 0x00000000413f7508 libcoreclr.so`___lldb_unnamed_symbol2152 + 28
frame #6: 0x00000000415b8400 libcoreclr.so`___lldb_unnamed_symbol8017 + 84
frame #7: 0x00000000415b7db0 libcoreclr.so`___lldb_unnamed_symbol8013 + 752
frame #8: 0x00000000415b7198 libcoreclr.so`___lldb_unnamed_symbol8011 + 1108
frame #9: 0x00000000415bb444 libcoreclr.so`___lldb_unnamed_symbol8050 + 336
frame #10: 0x00000000415bb71c libcoreclr.so`___lldb_unnamed_symbol8051 + 68
frame #11: 0x0000000041518964 libcoreclr.so`___lldb_unnamed_symbol6091 + 540
@Thefrank makes sense to build SDK in Debug to see where/why it's SIGSEGV or this is easy to fix now? :)
Crossgen2 packages, AFAIK, are no longer generated for FreeBSD as that ability got removed somewhere after net7p3. This is not ideal as those packages are needed for native builds (restore requires it)
Crossbuilt net7p5 for FreeBSD x64 with the package added back in looks like it generates incorrectly too:
Shipping/Microsoft.NETCore.App.Crossgen2.freebsd-x64.7.0.0-preview.5.22301.12/tools # ls | xargs file
coreclr_delegates.h: C source, ASCII text
corerun: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.3, FreeBSD-style, BuildID[sha1]=ea37edd26732ab0d86c7eff56bfd2dc4b97d7de5, with debug_info, not stripped
crossgen2: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.3, FreeBSD-style, BuildID[sha1]=f2abf1f880cce1b944515bb44ccf8778bd1e0d4d, stripped
dotnet: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=342e6dc49a1ef7d664ecf66554bd91f12377deb8, stripped
hostfxr.h: C source, ASCII text
libclrjit_universal_arm_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=5371055af15bb23ddab55f6d56ea05e68c4ec014, stripped
libclrjit_universal_arm64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=df1c176a5aedfe127ee1097723e5a9dd1ca0c9d6, stripped
libclrjit_unix_x64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=c0d6411b4d9dc86718543182049e390bebcc75c0, stripped
libclrjit_win_x64_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=c8498128119ad328a63433b0e113bf35a28b67b8, stripped
libclrjit_win_x86_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e1ef1e6c1d44175ebe3f2dd1ed7a22bae89dd3ea, stripped
libclrjit.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=b3050876013379e1abd2c8ab67a3b649c901550c, stripped
libcoreclr.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=beb05fea73aadf25e83d75f7cb06a638becb31bd, stripped
libhostfxr.a: current ar archive
libhostfxr.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=955d0bd7775d52bc8aba0124aa2e6f7b0e059ae1, stripped
libhostpolicy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=bd4f445a772be690487d98d63be8aa0d7013dad0, stripped
libjitinterface_x64.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=7aca7857d900236e5fd0925419afd2a1510541f5, stripped
libmscordaccore.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=fae13f6b4f0a887250f6e2f9bbfa986c0b493edd, stripped
libmscordbi.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e07ae986afb11b697f758994cb3dd069fe6cd801, stripped
libnethost.a: current ar archive
libnethost.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb0e1694b1521c04c74755a2210c8b8988298f81, stripped
libSystem.Globalization.Native.a: current ar archive
libSystem.Globalization.Native.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=411160e3995dfc8aa04d8929dc5958ae61e4d6c4, stripped
libSystem.IO.Compression.Native.a: current ar archive
libSystem.IO.Compression.Native.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e539b4dd805290e3a19c8551ab766cc7d7d0bafd, stripped
libSystem.IO.Ports.Native.a: current ar archive
libSystem.IO.Ports.Native.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=d41c2ed5987fd9f7ffe8f03c853673dc24f8a397, stripped
libSystem.Native.a: current ar archive
libSystem.Native.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e732c3d8b2d37c4584bcd8470f018a1493394bc0, stripped
libSystem.Net.Security.Native.a: current ar archive
libSystem.Net.Security.Native.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=3ecfe261dd743c77c13bc8ed53a811d1e0c4c6c3, stripped
libSystem.Security.Cryptography.Native.OpenSsl.a: current ar archive
libSystem.Security.Cryptography.Native.OpenSsl.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=b3f59bd1083874d1c0db7fc528b3c8452d11be21, stripped
nethost.h: C source, ASCII text
singlefilehost: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=7d2abe4df7dfc9c4f875865784d606267bc42c77, stripped,
net6RTM (the last crossbuild from that branch I have done) only contains one executable: crossgen2 and that is correctly FreeBSD
EDIT: did a quick crossbuild from Linux x86_64 -> FreeBSD ARM64. The contents of the Crossgen2 nupkg (and tarball) are just as odd: everything but dotnet and singlefilehost are FreeBSD ARM aarch64. Those two are Linux ARM aarch64
Broken System.Private.CoreLib those make it work OOTB (for runtime part) - 951c92a and 98d39cd
As for SDK to fail somewhere inside libunwind - @Thefrank would you mind doing SDK build in Debug or share the steps I can reproduce to crossbuild it myself (so I won't bug you) ? :)
As for those Linux files inside packages - that's weird, becuase there is FreeBSD version of singlefilehost and dotnet produced during build, but the Linux one is being packed inside - all of the below have their correct FreeBSD, but are inside dotnet-crossgen2-7.0.0-dev-freebsd-arm64.tar.gz:
./singlefilehost: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=000c66b0f0a90e33362b538857a1fe07858d95d2, stripped
./libhostpolicy.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f7f923736c5fa480faec666385f8ae7cb6191905, stripped
./libnethost.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ec8c97ecfbcf5bb77d4fa8a5fcb3e56cb7bc13e4, stripped
./dotnet: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=eb185bf4cb8984561b845c57ea43484c3ed2b40a, stripped
./libhostfxr.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=827719e673ae0b16ade2b80907cf2c39e6ae517a, stripped
At least crossgen2 inside is FreeBSD, bad news it doesn't work:
Assert failure(PID 42015 [0x0000a41f], Thread: 105128 [0x19aa8]): pThread
File: /home/ubuntu/runtime/src/coreclr/vm/threads.inl Line: 43
Image: /usr/home/sec/temp/cros/crossgen2
Abort (core dumped)
@sec https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/7.0.100-preview.7.22355.99-freebsd-arm64
I don't mind making the SDKs. This also reminds me that I should update my build script (from the above repo) to allow for arm64 builds.
For platforms like FreeBSD, the installer branch is basically a fancy fetch+sed+tar... unless you want to generate native binaries...which we don't under a crossbuild, as afaik, does not work for FreeBSD.
@Thefrank Thanks, but can't find libcoreclr.so.dbg anywhere :) (my guess it's not packed into SDK) - so the backtrace is still unknown.
Maybe someone more expierenced in this topic could help to debug the issue (something with threads, as crossgen2 fail in there) ? :)
@Thefrank Nope, libcoreclr.so.dbg is still not there:
sec@amper:~/sdk % tar --list --file dotnet-runtime-7.0.0-preview.6.22329.5-freebsd-arm64.tar.gz|grep libcoreclr
./shared/Microsoft.NETCore.App/7.0.0-preview.6.22329.5/libcoreclr.so
I don't think *.so.dbg are packed into tar's
I linked the wrong one dotnet-runtime-symbols-freebsd-arm64-7.0.0-preview.6.22329.5.tar.gz needs to say "symbols"
EDIT:
@sec on a different note: do you have any suggestions on cheap arm64 hardware that freebsd runs on?
sigh
Ok, so compiled libunwind in DEBUG also:
Process 38997 launched: '/home/sec/sdk7/dotnet' (aarch64)
Process 38997 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x00000000412f9f74 libunwind.so.8`access_mem(as=0x000000004132c690, addr=0, val=0x0000ffffffff89f0, write=0, arg=0x0000ffffffff8aa0) at Ginit.c:345:14
342 Debug (16, "mem[%016lx] -> invalid\n", addr);
343 return -1;
344 }
-> 345 *val = *(unw_word_t *) addr;
^
346 Debug (16, "mem[%lx] -> %lx\n", addr, *val);
347 }
348 return 0;
(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
* frame #0: 0x00000000412f9f74 libunwind.so.8`access_mem(as=0x000000004132c690, addr=0, val=0x0000ffffffff89f0, write=0, arg=0x0000ffffffff8aa0) at Ginit.c:345:14
frame #1: 0x00000000412fc5d8 libunwind.so.8`is_plt_entry(c=0x0000ffffffff8aa0) at Gstep.c:43:14
frame #2: 0x00000000412fbda8 libunwind.so.8`_ULaarch64_step(cursor=0x0000ffffffff8aa0) at Gstep.c:171:11
frame #3: 0x0000000041eedbec libcoreclr.so`::PAL_VirtualUnwind(context=0x0000ffffffff9750, contextPointers=0x0000ffffffff96b0) at seh-unwind.cpp:662:10
frame #4: 0x0000000041b71cfc libcoreclr.so`LazyMachState::unwindLazyState(baseState=0x0000ffffffffbd90, unwoundstate=0x0000ffffffff9b60, threadId=100776, funCallDepth=0, hostCallPreference=AllowHostCalls) at stubs.cpp:340:24
frame #5: 0x000000004181b2ec libcoreclr.so`HelperMethodFrame::InsureInit(this=0x0000ffffffffbd58, initialInit=false, unwindState=0x0000000000000000, hostCallPreference=AllowHostCalls) at frames.cpp:1813:9
frame #6: 0x000000004181b180 libcoreclr.so`HelperMethodFrame::GetFunction(this=0x0000ffffffffbd58) at frames.cpp:1732:5
frame #7: 0x0000000041b7e538 libcoreclr.so`ExceptionTracker::InitializeCrawlFrameForExplicitFrame(pcfThisFrame=0x0000ffffffffa840, pFrame=0x0000ffffffffbd58, pMD=0x00000000444478d8) at exceptionhandling.cpp:1319:35
frame #8: 0x0000000041b7dd98 libcoreclr.so`ExceptionTracker::ProcessOSExceptionNotification(this=0x0000000040d77020, pExceptionRecord=0x0000000040cf9c90, pContextRecord=0x0000000040cf9900, pDispatcherContext=0x0000ffffffffb310, dwExceptionFlags=0, sf=(SP = 281474976694080), pThread=0x0000000040c25010, STState=STS_NewException, pICFSetAsLimitFrame=0x0000000000000000) at exceptionhandling.cpp:1853:17
frame #9: 0x0000000041b7c780 libcoreclr.so`::ProcessCLRException(pExceptionRecord=0x0000000040cf9c90, MemoryStackFp=281474976694080, pContextRecord=0x0000000040cf9900, pDispatcherContext=0x0000ffffffffb310) at exceptionhandling.cpp:1062:28
frame #10: 0x0000000041b8342c libcoreclr.so`UnwindManagedExceptionPass1(ex=0x0000ffffffffbd20, frameContext=0x0000ffffffffb7f0) at exceptionhandling.cpp:4620:27
frame #11: 0x0000000041b83bec libcoreclr.so`DispatchManagedException(ex=0x0000ffffffffbd20, isHardwareException=false) at exceptionhandling.cpp:4806:17
frame #12: 0x0000000041a50938 libcoreclr.so`IL_Throw(obj=<unavailable>) at jithelpers.cpp:4022:5
Does this make any sense to anyone? :)
@janvorli Sorry to bother you (again), I'm trying to debug SDK crash issue and was reading about registers in arm64 - I may be wrong, but I think we might have the mapping done wrong:
#define MCREG_Cpsr(mc) (mc.mc_fpregs.fp_sr) - I read that CPSR (Current Program Status Register) is gone in aarch64 - and fp_sr is Floating Point Status Register
#define MCREG_Pc(mc) (mc.mc_gpregs.gp_elr) -elr is for Link Register, returning from exceptions from arm64 docs:
The Program Counter (PC) is not a general-purpose register in A64, and it cannot be used with data processing instructions. The PC can be read using:
ADR Xd, .
Here's the source I used - this and that - my knowledge about asm and aarch64 is almost zero, so that's a wild about why SDK could be crashing at exception handling...
Too bad that tests cannot be built and run without working SDK for current host :/
edit 1: looking at llvm source code, looks like elr is in fact pc
edit 2: Simple app with
Console.WriteLine("Hello, World!");
throw new Exception("Error throw by user code");when run with runtime fails exactly the same as SDK:
sec@amper:~/new_runtime % lldb ./dotnet ~/sample_app/console_app.dll
(lldb) target create "./dotnet"
Current executable set to '/home/sec/new_runtime/dotnet' (aarch64).
(lldb) settings set -- target.run-args "/home/sec/sample_app/console_app.dll"
(lldb) r
Process 88663 launched: '/home/sec/new_runtime/dotnet' (aarch64)
Hello, World!
Process 88663 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x00000000412f1f74 libunwind.so.8`access_mem(as=0x0000000041324690, addr=0, val=0x0000ffffffff9ba0, write=0, arg=0x0000ffffffff9c50) at Ginit.c:345:14
342 Debug (16, "mem[%016lx] -> invalid\n", addr);
343 return -1;
344 }
-> 345 *val = *(unw_word_t *) addr;
346 Debug (16, "mem[%lx] -> %lx\n", addr, *val);
347 }
348 return 0;
so that's something 100% connected with exception handling.
edit 3: fixed Spsr/Cpsr mapping in define's btw.
@sec I am sorry for a late response, I had vacation this week. As for the crash in the libunwind, can you please print the contents of the context argument at frame 3? That might give us a clue on where the 0 address stems from.
@janvorli No problem, hope you had a good break :) Here's the context (also note that Cpsr was previously mapped to fp_sr and I changed that to spsr from the ucontext structure):
frame #0: 0x0000000041ceddf8 libcoreclr.so`::PAL_VirtualUnwind(context=0x0000000040c4cc00, contextPointers=0x0000000000000000) at seh-unwind.cpp:662:10
...
(lldb) frame variable *context
(CONTEXT) *context = {
ContextFlags = 0
Cpsr = 65535
= {
= {
X0 = 1096611716
X1 = 1086869072
X2 = 4294967296
X3 = 281474976672648
X4 = 281474976672648
X5 = 281474976672400
X6 = 1097764784
X7 = 1086869072
X8 = 281474976672648
X9 = 281474976672432
X10 = 281474976672560
X11 = 1094257952
X12 = 0
X13 = 281474976672560
X14 = 281474976672560
X15 = 1104768280
X16 = 7115946263732781519
X17 = 281474976672512
X18 = 281474976672560
X19 = 0
X20 = 0
X21 = 0
X22 = 0
X23 = 281474976689496
X24 = 0
X25 = 0
X26 = 1134684072
X27 = 281474976690072
X28 = 2223034368
}
X = {
[0] = 1096611716
[1] = 1086869072
[2] = 4294967296
[3] = 281474976672648
[4] = 281474976672648
[5] = 281474976672400
[6] = 1097764784
[7] = 1086869072
[8] = 281474976672648
[9] = 281474976672432
[10] = 281474976672560
[11] = 1094257952
[12] = 0
[13] = 281474976672560
[14] = 281474976672560
[15] = 1104768280
[16] = 7115946263732781519
[17] = 281474976672512
[18] = 281474976672560
[19] = 0
[20] = 0
[21] = 0
[22] = 0
[23] = 281474976689496
[24] = 0
[25] = 0
[26] = 1134684072
[27] = 281474976690072
[28] = 2223034368
}
}
Fp = 281474976682224
Lr = 0
Sp = 281474976681632
Pc = 1099238688
V = {
[0] = (Low = 1075273936, High = 567518940)
[1] = (Low = 281474976672960, High = 281474976677216)
[2] = (Low = 5370241232, High = 1086488592)
[3] = (Low = 281474976672688, High = 1086488604)
[4] = (Low = 1086488592, High = 1086488592)
[5] = (Low = 8589896656, High = 1086488592)
[6] = (Low = 281474976672736, High = 1096069328)
[7] = (Low = 8589896896, High = 4294967296)
[8] = (Low = 281474976672968, High = 281474976672968)
[9] = (Low = 281474976672768, High = 1097002388)
[10] = (Low = 555955440, High = 281474976672968)
[11] = (Low = 281474976672992, High = 1096998048)
[12] = (Low = 1094257944, High = 567518940)
[13] = (Low = 0, High = 1)
[14] = (Low = 4294967296, High = 70368744177669)
[15] = (Low = 281474976673168, High = 1100453408)
[16] = (Low = 281474976673008, High = 281474976673128)
[17] = (Low = 1086857248, High = 1122775108)
[18] = (Low = 281474976675088, High = -545460846624)
[19] = (Low = 281474976684768, High = 0)
[20] = (Low = 1098430868, High = 0)
[21] = (Low = 9232388042942056450, High = 0)
[22] = (Low = 0, High = 0)
[23] = (Low = 0, High = 0)
[24] = (Low = 281474976673136, High = 1095960552)
[25] = (Low = 281474976698960, High = 281474976699184)
[26] = (Low = 1093749214, High = 0)
[27] = (Low = 281474976673256, High = 1105270056)
[28] = (Low = 4294967296, High = 70368744177669)
[29] = (Low = 281474976673392, High = 1100453408)
[30] = (Low = 281474976673232, High = 281474976673352)
[31] = (Low = 1086857248, High = 1122775108)
}
Fpcr = 4294931728
Fpsr = 65535
Bcr = ([0] = 4294967264, [1] = 4294967168, [2] = 4294941408, [3] = 65535, [4] = 4294929912, [5] = 65535, [6] = 1098430868, [7] = 0)
Bvr = ([0] = 0, [1] = 281474976673272, [2] = 281474976673272, [3] = 1104768280, [4] = 7115946263732781519, [5] = 281474976673232, [6] = 281474976673272, [7] = 281474976673272)
Wcr = ([0] = 1104785048, [1] = 0)
Wvr = ([0] = 2, [1] = 8589897208)
}
edit: this is first hit on breakpoint, I need to figure out how to print context from crash, gimme few minutes :)
edit 2: ok done :)
Ok, can you please also print the contents of the data structure pointed to by the "c" argument at frame 1?
Sure, hope that magic numbers make any sense :)
(lldb) frame variable *c
(dwarf_cursor) *c = {
as_arg = 0x0000ffffffff5d90
as = 0x000000004096f690
cfa = 281474976681632
ip = 0
args_size = 0
eh_args = ([0] = 281474976669320, [1] = 281474976669320, [2] = 4294926000, [3] = 281474976669360)
eh_valid_mask = 4294926000
loc = {
[0] = (val = 281474976671088)
[1] = (val = 281474976671096)
[2] = (val = 281474976671104)
[3] = (val = 281474976671112)
[4] = (val = 281474976671120)
[5] = (val = 281474976671128)
[6] = (val = 281474976671136)
[7] = (val = 281474976671144)
[8] = (val = 281474976671152)
[9] = (val = 281474976671160)
[10] = (val = 281474976671168)
[11] = (val = 281474976671176)
[12] = (val = 281474976671184)
[13] = (val = 281474976671192)
[14] = (val = 281474976671200)
[15] = (val = 281474976671208)
[16] = (val = 281474976671216)
[17] = (val = 281474976671224)
[18] = (val = 281474976671232)
[19] = (val = 281474976671240)
[20] = (val = 281474976671248)
[21] = (val = 281474976671256)
[22] = (val = 281474976671264)
[23] = (val = 281474976671272)
[24] = (val = 281474976671280)
[25] = (val = 281474976671288)
[26] = (val = 281474976671296)
[27] = (val = 281474976671304)
[28] = (val = 281474976671312)
[29] = (val = 281474976671320)
[30] = (val = 281474976671328)
[31] = (val = 281474976671336)
[32] = (val = 281474976671344)
[33] = (val = 281474976671352)
[34] = (val = 0)
[35] = (val = 0)
[36] = (val = 0)
[37] = (val = 0)
[38] = (val = 0)
[39] = (val = 0)
[40] = (val = 0)
[41] = (val = 0)
[42] = (val = 0)
[43] = (val = 0)
[44] = (val = 0)
[45] = (val = 0)
[46] = (val = 0)
[47] = (val = 0)
[48] = (val = 0)
[49] = (val = 0)
[50] = (val = 0)
[51] = (val = 0)
[52] = (val = 0)
[53] = (val = 0)
[54] = (val = 0)
[55] = (val = 0)
[56] = (val = 0)
[57] = (val = 0)
[58] = (val = 0)
[59] = (val = 0)
[60] = (val = 0)
[61] = (val = 0)
[62] = (val = 0)
[63] = (val = 0)
[64] = (val = 281474976672184)
[65] = (val = 281474976672200)
[66] = (val = 281474976672216)
[67] = (val = 281474976672232)
[68] = (val = 281474976672248)
[69] = (val = 281474976672264)
[70] = (val = 281474976672280)
[71] = (val = 281474976672296)
[72] = (val = 281474976672312)
[73] = (val = 281474976672328)
[74] = (val = 281474976672344)
[75] = (val = 281474976672360)
[76] = (val = 281474976672376)
[77] = (val = 281474976672392)
[78] = (val = 281474976672408)
[79] = (val = 281474976672424)
[80] = (val = 281474976672440)
[81] = (val = 281474976672456)
[82] = (val = 281474976672472)
[83] = (val = 281474976672488)
[84] = (val = 281474976672504)
[85] = (val = 281474976672520)
[86] = (val = 281474976672536)
[87] = (val = 281474976672552)
[88] = (val = 281474976672568)
[89] = (val = 281474976672584)
[90] = (val = 281474976672600)
[91] = (val = 281474976672616)
[92] = (val = 281474976672632)
[93] = (val = 281474976672648)
[94] = (val = 281474976672664)
[95] = (val = 281474976672680)
[96] = (val = 281474976669984)
}
stash_frames = 0
use_prev_instr = 1
pi_valid = 0
pi_is_dynamic = 0
pi = {
start_ip = 0
end_ip = 0
lsda = 0
handler = 0
gp = 0
flags = 0
format = 0
unwind_info_size = 0
unwind_info = 0x0000000000000000
extra = {}
}
hint = 0
prev_rs = 0
}
It is as I've thought. The ip member is 0 and doesn't reflect the 'Pcmember of theCONTEXT` struct that you've printed above. I believe it is the same issue as #68640. It is caused by a bug in the libunwind that incorrectly assumed that memory page size is always 4kB. On some Linux arm64 distros, it is 64kB. Can you please check the page size on your arm64 device?
Hm, looking at those, it looks like it's 4kB (this also confirm this https://wiki.freebsd.org/Memory):
sec@amper:~ % sysctl -a | grep pagesize
hw.pagesize: 4096
hw.pagesizes: { 4096, 2097152, 1073741824 }
p1003_1b.pagesize: 4096
libunwind is 1.6.2
Ok, so it is a different issue then. Let me take a look at the source of the PAL_VirtualUnwind, the initialization of the unwind context has several flavors, so maybe FreeBSD on arm64 is taking a wrong one there.
Can you please also dump the unwContext local at frame 3?
Sure, sorry my arm64 machine crashed :D had to reboot:
(lldb) print unwContext
(unw_context_t) $0 = {
uc_sigmask = {
__bits = ([0] = 4294927808, [1] = 65535, [2] = 1096182416, [3] = 0)
}
uc_mcontext = {
mc_gpregs = {
gp_x = {
[0] = 1096611716
[1] = 1086844448
[2] = 4294967296
[3] = 281474976672696
[4] = 281474976672696
[5] = 281474976672448
[6] = 1097764784
[7] = 1086844448
[8] = 281474976672696
[9] = 281474976672480
[10] = 281474976672608
[11] = 1094257952
[12] = 0
[13] = 281474976672608
[14] = 281474976672608
[15] = 1104768280
[16] = 4941924610348228111
[17] = 281474976672560
[18] = 281474976672608
[19] = 0
[20] = 0
[21] = 0
[22] = 0
[23] = 281474976689544
[24] = 0
[25] = 0
[26] = 1134487464
[27] = 281474976690120
[28] = 2223034368
[29] = 281474976682272
}
gp_lr = 0
gp_sp = 281474976681680
gp_elr = 1099238688
gp_spsr = 65535
gp_pad = -1
}
mc_fpregs = {
fp_q = {
[0] = 340282366920938463444927863359193545600
[1] = 91162418518660561966063991681896677376
[2] = 19986546559230420173681814368
[3] = 340282366842028248439668917311171985408
[4] = 1082450696
[5] = 20390428649778019886885240832
[6] = 19967690961520779449679740928
[7] = 20277386319564797028421599232
[8] = 1083635344
[9] = 5192296857817470644992079285977088
[10] = 5192296857817470644992087875911679
[11] = 20216902046738299275603568576
[12] = 3361175567287734928651563642085584
[13] = 20370631235103514796876588960
[14] = 3361195622772802028552419004082200
[15] = 20319681623119834194666022848
[16] = 20056267946918389089807853632
[17] = 20216886330112348475065591920
[18] = 20056267946918389089807853600
[19] = 5192296857817470644992156595388417
[20] = 5192296857825144490808217436121152
[21] = 20056268094492060230277398529
[22] = 20370448686124161367153797312
[23] = 163505428234035994541512652067176448003
[24] = 1846980251268587303469058
[25] = 18446744073709551632
[26] = 19967083620918896635309719552
[27] = 934176914327794729732800605
[28] = 20385497466152235853842219007
[29] = 20390395445638687215066617920
[30] = 20390395445638687210767605968
[31] = 20219001360000863717415676096
}
fp_sr = 1079318592
fp_cr = 0
fp_flags = 1105365552
fp_pad = 0
}
mc_flags = -38544
mc_pad = 65535
mc_spare = ([0] = 1104284236, [1] = 281474976672128, [2] = 1104284236, [3] = 3, [4] = 8863647025225728000, [5] = 281474976672160, [6] = 1104284236, [7] = 3)
}
uc_link = 0x000328030a000002
uc_stack = (ss_sp = 0x000000000001871d, ss_size = 1, ss_flags = 1105185552)
uc_flags = 1
__spare__ = ([0] = 0, [1] = 0, [2] = 0, [3] = 93)
}
@janvorli I've looked at ports and there is some custom patching for libunwind for arm64 - maybe I will try to apply the same for sources inside runtime and rebuild/check with that...
@sec that would be good to try before we dive deeper. From your last dump, I can see the gp_elr matches the Pc from the CONTEXT and that means that the code that copies the context works correctly. Then the problem likely happens in the unw_init_local call where the cursor is initialized. The code is a bit difficult to follow in sources as it used plenty of #defined macros. So I always end up stepping through the code when there is an issue there.
Applied the patch into runtime sources, rebuild - still the same crash (which would make sense, as it's linked to OS libunwind anyway). I tried to step that unw_init_local, but there are a few assign lines done and then it goes to -> 662 st = unw_step(&cursor); and crash on this...
I only did src/coreclr/build-runtime.sh && src/native/libs/build-native.sh and took System.Private.CoreLib.dl, System.Runtime.dll, System.Console.dll, System.Threading.dll from SDK built by @Thefrank (without libunwind fixes, but those are managed bits right) needed for corerun to execute and debug. Combined all that into bin and run :)
Thanks for your time and help, but I think I have no clue what to look for further - unless there's something more to try that comes to your mind :) Anyway, If/when you will have some spare time and will, I can grant you access to this box - just drop me an email (it's on my gh page) and we can work out the details.
Applied the patch into runtime sources
@sec, in which file did you apply that patch? Curious because you are using "system libunwind" and not src/native/external/libunwind.
@am11 Patch was taken from ports (so OS is using that) - I applied that to src/native/external/libunwind but that changed nothing, as I assume it's using the OS level one :)
Yes, I was suspecting it and it's probably libunwind/libunwind@d3583ed. On FreeBSD, we are not using HP libunwind. The OS libunwind in FreeBSD (and macOS) is a fork of llvm-libunwind, which has the same API but the rest of the implementation is very different.
From the stack trace above, it doesn't seem to be the case. It is referencing libunwind.8, which is the HP libunwind. And the internal functions on the stack trace match the ones in the HP libunwind, not the ones in the llvm libunwind.
It is linking to /usr/local/lib/libunwind.so.8 with CLR_CMAKE_USE_SYSTEM_LIBUNWIND=1. Applying a patch in tree won't have any effect. If FreeBSD indeed uses HP libunwind system-wide, then I think it is better to unset CLR_CMAKE_USE_SYSTEM_LIBUNWIND and use the in-tree copy with patches applied.
it is better to unset
CLR_CMAKE_USE_SYSTEM_LIBUNWINDand use the in-tree copy with patches applied.
It would be worth trying to see if it fixes the problem.
@am11 I think that it's the same - the version in ports it's fetching https://github.com/libunwind/libunwind/releases/download/v1.6.2/libunwind-1.6.2.tar.gz and applying that patch for arm64.
I will try with CLR_CMAKE_USE_SYSTEM_LIBUNWIND unset, rebuilt and check.
edit: try with in-tree unwind, with patch from ports applied (all hunk went ok):
Process 25067 launched: '/home/sec/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug/corerun' (aarch64)
Unhandled exception. Process 25067 stopped
* thread #1, name = 'corerun', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x0000000041e193d4 libcoreclr.so`access_mem(as=0x0000000041ef79f8, addr=0, val=0x0000ffffffff5f00, write=0, arg=0x0000ffffffff5fe0) at Ginit.c:342:14
339 Debug (16, "mem[%016lx] -> invalid\n", addr);
340 return -1;
341 }
-> 342 *val = *(unw_word_t *) addr;
^
343 Debug (16, "mem[%lx] -> %lx\n", addr, *val);
344 }
345 return 0;
(lldb) bt
* thread #1, name = 'corerun', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
* frame #0: 0x0000000041e193d4 libcoreclr.so`access_mem(as=0x0000000041ef79f8, addr=0, val=0x0000ffffffff5f00, write=0, arg=0x0000ffffffff5fe0) at Ginit.c:342:14
frame #1: 0x0000000041e1c954 libcoreclr.so`is_plt_entry(c=0x0000ffffffff5fe0) at Gstep.c:43:14
frame #2: 0x0000000041e1be6c libcoreclr.so`_ULaarch64_step(cursor=0x0000ffffffff5fe0) at Gstep.c:171:11
frame #3: 0x0000000041d782e8 libcoreclr.so`::PAL_VirtualUnwind(context=0x0000ffffffff6c80, contextPointers=0x0000ffffffff6be0) at seh-unwind.cpp:662:10
frame #4: 0x00000000419b0fe0 libcoreclr.so`LazyMachState::unwindLazyState(baseState=0x0000ffffffff91c0, unwoundstate=0x0000ffffffff7090, threadId=100475, funCallDepth=0, hostCallPreference=AllowHostCalls) at stubs.cpp:340:24
frame #5: 0x0000000041620880 libcoreclr.so`HelperMethodFrame::InsureInit(this=0x0000ffffffff9188, initialInit=false, unwindState=0x0000000000000000, hostCallPreference=AllowHostCalls) at frames.cpp:1813:9
frame #6: 0x00000000416206e0 libcoreclr.so`HelperMethodFrame::GetFunction(this=0x0000ffffffff9188) at frames.cpp:1732:5
frame #7: 0x00000000419be658 libcoreclr.so`ExceptionTracker::InitializeCrawlFrameForExplicitFrame(pcfThisFrame=0x0000ffffffff7d50, pFrame=0x0000ffffffff9188, pMD=0x0000000043a64db8) at exceptionhandling.cpp:1319:35
frame #8: 0x00000000419bde2c libcoreclr.so`ExceptionTracker::ProcessOSExceptionNotification(this=0x0000000040d02330, pExceptionRecord=0x0000000040d0d290, pContextRecord=0x0000000040d0cf00, pDispatcherContext=0x0000ffffffff87c0, dwExceptionFlags=0, sf=(SP = 281474976682784), pThread=0x0000000040c28010, STState=STS_FirstRethrowFrame, pICFSetAsLimitFrame=0x0000000000000000) at exceptionhandling.cpp:1853:17
frame #9: 0x00000000419bc5f0 libcoreclr.so`::ProcessCLRException(pExceptionRecord=0x0000000040d0d290, MemoryStackFp=281474976682784, pContextRecord=0x0000000040d0cf00, pDispatcherContext=0x0000ffffffff87c0) at exceptionhandling.cpp:1062:28
frame #10: 0x00000000419c3b98 libcoreclr.so`UnwindManagedExceptionPass1(ex=0x0000ffffffff9150, frameContext=0x0000ffffffff8ca0) at exceptionhandling.cpp:4620:27
frame #11: 0x00000000419c43f0 libcoreclr.so`DispatchManagedException(ex=0x0000ffffffff9150, isHardwareException=false) at exceptionhandling.cpp:4806:17
frame #12: 0x000000004187b844 libcoreclr.so`IL_Rethrow() at jithelpers.cpp:4048:5
edit 2: in-tree unwind without patch, don't compile
Tried libunwind with commit from libunwind/libunwind@0c44849 - same result :(
edit 2: in-tree unwind without patch, don't compile
It is expected. We keep track of patches we need (on top of the version we have in-tree) here https://github.com/dotnet/runtime/blob/main/src/native/external/libunwind-version.txt. Currently we are on latest release v1.6.2, but we still have a few patches from libunwind's master branch. You can apply more patches in similar way (keep track of upstream references).
Ok, thanks @sec for trying that. Then I think we should do source level debugging, stepping through the PAL_VirtualUnwind and its callees to see why the ip in the cursor is set to 0. The src\native\external\libunwind\src\aarch64\init.h. It is being set in the common_init function, the c->dwarf is what you've dumped above as dwarf_cursor. If you want me to try to debug that, please let me know.
@janvorli I tried to debug this, but that's beyond me - if you're up for some lldb fun, I will gladly grant you access to this machine to debug :)
some unwind debug log level 1:
>_ULaarch64_init_mem_validate: using msync to validate memory
>unw_init_local_common: (cursor=0xffffffff6030)
>_ULaarch64_step: (cursor=0xffffffff6030, ip=0x0000000041d78b80, cfa=0x0000ffffffff6bf0))
>_ULaarch64_step: Invalid address found in the call stack: 0x41d78b80
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 15a0e8, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414ded90
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffba80)
>_ULaarch64_step: (cursor=0xffffffffba80, ip=0x00000000419c4348, cfa=0x0000ffffffffc670))
>_ULaarch64_step: Invalid address found in the call stack: 0x419c4348
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 186818, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 4150b4c0
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffba80)
>_ULaarch64_step: (cursor=0xffffffffba80, ip=0x00000000416ec24c, cfa=0x0000ffffffffcae0))
>_ULaarch64_step: Invalid address found in the call stack: 0x416ec24c
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 176e40, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414fbae8
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffba80)
>_ULaarch64_step: (cursor=0xffffffffba80, ip=0x0000000041ad6f78, cfa=0x0000ffffffffcd60))
>_ULaarch64_step: Invalid address found in the call stack: 0x41ad6f78
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 10e6c8, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 41493370
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x0000000041ad7a38, cfa=0x0000ffffffffcfa0))
>_ULaarch64_step: Invalid address found in the call stack: 0x41ad7a38
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 10ec98, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 41493940
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x00000000417a8d8c, cfa=0x0000ffffffffcfc0))
>_ULaarch64_step: Invalid address found in the call stack: 0x417a8d8c
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 17aa68, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414ff710
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x00000000417a9890, cfa=0x0000ffffffffd030))
>_ULaarch64_step: Invalid address found in the call stack: 0x417a9890
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = b8160, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 4143ce08
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x00000000415400a0, cfa=0x0000ffffffffd3c0))
>_ULaarch64_step: Invalid address found in the call stack: 0x415400a0
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 566e8, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 413db390
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004155ce18, cfa=0x0000ffffffffd3e0))
>_ULaarch64_step: Invalid address found in the call stack: 0x4155ce18
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 16eab0, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f3758
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004155cbc8, cfa=0x0000ffffffffd5a0))
>_ULaarch64_step: Invalid address found in the call stack: 0x4155cbc8
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 5f508, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 413e41b0
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004155aa08, cfa=0x0000ffffffffd5c0))
>_ULaarch64_step: Invalid address found in the call stack: 0x4155aa08
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 16e7b8, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f3460
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004155a840, cfa=0x0000ffffffffd650))
>_ULaarch64_step: Invalid address found in the call stack: 0x4155a840
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 16e790, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f3438
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004155accc, cfa=0x0000ffffffffd740))
>_ULaarch64_step: Invalid address found in the call stack: 0x4155accc
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 16e7e0, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f3488
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x00000000415cb810, cfa=0x0000ffffffffda70))
>_ULaarch64_step: Invalid address found in the call stack: 0x415cb810
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 170ae0, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f5788
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000004152a6b4, cfa=0x0000ffffffffdd20))
>_ULaarch64_step: Invalid address found in the call stack: 0x4152a6b4
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 16cf58, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 414f1c00
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x0000000000124b68, cfa=0x0000ffffffffde10))
>_ULaarch64_step: Invalid address found in the call stack: 0x124b68
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 8d0c, segbase = 108844, debug_frame_base = 0, fde_addr = 111550
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x0000000000122838, cfa=0x0000ffffffffe3c0))
>_ULaarch64_step: Invalid address found in the call stack: 0x122838
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 8c6c, segbase = 108844, debug_frame_base = 0, fde_addr = 1114b0
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffffb3e0)
>_ULaarch64_step: (cursor=0xffffffffb3e0, ip=0x000000000012253c, cfa=0x0000ffffffffe4e0))
>_ULaarch64_step: Invalid address found in the call stack: 0x12253c
>_ULaarch64_dwarf_search_unwind_table: IP 12253b inside range 12242c-13fe20, but no explicit unwind info found
>_ULaarch64_step: dwarf_step()=-10
Unhandled exception. >unw_init_local_common: (cursor=0xffffffff97e0)
>_ULaarch64_step: (cursor=0xffffffff97e0, ip=0x0000000041d78b80, cfa=0x0000ffffffffa3a0))
>_ULaarch64_step: Invalid address found in the call stack: 0x41d78b80
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff9710)
>_ULaarch64_step: (cursor=0xffffffff9710, ip=0x00000000419c4348, cfa=0x0000ffffffffa300))
>_ULaarch64_step: Invalid address found in the call stack: 0x419c4348
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff9710)
>_ULaarch64_step: (cursor=0xffffffff9710, ip=0x0000000041939250, cfa=0x0000ffffffffa770))
>_ULaarch64_step: Invalid address found in the call stack: 0x41939250
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 183898, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 41508540
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff7d50)
>_ULaarch64_step: (cursor=0xffffffff7d50, ip=0x0000000041d78b80, cfa=0x0000ffffffff8910))
>_ULaarch64_step: Invalid address found in the call stack: 0x41d78b80
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff7c20)
>_ULaarch64_step: (cursor=0xffffffff7c20, ip=0x00000000419c4348, cfa=0x0000ffffffff8810))
>_ULaarch64_step: Invalid address found in the call stack: 0x419c4348
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff7c20)
>_ULaarch64_step: (cursor=0xffffffff7c20, ip=0x000000004187b844, cfa=0x0000ffffffff8c80))
>_ULaarch64_step: Invalid address found in the call stack: 0x4187b844
>_ULaarch64_dwarf_search_unwind_table: e->fde_offset = 180058, segbase = 41384ca8, debug_frame_base = 0, fde_addr = 41504d00
>_ULaarch64_step: dwarf_step()=1
>unw_init_local_common: (cursor=0xffffffff5c00)
>_ULaarch64_step: (cursor=0xffffffff5c00, ip=0x000000004187b5a8, cfa=0x0000ffffffff8c80))
>_ULaarch64_step: Invalid address found in the call stack: 0x4187b5a8
>_ULaarch64_step: dwarf_step()=-10
With help from @janvorli (tbh he did debug and hack part) - it turns out, that crashing was due missing implementation of unw_is_signal_frame function on FreeBSD aarch64 host - it was returing -UNW_ENOINFO - with change to return 0 (I think @janvorli did a fair roll of dice here), sample app with Writeline/throw new exception is finishing with:
Hello, World!
Unhandled exception. System.Exception: Error throw by user code
at Program.<Main>$(String[] args) in C:\temp\console_app\Program.cs:line 3
Abort (core dumped)
If I compile the same app using SDK on freebsd/aarch64 host, output is the same, but without core dumped
The crash is from app compiled under Windows and when run by SDK/corerun (with unwind hack), bt is now inside libc - @janvorli I've added some needed DLL's to your rebuild script, if you think it make sense to investigate.
For the SDK part - I did quick dotnew new console and dotnet new mvc and was able to create, restore, build and run both with success - more testing/attempt to native build under FreeBSD/aarch64 I can do, when I will return home from vacation as my net access here is poor :(
Are there any chances to get #71486 into main with current status?
@Thefrank I think your scripts are failing/missing the part to add freebsd-arm64 into some sections in Microsoft.NETCoreSdk.BundledVersions.props - as I needed to patch this by hand to be able to use SDK under aarch64 host.
@sec unhandled exception are supposed to call abort, which in turn can generate core dump if enabled on the system. So it seems the behavior is correct. Provided the exception is supposed to be unhandled.
@sec I might have missed something in the patching for installer especially for net7.
Things there typically don't change enough to break patches but I will take another look at it as net7rtm should be coming soon(tm)
Few things to update:
- I've did sync with main branch of runtime and checked it still compile under Linux (cross) and native (FreeBSD aarch64 host, only runtime and native bits) - cross compiled runtime works under FreeBSD/aarch64 host. Hope that PR will get into main at some point.
- @Thefrank - I've made system wide "quick fix" to libunwind (@janvorli
return 0) and did check of yourhttps://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/7.0.100-preview.7.22355.99-freebsd-arm64again - works, was able todotnet new mvcand run the sample app. There's only minro thing to patch inMicrosoft.NETCoreSdk.BundledVersions.props- don't know how you did that SDK, but maybe something like this was missing or failed - @Thefrank how do you make cross SDK under Linux - for installer I've tried
./build.sh -c Debug -pack --runtime-id freebsd-arm64 /p:OSName=freebsd /p:IncludeAspNetCoreRuntime=false --architecture arm64- but it's still packing Linux binaries inside... - Given that now I have (as I assume) working SDK, I can go and try to do native build of
runtimeunder FreeBSD/aarch64 host (given SDK is in Debug, this can take a while :D) edit: not gonna happen, missingMicrosoft.AspNetCore.App.Runtime.freebsd-arm64
2_patching RID support for net7 is broken in my script. I plan on moving it to sed vs patch/diff as that should be more resilient
3_https://github.com/Thefrank/dotnet-freebsd-crossbuild/blob/main/patches/0001-freebsd-support.patch should fix the issue in installer. I suggest using something like ./build.sh -c Release -ci -pack --runtime-id freebsd-x64 /p:OSName=freebsd /p:CrossgenOutput=false /p:OfficialBuildId=$OFFICIALBUILDID /p:IncludeAspNetCoreRuntime=True /p:DISABLE_CROSSGEN=True so you include aspnetcore and it makes sure that it will not try and crossgen the product
@Thefrank Thanks for the tips - managed to cross compile full SDK for FreeBSD/aarch64 (all from main branches), which works (with "patched" libunwind):
sec@amper:~/sec_rc/test % ../dotnet --info
.NET SDK:
Version: 7.0.100-rc.1.22412.99
Commit: b47c93b9f8
Runtime Environment:
OS Name: FreeBSD
OS Version: 13
OS Platform: FreeBSD
RID: freebsd.13-arm64
Base Path: /usr/home/sec/sec_rc/sdk/7.0.100-rc.1.22412.99/
Host:
Version: 7.0.0-rc.1.22375.6
Architecture: arm64
Commit: e1aa560130
.NET SDKs installed:
7.0.100-rc.1.22412.99 [/usr/home/sec/sec_rc/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 7.0.0-rc.1.22404.6 [/usr/home/sec/sec_rc/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.0-rc.1.22410.9 [/usr/home/sec/sec_rc/shared/Microsoft.NETCore.App]
sec@amper:~/sec_rc/test % ../dotnet new mvc
The template "ASP.NET Core Web App (Model-View-Controller)" was created successfully.
This template contains technologies from parties other than Microsoft, see https://aka.ms/aspnetcore/7.0-third-party-notices for details.
Processing post-creation actions...
Restoring /usr/home/sec/sec_rc/test/test.csproj:
Determining projects to restore...
Restored /usr/home/sec/sec_rc/test/test.csproj (in 3 ms).
Restore succeeded.
sec@amper:~/sec_rc/test % ../dotnet run
Building...
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://localhost:5261
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
Content root path: /usr/home/sec/sec_rc/test
One more question though - when doing runtime from main branch - any way to force version output to let's say v7.0.0-preview.7.22376.6
@sec You can use https://github.com/Thefrank/dotnet-freebsd-crossbuild/blob/main/common.sh
It's a bash script so you will need to source common.sh. It also has set -e which, if outside of automation, you might want to remove.
After that you can calculate_build_id $(git -C runtime tag --points-at HEAD) this will return a value to $OFFICIALBUILDID. Pass this to the build as /p:OfficialBuildId=$OFFICIALBUILDID Things that are not preview|rc|rtm it will fall back to $(date +%Y%m%d).99
@Thefrank That I already know and use :) What I had in mind was the preview.7 part - when doing build from main it's rc.1 - was wondering where's that part set.
@sec under eng/Versions.props and its usually in two parts <PreReleaseVersionLabel>rc</PreReleaseVersionLabel> and <PreReleaseVersionIteration>1</PreReleaseVersionIteration> both typically found in the first few lines
Update on that one, #71486 got merged into main, so it's now possible to build (or crossbuild) for freebsd-arm64 and run (atm patch is needed for libunwind to make it work) - anyone interested in testing can find builds, scripts on my page.
Small update, with the recent libunwind commit dotnet is working without need for "fake patched" libunwind anymore - hope this will get released soon and backported into FreeBSD ports, but building and installing from source is not a problem.
Latest preview.7 is failing during build (can get past that with PublishReadyToRun=false) and/but then also on runtime on aarch64 with:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Microsoft.Build.Evaluation.Expander`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExtractFunctionArguments(Microsoft.Build.Shared.IElementLocation, System.String, System.String)
at Microsoft.Build.Evaluation.Expander`2+Function`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExtractPropertyFunction(System.String, Microsoft.Build.Shared.IElementLocation, System.Object, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertyBody(System.String, System.Object, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertiesLeaveTypedAndEscaped(System.String, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertiesLeaveEscaped(System.String, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.Expander`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandIntoStringLeaveEscaped(System.String, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.StringExpressionNode.GetExpandedValue(IConditionEvaluationState, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.StringExpressionNode.TryBoolEvaluate(IConditionEvaluationState, Boolean ByRef, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.NotExpressionNode.BoolEvaluate(IConditionEvaluationState, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.OperatorExpressionNode.TryBoolEvaluate(IConditionEvaluationState, Boolean ByRef, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.AndExpressionNode.BoolEvaluate(IConditionEvaluationState, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.OperatorExpressionNode.TryBoolEvaluate(IConditionEvaluationState, Boolean ByRef, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.GenericExpressionNode.Evaluate(IConditionEvaluationState, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.ConditionEvaluator.EvaluateConditionCollectingConditionedProperties[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.String, Microsoft.Build.Evaluation.ParserOptions, Microsoft.Build.Evaluation.Expander`2<System.__Canon,System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.List`1<System.String>>, System.String, Microsoft.Build.Construction.ElementLocation, Microsoft.Build.BackEnd.Logging.ILoggingService, Microsoft.Build.Framework.BuildEventContext, Microsoft.Build.Shared.FileSystem.IFileSystem, Microsoft.Build.Evaluation.ProjectRootElementCacheBase, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.ConditionEvaluator.EvaluateCondition[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.String, Microsoft.Build.Evaluation.ParserOptions, Microsoft.Build.Evaluation.Expander`2<System.__Canon,System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, System.String, Microsoft.Build.Construction.ElementLocation, Microsoft.Build.BackEnd.Logging.ILoggingService, Microsoft.Build.Framework.BuildEventContext, Microsoft.Build.Shared.FileSystem.IFileSystem, Microsoft.Build.Evaluation.ProjectRootElementCacheBase, Microsoft.Build.BackEnd.Logging.LoggingContext)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].EvaluateCondition(System.String, Microsoft.Build.Construction.ProjectElement, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Evaluation.ParserOptions, Microsoft.Build.Evaluation.Expander`2<System.__Canon,System.__Canon>, Microsoft.Build.Evaluation.LazyItemEvaluator`4<System.__Canon,System.__Canon,System.__Canon,System.__Canon>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+LazyItemOperation[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].DecorateItemsWithMetadata(System.Collections.Generic.IEnumerable`1<ItemBatchingContext<System.__Canon,System.__Canon,System.__Canon,System.__Canon>>, System.Collections.Immutable.ImmutableArray`1<Microsoft.Build.Construction.ProjectMetadataElement>, System.Nullable`1<Boolean>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+UpdateOperation[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ApplyImpl(Builder<System.__Canon,System.__Canon,System.__Canon,System.__Canon>, System.Collections.Immutable.ImmutableHashSet`1<System.String>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+LazyItemOperation[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Apply(Builder<System.__Canon,System.__Canon,System.__Canon,System.__Canon>, System.Collections.Immutable.ImmutableHashSet`1<System.String>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+MemoizedOperation[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Apply(Builder<System.__Canon,System.__Canon,System.__Canon,System.__Canon>, System.Collections.Immutable.ImmutableHashSet`1<System.String>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+LazyItemList[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ComputeItems(LazyItemList<System.__Canon,System.__Canon,System.__Canon,System.__Canon>, System.Collections.Immutable.ImmutableHashSet`1<System.String>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+LazyItemList[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetItemData(System.Collections.Immutable.ImmutableHashSet`1<System.String>)
at Microsoft.Build.Evaluation.LazyItemEvaluator`4+<>c[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<GetAllItemsDeferred>b__26_0(LazyItemList<System.__Canon,System.__Canon,System.__Canon,System.__Canon>)
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.Build.Evaluation.LazyItemEvaluator`4+ItemData[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ToArray()
at System.Linq.Buffer`1[[Microsoft.Build.Evaluation.LazyItemEvaluator`4+ItemData[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(System.Collections.Generic.IEnumerable`1<ItemData<System.__Canon,System.__Canon,System.__Canon,System.__Canon>>)
at System.Linq.OrderedEnumerable`1+<GetEnumerator>d__19[[Microsoft.Build.Evaluation.LazyItemEvaluator`4+ItemData[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at Microsoft.Build.Evaluation.Evaluator`4[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Evaluate()
at Microsoft.Build.Evaluation.Evaluator`4[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Evaluate(Microsoft.Build.Evaluation.IEvaluatorData`4<System.__Canon,System.__Canon,System.__Canon,System.__Canon>, Microsoft.Build.Evaluation.Project, Microsoft.Build.Construction.ProjectRootElement, Microsoft.Build.Evaluation.ProjectLoadSettings, Int32, Microsoft.Build.Collections.PropertyDictionary`1<Microsoft.Build.Execution.ProjectPropertyInstance>, Microsoft.Build.BackEnd.Logging.ILoggingService, Microsoft.Build.Evaluation.IItemFactory`2<System.__Canon,System.__Canon>, Microsoft.Build.Evaluation.IToolsetProvider, Microsoft.Build.FileSystem.IDirectoryCacheFactory, Microsoft.Build.Evaluation.ProjectRootElementCacheBase, Microsoft.Build.Framework.BuildEventContext, Microsoft.Build.BackEnd.SdkResolution.ISdkResolverService, Int32, Microsoft.Build.Evaluation.Context.EvaluationContext, Boolean)
at Microsoft.Build.Execution.ProjectInstance.Initialize(Microsoft.Build.Construction.ProjectRootElement, System.Collections.Generic.IDictionary`2<System.String,System.String>, System.String, System.String, Int32, Microsoft.Build.Execution.BuildParameters, Microsoft.Build.BackEnd.Logging.ILoggingService, Microsoft.Build.Framework.BuildEventContext, Microsoft.Build.BackEnd.SdkResolution.ISdkResolverService, Int32, System.Nullable`1<Microsoft.Build.Evaluation.ProjectLoadSettings>, Microsoft.Build.Evaluation.Context.EvaluationContext, Microsoft.Build.FileSystem.IDirectoryCacheFactory)
at Microsoft.Build.Execution.ProjectInstance..ctor(System.String, System.Collections.Generic.IDictionary`2<System.String,System.String>, System.String, Microsoft.Build.Execution.BuildParameters, Microsoft.Build.BackEnd.Logging.ILoggingService, Microsoft.Build.Framework.BuildEventContext, Microsoft.Build.BackEnd.SdkResolution.ISdkResolverService, Int32, System.Nullable`1<Microsoft.Build.Evaluation.ProjectLoadSettings>)
at Microsoft.Build.BackEnd.BuildRequestConfiguration+<>c__DisplayClass61_0.<LoadProjectIntoConfiguration>b__0()
at Microsoft.Build.BackEnd.BuildRequestConfiguration.InitializeProject(Microsoft.Build.Execution.BuildParameters, System.Func`1<Microsoft.Build.Execution.ProjectInstance>)
at Microsoft.Build.BackEnd.BuildRequestConfiguration.LoadProjectIntoConfiguration(Microsoft.Build.BackEnd.IBuildComponentHost, Microsoft.Build.Execution.BuildRequestDataFlags, Int32, Int32)
at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
at Microsoft.Build.BackEnd.RequestBuilder.<StartBuilderThread>b__53_2()
at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
at System.Threading.Tasks.Task.ExecuteEntry()
at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
Abort (core dumped)
stacktrace is diffrent for each run, ex.:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Microsoft.Build.Shared.EscapingUtilities.UnescapeAll(System.String, Boolean)
at Microsoft.Build.Evaluation.Expander`2+Function`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Execute(System.Object, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertyBody(System.String, System.Object, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertiesLeaveTypedAndEscaped(System.String, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem, Microsoft.Build.BackEnd.Logging.LoggingContext)
but it's around Microsoft.NET.StringTools.Strings.GetSpanBasedStringBuilder :)
preview.6 don't have this issue.
NB: preview.7 is working fine under FreeBSD amd64
NativeAOT or just Crossgen2?
edit: are you sure the memory is not corrupt? ;)
During build it fail at this stage
Generating native code
crossgen2 -> /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/coreclr/freebsd.arm64.Release/crossgen2/freebsd-arm64/publish/
Emitting R2R PE file: /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/obj/Microsoft.NETCore.App.Crossgen2/Release/net8.0/freebsd-arm64/S.P.C.tmp
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/obj/Microsoft.NETCore.App.Crossgen2/Release/net8.0/freebsd-arm64/output/ -> /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping//dotnet-crossgen2-8.0.0-preview.7.23375.6-freebsd-arm64.tar.gz
The package Microsoft.NETCore.App.Crossgen2.freebsd-arm64.8.0.0-preview.7.23375.6 is missing a readme. Go to https://aka.ms/nuget/authoring-best-practices/readme to learn why package readmes are important.
Successfully created package '/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Crossgen2.freebsd-arm64.8.0.0-preview.7.23375.6.nupkg'.
Successfully created package '/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Crossgen2.freebsd-arm64.8.0.0-preview.7.23375.6.symbols.nupkg'.
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/obj/dotnet-hostfxr/Release/net8.0/freebsd-arm64/output/ -> /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping//dotnet-hostfxr-internal-8.0.0-preview.7.23375.6-freebsd-arm64.tar.gz
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/obj/dotnet-nethost/Release/net8.0/freebsd-arm64/output/ -> /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping//dotnet-nethost-8.0.0-preview.7.23375.6-freebsd-arm64.tar.gz
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/obj/dotnet-nethost/Release/net8.0/freebsd-arm64/symbols/ -> /mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/packages/Release/Shipping//dotnet-nethost-symbols-freebsd-arm64-8.0.0-preview.7.23375.6.tar.gz
Microsoft.NETCore.App.Runtime ->
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/Crossgen2Tasks/Debug/net8.0/Microsoft.NET.CrossGen.targets(469,5): error : Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. [/mnt/sec/dotnet-core-freebsd-source-build/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.sfxproj]
##vso[task.logissue type=error;sourcepath=/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/Crossgen2Tasks/Debug/net8.0/Microsoft.NET.CrossGen.targets;linenumber=469;columnnumber=5;code=;](NETCORE_ENGINEERING_TELEMETRY=Build) Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/Crossgen2Tasks/Debug/net8.0/Microsoft.NET.CrossGen.targets(469,5): error : at ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver+TokenResolverProvider.Rent(ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver, Internal.TypeSystem.Ecma.IEcmaModule) [/mnt/sec/dotnet-core-freebsd-source-build/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.sfxproj]
##vso[task.logissue type=error;sourcepath=/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/Crossgen2Tasks/Debug/net8.0/Microsoft.NET.CrossGen.targets;linenumber=469;columnnumber=5;code=;](NETCORE_ENGINEERING_TELEMETRY=Build) at ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver+TokenResolverProvider.Rent(ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver, Internal.TypeSystem.Ecma.IEcmaModule)
...
/mnt/sec/dotnet-core-freebsd-source-build/runtime/artifacts/bin/Crossgen2Tasks/Debug/net8.0/Microsoft.NET.CrossGen.targets(357,5): error NETSDK1096: Optimizing assemblies for performance failed. You can either exclude the failing assemblies from being optimized, or set the PublishReadyToRun property to false. [/mnt/sec/dotnet-core-freebsd-source-build/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.sfxproj]
Setting PublishReadyToRun=false will complete the build of runtime, aspnet will go fine and the final sdk is failing at restore stage
sec@amper:~/aa % ./dotnet new console --force
The template "Console App" was created successfully.
Processing post-creation actions...
Restoring /usr/home/sec/aa/aa.csproj:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Microsoft.Build.Shared.EscapingUtilities.UnescapeAll(System.String, Boolean)
at Microsoft.Build.Evaluation.Expander`2+Function`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Execute(System.Object, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation)
at Microsoft.Build.Evaluation.Expander`2+PropertyExpander`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExpandPropertyBody(System.String, System.Object, Microsoft.Build.Evaluation.IPropertyProvider`1<System.__Canon>, Microsoft.Build.Evaluation.ExpanderOptions, Microsoft.Build.Shared.IElementLocation, Microsoft.Build.Evaluation.UsedUninitializedProperties, Microsoft.Build.Shared.FileSystem.IFileSystem)
And yes, I'm pretty sure the memory is ok :)
Did git bisect fun and a3981dde2de089061fe7c857d8bd5da573546a3f is the first bad commit - a3981dd with #87082 - to be sure, I also checked on another arm64 box, to make sure memory is not corrupt :)
@kunalspathak maybe you will figure out why it started to not-work on arm64 FreeBSD box? :)
Yup, I think we can/should skip inlining TLS for crossbuild, at least for non-Linux Unices (ref: #89472 (comment)).
This is happening during native arm64 build under FreeBSD - during crossbuild, as @Thefrank pointed, this is not triggering, as it's not run. It also happen when using crossbuilded (or build without crossgen done on arm64) SDK, for ex. during restore stage (as it access https I assume).
as I don't have knowledge to dig this deeper (too much ifdefn in there :) , but simple edit of src/coreclr/vm/jitinterface.cpp and setting optimizeThreadStaticAccess = false make the preview.7 runtime and SDK work/build fine on FreeBSD arm64.
For x64, this fixed the issue for me yesterday a9d74ac#diff-a17477bd3f8027a1ed66b7c0af64da44d84489985b2846a9114f9e56f0a06a24 (it basically turned __tls_get_addr into __tls_get_addr@plt on x64; fix is in the main branch).
@sec Thanks a lot for https://github.com/sec/dotnet-core-freebsd-source-build !!! The other day I created a FreeBSD 14.0 x64 self-hosted github runner on AWS with .NET 7.0 and 8.0 - everything works as it should! However I'm getting a segmentation fault for ARM64 inside libunwind and the latest libunwind-20230730 doesn't help. Above you wrote that libunwind/libunwind#407 can solve all problems. When should we expect this fix in FreeBSD ports? Should we even expect FreeBSD to be included in the list of officially supported Microsoft platforms or will it all remain on the conscience of enthusiasts?
@ww898 Yes getting libunwind with that fix should resolve segfault. It's not released yet (I saw 1.8-rc is having this included). You can either compile it from latest sources (main or rc branch) or drop in the patch with that fix (generate diff and put in ports) and then compile and replace the version from packages/ports you have.
I wouldn't count on MS support soon :)
Should we even expect FreeBSD to be included in the list of officially supported Microsoft platforms or will it all remain on the conscience of enthusiasts?
They are referred to as "community supported platforms". Although linux binaries are available by Microsoft, distors (such as Alpine, Debian, Fedora, Gentoo etc.) also create their own dotnet package (by building https://github.com/dotnet/dotnet virtual repo which includes all components). ). AFAIK, it's on source-build team's roadmap to make dotnet/dotnet repo build for non-linux operating systems.
AFAIK, it's on source-build team's roadmap to make dotnet/dotnet repo build for non-linux operating systems.
It already builds on FreeBSD with a community supported bootstrap SDK :)
@arrowd I've started to prepare box for testing port building .net, but atm have only access to 14, so ===> compat13x-aarch64-13.2.1302001.20231227 is only for amd64 i386, while you are running aarch64. failed - I will workaround that, but I assume getting aarch64 version into this port shouldn't be a problem? :)
I will workaround that, but I assume getting aarch64 version into this port shouldn't be a problem? :)
Yep, these files are simply taken from the vanilla FreeBSD 13.2 installation.