Support for ARMv8.3+ / ARM64e targets

Question

Support for ARMv8.3+ / ARM64e targets

Absolucy opened this issue 5 years ago · 45 comments

Currently, a blocker in a project of mine - Crabapple - is the lack of native arm64e support in Rust.

The aarch64-apple-ios target only emits arm64 (ARMv8) code, which will not work within an arm64e (ARMv8.3+) environment due to it's lack of awareness of Pointer Authentication, leading to segfaults when trying to access a signed pointer.

It is currently possible to work around this, by using a backtrace=false rustc, compiling with --emit=llvm-ir -Clto, and running the resulting IR through an arm64e-aware LLVM (such as apple/llvm-project). Pointer authentication may be worked around by linking to an FFI function that runs ptrauth_strip.

This may require LLVM changes if done, and quite possibly a new target (arm64e-apple-ios, aarch64-apple-ios-armv83, etc)

Answer 1 · 2020-06-22T19:00:34.000Z

In addition, arm64e support important due to the upcoming Mac desktops running Bionic processors (such as an A12Z).

Answer 2 · 2020-06-22T19:06:59.000Z

Upstream LLVM supports armv8.3, and most likely the Rust LLVM as well.

Answer 3 · 2020-06-22T19:11:05.000Z

Upstream LLVM supports armv8.3, and most likely the Rust LLVM as well.

Weren't you working on building a custom LLVM since upstream doesn't support this yet? The link just goes to an RFC thread on the mailing list with no replies.

Answer 4 · 2020-06-22T19:14:07.000Z

@jonas-schievink I was poorly attempting to jury-rig apple/llvm-project onto rustc. In short, I got arm64e and armv8.3 mixed up.

A quick grep and dig through rust-project/llvm reveals that it has the code for ARMv8.3 and Pointer Authentication support.

Answer 5 · 2020-06-22T20:28:12.000Z

ARMv8.5 support would also be good, considering that there's a high likelihood than ARM Macs will run on ARMv8.5.

Answer 6 · 2020-06-22T20:30:30.000Z

~~It is worth noting that the Rust LLVM branch has support for PAC (ARMv8.3) and BTI (ARMv8.5)~~

This isn't upstreamed, my bad.

Answer 7 · 2020-06-24T00:19:46.000Z

Note that the macOS 11 SDK has both arm64-macos and arm64e-macos targets, although the arm64 variants don't have an uuid in the system library TBD files.

Answer 8 · 2020-06-24T00:22:29.000Z

Upstream LLVM supports armv8.3, and most likely the Rust LLVM as well.

That's not merged in LLVM upstream.

Answer 9 · 2020-06-24T00:44:02.000Z

@glandium my bad.

Answer 10 · 2020-06-25T13:44:30.000Z

I'm interested in this as well because of Apple Silicon (i.e arm64 on macOS)

Going forward it will be mandated by Apple that executable released (at least for the App Store) supports both x86_64 and amd64 target. On macOS there is "fat binary" support which makes it possible to have both targets in the same exe/obj/dylib etc file. Clang for example supports that you can have -arch x86_64 and -arch amd64 on the command line to generate the correct binary for this to work. Then the OS will load the correct version depending on the OS version running.

There is a command line tool called lipo that can be used to combine two different compiles, but this (according to Apple) should only be used as a work-around and I think for Rust it would be better to support building for both targets at the same time.

Answer 11 · 2020-06-25T13:51:51.000Z

We (Embark) are interested in it as well, should we track having a Darwin Mac on ARM target in this issue or file an additional one specifically for it (with some overlap with this)?

Answer 12 · 2020-06-25T18:21:00.000Z

We'd need someone with a Developer Transition Kit to truly test any sort of support for this.

Answer 13 · 2020-06-26T11:46:01.000Z

The aarch64-apple-ios triple seems slightly ambiguous once there are arm64e targets as pointer-authentication would break unauthenticated environments. Given that arm64e-darwin is on its way, two new triples, aarch64e-apple-darwin and aarch64e-apple-ios, would probably make more sense?

Edit: sorry, this was a typo. I meant to write arm64e-apple-darwin and arm64e-apple-ios.

Answer 14 · 2020-06-26T14:05:39.000Z

@19h aarch64e would be ENTIRELY inaccurate of a name.

Answer 15 · 2020-06-26T14:38:04.000Z

@Luxxxxy It's unfortunate that both arm64 and aarch64 refer to the same ISA, whereas the former was the result of proprietary work by Apple that only was open-sourced after other vendors published their work on the latter.

Aarch64 is an alias for the ARMv8-A ISA, and it would be confusing to have arm64e coexist with aarch64. But given that arm64e actually references a proprietary extension of the ARMv8-A ISA (it is actually an ABI), it might be reasonable to have it named in a way that clearly separates it from the other.

Answer 16 · 2020-07-02T20:18:37.000Z

@Luxxxxy If one had such a device, would you be interested in testing something on it?

Answer 17 · 2020-07-02T20:20:01.000Z

I have an arm64e phone (iPhone 11, iOS 13.5)

I'd be glad to test something on it, I've been using it to get rustc/cargo working on iOS.

Answer 18 · 2020-07-23T01:19:26.000Z

I am currently adding PAC/arm64e to a fork using Apple's LLVM. Hopefully some of my work can be reused whenever PAC/arm64e support is upstreamed into LLVM.

Answer 19 · 2020-07-23T01:32:11.000Z

that's work on jailbreak only?

Answer 20 · 2020-07-23T02:04:48.000Z

that's work on jailbreak only?

That's how I'm testing it. However, most likely, this will also work on Apple Silicon (ARM) macs. Not sure if iOS lets you submit arm64e slice apps to the app store, though.

Answer 21 · 2020-09-11T10:58:29.000Z

arm64e is currently required when building kernel modules for macOS on Apple Silicon machines, but arm64e binaries will not run in userspace (currently arm64, or running x86_64 under Rosetta emulation is required).

Answer 22 · 2020-10-31T10:15:56.000Z

@hjmallon Hello,

The arm64e ABI is currently not considered as final. To run arm64e binaries on macOS on Apple Silicon, add the -arm64e_preview_abi argument to boot-args.

Arm 64-bit macOS supports arm64 binaries, and that's the only supported configuration. Currently, arm64e is only for bundled in the OS applications and libraries, as you can load arm64e binaries in an arm64 process.

Answer 23 · 2020-12-16T09:15:24.000Z

In some cases, the arm64e arch might be "different". For example:

thread_set_state might fail with (os/kern) protection failure if we try to call it from arm64 process to arm64e process.
The returning value of dlsym is PAC signed on arm64e, while left untouched on arm64
Some function like pthread_create_from_mach_thread requires a PAC signed function pointer on arm64e, which is not required on arm64.

It seems that Apple just simply disables PAC on arm64 process to accomplish 2 and 3. However, case 1 can still not be covered if there is no arm64e target. I am not sure whether there is another case of ”cross arch“ interaction like case 1.

Answer 24 · 2021-08-27T13:16:07.000Z

I believe that we may see proper PAC support soon, with #88354

Answer 25 · 2022-09-13T14:02:13.000Z

Any news on this?

Answer 26 · 2022-12-19T23:42:55.000Z

Any progress on this issue?

Answer 27 · 2024-01-26T00:12:50.000Z

What's missing after #115526 to have rustup target add arm64e-apple-ios work out of the box? (IOW, what is missing for artifacts for these tier 3 targets to be available?)

Answer 28 · 2024-01-27T17:52:56.000Z

@glandium It needs to be promoted to tier 2 AFAIK

Answer 29 · 2024-01-27T18:10:24.000Z

I sent a MCP to do the promotion for arm64-darwin.
rust-lang/compiler-team#717

Answer 30 · 2024-01-31T21:48:04.000Z

For what it's worth, LLVM doesn't support CPU_SUBTYPE_PTRAUTH_ABI yet, which is required by Xcode's linker to link arm64e code.

Answer 31 · 2024-10-18T09:56:36.000Z

Summary:

Support for CPU_SUBTYPE_PTRAUTH_ABI in LLVM is required. Blocks the following issues:
- #130085
- #130834
Support for arm64e in LLD is required.
Support for arm64e in mold is required.
Support for arm64e in jemallocator is required.

Answer 32 · 2024-11-11T05:01:51.000Z

I'll add a few more things:

Given that Apple haven't completed upstreaming their LLVM patches for this yet, it is a bit unclear what kind of ABI stability one can expect between compilers (e.g. linking Rust and C code), so I'd be quite weary of making this tier 2 until we are certain in that.
@thejpster seemed to have reservations about the architecture name used in the triple being arm64e, do those still apply? And if so, what should we name the targets then? aarch64-apple-darwin-pointerauth, aarch64-apple-ios-simpointerauth, etc.?
We'd want to ensure that our asm! supports the necessary pointer authentication extensions (.arch_extension pauth? Dunno how it works), or at least have a test for / documentation that it doesn't.
How do we expose this to user code? cfg!(target_feature = "pointer-auth")? Various headers seems to use either #if __has_feature(ptrauth_calls) or #if defined(__arm64e__).
The opsem semantics of pointer authentication needs to be discussed a bit more.
Possibly guaranteed assembler support for .arch_extension pauth?

And link a few things:

@rustbot label O-apple

Answer 33 · 2025-05-20T22:37:34.000Z

Support for CPU_SUBTYPE_PTRAUTH_ABI in llvm/llvm-project#80200 is required.

Note this was fixed in LLVM 20, which means rustc 1.87 has it.

Current nightly can build a hello world without LLD (lld hasn't been fixed yet to support arm64e properly):

% cargo +nightly build --target arm64e-apple-darwin -Zbuild-std=std
   Compiling compiler_builtins v0.1.159
   Compiling core v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/core)
   Compiling libc v0.2.172
   Compiling object v0.36.7
   Compiling std v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/std)
   Compiling rustc-std-workspace-core v1.99.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc)
   Compiling cfg-if v1.0.0
   Compiling memchr v2.7.4
   Compiling adler2 v2.0.0
   Compiling panic_abort v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/panic_abort)
   Compiling rustc-demangle v0.1.24
   Compiling unwind v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/unwind)
   Compiling rustc-std-workspace-alloc v1.99.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/panic_unwind)
   Compiling gimli v0.31.1
   Compiling miniz_oxide v0.8.8
   Compiling hashbrown v0.15.3
   Compiling std_detect v0.1.5 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/stdarch/crates/std_detect)
   Compiling addr2line v0.24.2
   Compiling rustc-std-workspace-std v1.99.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/rustc-std-workspace-std)
   Compiling rustc-literal-escaper v0.0.2
   Compiling proc_macro v0.0.0 (/Users/glandium/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/proc_macro)
   Compiling foo v0.1.0 (/Users/glandium/foo)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.08s

The resulting binary, however, doesn't work:

% lldb target/arm64e-apple-darwin/debug/foo
(lldb) target create "target/arm64e-apple-darwin/debug/foo"
Current executable set to '/Users/glandium/foo/target/arm64e-apple-darwin/debug/foo' (arm64e).
(lldb) r
Process 2300 launched: '/Users/glandium/foo/target/arm64e-apple-darwin/debug/foo' (arm64e)
Process 2300 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x605a00018fa440f8)
    frame #0: 0x000000018fa440f8 libdyld.dylib`_tlv_get_addr
libdyld.dylib`_tlv_get_addr:
->  0x18fa440f8 <+0>:  ldr    w16, [x0, #0x8]
    0x18fa440fc <+4>:  mrs    x17, TPIDRRO_EL0
    0x18fa44100 <+8>:  and    x17, x17, #0xfffffffffffffff8
    0x18fa44104 <+12>: ldr    x17, [x17, x16, lsl #3]
Target 0: (foo) stopped.
(lldb) register read $x0
      x0 = 0x00000001001b89f0  foo`std::thread::current::CURRENT::h46ed779abe02c7ae

Answer 34 · 2025-09-12T02:49:50.000Z

I've made some progress on adding support for arm64e instructions in oskarwirga@8a79ed2

Currently, it depends on oskarwirga/llvm-project@485d772

I am still testing it locally, but I will submit as a PR with test cases (for both commits) soon.

Answer 35 · 2025-09-12T23:54:13.000Z

I've made some progress on adding support for arm64e instructions in oskarwirga@8a79ed2

I think it is important to separate things a bit as arm64e here is actually misleading :)

TL;DR: One cannot obtain arm64e support right now using mainline LLVM.

Here are the details:

There are pointer authentication (pauth) instructions in AArch64 v8.3. ISA
arm64e is an ABI available on Apple platforms that uses pauth instructions (see slides in More information). So far, arm64e end-to-end support (swift, C/C++ frontend, midend and backend) is available only in Apple downstream fork of clang / llvm.
There is a "backward compatible" subset of pauth ABI that is used to sign / protect return addresses that is normally referred to as PAC-RET. This is already available in LLVM mainline and does not depend on anything (e.g. source language or frontend) and can be just enabled.
Recently there were efforts in LLVM mainline for:
- Backport required pauth ISA and C/C++ pauth ABI bits from Apple downstream forks, including clang frontend bits, something in midend and generic backend things (see https://clang.llvm.org/docs/PointerAuthentication.html)
- Add support for pointer authentication for ELF platforms as specified in https://github.com/ARM-software/abi-aa/blob/main/pauthabielf64/pauthabielf64.rst, both in frontend (via "test" pauthest ABI and LLVM backend).
As a result, LLVM 21 has more or less complete support for pointer authentication features for ELF targets. Important point is that not all MachO backend bits are there yet, so one cannot have end-to-end support for pointer authentication for Apple platforms via LLVM mainline yet. Also, not all details of arm64e ABI are aligned between mainline clang and Apple downstream clang. And work is currently in progress to close the gap between these, there is no ETA though.

So, essentially arm64e support is Rust is mostly about C/C++ interop as arm64e is a platform ABI. One cannot obtain this now, as mainline LLVM does not have all necessary bits including backend support for MachO.

What is possible:

Provide necessary hooks and ensure C/C++ interop uses appropriate pointer signing schema from as specified by platform
Have pointer authentication enabled for ELF-based platforms (read: Linux). So far no publicly available platform uses pointer authentication as a part of its ABI. Though it is possible to have an isolated sysroot (the missed bits are dynamic loader and pauth-aware libunwind, the latter hopefully will be available from LLVM mainline soon, for the former there is proof-of-concept reference implementation for musl).
Define & implement pointer authentication ABI for Rust code. After all there are indirect calls here as well :)

That said, we're going to work on the things outlined above soon. This eventually will give in addition arm64e support as soon as Apple will finish opening their downstream changes around pauth to LLVM mainline.

Hope I made picture a bit more cleaner :)

More information:

Answer 36 · 2025-09-13T13:13:37.000Z

Thanks for explaining the current situation @asl !
I wonder if anyone here knows when/if Apple will let 3rd party executables run arm64e any time soon (on Macbooks)?

Answer 37 · 2025-09-14T00:39:33.000Z

@aviramha This is now possible on macOS/iOS/... 26. See https://developer.apple.com/documentation/xcode/enabling-enhanced-security-for-your-app

Answer 38 · 2025-09-14T05:55:03.000Z

@aviramha This is now possible on macOS/iOS/... 26. See developer.apple.com/documentation/xcode/enabling-enhanced-security-for-your-app

I searched for that just 2 weeks ago and didn't find - I guess my search was around aarch64/arm64e and the doc doesn't have any of those keywords.

Reading more into that, I see this note:

This setting is available on iOS 26 and later on iPhone 17, iPhone 17 Pro, iPhone 17 Pro Max, or iPhone Air.

Are you sure it'll also for macOS? I knew arm64e executables were supported on iPhones

Answer 39 · 2025-09-14T06:08:06.000Z

@aviramha that's about MTE specifically, which isn't in any announced Mac right now.

Answer 40 · 2025-09-14T06:21:02.000Z

@aviramha that's about MTE specifically, which isn't in any announced Mac right now.

Based on the docs it does make sense, but wayback archive doesn't work on the page so I can't know if that's a recent change and I don't see anything in the Tahoe changelog 👀

Answer 41 · 2025-09-14T06:23:49.000Z

See the Xcode changelog: https://developer.apple.com/documentation/xcode-release-notes/xcode-26-release-notes

Answer 42 · 2025-09-14T06:25:53.000Z

See the Xcode changelog: developer.apple.com/documentation/xcode-release-notes/xcode-26-release-notes

It says it supports it but it supported building even before, but you need to enable the preview abi in boot to run the process.
I'm upgrading to the beta to see if I can run a 3rd party arm64e binary
I wish they were less ambiguous about it :(

Answer 43 · 2025-09-14T06:47:26.000Z

It's not a preview ABI anymore in the *OS 26 releases.

However, make sure to build with the newer toolchain. Older toolchains will produce preview ABI binaries that will still be blocked from launch on the new macOS release.

Answer 44 · 2025-09-14T07:50:38.000Z

Thanks for the heads up! It indeed works. Do you know if arm64e binary built with Xcode 26 will be able to run on pre-Tahoe (26) macOS? Wondering what kind of compatibility is needed to be considered.

Answer 45 · 2025-09-14T07:56:21.000Z

I provided an arm64e binary built on Xcode 26 to macOS 16 user and it ran :) very cool