Can't load C ext on Apple M1
Closed this issue · 26 comments
Traceback (most recent call last):
23: from /usr/local/bin/bacon:23:in `<main>'
22: from /usr/local/bin/bacon:23:in `load'
21: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:115:in `<top (required)>'
20: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:115:in `each'
19: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:116:in `block in <top (required)>'
18: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:116:in `load'
17: from spec/specification_spec.rb:1:in `<top (required)>'
16: from spec/specification_spec.rb:1:in `require'
15: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:60:in `<top (required)>'
14: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:15:in `silence_warnings'
13: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:28:in `with_warnings'
12: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:15:in `block in silence_warnings'
11: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:61:in `block in <top (required)>'
10: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:61:in `require'
9: from /Users/jasl/Workspaces/Ruby/Core/lib/cocoapods-core/cdn_source.rb:6:in `<top (required)>'
8: from /Users/jasl/Workspaces/Ruby/Core/lib/cocoapods-core/cdn_source.rb:6:in `require'
7: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async.rb:25:in `<top (required)>'
6: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async.rb:25:in `require_relative'
5: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/reactor.rb:25:in `<top (required)>'
4: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/reactor.rb:25:in `require_relative'
3: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/wrapper.rb:23:in `<top (required)>'
2: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/wrapper.rb:23:in `require'
1: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `require': dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle (LoadError)
BTW, NIO4R_PURE=true works
In addition, I found custom build Ruby on M1 Mac can load c ext properly, but in some cases, it will crash, I don't know how to dig this, so I push a crash log here cocoapods.log
Maybe we have to force to use pure Ruby fallback on M1 Mac
missing compatible arch looks like it's not being compiled correctly.
@nobu told me:
A Rubygem's issue. Rubygems puts the last compiled bundle in the architecture neutral path (/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle). The correct bundle should be another architecture dependent path, if it has been compiled.
@hsbt said:
Do not use system ruby
@jasl Can you check this advice and let me know if it helps?
@nobu told me:
A Rubygem's issue. Rubygems puts the last compiled bundle in the architecture neutral path (/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle). The correct bundle should be another architecture dependent path, if it has been compiled.
@hsbt said:
Do not use system ruby
@jasl Can you check this advice and let me know if it helps?
I'm afraid these won't help...
- This issue was found by CocoaPods, this tool is aim for iOS developers, most of them don't have motivation to install Ruby...
- I know the M1 macOS bundled Ruby looks very strange, it can be treated as both x86 and arm64
For missing compatible arch I suggest CocoaPods set NIO4R_PURE=true, and it works well
BUT there is another problem,
In addition, I found custom build Ruby on M1 Mac can load c ext properly, but in some cases, it will crash, I don't know how to dig this, so I push a crash log here cocoapods.log
This case also sovled by set NIO4R_PURE=true.
But it made me concern: previously, I was only thought ffi has compatibility issue with M1 macOS, now nio4r too, maybe more c-ext gems have protential compatibility issue.
Question from a 'macOS' challenged user - can one build 'ARM-based macOS' in 'the cloud'? More importantly, run CI with it?
@jasl why can't we build fat binary?
This would be better, but it also requires MRI fat binary too?
now only macOS system Ruby is fat binary, is it easy to made MRI become fat binary?
Question from a 'macOS' challenged user - can one build 'ARM-based macOS' in 'the cloud'? More importantly, run CI with it?
I'm afraid not, first macOS isn't opensource... second the License issue, we have to waiting AWS/Azure/TravisCI update their Mac Mini to M1 Mac...
AFAIK, HomeBrew team got some M1 Macs they're running ARM-related jobs locally,
Someone sponsored CocoaPods a M1 Mac Mini as CI via BuildKite
If someone wants to sponsor a Mac Mini M1 I will set up a build server for anyone in the Ruby community who wants to use it via GitHub actions (remote instance) within reason.
If someone wants to sponsor a Mac Mini M1 I will set up a build server for anyone in the Ruby community who wants to use it via GitHub actions (remote instance) within reason.
I just sponsor you from GitHub sponsor for $100 a month for 1 year, so it should be $1200 in total that should enough to buy a 16G memory 512G SSD M1 Mac Mini!
Please help to ensure Ruby can working well with M1 macOS.
BTW, I use GH sponsor becasue I don't have a easy way to transfer you the money.
Hey, that's amazing. Thanks for your contribution. I'll get the hardware ordered tomorrow!
Okay, I was excited, so I ordered it already. It will take 2-3 weeks to arrive so please be patient.
FWIW it seems to work for me on an M1 Mini:
~ $ ruby -v
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [arm64-darwin20]
~ $ gem install nio4r
Fetching nio4r-2.5.4.gem
Building native extensions. This could take a while...
Successfully installed nio4r-2.5.4
1 gem installed
~ $ irb
irb(main):001:0> require "nio"
=> true
I built Ruby using ruby-install, installed via homebrew.
@jasl Are you doing anything other than require "nio" to cause the failure you're seeing? Any other tips for reproducing?
FWIW it seems to work for me on an M1 Mini:
~ $ ruby -v ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [arm64-darwin20] ~ $ gem install nio4r Fetching nio4r-2.5.4.gem Building native extensions. This could take a while... Successfully installed nio4r-2.5.4 1 gem installed ~ $ irb irb(main):001:0> require "nio" => trueI built Ruby using
ruby-install, installed via homebrew.@jasl Are you doing anything other than
require "nio"to cause the failure you're seeing? Any other tips for reproducing?
this issue only occurred on System Ruby, not affect on self-build Ruby
@jasl you said there were crashes here: #259 (comment)
Can you reproduce any of those?
@jasl you said there were crashes here: #259 (comment)
Can you reproduce any of those?
Just re-confirm the crash still happens, I push a reproducible branch https://github.com/jasl/CocoaPods/tree/m1-crash-case
Step (on a M1 Mac)
- Clone
https://github.com/jasl/CocoaPods.gitthen switch tom1-crash-casebranch - Use self-build Ruby 3.0 and
bundle bundle exec rake spec
In nio4r repository, on M1 Mac, I tried to build native extension, and it did seem to correctly build fat binary:
otool -fah nio4r_ext.bundle
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
cputype 16777223
cpusubtype 3
capabilities 0x0
offset 16384
size 16432
align 2^14 (16384)
architecture 1
cputype 16777228
cpusubtype 2
capabilities 0x0
offset 49152
size 16738
align 2^14 (16384)
nio4r_ext.bundle (architecture x86_64):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 8 12 648 0x00000085
nio4r_ext.bundle (architecture arm64e):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777228 2 0x00 8 14 648 0x00000085
All tests passed (bundle exec rspec).
So, I'll try to reproduce CocoaPods issue using only native Ruby.
Okay, so I tried m1-crash-branch of CocoaPods as outlined above.
I installed gems by deleting Gemfile.lock (don't have bundler 2 on native Ruby).
Then, checkout all submodule using git submodule update --init --recursive
I then did bundle install.
Finally try to run tests with bundle exec rake:
bundler: failed to load command: bacon (/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/bin/bacon)
LoadError: dlopen(/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle, 0x0009): missing compatible arch in /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle - /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle
Strangely, I checked this binary and it does appear to have the right architectures:
samuel@Motoko ~/D/P/t/CocoaPods (m1-crash-case) [1]> otool -fah /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
cputype 16777223
cpusubtype 3
capabilities 0x0
offset 16384
size 16432
align 2^14 (16384)
architecture 1
cputype 16777228
cpusubtype 2
capabilities 0x0
offset 49152
size 16738
align 2^14 (16384)
/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle (architecture x86_64):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 8 12 648 0x00000085
/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle (architecture arm64e):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777228 2 0x00 8 14 648 0x00000085
I tried to look for system library but it also seems okay:
samuel@Motoko ~/D/P/t/CocoaPods (m1-crash-case) [1]> otool -fah /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
cputype 16777223
cpusubtype 3
capabilities 0x0
offset 16384
size 94816
align 2^14 (16384)
architecture 1
cputype 16777228
cpusubtype 2
capabilities 0x80
offset 114688
size 110976
align 2^14 (16384)
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle (architecture x86_64):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 8 15 1688 0x00000085
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle (architecture arm64e):
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777228 2 0x80 8 16 1672 0x00000085
I found some useful discussion here: rubygems/rubygems#4234
I can definitely reproduce part of the issue:
samuel@Motoko ~ [1]> sudo gem install nio4r --version 2.5.4
Fetching nio4r-2.5.4.gem
Building native extensions. This could take a while...
Successfully installed nio4r-2.5.4
Parsing documentation for nio4r-2.5.4
Installing ri documentation for nio4r-2.5.4
Done installing documentation for nio4r after 0 seconds
1 gem installed
samuel@Motoko ~> irb
WARNING: This version of ruby is included in macOS for compatibility with legacy software.
In future versions of macOS the ruby runtime will not be available by
default, and may require you to install an additional package.
irb(main):001:0> require 'nio'
Traceback (most recent call last):
10: from /usr/bin/irb:23:in `<main>'
9: from /usr/bin/irb:23:in `load'
8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
7: from (irb):1
6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
When I checked, the extension is compiled correctly:
> lipo -archs /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle
x86_64 arm64e
But what's even more odd:
> arch -x86_64 irb
WARNING: This version of ruby is included in macOS for compatibility with legacy software.
In future versions of macOS the ruby runtime will not be available by
default, and may require you to install an additional package.
irb(main):001:0> require 'nio'
Traceback (most recent call last):
10: from /usr/bin/irb:23:in `<main>'
9: from /usr/bin/irb:23:in `load'
8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
7: from (irb):1
6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlsym(0x7ffdaa474a80, Init_nio4r_ext): symbol not found - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
> arch -arm64e irb
WARNING: This version of ruby is included in macOS for compatibility with legacy software.
In future versions of macOS the ruby runtime will not be available by
default, and may require you to install an additional package.
irb(main):001:0> require 'nio'
Traceback (most recent call last):
10: from /usr/bin/irb:23:in `<main>'
9: from /usr/bin/irb:23:in `load'
8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
7: from (irb):1
6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
Okay, I think I found the problem:
> nm nio4r_ext.bundle
nio4r_ext.bundle (for architecture x86_64):
U dyld_stub_binder
nio4r_ext.bundle (for architecture arm64e):
no symbols
There are no symbols in the native extension :p
linking shared-object nio4r_ext.bundle
ld: warning: directory not found for option '-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.2.Internal.sdk/usr/local/lib'
ld: warning: ignoring file monitor.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file bytebuffer.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file nio4r_ext.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file selector.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: directory not found for option '-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.2.Internal.sdk/usr/local/lib'
ld: warning: ignoring file bytebuffer.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file monitor.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file nio4r_ext.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file selector.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
Linking errors.
Okay, I found the DLDFLAGS are broken at least for M1.
Adding the following seems to make it build correctly:
# extconf.rb
# ... snip ...
$DLDFLAGS.gsub!(/\-arch\s+[^\s]+/, "")
dir_config "nio4r_ext"
create_makefile "nio4r_ext"
The problem seems to be that the linker gets confused what to do when you specify the arch flags.
This is now released and I tested it locally.