Incompatible OpenSSL library prevents GD from running on F28
vladionescu opened this issue · 14 comments
Trying to run grocery-delivery results in an unhandled exception:
# /opt/opscode/embedded/bin/grocery-delivery --help
Traceback (most recent call last):
11: from /opt/opscode/embedded/bin/grocery-delivery:23:in `<main>'
10: from /opt/opscode/embedded/bin/grocery-delivery:23:in `load'
9: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:21:in `<top (required)>'
8: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
7: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
6: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/between_meals-0.0.9/lib/between_meals/repo/git.rb:20:in `<top (required)>'
5: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
4: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
3: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged.rb:6:in `<top (required)>'
2: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged.rb:10:in `rescue in <top (required)>'
1: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so: undefined symbol: OPENSSL_sk_num - /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so (LoadError)
It looks like rugged depends on libssl but is expecting a newer version than what is vendored in /opt/opscode/.
# ldd /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so
linux-vdso.so.1 (0x00007ffcbf2f0000)
librt.so.1 => /lib64/librt.so.1 (0x00007f2768887000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2768668000)
libssl.so.1.0.0 => /opt/opscode/embedded/lib/libssl.so.1.0.0 (0x00007f27683fc000)
libz.so.1 => /opt/opscode/embedded/lib/libz.so.1 (0x00007f27681e5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f2767fe1000)
libcrypto.so.1.0.0 => /opt/opscode/embedded/lib/libcrypto.so.1.0.0 (0x00007f2767b32000)
libruby.so.2.5 => /opt/opscode/embedded/lib/libruby.so.2.5 (0x00007f27675fa000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f27673c1000)
libm.so.6 => /lib64/libm.so.6 (0x00007f276702d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f2766c6e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2768db6000)
# objdump -T /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so | grep OPENSSL_sk_num
0000000000000000 D *UND* 0000000000000000 OPENSSL_sk_num
If I try overwriting /opt/opscode/embedded/lib/libssl.so.1.0.0 with the system's /usr/lib64/libssl.so.1.1.0i that LoadError exception goes away (another exception prevents gd from working) indicating it's expecting a newer libssl than what's in opscode.
I tried a /opt/opscode/embedded/bin/gem update as well as uninstalling openssl, rugged, grocery_delivery and installing them again in order to rebuild the .so files, but that didn't help.
This is a fresh install of chef workstation and server on Fedora 28. I installed grocery_delivery via /opt/opscode/embedded/bin/gem install grocery_delivery from https://clburlison.com/chef-grocery-delivery-setup/.
# dnf list | grep chef
chef-server-core.x86_64 12.18.14-1.el7 @@commandline
chef-workstation.x86_64 0.2.43-1.el6 @@commandline
rubygem-chef-zero.noarch 2.2-6.fc28 fedora
rubygem-chef-zero-doc.noarch 2.2-6.fc28 fedora
# chef --version
Chef Workstation: 0.2.43
chef-run: 0.2.4
chef-client: 14.8.12
delivery-cli: master (5fb4908da53579c9dcf4894d4acf94d2d9ee6475)
berks: 7.0.7
test-kitchen: 1.24.0
inspec: 3.2.6
Try running with ChefDK. g-d was kinda developed using it. I've tried in the past using regular Chef client with the gems installed and had weird problems surrounding rugged.
That makes sense.
I uninstalled the chef-workstation package, and uninstalled rugged grocery_delivery and between_meals from opscode gems.
Then I installed the latest chefdk el7 package, setup my env to use chefdk's Ruby (per chefdk setup https://docs.chef.io/chefdk_setup.html) and ran gem install grocery_delivery.
But I'm getting another undefined symbol error when trying to run gd.
# which gem
/opt/chefdk/embedded/bin/gem
# gem install grocery_delivery
Fetching: rugged-0.27.7.gem (100%)
Building native extensions. This could take a while...
Successfully installed rugged-0.27.7
Fetching: between_meals-0.0.9.gem (100%)
Successfully installed between_meals-0.0.9
Fetching: grocery_delivery-0.0.8.gem (100%)
Successfully installed grocery_delivery-0.0.8
3 gems installed
# grocery-delivery --help
/opt/chefdk/embedded/bin/ruby: symbol lookup error: /root/.chefdk/gem/ruby/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so: undefined symbol: OPENSSL_init_ssl
Try installing rugged 0.26.6 instead of letting it use the latest rugged. I believe 0.27 is when I hit the SSL errors similar to you.
that is, erase the current rugged, manually install 0.26.6, then finish the g-d install. Once the dependency is satisfied it shouldn't try to upgrade to the 0.27 anymore
[root@basic07 ~]# /opt/chefdk/embedded/bin/gem list | grep rugg
rugged (0.26.6)
Thanks, I just gave that a shot.
Still nothing 😢
# which gem
/opt/chefdk/embedded/bin/gem
# gem uninstall grocery_delivery between_meals rugged
Gem 'grocery_delivery' is not installed
Gem 'between_meals' is not installed
Gem 'rugged' is not installed
# gem install rugged --version '= 0.26.6'
Fetching: rugged-0.26.6.gem (100%)
Building native extensions. This could take a while...
Successfully installed rugged-0.26.6
1 gem installed
# gem install grocery_delivery
Fetching: between_meals-0.0.9.gem (100%)
Successfully installed between_meals-0.0.9
Fetching: grocery_delivery-0.0.8.gem (100%)
Successfully installed grocery_delivery-0.0.8
2 gems installed
# gem list | grep rugg
rugged (0.26.6)
# grocery-delivery --help
/opt/chefdk/embedded/bin/ruby: symbol lookup error: /root/.chefdk/gem/ruby/2.5.0/gems/rugged-0.26.6/lib/rugged/rugged.so: undefined symbol: OPENSSL_init_ssl
Hmm, this isn't the issue I was thinking of... this is odd.
It looks like rugged gets compiled against the openssl headers/libs on your system, not the ones in your gem dir, but then when you run it, it runs against the one in the gem dir.
What' does gem list -l show? Also dpkg -l | grep openssl ?
# which gem
/opt/chefdk/embedded/bin/gem
# gem list -l
*** LOCAL GEMS ***
activesupport (5.2.2)
addressable (2.5.2)
app_conf (0.4.2)
appbundler (0.11.6)
artifactory (3.0.0)
ast (2.4.0)
aws-sdk (2.11.212)
aws-sdk-core (2.11.212)
aws-sdk-resources (2.11.212)
aws-sigv4 (1.0.3)
axiom-types (0.1.1)
azure_graph_rbac (0.17.0)
azure_mgmt_key_vault (0.17.2)
azure_mgmt_network (0.18.3)
azure_mgmt_resources (0.17.2)
backports (3.11.4)
bcrypt_pbkdf (1.0.0)
berkshelf (7.0.7)
between_meals (0.0.9)
bigdecimal (default: 1.3.4)
binding_of_caller (0.8.0)
builder (3.2.3)
bundler (1.17.3, default: 1.16.6)
byebug (10.0.2)
chef (14.10.9)
chef-api (0.9.0)
chef-apply (0.2.8)
chef-config (14.10.9)
chef-dk (3.7.23)
chef-provisioning (2.7.4)
chef-provisioning-aws (3.0.7)
chef-provisioning-fog (0.26.3)
chef-sugar (5.0.0)
chef-telemetry (0.1.8)
chef-vault (3.4.3)
chef-zero (14.0.11)
cheffish (14.0.4)
chefspec (7.3.4)
chefstyle (0.11.2)
citrus (3.0.2)
cleanroom (1.0.0)
cmath (default: 1.0.0)
coderay (1.1.2)
coercible (1.0.0)
colorize (0.8.1)
concurrent-ruby (1.1.4)
cookbook-omnifetch (0.8.0)
cookstyle (3.0.2)
csv (default: 1.0.0)
cucumber (3.1.2)
cucumber-core (3.2.1)
cucumber-expressions (6.0.1)
cucumber-tag_expressions (1.1.1)
cucumber-wire (0.0.1)
date (default: 1.0.0)
dco (1.0.1)
debug_inspector (0.0.3)
declarative (0.0.10)
declarative-option (0.1.0)
dep-selector-libgecode (1.3.1)
dep_selector (1.0.6)
descendants_tracker (0.0.4)
did_you_mean (1.2.0)
diff-lcs (1.3)
diffy (3.3.0)
docker-api (1.34.2)
domain_name (0.5.20180417)
droplet_kit (2.8.0)
equalizer (0.0.11)
equatable (0.5.0)
erubis (2.7.0)
etc (default: 1.0.0)
excon (0.62.0)
faraday (0.15.4)
faraday-cookie_jar (0.0.6)
faraday_middleware (0.12.2)
fauxhai (6.10.0)
fcntl (default: 1.0.0)
ffi (1.10.0)
ffi-libarchive (0.4.6)
ffi-rzmq (2.0.7)
ffi-rzmq-core (1.0.7)
ffi-yajl (2.3.1)
fiddle (default: 1.0.0)
filesize (0.1.1)
fileutils (default: 1.0.2)
fog-aws (2.0.1)
fog-core (1.45.0)
fog-digitalocean (0.4.0)
fog-joyent (0.0.1)
fog-json (1.2.0)
fog-openstack (0.3.10)
fog-rackspace (0.1.6)
fog-scaleway (0.5.0)
fog-softlayer (1.1.4)
fog-xenserver (1.0.0)
fog-xml (0.1.3)
foodcritic (15.1.0)
formatador (0.2.5)
fuzzyurl (0.9.0)
gcewinpass (1.1.0)
gherkin (5.1.0)
git (1.5.0)
google-api-client (0.23.9)
google-protobuf (3.5.2)
googleauth (0.6.7)
grocery_delivery (0.0.8)
gssapi (1.2.0)
guard (2.15.0)
gyoku (1.3.1)
hashie (3.6.0)
highline (1.7.10)
htmlentities (4.3.4)
http (2.2.2)
http-cookie (1.0.3)
http-form_data (1.0.3)
http_parser.rb (0.6.0)
httpclient (2.8.3)
i18n (1.5.3)
ice_nine (0.11.2)
inifile (3.0.0)
iniparse (1.4.4)
inspec (3.4.1)
io-console (default: 0.4.6)
ipaddr (default: 1.2.0)
ipaddress (0.8.3)
jmespath (1.4.0)
json (default: 2.1.0)
jwt (2.1.0)
kartograph (0.2.7)
kitchen-azurerm (0.14.8)
kitchen-digitalocean (0.10.1)
kitchen-dokken (2.6.7)
kitchen-ec2 (2.4.0)
kitchen-google (2.0.1)
kitchen-hyperv (0.5.2)
kitchen-inspec (1.0.1)
kitchen-vagrant (1.4.0)
knife-acl (1.0.3)
knife-cloud (1.2.3)
knife-ec2 (0.19.16)
knife-google (3.3.7)
knife-opc (0.4.1)
knife-push (1.1.2)
knife-spork (1.7.2)
knife-tidy (2.0.0)
knife-vsphere (2.1.1)
knife-windows (1.9.6)
libyajl2 (1.2.0)
listen (3.1.5)
little-plugger (1.1.4)
lockfile (2.1.3)
logging (2.2.2)
logify (0.2.0)
lumberjack (1.0.13)
memoist (0.16.0)
method_source (0.9.2)
mime-types (3.2.2)
mime-types-data (3.2018.0812)
mini_portile2 (2.4.0)
minitar (0.8)
minitest (5.10.3)
mixlib-archive (0.4.19)
mixlib-authentication (2.1.1)
mixlib-cli (1.7.0)
mixlib-config (2.2.18)
mixlib-install (3.11.5)
mixlib-log (2.0.9)
mixlib-shellout (2.4.4)
mixlib-versioning (1.2.7)
molinillo (0.6.6)
ms_rest (0.7.3)
ms_rest_azure (0.11.0)
multi_json (1.13.1)
multi_test (0.1.2)
multipart-post (2.0.0)
necromancer (0.4.0)
nenv (0.3.0)
net-scp (1.2.1)
net-sftp (2.1.2)
net-ssh (4.2.0)
net-ssh-gateway (1.3.0)
net-ssh-multi (1.2.1)
net-telnet (0.1.1)
netaddr (1.5.1)
nokogiri (1.10.1)
nori (2.6.0)
notiffany (0.1.1)
octokit (4.13.0)
ohai (14.8.10)
openssl (default: 2.1.2)
opscode-pushy-client (2.4.11)
os (1.0.0)
paint (1.0.1)
parallel (1.13.0)
parser (2.6.0.0)
parslet (1.8.2)
pastel (0.7.2)
plist (3.5.0)
polyglot (0.3.5)
power_assert (1.1.1)
powerpack (0.1.2)
proxifier (1.0.3)
pry (0.12.2)
pry-byebug (3.6.0)
pry-remote (0.1.8)
pry-stack_explorer (0.4.9.3)
psych (default: 3.0.2)
public_suffix (3.0.3)
r18n-core (3.2.0)
r18n-desktop (3.2.0)
rack (2.0.6)
rainbow (3.0.0)
rake (12.3.0)
rb-fsevent (0.10.3)
rb-inotify (0.10.0)
rb-readline (0.5.5)
rbnacl (6.0.0)
rbnacl-libsodium (1.0.16)
rbvmomi (1.13.0)
rdoc (default: 6.0.1)
representable (3.0.4)
resource_kit (0.1.7)
retriable (3.1.2)
retryable (2.0.4)
rspec (3.8.0)
rspec-core (3.8.0)
rspec-expectations (3.8.2)
rspec-its (1.2.0)
rspec-mocks (3.8.0)
rspec-support (3.8.0)
rspec_junit_formatter (0.2.3)
rubocop (0.55.0)
ruby-prof (0.17.0)
ruby-progressbar (1.10.0)
ruby-shadow (2.5.0)
ruby_dep (1.5.0)
rubygems-update (2.7.8)
rubyntlm (0.6.2)
rubyzip (1.2.2)
rufus-lru (1.1.0)
rugged (0.26.6)
sawyer (0.8.1)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
semverse (3.0.0)
serverspec (2.41.3)
sfl (2.3)
shellany (0.0.1)
signet (0.11.0)
slop (3.6.0)
solve (4.0.2)
specinfra (2.76.9)
sshkey (1.9.0)
sslshake (1.3.0)
stove (7.1.0)
stringio (default: 0.0.1)
strings (0.1.4)
strings-ansi (0.1.0)
strscan (default: 1.0.0)
syslog-logger (1.6.8)
systemu (2.6.5)
term-ansicolor (1.7.1)
test-kitchen (1.24.0)
test-unit (3.2.7)
thor (0.20.3)
thread_safe (0.3.6)
timeliness (0.3.8)
timers (4.3.0)
tins (1.20.2)
toml-rb (1.1.2)
tomlrb (1.2.8)
train (1.7.1)
treetop (1.6.10)
trollop (2.9.9)
tty-color (0.4.3)
tty-cursor (0.6.0)
tty-prompt (0.18.1)
tty-reader (0.5.0)
tty-screen (0.6.5)
tty-spinner (0.9.0)
tty-table (0.10.0)
tzinfo (1.2.5)
uber (0.1.0)
ubuntu_ami (0.4.2)
unf (0.1.4)
unf_ext (0.0.7.5)
unicode-display_width (1.4.1)
unicode_utils (1.4.0)
uuidtools (2.1.5)
virtus (1.0.5)
webrick (default: 1.4.2)
winrm (2.3.1)
winrm-elevated (1.1.1)
winrm-fs (1.3.2)
wisper (2.0.0)
wmi-lite (1.0.2)
xmlrpc (0.3.0)
yard (0.9.18)
zlib (default: 1.0.0)
# rpm -qa | grep openssl
rubygem-openssl-2.1.2-95.fc28.x86_64
openssl-libs-1.1.0i-1.fc28.x86_64
compat-openssl10-1.0.2o-1.fc28.x86_64
openssl-pkcs11-0.4.8-2.fc28.x86_64
apr-util-openssl-1.6.1-8.fc28.x86_64
openssl-devel-1.1.0i-1.fc28.x86_64
xmlsec1-openssl-1.2.25-4.fc28.x86_64
openssl-1.1.0i-1.fc28.x86_64
I think this looks like it tries to load the gem dir openssl. I'm not familiar with Ruby internals to say for sure.
# strace /opt/chefdk/embedded/bin/ruby -e "require 'rugged'" 2>&1 | grep ssl
stat("/opt/chefdk/embedded/lib/ruby/gems/2.5.0/specifications/default/openssl-2.1.2.gemspec", {st_mode=S_IFREG|0644, st_size=4373, ...}) = 0
openat(AT_FDCWD, "/opt/chefdk/embedded/lib/ruby/gems/2.5.0/specifications/default/openssl-2.1.2.gemspec", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/opt/chefdk/embedded/lib/ruby/gems/2.5.0/specifications/default/openssl-2.1.2.gemspec", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/opt/chefdk/embedded/lib/ruby/gems/2.5.0/specifications/sslshake-1.3.0.gemspec", O_RDONLY|O_CLOEXEC) = 7
...
write(2, "/opt/chefdk/embedded/bin/ruby: s"..., 157/opt/chefdk/embedded/bin/ruby: symbol lookup error: /root/.chefdk/gem/ruby/2.5.0/gems/rugged-0.26.6/lib/rugged/rugged.so: undefined symbol: OPENSSL_init_ssl
There are a few grocery-delivery binaries around, all seem to be different, but they all exit with the same error.
# which grocery-delivery
/root/.chefdk/gem/ruby/2.5.0/bin/grocery-delivery
# find /opt -name grocery-delivery
/opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery
/opt/opscode/embedded/bin/grocery-delivery
# ls -lash /root/.chefdk/gem/ruby/2.5.0/bin/grocery-delivery /opt/opscode/embedded/bin/grocery-delivery /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery
4.0K -rwxr-xr-x 1 root root 672 Feb 3 10:57 /opt/opscode/embedded/bin/grocery-delivery
12K -rwxr-xr-x 1 root root 9.1K Feb 3 10:57 /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery
4.0K -rwxr-xr-x 1 root root 671 Feb 3 13:04 /root/.chefdk/gem/ruby/2.5.0/bin/grocery-delivery
# md5sum /root/.chefdk/gem/ruby/2.5.0/bin/grocery-delivery /opt/opscode/embedded/bin/grocery-delivery /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery
64c1587e9ee2af816ec0adaa2e5849fb /root/.chefdk/gem/ruby/2.5.0/bin/grocery-delivery
d92786bf5bcead52c69be1752253bd7c /opt/opscode/embedded/bin/grocery-delivery
81d1e0c5857ab8d753cf4114d6da1f6e /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery
You'll want to clean up all the cruft in your chef-server install. I'll repro the chefdk part and get back to you soon!
I can confirm a repro of this on Debian with chefdk2, chefdk3, rugged 0.26 and rugged 0.27.
My suspicion is that this is debian/ubuntu-specific, but I'll continue to debug. I'm about to get on a plane, so may be a day or two.
Yeah rugged is getting compiled wrong. The system one has it:
[phild@fuel lib]$ objdump -T /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep init_ssl
0000000000035ca0 g DF .text 000000000000010e OPENSSL_1_1_0 OPENSSL_init_ssl
But the embedded one does not:
[phild@fuel lib]$ objdump -T /opt/chefdk/embedded/lib/libssl.so.1.0.0 | grep init_ssl
[phild@fuel lib]$
And the problem here is that the build process is referencing the header files from the system. You can see this if you apt-get remove libssl-dev, you will no longer be able to /opt/chefdk/embedded/bin/gem install rugged, it'll fail to find the headers.
However, they do distribute the include files in /opt/chefdk/embedded/include/openssl - they just don't get used in the installation.
You can work around this like so:
root@fuel:/opt/chefdk/embedded# OPENSSL_ROOT_DIR=/opt/chefdk/embedded /opt/chefdk/embedded/bin/gem install --no-user-install rugged
Fetching: rugged-0.27.7.gem (100%)
Building native extensions. This could take a while...
Successfully installed rugged-0.27.7
1 gem installed
root@fuel:/opt/chefdk/embedded# /opt/chefdk/embedded/bin/grocery-delivery --help
Usage: grocery-delivery [options]
-n, --dry-run Dryrun mode
-v, --verbosity Verbosity level. Twice for debug.
-c, --config-file FILE config file
-l, --lockfile FILE lockfile
--stdout Log to stdout as well.
-p, --pidfile FILE pidfile
root@fuel:/opt/chefdk/embedded#
I will file an upstream bug. It's likely a bug in omnibus.
I filed chef/omnibus#876 - it's unclear if it's omnibus or chefdk, but that'll sort it out, and you have a work around in the meantime. Sorry for the delay!
Your workaround works on F28! Thanks a lot for the quick turnaround here and for the upstream bug 🥇
Try installing rugged 0.26.6 instead of letting it use the latest rugged. I believe 0.27 is when I hit the SSL errors similar to you.
that is, erase the current rugged, manually install 0.26.6, then finish the g-d install. Once the dependency is satisfied it shouldn't try to upgrade to the 0.27 anymore
[root@basic07 ~]# /opt/chefdk/embedded/bin/gem list | grep rugg rugged (0.26.6)
This fixed my issue on Ubuntu 18.04.3! Thanks.