Azure/WALinuxAgent

[2.2.12] Fail to run Extension in FIPS mode

yuxisun1217 opened this issue · 6 comments

yuxisun1217 commented

Hi,

In FIPS mode, the Extension doesn't work sell.
It also impact the VM provisioning if authenticate with ssh key in FIPS mode.

Packages:
RHEL-7.4
WALA-2.2.12
openssl-1.0.2k-8.el7.x86_64

Steps to Reproduce:

  1. Prepare a VM in Azure. Enable FIPS:
    1). yum install dracut-fips
    2). mv -v /boot/initramfs-$(uname -r).img{,.bak}
    dracut
    3). grubby --update-kernel=$(grubby --default-kernel) --args=fips=1
    uuid=$(findmnt -no uuid /boot)
    [[ -n $uuid ]] && grubby --update-kernel=$(grubby --default-kernel) --args=boot=UUID=${uuid}
    4). reboot
  2. Run "reset remote access" to install an Extension into the VM. There's no error logs in waagent.log
  3. Set "OS.EnableFIPS=y" in /etc/waagent.conf. Restart waagent service
  4. Check if the extension works. Check /var/log/waagent.log

Actual results:
The extension doesn't work. The waagent -run-exthandler process keeps restarting.

The error logs in waagent.log: (Seems the same as #668 )

2017/06/09 18:41:23.406056 WARNING Server preferred version:2015-04-05
2017/06/09 18:41:28.146195 ERROR Command: '/usr/bin/openssl cms -decrypt -in /var/lib/waagent/Certificates.p7m -inkey /var/lib/waagent/TransportPrivate.pem -recip /var/lib/waagent/TransportCert.pem | /usr/bin/openssl pkcs12 -nodes -password pass: -out /var/lib/waagent/Certificates.pem'
2017/06/09 18:41:28.184821 ERROR Return code: 1
2017/06/09 18:41:28.195972 ERROR Result: MAC verified OK
Error outputting keys and certificates
140308542494624:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140308542494624:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140308542494624:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

2017/06/09 18:41:28.306785 ERROR Failed to run 'run-exthandlers': Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/agent.py", line 147, in main
    agent.run_exthandlers()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/agent.py", line 117, in run_exthandlers
    update_handler.run()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/ga/update.py", line 236, in run
    get_monitor_handler().run()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/ga/monitor.py", line 96, in run
    self.init_sysinfo()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/ga/monitor.py", line 121, in init_sysinfo
    protocol = self.protocol_util.get_protocol()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/util.py", line 261, in get_protocol
    self.protocol = self._detect_protocol(protocols)
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/util.py", line 183, in _detect_protocol
    return self._detect_wire_protocol()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/util.py", line 152, in _detect_wire_protocol
    protocol.detect()
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/wire.py", line 99, in detect
    self.client.update_goal_state(forced=True)
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/wire.py", line 729, in update_goal_state
    self.update_certs(goal_state)
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/wire.py", line 688, in update_certs
    self.certs = Certificates(self, xml_text)
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/wire.py", line 1147, in __init__
    self.parse(xml_text)
  File "/usr/lib/python2.7/site-packages/azurelinuxagent/common/protocol/wire.py", line 1218, in parse
    thumbprint = thumbprints[pubkey]
KeyError: u'-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAocW4DnlCqiI8MrQAj8ec\nZACpCKUwPCPg3vDYGLdwqvKs9H9bMxy1cXzgGFnPgfG/azfyzB3kbDlW+I9DMLq9\nw2ntdRdDn2esLlToWymQcQjs0FesvJhppgJSe0hOlUCBBgmWqFC1Lfom+SGDnxeR\nkc6z42ExX4VPRvNKeU7yZwoOqpTZmy2FXNxVe3db0nB87ZRRy15gXjHICFPMG4HV\nsPI/xDttaqTLlzmmGVh36oxE8WVCNiTarTOTNfA4udNmk07Xw2Y3lrms28jr2AKj\ngxpI+IUraN8reLUVNmkumeNwEl0ttdv6ngltkGCoNh+3lKVpnugahB+GCQ5hamCe\nGQIDAQAB\n-----END PUBLIC KEY-----\n'

I run the command manually and also get error messages. My steps:

  1. export OPENSSL_FIPS=1
  2. Run command:
#/usr/bin/openssl cms -decrypt -in /var/lib/waagent/Certificates.p7m -inkey /var/lib/waagent/TransportPrivate.pem -recip /var/lib/waagent/TransportCert.pem | /usr/bin/openssl pkcs12 -nodes -password pass: -out /var/lib/waagent/Certificates.pem
MAC verified OK
Error outputting keys and certificates
139851566958496:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
139851566958496:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
139851566958496:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

I'm not sure if the openssl pkcs12 is supported in FIPS mode...

sriramsa commented

Seen in Centos 6.7 too. Agent version WALiuxAgent-2.2.13 .

I found some solutions recommending using -descert option when dealing with PKCS12.
https://community.rsa.com/docs/DOC-51951

[  OK  ]Starting puppetmaster: 2017/06/22 06:59:00.412282 ERROR Command: '/usr/bin/openssl cms -decrypt -in /var/lib/waagent/Certificates.p7m -inkey /var/lib/waagent/TransportPrivate.pem -recip /var/lib/waagent/TransportCert.pem | /usr/bin/openssl pkcs12 -nodes -password pass: -out /var/lib/waagent/Certificates.pem'
2017/06/22 06:59:00.412560 ERROR Return code: 1
2017/06/22 06:59:00.412835 ERROR Result: MAC verified OK
Error outputting keys and certificates
139760389863240:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:186:
139760389863240:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:83:
139760389863240:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:130:2017/06/22 06:59:00.585337 ERROR Failed to run 'run-exthandlers': Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/agent.py", line 147, in main
    agent.run_exthandlers()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/agent.py", line 117, in run_exthandlers
    update_handler.run()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/ga/update.py", line 236, in run
    get_monitor_handler().run()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/ga/monitor.py", line 96, in run
    self.init_sysinfo()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/ga/monitor.py", line 121, in init_sysinfo
    protocol = self.protocol_util.get_protocol()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/util.py", line 256, in get_protocol
    self.protocol = self._detect_protocol(protocols)
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/util.py", line 178, in _detect_protocol
    return self._detect_wire_protocol()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/util.py", line 152, in _detect_wire_protocol
    protocol.detect()
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/wire.py", line 99, in detect
    self.client.update_goal_state(forced=True)
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/wire.py", line 729, in update_goal_state
    self.update_certs(goal_state)
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/wire.py", line 688, in update_certs
    self.certs = Certificates(self, xml_text)
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/wire.py", line 1147, in __init__
    self.parse(xml_text)
  File "/usr/lib/python2.6/site-packages/azurelinuxagent/common/protocol/wire.py", line 1218, in parse
    thumbprint = thumbprints[pubkey]
KeyError: u'-----BEGIN PUBLIC 
2017/06/22 06:59:01.000 INFO Event: ame=WALinuxAgent, op=sage=Agent WALiuxAgent-2.2.13 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' is successfully running
hglkrijger commented

I investigated this, and the certificates we receive are not FIPS compliant, so from the agent perspective there is nothing to be done here.

sbohlen commented

@hglkrijger Are you certain that its only the cert that's the issue here? These issues from RHEL Bugzilla seem to imply its also the encryption being applied:

https://bugzilla.redhat.com/show_bug.cgi?id=1460671
https://bugzilla.redhat.com/show_bug.cgi?id=1461243

Even with a different cert, am I misunderstanding that the same algo would still be used, seemingly resulting in a (still) invalid encryption for FIPS mode? Note this isn't my area of expertise, so it well be that I'm wrong here....

yuxisun1217 commented

Hi @hglkrijger ,
Do you mean that WALA cannot support FIPS mode? Thanks!

jasonzio commented

@yuxisun1217 that's pretty much correct.

yuxisun1217 commented

@jasonzio OK. Thank you so much :)