Faucet v 1.10.4 "SSL WRONG VERSION NUMBER"
lhchang opened this issue · 13 comments
Hello. We are trying to run Faucet in our Kubernetes cluster (v 1.10.4) and running into issues when trying to load the certs/private keys. We are running Faucet with the following args:
--ryu-ctl-privkey /etc/ryu/ssl/ControllerCertPrivateKey.pem --ryu-ctl-cert /etc/ryu/ssl/ControllerCert.pem --ryu-ca-certs /etc/ryu/ssl/RootCAcert.pem
These were different versus the ones in this example (https://docs.faucet.nz/en/1.10.4/installation.html?highlight=ssl#additional-arguments) but that was what was available in the listed available args. After running, we encounter this error.
loading app faucet.faucet
loading app os_ken.controller.ofp_handler
instantiating app None of DPSet
creating context dpset
instantiating app faucet.faucet of Faucet
instantiating app os_ken.controller.ofp_handler of OFPHandler
May 10 21:40:27 faucet INFO version 1.10.4
May 10 21:40:27 faucet INFO Reloading configuration
May 10 21:40:27 faucet INFO configuration /etc/faucet/faucet.yaml changed, analyzing differences
May 10 21:40:27 faucet INFO Add new datapath DPID 208962001918 (0x30a71b1bfe)
(1) wsgi starting up on http://0.0.0.0:9302
hub: uncaught exception: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/os_ken/lib/hub.py", line 69, in _launch
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/os_ken/lib/hub.py", line 150, in wrap_and_handle_ctx
handle(ctx.wrap_socket(sock, **ssl_args), addr)
File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 446, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 140, in __init__
self.do_handshake()
File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
return self._call_trampolining(
File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)
Some things we have tried:
- Rolling back to a older version of Faucet
- We have checked that our ControllerCert.pem format is valid through
openssl x509 -in ControllerCert.pem -text -noout - Checking pub/priv keys match with
openssl pkey -in ControllerCertPrivateKey.pem -pubout -outform pem | sha256sumand
openssl x509 -in ControllerCert.pem -pubkey -noout -outform pem | sha256sum
Thank you
Thanks for reporting this! Please would you try the self signed example at https://eventlet.net/doc/ssl.html? We'll also investigate.
Thanks anarkiwi.
Client side:
nc 127.0.0.1 8443
asdf
Running the Python code with our key/certs:
Traceback (most recent call last):
File "mytest.py", line 22, in <module>
print(client_conn.read(100))
File "/.local/lib/python3.8/site-packages/eventlet/green/OpenSSL/SSL.py", line 55, in read
return self.fd.read(size)
File "/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1865, in recv
self._raise_ssl_error(self._ssl, result)
File "/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1700, in _raise_ssl_error
_raise_current_error()
File "/.local/lib/python3.8/site-packages/OpenSSL/_util.py", line 55, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', '', 'wrong version number')]
I think you're getting this error because you're trying to connect to an SSL/TLS socket with a plain-text connection.
Can you try using openssl s_client instead of nc to talk to your encrypted socket listening on port 8443:
openssl s_client -connect localhost:8443
Ah you are right, thanks for the suggestion. Heres my output when using openssl s_client
openssl s_client -connect localhost:8443
CONNECTED(00000003)
140234030507328:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
On the server side:
Traceback (most recent call last):
File "mytest.py", line 22, in <module>
print(client_conn.read(100))
File "/home/.local/lib/python3.8/site-packages/eventlet/green/OpenSSL/SSL.py", line 55, in read
return self.fd.read(size)
File "/home/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1865, in recv
self._raise_ssl_error(self._ssl, result)
File "/home/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1700, in _raise_ssl_error
_raise_current_error()
File "/home/.local/lib/python3.8/site-packages/OpenSSL/_util.py", line 55, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', '', 'no shared cipher')]
Is this indicating the cert itself maybe missing something?
Sounds like there is a mismatch in ciphers between your version of openssl and pyopenssl/eventlet.
I tried running the test code below with python 3.8.10 on ubuntu 20.04, using eventlet==0.33.1 and pyOpenSSL==22.0.0:
https://gist.github.com/gizmoguy/0372afad6e34170a693751859e3c0302
And it worked fine, could you confirm what version of linux you are using and your python / python library versions?:
$ python3 eventlet-ssl-test.py
b'HELLO\n'
$ openssl s_client -connect localhost:8443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=66:EE certificate key too weak
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICZjCCAc+gAwIBAgIUHBS3xl7XeAChb+hPzRHtvTL7y5owDQYJKoZIhvcNAQEF
BQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjA1MjUwMjA5MDhaFw0yMzA1
MjUwMjA5MDhaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw
HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwgZ8wDQYJKoZIhvcNAQEB
BQADgY0AMIGJAoGBALG8UnUQuDbWAOj5L3IZUYsSlbi3d+/zyCobw6l0EBDdSLZG
xdQrc80p6icAs16vV9VNkDEkvYh7iE/nDGreULvZp72qKltudEW2tKNIq43tvtCB
knTI6PohBr5s7re/jOEnKhZro8skNaRNacDRT1aAYElNRMNP/VhMhvCd2ctBAgMB
AAGjUzBRMB0GA1UdDgQWBBRe2ipeaKtXaLS6gV1iGPBZBZNUJTAfBgNVHSMEGDAW
gBRe2ipeaKtXaLS6gV1iGPBZBZNUJTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBBQUAA4GBAJsyLXMp4XUkSAy4xJAHwleNNWUzKAraJbxMKoLdmfZ3A7At16gj
XM1mPSX6A9dHpCdOMgJCgsfP7lwCEydtHaS/fh0mTGNJI9O8eEmJq4OBhXBhJqQj
rqf+5XEbcz6LSJlBkt4InLQ21HNCaSuLPvfDFHhQHiPXO9ayW6J0Johu
-----END CERTIFICATE-----
subject=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
issuer=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1046 bytes and written 363 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 744815FE250C33938A99B0C46435D093C91A2EC83987260D99680EFA27037218
Session-ID-ctx:
Resumption PSK: 65B14DFC2B8E77A1C7DB6AC861CC4B75E994657EE8A48FDBDC210D38798D3290EE45EB83F742EA172A8532AF88667880
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 5f 87 a9 8e f0 65 b3 f3-3c dd 15 65 33 7c 67 98 _....e..<..e3|g.
0010 - 15 f0 c2 ee 46 01 35 14-93 ae 9b f1 d1 dd bc 17 ....F.5.........
0020 - fe b4 31 d1 0a 98 66 66-1d 2a 3d 4d f8 80 c9 b4 ..1...ff.*=M....
0030 - 36 7d d5 ff 74 42 5c 16-bf 2b 70 09 94 b7 b1 d0 6}..tB\..+p.....
0040 - cd 53 e7 28 11 39 4d 62-2f aa fd 4f 43 9e 40 59 .S.(.9Mb/..OC.@Y
0050 - ab cb d4 6a 78 95 63 44-ff 2f 50 ca 5c bc 5d 5c ...jx.cD./P.\.]\
0060 - fa 25 cf b1 be b6 d5 e0-73 1a 2e 36 e0 f7 52 45 .%......s..6..RE
0070 - d7 2e 27 13 af 64 6d 9a-f8 df 37 e7 b4 0b dc 8f ..'..dm...7.....
0080 - a9 70 e1 13 82 f9 6d f5-b9 b0 71 cc f2 09 73 77 .p....m...q...sw
0090 - 18 1a e0 92 3d 78 db 66-8d 1b a4 6a a2 6c 42 1a ....=x.f...j.lB.
00a0 - 6a cd 13 ec 89 e0 7f 8f-8d b1 fa f9 0d 09 01 ba j...............
00b0 - af 2f 6d 29 44 b5 83 d9-66 ba 3f aa 21 8c a0 09 ./m)D...f.?.!...
00c0 - 12 50 48 82 3d 67 4b b2-1f 3c bd da d4 3e 8a a4 .PH.=gK..<...>..
Start Time: 1653444942
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 7A9161448040884056075587BF14A37B45B40855B24B141E376A876418E292FC
Session-ID-ctx:
Resumption PSK: 056136C680B11569ECEB6398C57F45ACECC8E1AAB086675929065F20A0C12B83EB3AF3DCE950F886A7F711870E8221B9
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 5f 87 a9 8e f0 65 b3 f3-3c dd 15 65 33 7c 67 98 _....e..<..e3|g.
0010 - c0 30 c5 fd 84 af bf e8-9e 06 c8 8d a7 51 0d e2 .0...........Q..
0020 - 85 b4 0b bd 77 7f 8b 1c-ec 4d 29 79 67 95 45 eb ....w....M)yg.E.
0030 - 6b fe 70 95 85 a1 b2 de-57 51 27 01 a6 0a aa 10 k.p.....WQ'.....
0040 - 5c cb f7 64 00 cd 2a 65-31 12 b5 fa fe 85 1c c1 \..d..*e1.......
0050 - 1e f2 ae a4 4f 18 78 55-86 bf bc 4e 2a 86 8a 3a ....O.xU...N*..:
0060 - b7 56 04 07 e2 fb 99 32-5e ed af 0f 36 44 a1 68 .V.....2^...6D.h
0070 - 9a 71 d4 1c 1d 1d 6a a2-42 0d 3e 54 29 98 2d eb .q....j.B.>T).-.
0080 - 88 43 66 ff b7 fb 68 1b-c7 8a 5f 67 0b 91 14 7d .Cf...h..._g...}
0090 - b5 43 91 67 41 7c dd 35-98 d8 a7 cd 12 3c 29 f3 .C.gA|.5.....<).
00a0 - cf 8a 71 ee 5d 18 ce 7d-33 da e8 11 41 f1 20 aa ..q.]..}3...A. .
00b0 - fa e0 b9 c9 5c 3c f1 32-5d 0d bf 6f 7e 43 d6 98 ....\<.2]..o~C..
00c0 - 21 02 c0 fe 75 f7 f9 7f-6e 92 f9 ec e2 9c 1e d9 !...u...n.......
Start Time: 1653444942
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
HELLO
On our local machine using the eventlet example, it looks like we are running Python3.8, Ubuntu 20.04.1. The Faucet container seems to be using Python3.9, at least for 1.10.4.
For your local machine running the eventlet example can you try upgrading to ubuntu 20.04.4? This is what I am using here and it works fine.
I can confirm that using another generated cert that I can get the same results as your example output, but still not with the cert/keys we had on hand which were generated from another controller.
We tried pivoting a bit to see if we could get Faucet to talk TLS with OVS using the process described in https://docs.openvswitch.org/en/latest/howto/ssl/, we get the following output.
sudo faucet --ryu-ctl-privkey ctl-privkey.pem --ryu-ctl-cert ctl-cert.pem --ryu-ca-certs controllerca/cacert.pem --verbose
loading app faucet.faucet
loading app os_ken.controller.ofp_handler
instantiating app None of DPSet
creating context dpset
instantiating app faucet.faucet of Faucet
instantiating app os_ken.controller.ofp_handler of OFPHandler
BRICK dpset
PROVIDES EventDP TO {'Faucet': {'dpset'}}
PROVIDES EventDPReconnected TO {'Faucet': {'dpset'}}
CONSUMES EventOFPStateChange
CONSUMES EventOFPPortStatus
CONSUMES EventOFPSwitchFeatures
BRICK Faucet
CONSUMES EventFaucetEventSockHeartbeat
CONSUMES EventFaucetMaintainStackRoot
CONSUMES EventFaucetFastAdvertise
CONSUMES EventFaucetAdvertise
CONSUMES EventFaucetFastStateExpire
CONSUMES EventFaucetStateExpire
CONSUMES EventFaucetResolveGateways
CONSUMES EventDP
CONSUMES EventOFPDescStatsReply
CONSUMES EventOFPErrorMsg
CONSUMES EventOFPSwitchFeatures
CONSUMES EventOFPFlowRemoved
CONSUMES EventFaucetMetricUpdate
CONSUMES EventOFPPacketIn
CONSUMES EventOFPPortDescStatsReply
CONSUMES EventOFPPortStatus
CONSUMES EventDPReconnected
CONSUMES EventReconfigure
BRICK ofp_event
PROVIDES EventOFPStateChange TO {'dpset': {'dead', 'main'}}
PROVIDES EventOFPPortStatus TO {'dpset': {'main'}, 'Faucet': {'main'}}
PROVIDES EventOFPSwitchFeatures TO {'dpset': {'config'}, 'Faucet': {'config'}}
PROVIDES EventOFPDescStatsReply TO {'Faucet': {'main'}}
PROVIDES EventOFPErrorMsg TO {'Faucet': {'main'}}
PROVIDES EventOFPFlowRemoved TO {'Faucet': {'main'}}
PROVIDES EventOFPPacketIn TO {'Faucet': {'main'}}
PROVIDES EventOFPPortDescStatsReply TO {'Faucet': {'config'}}
CONSUMES EventOFPEchoReply
CONSUMES EventOFPEchoRequest
CONSUMES EventOFPErrorMsg
CONSUMES EventOFPHello
CONSUMES EventOFPPortDescStatsReply
CONSUMES EventOFPPortStatus
CONSUMES EventOFPSwitchFeatures
Terminated
Not sure if this is a related issue but it doesn't seem like the process stays up when fed the certs/keys from OVS PKI. Ideally we would like to get this to work with our original switch but if there is something I missed with the OVS method hopefully that may help with our original problem?
That log doesn't have the error that occurred, what is in your faucet_exception.log file?
Found the issue, had another instance running at the time. However after resolving that I am running back into another SSL issue:
hub: uncaught exception: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/os_ken/lib/hub.py", line 69, in _launch
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/os_ken/lib/hub.py", line 150, in wrap_and_handle_ctx
handle(ctx.wrap_socket(sock, **ssl_args), addr)
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 445, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 139, in __init__
self.do_handshake()
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
return self._call_trampolining(
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 161, in _call_trampolining
return func(*a, **kw)
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1131)
This is using certs/keys generated from the OVS PKI tutorial
What are you using to connect to the SSL/TLS port of faucet? netcat? openssl s_client? an openflow dataplane?
Have you checked whatever client you are using is correctly configured to speak SSL/TLS rather than plaintext?
In the context of my previous comment we were attempting to use Open vSwitch to connect to Faucet. In the context of the original issue, we were trying to connect a physical OpenFlow switch to Faucet using certificates/keys pulled from another SDN controller that we had originally adopted the switches with. We had only used openssl s_client/netcat to test the Python eventlet code that was linked before.
Can you try using openssl s_client to connect to the faucet SSL/TLS port instead of OVS?