apple/foundationdb

Setting FDBNetworkOption::FDB_NET_OPTION_TLS_KEY_BYTES has no effect

c4pQ opened this issue · 1 comments

c4pQ commented

Hi!

For some reason setting FDBNetworkOption::FDB_NET_OPTION_TLS_KEY_BYTES and similar TLS_CERT_BYTES, TLS_CA_BYTES seems like does not have any effect. We use fdb_network_set_option() for this matter.

After setting everything we receive transaction timeouts and the following in client logs:

<Event Severity="20" Time="1707906229.507757" DateTime="2024-02-14T10:23:49Z" Type="N2_ConnectHandshakeError" ID="8f39f794bfafa886" SuppressedEventCount="1" ErrorCode="337047686" Message="certificate verify failed (SSL routines, tls_process_server_certificate)" WhichMeans="error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.507757" DateTime="2024-02-14T10:23:49Z" Type="ConnectionTimedOut" ID="8f39f794bfafa886" SuppressedEventCount="1" PeerAddr="10.233.155.166:4500:tls(fromHostname)" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.507757" DateTime="2024-02-14T10:23:49Z" Type="ConnectionClosed" ID="8f39f794bfafa886" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="1" PeerAddr="10.233.155.166:4500:tls(fromHostname)" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.507757" DateTime="2024-02-14T10:23:49Z" Type="TooManyConnectionsClosedMarkFailed" ID="0000000000000000" Dest="10.233.155.166:4500:tls(fromHostname)" ClosedCount="6" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.507757" DateTime="2024-02-14T10:23:49Z" Type="NotifyAddressFailed" ID="0000000000000000" SuppressedEventCount="0" Address="10.233.155.166:4500:tls(fromHostname)" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.516344" DateTime="2024-02-14T10:23:49Z" Type="AlreadyDisconnected" ID="0000000000000000" SuppressedEventCount="3" Addr="10.232.155.164:4500:tls(fromHostname)" Reason="Disconnected" Tok="ffffffffffffffff" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />
<Event Severity="10" Time="1707906229.931201" DateTime="2024-02-14T10:23:49Z" Type="PingLatency" ID="0000000000000000" Elapsed="0.460624" PeerAddr="10.232.155.164:4500:tls(fromHostname)" MinLatency="0" MaxLatency="0" MeanLatency="0" MedianLatency="0" P90Latency="0" Count="0" BytesReceived="0" BytesSent="0" TimeoutCount="0" ConnectOutgoingCount="3" ConnectIncomingCount="0" ConnectFailedCount="2" ConnectMinLatency="0" ConnectMaxLatency="0" ConnectMeanLatency="0" ConnectMedianLatency="0" ConnectP90Latency="0" ThreadID="12547063887027460357" Machine="10.109.161.182:108" LogGroup="ss-write" ClientDescription="primary-7.1.51-12547063887027460357" />

On the other hand setting environment variables FDB_TLS_CERTIFICATE_FILE, FDB_TLS_KEY_FILE, FDB_TLS_CA_FILE with the paths to the files with same content does the trick and everything works.

We've also tried setting FDBNetworkOption::FDB_NET_OPTION_TLS_KEY_PATH and other _PATH options with no effect as well.

Environment:

  • FDB 7.1.51
  • Client in Docker container Linux c31802bb42bc 6.5.0-17-generic #17-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 11 14:01:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
c4pQ commented

The route cause is that setup was called before setting the options.

But one thing is still confusing - how we managed to apply those settings at some point and successfully access FDB while setup-set_network_option order was incorrect.