segmentation fault might occur when SSL renegotiation happens
tokers opened this issue · 4 comments
Hello!
Recently I found one of our Nginx worker process exited abnormally (segmentation fault). The backtrace is like:
nginx: worker process(ngx_http_lua_ssl_cert_handler+0x1c2) [0x5b8313]
nginx: worker process(tls_post_process_client_hello+0x1be) [0x7302e9]
nginx: worker process(ossl_statem_server_post_process_message+0x6c) [0x72dd0f]
nginx: worker process(read_state_machine) [0x71b25d]
nginx: worker process(state_machine) [0x71abc1]
nginx: worker process(ossl_statem_accept+0x1d) [0x71a74d]
nginx: worker process(ssl3_read_bytes+0xed4) [0x6e9846]
nginx: worker process(ssl3_read_internal) [0x6f33e9]
nginx: worker process(ssl3_read+0x38)
nginx: worker process(ssl_read_internal+0x165)
nginx: worker process(SSL_read+0x5b)
nginx: worker process(ngx_ssl_recv+0xc5)
nginx: worker process(ngx_http_v2_read_handler)
nginx: worker process(ngx_http_v2_idle_handler+0x116)
nginx: worker process(ngx_epoll_process_events)
nginx: worker process(ngx_process_events_and_timers+0xd3)
The coredump point is inside function ngx_http_lua_ssl_cert_handler.
You can reproduce this problem by the following way.
server {
listen 8443 ssl http2;
ssl_certificate_by_lua_block {
return;
}
ssl_certificate /path/to/cert;
ssl_certificate_key /path/to/pkey;
location / {
return 200;
}
}
openssl s_client -connect 127.0.0.1:8443 -alpn h2 -reconnectThen type "R" in the interactive mode (trigger the TLS reneogatitaion).
ALPN protocol: h2
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 8C3B8082AEEDC0AF9894C28A7ED50F2B52AA5C2274B4E15509DFEA097B12CB42
Session-ID-ctx:
Master-Key: DABD0D31B6A799DB05E199DC89045833480E47EF8F43DC2DDD36A4851C2026DB05A242580928135214395734343EADF7
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 92 bf 74 6e 7f 3d 7a 0b-34 4a 48 18 11 7f 60 31 ..tn.=z.4JH...`1
0010 - 1e 9b 46 8f 11 62 d6 b2-83 ea 2e d5 45 26 a6 46 ..F..b......E&.F
0020 - a5 c8 13 48 35 0b 87 fd-26 cd 6e aa c1 0a 35 c9 ...H5...&.n...5.
0030 - 1b 2f 93 5a d6 9d 11 b6-c2 5a df b0 53 23 b3 b4 ./.Z.....Z..S#..
0040 - de d8 73 50 b6 61 e5 1b-08 b9 9b b4 22 18 cc e0 ..sP.a......"...
0050 - 3a 13 c0 d0 db f1 7e 3f-2a 01 14 61 4c 08 0e 4c :.....~?*..aL..L
0060 - 63 cb 02 93 92 80 03 64-33 c9 aa b0 d8 63 c8 33 c......d3....c.3
0070 - 92 b5 c9 4c 35 65 fd 62-f0 fd 53 cf 0b 2f 97 c6 ...L5e.b..S../..
0080 - b7 bc f3 e0 83 15 3a 94-1a 95 78 68 33 99 dc df ......:...xh3...
0090 - 67 b5 e6 4a 16 ca 20 e1-43 85 eb fc 6a 52 01 d7 g..J.. .C...jR..
00a0 - 9b 51 18 b3 e1 3d 32 fd-80 a3 4e 40 7c 45 7f 0b .Q...=2...N@|E..
00b0 - 52 06 b8 07 7e ee c6 d3-16 8b df b4 24 f7 f8 2c R...~.......$..,
00c0 - 58 0f e9 86 6a a3 84 70-0e 08 87 b0 84 22 51 a5 X...j..p....."Q.
Start Time: 1532434389
Timeout : 7200 (sec)
Verify return code: 10 (certificate has expired)
Extended master secret: yes
---
R
RENEGOTIATING
write:errno=0`
Now open Nginx's error log and we can see:
2018/07/24 19:13:25 [alert] 7573#7573: worker process 7574 exited on signal 11 (core dumped)
After analysis, I found the reason finally. This problem occurs when client performed the SSL renegotiation and the current connection is already upgraded to HTTP/2, furthermore, current connection should reuse a TLS session.
When TLS renegotiation happens, the ngx_http_lua_ssl_cert_handler will be called.
Since current connection reused a TLS session, so the cctx inside this function is still NULL, the following if block will not be executed:
if (cctx && cctx->entered_cert_handler) {
/* not the first time */
if (cctx->done) {
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0,
"lua_certificate_by_lua: cert cb exit code: %d",
cctx->exit_code);
dd("lua ssl cert done, finally");
return cctx->exit_code;
}
return -1;
}Then c->data will be treated as the ngx_http_connection_t, we know when the functionngx_http_v2_init called, the c->data will be set to ngx_http_v2_connection_t (rather than the original ngx_http_connection_t). While ngx_http_lua_ssl_cert_handler doesn't distinguish this situation. It just uses the "ngx_http_v2_connection_t" as "ngx_http_connection_t", and some invalid address will be referenced.
couldn't reproduce it with nginx-1.13.6+lua-nginx-module-0.10.12rc2+openssl-1.1.0g