tokio-rs/axum

graceful shutdown does not shutdown when i use Google Chrome。

Closed this issue · 12 comments

maojinhua commented

Bug Report

Version

axum 0.7.4

Platform

macOs Big Sur 11.5.1

Crates

Description

After clone axum source code, run the following command

cargo run -p example-graceful-shutdown

When the server is up, do the follows in Geegle Chrome:

1.open the serving page http://127.0.0.1:3000/ in browser (this is a must step to reproduce the problem)
2.go back to your terminal and press ctrl+c to stop the server
3.and you see that the server is hanging up but not stop even after a long time
4.After closing Google Chrome, the server shuts down immediately.

this is the log.
I found that when Google Chrome initiates a request, it creates two connections. After the request is completed, only one connection is closed, but the other connection remains active。

2024-02-20T16:55:44.219277+08:00  INFO resource_server: listening on 0.0.0.0:8998
2024-02-20T16:55:49.452449+08:00 TRACE axum::serve: connection 127.0.0.1:49501 accepted
2024-02-20T16:55:49.452611+08:00 TRACE axum::serve: connection 127.0.0.1:49502 accepted
2024-02-20T16:55:49.456245+08:00 DEBUG request: tower_http::trace::on_request: started processing request method=GET uri=/slow version=HTTP/1.1
^C2024-02-20T16:55:52.647503+08:00 TRACE axum::serve: received graceful shutdown signal. Telling tasks to shutdown
2024-02-20T16:55:52.647611+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T16:55:52.647609+08:00 TRACE axum::serve: signal received, not accepting new connections
2024-02-20T16:55:52.647668+08:00 TRACE axum::serve: waiting for 2 task(s) to finish
2024-02-20T16:55:52.647629+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T16:55:54.461719+08:00 DEBUG request: tower_http::trace::on_response: finished processing request latency=5005 ms status=200 method=GET uri=/slow version=HTTP/1.1
2024-02-20T16:55:54.462067+08:00 TRACE axum::serve: connection 127.0.0.1:49501 closed
2024-02-20T16:56:50.811452+08:00 TRACE axum::serve: connection 127.0.0.1:49502 closed

The following is the request log sent using Safari browser or Postman. You can see that the server shuts down normally, and there is only one connection initiated for each request sent by the browser.

2024-02-20T17:16:31.602835+08:00  INFO resource_server: listening on 0.0.0.0:8998
2024-02-20T17:16:36.918718+08:00 TRACE axum::serve: connection 127.0.0.1:55361 accepted
2024-02-20T17:16:36.935941+08:00 DEBUG request: tower_http::trace::on_request: started processing request method=GET uri=/slow version=HTTP/1.1
^C2024-02-20T17:16:38.867539+08:00 TRACE axum::serve: received graceful shutdown signal. Telling tasks to shutdown
2024-02-20T17:16:38.867729+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T17:16:38.867749+08:00 TRACE axum::serve: signal received, not accepting new connections
2024-02-20T17:16:38.867926+08:00 TRACE axum::serve: waiting for 1 task(s) to finish
2024-02-20T17:16:41.938476+08:00 DEBUG request: tower_http::trace::on_response: finished processing request latency=5002 ms status=200 method=GET uri=/slow version=HTTP/1.1
2024-02-20T17:16:41.938716+08:00 TRACE axum::serve: connection 127.0.0.1:55361 closed

How can I ensure that Google Chrome also doesn't encounter such an issue? It seems like this problem could lead to the server being unable to shut down for extended periods, which appears to be a bug.

mladedav commented

I've been able to reproduce it. It's true that chrome keeps the connection open even after the user has closed the site. However, it's true hyper could try and close connections were there are no inflight requests.

I can see you've already opened a bug in hyper which I believe is the right place.

dswij commented

Should be fixed by hyperium/hyper-util#101 once that's out.

  • 👍3
estnml commented

still encountering this in chromium Version 123.0.6312.86, i think its not in the release version yet?

  • x86_64 GNU/Linux.
  • axum = "0.7.5"
  • chrono = { version = "0.4.37", features = ["now"] }
  • serde = "1.0.197"
  • tokio = { version = "1.37.0", features = ["rt-multi-thread", "signal"] }
  • tower-http = { version = "0.5.2", features = ["trace", "timeout"] }
  • tracing = "0.1.40"
  • tracing-appender = "0.2.3"
  • tracing-subscriber = { version = "0.3.18", features = ["env-filter","fmt","std","chrono",] }
mladedav commented

@estnml can you try running cargo tree | grep hyper? If hyper-util has a version lower than 1.0.3, try running cargo update that should fix it.

In either case please do share the version and if update helped.

estnml commented

hi @mladedav output of "cargo tree | grep hyper" :

│ ├── hyper v1.2.0
│ ├── hyper-util v0.1.3
│ │ ├── hyper v1.2.0 (*)

i did run cargo update but versions not changed

kod-kristoff commented

The fix in hyper-util is merged to master, but hasn't been released yet, for what I can see.
Last release 0.1.3 was released 2024-01-31.
PR merged 2024-02-26.
So we have to wait for the release :)

czy-29 commented

Is there any way we can hot-fix or workaround this bug before hyper-util is updated and released?

mladedav commented

You can patch your dependencies with the unreleased version of hyper-util from git.

czy-29 commented

You can patch your dependencies with the unreleased version of hyper-util from git.

Thanks! This works, but the current master branch of hyper-util (24.05.08) fails to compile when working with crates with the client feature enabled (such as reqwest). I found a stable commit with no bugs. For anyone who encounters the same problem, you can just paste the following code into your Cargo.toml and everything will be solved:

[patch.crates-io]
hyper-util = { git = "https://github.com/hyperium/hyper-util.git", rev = "a77d866ff65a68ea5d048d058108552c39ec470d" }
czy-29 commented

hyper-util 0.1.4 is released!
https://crates.io/crates/hyper-util/0.1.4
We should update as soon as possible and finally close this issue.

jplatte commented

There's nothing axum needs to do. You can remove the patch section, cargo update or cargo update -p hyper-util, and you'll get the latest version.

pac-work commented

Hello @jplatte . The example-graceful-shutdown indeed seems to run properly with the current hyper-util v0.1.5. However, example-tls-graceful-shutdown seems to suffer from the same problem. At least when looking at the outside behavior. Steps to reproduce:

# in terminal 1:
cargo run -p example-tls-graceful-shutdown
# in terminal 2:
nc 127.0.0.1 3000
# in terminal 1 press Ctrl-C - things are stuck for the whole 10 s period

Repeating with nc 127.0.0.1 7878 does not perform any waiting. Is this expected behavior? I understand that I can just call handle.shutdown() without the graceful part, but it seems like there is an inconsistency about what graceful means then... because for HTTP, the graceful does not wait for opened but inactive connections, for HTTPS, it does wait for them.