`higher-ranked lifetime error` while spawning server in Tokio

[jgeluk]

Describe the bug
When trying to not configure and start the Salvo server on the main thread but in a spawned thread the compiler fails with a higher-ranked lifetime error and messages like note: could not prove [async block@crate/server/src/server_builder.rs:115:22: 120:10]: Send.

To Reproduce
Steps to reproduce the behavior:

        let router = self.router()?;
        let rustls_config = rustls_config();
        let (tx, rx) = oneshot::channel();

        let runtime = tokio::runtime::Builder::new_multi_thread()
            .enable_all()
            .build()
            .unwrap();

        let _guard = runtime.enter();

        let create_acceptor = async move {
            let listener = TcpListener::new(local_address.clone()).rustls(rustls_config.clone());

            let acceptor = QuinnListener::new(rustls_config, local_address.clone())
                .join(listener)
                .try_bind()
                .await?;

            Ok::<_, Error>(acceptor)
        };

        let acceptor = runtime.block_on(create_acceptor)?;

        let server = salvo::Server::new(acceptor);

        let serve1 = server.serve_with_graceful_shutdown(
            router,
            async {
                tracing::info!("Waiting for server to stop");
                rx.await.ok();
            },
            None,
        );

        let serve2 = async move {
            tracing::info!("server.await begin");
            // let x = force_send_sync::Send::new(serve);
            serve1.await;
            tracing::info!("server.await end");
        };

        runtime.spawn(serve2);  // <<< This is the line where the error occurs.

Additional context

• rustc 1.68.0-nightly

[jgeluk]

It seems to be related to the .rustls() call, if that's all left out (and the QuinnListener), it compiles

[chrislearn]

Can you give me the test project source code or repository url?

[jgeluk]

If you change examples/tls-rust/main.rs as follows and then do cargo run in that directory you'll see the same error:

use std::thread::spawn;
use salvo::conn::rustls::{Keycert, RustlsConfig};
use salvo::prelude::*;

#[handler]
async fn hello(res: &mut Response) {
    res.render(Text::Plain("Hello World"));
}

async fn run_server() {
    let router = Router::new().get(hello);
    let config = RustlsConfig::new(
        Keycert::new()
            .with_cert(include_bytes!("../certs/cert.pem").as_ref())
            .with_key(include_bytes!("../certs/key.pem").as_ref()),
    );
    let acceptor = TcpListener::new("127.0.0.1:7878").rustls(config).bind().await;
    Server::new(acceptor).serve(router).await;
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt().init();

    let _ = spawn(run_server);

    tokio::task::yield_now().await;
    println!("main task done!");
}

Error:

[jgeluk]

Or here's a version using tokio::task::spawn rather than std::thread::spawn:

use salvo::conn::rustls::{Keycert, RustlsConfig};
use salvo::prelude::*;

#[handler]
async fn hello(res: &mut Response) {
    res.render(Text::Plain("Hello World"));
}

async fn run_server() {
    let router = Router::new().get(hello);
    let config = RustlsConfig::new(
        Keycert::new()
            .with_cert(include_bytes!("../certs/cert.pem").as_ref())
            .with_key(include_bytes!("../certs/key.pem").as_ref()),
    );
    let acceptor = TcpListener::new("127.0.0.1:7878").rustls(config).bind().await;
    Server::new(acceptor).serve(router).await;
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt().init();

    let _ = tokio::task::spawn(run_server());

    tokio::task::yield_now().await;
    println!("main task done!");
}

Same error.

I don't know whether it's a good idea to spawn the whole server as a tokio task or not. I'm trying to get it to run in the context of a Tauri plugin (https://tauri.app/v1/guides/features/plugin/) for an app that we're building where the salvo server (that usually runs as the backend server for the web-version of the app) is embedded in the tauri-app. Since Tauri wraps around a Tokio runtime itself and needs to start from a non-tokio main thread, we have to spawn the salvo server in a background thread, ideally managed by the same tokio runtime that tauri uses.

[jgeluk]

Point is that this example works when you replace run_server() with:

async fn run_server() {
    let router = Router::new().get(hello);
    // let config = RustlsConfig::new(
    //     Keycert::new()
    //         .with_cert(include_bytes!("../certs/cert.pem").as_ref())
    //         .with_key(include_bytes!("../certs/key.pem").as_ref()),
    // );
    // let acceptor = TcpListener::new("127.0.0.1:7878").rustls(config).bind().await;
    let acceptor = TcpListener::new("127.0.0.1:7878").bind().await;
    Server::new(acceptor).serve(router).await;
}

So it seems that RustlsConfig is not Send? I checked that, I tried to add more Send annotations to it, even implementing Send for RustlsConfig and Keycert etc, but couldn't get that to work...

[chrislearn]

I have checked all futures in run_server is Send, hope the compiler will give more details about this error in future.

[chrislearn]

rust-lang/rust#102211

[jgeluk]

Saw that article yes. The suggested solution is hard to do though, boxing the future created by .rustls(config).bind() does not work since it's not Unpin...

[chrislearn]

RustlsAcceptor has a field config_stream, this issue may caused by it.
This may rust bug, I am not sure. this issue very similar to this: rust-lang/rust#102211 (comment)

[jgeluk]

It's not just RustlsConfig, NativeTlsConfig shows the same problem, if you replace examples/tls-native-tls/src/main.rs with this version it gives the higher-ranked lifetime error as well on line 22:

use salvo::conn::native_tls::NativeTlsConfig;
use salvo::prelude::*;

#[handler]
async fn hello(res: &mut Response) {
    res.render(Text::Plain("Hello World"));
}

async fn run_server() {
    let router = Router::new().get(hello);
    let config = NativeTlsConfig::new()
        .with_pkcs12(include_bytes!("../certs/identity.p12").to_vec())
        .with_password("mypass");
    let acceptor = TcpListener::new("127.0.0.1:7878").native_tls(config).bind().await;
    Server::new(acceptor).serve(router).await;
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt().init();

    let _ = tokio::task::spawn(run_server());

    tokio::task::yield_now().await;
    println!("main task done!");
}

Leaving out the call to native_tls() makes it compile happily...

[jgeluk]

So the question is, is this due to multiple causes? Like the one you mentioned regarding config_stream? Or is this due to one other cause, shared by both RustlsConfig and NativeTlsConfig?

I saw your latest changes (adding Send + 'static at various places), I tried the same thing but the above examples still give the same error, unfortunately. The problem with this very opaque error message is that it's a bit like stabbing in the dark, I update the compiler every day hoping that the error message will get more precise but no luck so far.

[aoiryc]

I've simplified it to a self-contained snippet:

trait Stream {}
impl<T: ?Sized> Stream for T {}

trait Acceptor {
    fn accept(&mut self) -> impl Future<Output = ()> + Send;
}
struct RustlsAcceptor<S> { config_stream: S }
impl<S: Stream + Send + 'static> Acceptor for RustlsAcceptor<S>  {
    async fn accept(&mut self) {}
}

struct Server<A> { acceptor: A }
impl<A: Acceptor + Send> Server<A> {
    async fn try_serve(mut self)  {
        self.acceptor.accept().await;
    }
}

let _: &dyn Send = &async { // higher-ranked lifetime error
    let acceptor = RustlsAcceptor {
        config_stream: Box::new(()) as Box<dyn Stream + Send + 'static>,
    };
    Server { acceptor }.try_serve().await;
};

Though it involves some tricky compiler bugs, I've come up with three workarounds:

1. Use S::Stream instead of BoxStream in Listener.
2. Use BoxStream instead of the generic S in Accepter.
3. Explicitly add + Send to the future returned by Server::try_serve.

I'm in favor of the 3rd one as it doesn't introduce any breaking changes and is protocol insensitive.