Sending HTTP traffic over Hyperswarm. Should we consider supporting multiplexing in the protocol?
Closed this issue · 25 comments
Sending HTTP traffic over Hyperswarm sockets is a use-case I'm currently exploring. This leads to a small challenge:
HTTP semantics depend on the connection life-cycle. After a response is finished, the server is expected to close the connection. This seems like a problem for Hyperswarm, because connection setup isn't as quick as it is for traditional TCP. We don't want to open and close connections for every request.
One option is to use empty messages to indicate the end of a request or response, without closing the stream. This should work so long as responses are sent in the same order that requests are sent, but that's a performance penalty.
We can improve on that by using multiplexing such as in libp2p-mplex. This would allow multiple in-flight HTTP request/responses, and the "channels" could be freed for reuse after the req/res sessions end on each of them.
I'm opening this issue for two reasons:
- To confirm that I understand the situation correctly. (not a given)
- If I'm correct, I wonder if there'd be any merit to supporting multiplexing natively in the protocol? I have two reasons for suggesting that: A) It might be useful in other contexts, and B) I suspect that sending HTTP over Hyperswarm sockets will be really common, and since Hyperswarm sockets don't have any "standard header" to declare the message format they're about to send, having multiplexing on all sockets would simplify things.
Another option I might look into is using HTTP/2, which supports multiplexing.
EDIT: One consideration is that HTTP/2 does not, sadly, support "P2P". That means the client/server relationship can't be inverted on an existing connection. If HTTP/2 is the solution, we'll need at least 2 hyperswarm sockets (or an additional multiplexer) to do full p2p messaging.
The hyperswarm layer will likely never support any high level protocols. Support is coming for TCP in addition to UTP and we are open to adding more "raw protocols". If you wanna run any multiplexing you should do that on top of your raw connection. The only exception to this might be a websocket interface for browser compat, but I'd prefer not doing that here as well for the same reasons. For the same reasons headers don't make much sense, it's just an encrypted raw socket.
In addition connections are always handshaked seperately through the dht for holepunching reasons.
I'd suggest the following:
- Quantify what "isn't as quick" means - it's a testnet that is clustered in europe atm - do you mean high latency? Whats the numbers? Where are you connecting to/from.
- Contribute low level benchmarks that help us track this over time
In my experiments when relaying browser communications to a hyperswarm relayed http server, (logical) multiplexing and keepalive was happening, presumably negotiated between the browser and web server, this was also confirmed and improved on by adding and correctly configuring two http proxies on the chain, nginx for ssl addition and an npm implementation for programmability.
@mafintosh I expected that would be the case. That's fine, just wanted to ask.
Quantify what "isn't as quick" means
See https://github.com/pfrazee/hyperswarm-bench. Summarized results:
- 10 TCP connections: 922ms
- 10 HS connections: 7036ms
In my experiments when relaying browser communications to a hyperswarm relayed http server, (logical) multiplexing and keepalive was happening
I need to understand how that multiplexing is happening. HTTP1.* doesn't support it AFAIK, so either nginx is using HTTP2 in its reverse proxy or there's some other mechanic involved that I'm not aware of.
Doesn't matter where the nodes are, I'd imagine the bulk of the current latency is speed of light to the testnet DHT which is all running in northern germany. All handshakes go through the DHT always - that's how you holepunch.
With a more widely deployed DHT this might be better or worse. For both your localities I'd imagine it can only get better as it's really far geographically to the testnet.
@mafintosh yeah that's fine. Like I said, this is what I expected. Now I'm just trying to find the right approach to work with these properties -- even if the handshake got down to 100ms, that would still be too high for HTTP proxies.
@mafintosh @lanmower My guess is that we're going to want to establish stable connections between peers and reuse them until some amount of inactive-time passes. What I'm trying to figure out now is, how do the protocols we're going to use behave relative to the connection lifecycle?
If we have a lot of protocols like HTTP which expect to use 1 connection per exchange, the next step might be to find/create a wrapper protocol. That's why I'm trying to thoroughly understand HTTP's multiplexing behavior, because ideally we wouldnt create a meta protocol. We may have to though.
Very much doubt that we'll get everywhere close to 100ms on average even when the network is deployed. DHTs multiple require roundtrips, none of which is necessarily optimised for geographic closeness. That's not saying it wont get much faster than it is, but it'll always be much slower, relatively speaking, than direct connecting to anything.
Right! You're debating me on something we agree on. The 100ms was my example of "wow somehow we magically got the DHT to be that fast." My point was, even then it would be too slow for the use-case of naive HTTP proxying.
If you make tons of connections to different keys yea. For HTTP/1.1 keep-alive should solve it for you per default.
(closing for now as there isn't anything actionable for hyperswarm to do here, but happy to keep the discussion going)
Right, okay. Keep-alive was what I was missing about HTTP 1.1. It's not multiplexing so your requests end up getting de-parallelized, but it does reuse the connection.
Given that making a wrapper protocol is complex, here's what I'll do next:
- For Web traffic: Use HTTP2 when possible, HTTP1.1 when necessary.
- Open two Hyperswarm connections when server-to-server HTTP is required. (HTTP 1 and 2 both require a fixed server/client assignment on a connection.)
- ...but look into how gRPC does bi-directional RPC, because some writing suggests that they use event-streams or server push to accomplish that.
You should look into HTTP pipelining, it's basically multiplexing for 90% of all use cases.
Also making multiple connections to the SAME host is very optimisable (read competive to direct connections)
You should look into HTTP pipelining, it's basically multiplexing for 90% of all use cases.
Yeah it seems to have pretty mixed support these days because HTTP gives no clear way to establish Pipelining support, and I'm not sure if you can elegantly handle a "wrong guess." I'll check into it though. HTTP/2 may just be the simplest solution.
Also making multiple connections to the SAME host is very optimisable (read competive to direct connections)
By Hyperswarm? That's what we're looking at here, so if Hyperswarm can optimize that to within 50-100ms, then that plus keep-alive means we're golden.
If you are making a proxy, you're implementing the pipelining yourself as the multiplex layer so I don't think generel supports matter - this is just a simple plug'n'play solution (i think node core has support even, but maybe i'm remembering wrong)
Yea by Hyperswarm. Pretty easy but not high on the list, so wouldn't happen for a while.
Okay cool. Thanks for the help!
Just confirmed through some googling that node's http server HAS pipelining but the client doesn't. https://github.com/nodejs/undici does tho.
Okay thanks for checking that.
@lanmower I'm guessing your hyper-web-relay package is going to get different kinds of HTTP traffic from nginx than it would from a browser directly accessing it.
That being said, all browsers multiplex.
@pfrazee can you link me to where you do multiplexing? maybe I can blatantly steal your technique for hyper-web-relay :)
I ended up solving it by putting an nginx reverse proxy in front with keep-alive enabled