SPDY is deprecated. Switch to HTTP/2.
lavalamp opened this issue · 64 comments
Filing for tracking purposes.
See #7392 where the spdy package we use doesn't even exist any more. (And btw it has bugs anyway, possibly security bugs-- see the trophies here: https://github.com/dvyukov/go-fuzz)
Blocked on github.com/docker/spdystream switching to HTTP/2.
(Note: HTTP/2 is based on SPDY, with just a few differences-- hopefully this transition will be easy.)
Waiting on HTTP/2 support for Go that includes low level client/server ability to create and handle streams (https://github.com/bradfitz/http2).
moby/spdystream#53 pulls the upstream spdy code into spdystream.
Ah, after moby/spdystream#53 is merged, we can at least make godep happy again. :)
@thockin @lavalamp it looks like the Go HTTP/2 issue was just closed (golang/go#6891 (comment)). I glanced quickly through the code, but I don't see a way to hijack the connection and create our own streams, like we do with spdystream. Hopefully we can get this functionality in there at some point...
I created golang/go#13444 as an RFE for end-user stream support
What's the plan for the deprecation? We are using nginx in front of apiserver which drops support for SPDY since version 1.9.5. Our workaround is to use an older nginx, but would like to see http/2 support go upstream :)
I prototyped this with a prerelease version of go1.6. It worked somewhat, but there were some rough spots (probably stuff I was doing incorrectly). We can't move to http/2 until go1.6 at the earliest, and even then we'll have to do R&D to see if it's technically feasible. I had been hoping that the http/2 support in go1.6 would allow me to create streams just like we can do in spdystream, but that isn't exposed via the API. For my prototype, I ended up writing a simple multiplexer just like @smarterclayton wrote for the websocket support we have in Kubernetes now.
Thanks for the heads up. Looking at the linked comments, there is no intention to add the support in go1.6, so I guess it's probably still months away.
We can't move to http/2 until go1.6 at the earliest
Golang 1.7 is out now. Does that mean we can make progress on this issue now? Anything else stopping us?
A usable HTTP/2 framer in Go that allows us hijack control and the ability
to stand up the connection. Last I saw we were still blocked without
explicitly forking the Go implementation.
In the meantime, websockets continue to be an option (with portforward
being the last bit remaining).
On Oct 29, 2016, at 2:22 PM, Jens Rantil notifications@github.com wrote:
We can't move to http/2 until go1.6 at the earliest
Golang 1.7 is out now. Does that mean we can make progress on this issue
now? Anything else stopping us?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7452 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p-fe3BrfKe4VjAQyYLOgK7AjvhLkks5q447ngaJpZM4EK7sh
.
We should be able to use HTTP/2 without any changes to the Go implementation. But we would have to write our own code to do multiplexing of multiple data streams over a single request/response body pair, which we'd rather not do if we can avoid it.
@ncdc, if it helps, https://godoc.org/golang.org/x/build/revdial is used by the Go build system to multiplex multiple net.Conns and a net.Listener all over a single net.Conn.
@bradfitz Thanks for the pointer, but it doesn't fix the underlying issue. While SPDY is being used, it is not following HTTP request/response semantics. A logical connection is created and then data is sent in either direction as needed. It would be equivalent to the client sending an HTTP request and the server doing a Push whenever.
@fraenkel I don't think there's any technical limitation in the facilities provided by the Go HTTP/2 implementation. As I wrote above, we would just need to write our own muxing/framing code, which we'd rather avoid (or find a library that can do it?).
@ncdc It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done.
Ping, any update on this?
We at Rancher Labs are also interested in this change.
We can actually just move to web sockets in the near term. The server has supported it for a while and HOL blocking is not an issue
Tracking #48633 to move kubectl to web sockets.
General proposal:
Mark SPDY deprecated in 1.8. Clients (#48633) switch to using websockets in 1.8/1.9/1.10. We stop adding new streaming support for SPDY.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Assuming @smarterclayton is still working on #50428
Is there any status update on this issue?
Please ignore my reference, it has nothing to do with that issue, if I could I would remove it, sorry.
We run a http/2 reverse proxy to expose a on-prem api-server. This issues breaks kubectl exec and thereby also tools like helm.
long-term-issue (note-to-self)
I've started working on streaming over h2 here:
https://github.com/kubernetes/kubernetes/compare/master...mikedanese:h2?expand=1
I hope to make some progress in 1.14
I'd like to see this happen in two parts:
- support all streaming codecs over http/2.
- deprecate spdy streaming codecs.
I think (1) is probably the biggest win.
I have a rough prototype of (1) working e2e from kubectl to CRI. Migration will be tricky. I haven't thought of a way for clients to discover h2 streaming capabilities since the entire stack (kubectl -> apiserver -> kubelet -> cri) needs to support streaming over h2. With a significantly more complex implementation, the upgrade aware proxy could translate between clients and servers speaking different protocols. If that were the case, we'd likely only need single hop discovery to transition smoothly.
#71411 is an example of the dangers of dropping an http connection to TCP. There's a category of bugs to be avoided but this one was particularly insidious.
Any update on this?
We are blocked on support for https://tools.ietf.org/html/rfc8441 in x/net/http2 if we don't want to implement something non-standard again.
Is it possible for the API server to support HTTP2 for watches etc... while still using HTTP1 for everything else? xref kubernetes/client-go#374. Forgive me if this question is a little ignorant, im not up on the latest API stuff :).
Has there been any progress on this since May? The use of SPDY for kubectl exec continues to be a pain point because it's no longer supported by common network proxies (nginx, haproxy, envoy, and so on).
Naively, I would expect something like interactive sessions to be implemented on top of gRPC or another thin layer on the normal HTTP/2 framing. Websockets, CONNECT, and other ways to get a raw byte stream have a difficult security model and need much more support in intermediate software compared to plain HTTP/2.
Hi,
I have a setup where proxying kubectl exec is working, using openresty (docker image openresty/openresty:1.15.8.1-2-stretch to be precise).
It should also work with nginx as openresty is based on nginx.
The relevant config in the location block to have kubectl exec proxied correctly is to add:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
this is the listen line in the server block:
listen 443 http2 ssl
The proxied kube-apiserver is running 1.14.3-gke.11 (so, k8s 1.14 with gke modified version).
Hello all, any progress on this issue late 2019 / early 2020? We are using an AWS ALB for the rest of our application, but since AWS ALBs do not support the SPDY protocol we have to use a different load balancer solution just for the k8s API.
Hey @brooksmtownsend i am sure switching to AWS NetworkLoadBalancer is the only solution at the moment.
I tried for days to get it running, ALB and ELB don't support SPDY and i can't imagine AWS will add support. So please safe time and use NLB.
Is there any status update on this issue?
any update on this ??
Hi,
I have a setup where proxying
kubectl execis working, using openresty (docker imageopenresty/openresty:1.15.8.1-2-stretchto be precise).
It should also work with nginx as openresty is based on nginx.The relevant config in the
locationblock to havekubectl execproxied correctly is to add:proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade";this is the
listenline in theserverblock:listen 443 http2 sslThe proxied kube-apiserver is running
1.14.3-gke.11(so, k8s 1.14 with gke modified version).
You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?
You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?
... or kong? as it's based on openresty somehow.
still relevant! known tech debt but we don't have anyone working on fixing it :(
/kind cleanup
Just want to mention that nginx does not support HTTP/2 when proxying. Are you guys sure you want to use HTTP/2?
Specifically:
client (http2) -> nginx -> (http2 does not work here) kube-api-server
We are running into essentially the same issue with our API (picking a good bidirectional streaming protocol) and we're trying to pick a good option that works well with proxying servers like nginx. We implemented HTTP/2 initially and then realized, oops, can't use it with nginx so now we are abandoning the implementation.
We are looking at replacing HTTP/2 with HTTP/1 + WebSockets which seem to work well with nginx.
Thanks -- at least some of our protocols are already available via websockets and that's a good reason to go all in on that.
Note that there is also an apiserver <-> kubelet / container runtime leg of some calls (exec, attach), and this also needs to not use SPDY any more, and it doesn't need to go through a proxy. So that needn't use websockets (although it could).
I believe that would work out of the box with cloudflare
some calls (exec, attach) ... and it doesn't need to go through a proxy
@lavalamp Why is that?
I'm a user of Kubernetes so the below is my own viewpoint as an API consumer, not a Kubernetes contributor.
some calls (exec, attach) ... and it doesn't need to go through a proxy
Why is that?
When the API server makes HTTP calls to a particular kubelet, the connection is generally made without an HTTP proxy in the way; the kubelets each have their own TLS certificates and the apiserver checks this encryption, so installing an HTTP proxy in between would introduce MITM-like security concerns.
This connectivity of course may need to be relayed e.g. across the internet and this is done at the TCP level. For example GKE runs the apiserver->kubelet traffic through long-lived SSH tunnels, and the new system to replace those SSH tunnels, Konnectivity, is also a TCP-level proxy.
The client -> apiserver traffic is definitely more important for compatibility and this is where Websockets are already offered. Unfortunately the websocket interop isn't perfect:
- The
new WebSocket()constructor in browsers don't allow attaching anAuthorizationheader, so token auth isn't possible from an actual web client. - Each WebSocket is historically its own TCP connection, so
kubectl port-forwardfunctionality would need to open a new TCP connection to the API Server for each new tunnel client if it were using WebSockets. RFC 8441 addresses this, but similarly, nginx seems to have no plans to implement websocket over HTTP2.
I've been working on a WebSocket client to Kubernetes and you can get things done with it. If the websocket subprotocol used by Kubernetes was extended to support functions like dynamically opening and closing port-forward streams, I could see it replacing SPDY. I just don't know if creating a more special-cased subprotocol in Websocket would be worth it vs. leveraging the http2 standard.
Thanks for the websocket feedback. I don't know if anyone is working on this right now, though. It may be a good idea to file a separate issue about that for visibility.
@danopia doesn't it make sense for the nevermind, it seems I misunderstood what you said.Authorization header go side-by-side with an Upgrade header in the same HTTP request to kube-apiserver and then, provided the requestor is authorized to proxy, upgrade to Websocket (not needing the Authorization header)?
The new WebSocket() constructor in browsers don't allow attaching an Authorization header, so token auth isn't possible from an actual web client.
FYI it seems like there's a way to do this using the websocket subprotocol header: #47740
FYI it seems like there's a way to do this using the websocket subprotocol header: #47740
That hack looks questionable at best, despicable or insecure at worst, and I'm sure not at all documented. (If the Kubernetes websocket subprotocols are documented somewhere I haven't seen it)
I've verified that the hack works for my usecase. Thanks for the link. 😄
I solved the issue, see my article
https://dev.to/aws-builders/kubectl-execport-forward-with-aws-alb-and-nginx-ingress-controller-4fn0
Hi, is this issue actively being worked on?
No --
We know the user facing path should be swapped out for websockets, but I don't know if there is any consensus on what should happen for the apiserver -> kubelet path.
kubernetes/enhancements#3401 was closed, noting issues with implementing an application-layer multiplexing system for a websocket stream.
I'm really unsure as to how well this is ready for the k8s ecosystem, but it might be worth keeping an eye on WebTransport (W3C), as it seems to natively support many of the desired features missing from WebSockets (multiplexing, bidirectional+unidirectional streams). However, afaik it's still being standardized, and depends on HTTP/3 (QUIC), which is (newer?) to k8s.
Notably, WebTransport should support the HTTP/3 (QUIC) STOP_SENDING and RESET_STREAM which (might?) implement what the KEP was looking for.
BTW, the title of this issue scans to "Rudolph the red nosed reindeer".
Although WebTransport is not yet standardized, we do have both WebSocket and WebRTC Datachannels as (fairly) stable things.
WebSockets over HTTP/3 over QUIC is standardized albeit with few browser implementations. That could nonetheless work fine for communication between code we write (eg kubectl talking to the Kubernetes API server), with a fallback to an earlier HTTP version and TCP.
I'm going to close this, since KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets moved to websockets that is widely available I don't think we are going to do another protocol change
/close
@aojea: Closing this issue.
In response to this:
I'm going to close this, since KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets moved to websockets that is widely available I don't think we are going to do another protocol change
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.