coryodaniel/k8s

Http2 Adapter causing 500 requests

canuc opened this issue · 10 comments

canuc commented

The http2 adapter gets closed after some time, and causes the process to shutdown, if there are messsages in the inbox they get flushed and the phoenix endpoint returns 500 reproducable locally.

The workaround is to disable http2 and always return http1 for our usecase.

[warning] K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process.
[debug] K8s.Client.Mint.HTTPAdapter Terminating HTTPAdapter GenServer #PID<0.2321.0>
[debug] K8s.Client.Mint.ConnectionRegistry DOWN of process #PID<0.2321.0> received.
[error] #PID<0.2350.0> running Phoenix.Endpoint.SyncCodeReloadPlug (connection #PID<0.2082.0>, stream id 9) terminated
Server: localhost:4000 (http)
Request: POST {
** (exit) exited in: GenServer.call(#PID<0.2321.0>, {:recv, #Reference<0.2495543023.3533963267.100249>}, :infinity)
    ** (EXIT) shutdown: :closed
mruoss commented

Hi @canuc

This sounds like a bug. But you're not giving me much context to reproduce this. What version of k8s is this? What is the code that leads to this?

Thanks

razielgn commented

I'm also having this issue, using version 2.4.1 (mint 1.5.1). It happens every 3/4 days, sometimes multiple times in a single day.
What I have is a cronjob via Oban that periodically does

K8s.Conn.from_service_account()
|> K8s.Client.run(K8s.Client.list("argoproj.io/v1alpha1", "workflow", namespace: "argo-workflows"))

Here's the stacktrace:

** (exit) exited in: GenServer.call(#PID<0.292039.0>, {:recv, #Reference<0.4064860026.2373451778.104924>}, :infinity)
    ** (EXIT) shutdown: :closed

lib/gen_server.ex in GenServer.call/3 at line 1074
lib/k8s/client/mint_http_provider.ex in anonymous fn/3 in K8s.Client.MintHTTPProvider.stream/5 at line 41
lib/stream.ex in Stream.do_resource/5 at line 1626
lib/stream.ex in Enumerable.Stream.do_each/4 at line 1828
lib/enum.ex in Enum.reduce/3 at line 4387
lib/k8s/client/mint_http_provider.ex in K8s.Client.MintHTTPProvider.request/5 at line 20
lib/app/workers/workflows_refresher.ex in App.Workers.WorkflowsRefresher.perform/1 at line 19

Let me know if I should provide more info, thanks.

mruoss commented

Sorry, I totally forgot about this issue. It reached me while on vacation. I have an idea of what could be the problem, but I need a quiet moment to wrap my head around it.

Note to self: Double check the meaning of the type option of Mint.open?() - docs don't seem obvious right now.

mruoss commented

Hey @canuc and @razielgn

I think #285 actually solves this. Would either of you be willing to test your code with the version of that PR?

  • ❤️1
razielgn commented

Hello, yes I'm going to deploy this patch today and let it run for a few days. I think by Monday it'll be clear. Thank you!

  • ❤️1
mruoss commented

@razielgn all quiet so far?

razielgn commented

All quiet so far!

mruoss commented

Nice. Thanks for testing it. I will merge and release.

razielgn commented

Thanks!

canuc commented

This is amazing! thanks so much!!!

  • 👍1