nabsul/kcert

Ingress watching failed unexpectedly

tkaur6494 opened this issue · 8 comments

tkaur6494 commented

I am getting email as below error

Ingress watching failed unexpectedly

The request was aborted.

at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span1 buffer, Boolean partOfSyncRead) at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
at System.IO.StreamReader.ReadLineAsyncInternal()
at k8s.Watcher1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at k8s.Watcher1.CreateWatchEventEnumerator(Func1 streamReaderCreator, Action1 onError, CancellationToken cancellationToken)+MoveNext()
at k8s.Watcher1.CreateWatchEventEnumerator(Func1 streamReaderCreator, Action1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at KCert.Services.K8sClient.WatchIngressesAsync(Func4 callback, CancellationToken tok) in /build/Services/K8sClient.cs:line 44
at KCert.Services.K8sClient.WatchIngressesAsync(Func`4 callback, CancellationToken tok) in /build/Services/K8sClient.cs:line 44
at KCert.Services.IngressMonitorService.WatchIngressesAsync(CancellationToken tok) in /build/Services/IngressMonitorService.cs:line 53

nabsul commented

Hi, two questions:

  • Are you getting many of these emails, or just one?
  • Do you get the email as soon as you deploy KCert, or did it happen after it ran for a while?
tkaur6494 commented

@nabsul

  1. Many emails
  2. It is happening after a while, Once i get this error, pod becomes unresponsive
nabsul commented

Interesting. It sounds like at least one actionable thing is to change KCert to only report the error once, or retry but after a wait time.

But unresponsive is strange. Do you mean the web UI stops works? Or just that it doesn't respond to ingress create/edit operations?

Can you tell me more about your setup? Where are you running Kubernetes? Anything else that might be relevant?

tkaur6494 commented

@nabsul :

I am using Kubernetes on Azure with 1.19 version. I Deployed Ingress-nginx 0.14 (old version). After Deploying Kcert it created my certificate but after few minutes it went in un-responsive stage(webUI stopped working as well as it failed to to watch new ingress).

I recently upgraded my Ingress-nginx to 1.1.3 to try again.

nabsul commented

Sounds good, let me know how it goes. For reference, I've been running KCert in my DigitalOcean cluster for a couple of months now and have not seen such an error.

That said. I think there is at least one bug fix that we can extract so far from this issue: #45

tkaur6494 commented

@nabsul : still same behaviours, I observed one more thing, Once SSL certificate is allocated to me i am not able to open my website on TLS. If i change tls secret to my own generated secret it is working fine, but if I am using secret generated by kcert website goes in hang state.

nabsul commented

This is extremely strange and unexpected. Is it possible that Azure is managing your certs, and therefore it blocks/conflicts with KCert? Are you using a load balancer that already provides TLS termination?

Maybe I can spot something in your Ingress and Secrets? could you run (but please strip out sensitive data before sharing!) kubectl get -o yaml on your ingress and the two secrets (the one that works and the one that doesn't)?

If you do share this information, be sure to delete the "data" section of your secrets, as well as any metadata that identifies your site/username/etc.

tkaur6494 commented

@nabsul i found out the issue was in my backend as i was using old version of Openssl 1.0.1t and after september 2021 Lets encrypt has changed its root cert which was making my internal call failed and no response was returning to ingress. after upgrading version to 1.1.0 it was fixed.

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Thanks !

  • 👍1