istio/istio

Envoy init hang: EDS must respond to all requests

Closed this issue · 10 comments

mandarjog commented

Envoy goes thru the following init sequence

  1. Load clusters with CDS
  2. Load endpoint with EDS (bidi grpc stream)
  3. lds
  4. rds

Until this sequence finishes envoy is not operational.

If Pilot fails to respond to an EDS request, envoy waits indefinitely and init process never finishes.
Such an envoy has no listeners and it will reject all inbound and outbound traffic.

  1. Pilot must ensure that every EDS request has a response.
  2. Envoy must have a timeout during the init process. If init process does not finish in a reasonable amount of time, envoy should either
    a. Error out and die.
    b. Continue with what it has
    c. or configurable.

Note: Check and fix the same issue in ADS.

mandarjog commented

Under high transience, this is more likely to occur.

This has been reported by several customers. The symptom is that some or many proxy-injected pods start crash looping.

kyessenov commented
mandarjog commented

If EDS blocks or deadlocks

  1. Existing envoys will stop receiving new configuration. Envoys keep on operating using stale configuration.
  2. New envoys that start will never finish initialization and stop working.

@Stono

  • 👍2
Stono commented

Sounds exactly like what I saw!

hklai commented

Fixed by #5389. Closing.

mandarjog commented

We are back porting the fix to 0.7.
We will close the issue after that is done.

mandarjog commented

Since the PR is merged into 0.8, this is no longer blocking the 0.8 release.

sakshigoel12 commented

@mandarjog what is pending to close this issue?

mandarjog commented

Potentially back porting to 0.7.x. If we are not going to backport, then let's close this.

sakshigoel12 commented

0.8 is fine for now. We can consider 0.7.x if someone explicitly asks for it.