Egress rule must support Mongo/Redis/TCP services

[kyessenov]

Context: production readiness proposal and plan

The feature “cluster egress” is identified as incomplete test coverage. As a measure to reach Istio production quality in 0.3, this issue is opened to track the progress of providing sufficient test coverage for the feature in question.

This issue is initially assigned to the working group leads.

Recommendations:

• Come up with a test plan for how the feature should be tested. Identify and document (in this issue, or a separate document this issue will link to) test cases and whether each will be covered by an integration test (with some level of mocking and/or starting up selected components in test environment), or end to end test (aka system test, which brings up the whole Istio to exercise the feature; currently live in the istio/istio repo).
• If you expect the test plan is complex and controversial, consider having it formally reviewed.
• State an ETA in the issue, and assign this to a developer who will execute the test plan.
• Execute the test plan as the highest priority for 0.3.
• For any bug or defect identified by the new test cases, file a new issue and link it from this one.
• Close this issue only when the test plan is completely executed, and all additional issues identified in step 3 are fixed.

If you see a risk of not fixing this issue by end of 0.3, please contact istio-production-quality@googlegroups.com as soon as possible. We will also periodically ping the issue for update.

[rshriram]

Need to be able to specify external mongo/redis or generic TCP services, and be able to collect envoy metrics for connections to these services, and be able to apply circuit breakers for connections to these services

[vadimeisenbergibm]

Proposed syntax - using CIDR notation in service field.

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: tcp-egress-rule
spec:
  destination:
      service: 9.148.1.0/24
  ports:
    - port: 30000
      protocol: tcp

This rule will allow tcp connection to port 30000 for tcp protocol. Note that CIDR notation will be allowed for tcp protocol only, for example the following rule will be invalid:

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: tcp-egress-rule
spec:
  destination:
      service: 9.148.1.0/24
  ports:
    - port: 30000
      protocol: tcp
    - port: 80
      protocol: http <-- unable to use http protocol for CIDR.

@rshriram @kyessenov what do you think?

[vadimeisenbergibm]

Continuing with #1417 (comment), domain notation could not be used for tcp, so the following rule will not be valid:

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: google-egress-rule
spec:
  destination:
    service: *.google.com
  ports:
    - port: 80
      protocol: http
    - port: 30000
      protocol: tcp <- unable to use FDQN for tcp rules

[rshriram]

@vadimeisenbergibm this sounds pretty good. Also, the restriction applies IF and ONLY If there is a CIDR notation. If user specifies simple IP address, then she should still be able to define HTTP protocol.

Another thing that is missing is mongo, redis. I think for Mongo, the same principle as TCP would apply (use CIDR notation). But for Redis, the client connects to Envoy and envoy picks an upstream redis instance, or client accesses some IP address, we trap it on port 2379 and pick an upstream host based on it.
I think for both of these cases, the user would probably specify service: * and use protocol: mongo or protocol: redis to tell us that its a mongo/redis service. In this case, we should just create a listener on the user specified port, with the appropriate filter.

@nmnellis is actually using an IP address based egress rule on AWS, to monitor accesses to the metadata service (http).

[nmnellis]

surprisingly this worked for us when talking to the EC2 metadata service

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: ec2-instance-metadata-rule
  namespace: istio-system
spec:
  destination:
    service: "169.254.169.254"
  ports:
    - port: 80
      protocol: http

[vadimeisenbergibm]

@nmnellis - this is by design, according to the envoy's virtual host configuration. CIDR notation would not work, e.g. "169.254.169.0/24".

[vadimeisenbergibm]

@rshriram I confused with regard to mongo and redis. How the mongo and redis clusters will be configured, in the envoy configuration? Will the istio user provide custom envoy configuration for Istio, that will contain mongo and redis clusters?

[rshriram]

We already support Mongo and redis. The user would do something like

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: mongo-rule
  namespace: istio-system
spec:
  destination:
    service: "169.254.169.254"
  ports:
    - port: 3308
      protocol: Mongo

or in a simpler way:

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: ec2-instance-metadata-rule
  namespace: istio-system
spec:
  destination:
    service: *
  ports:
    - port: 3308
      protocol: Mongo

We would setup a listener on port 3308, with the Mongo filter installed and use the orig_dst cluster. Business as usual :). The benefit is that we get metrics for mongo/redis.

Now, there is one caveat here: using TLS origination. One option is to add a new protocol MONGO-TLS.. Or we just add a simple use_tls: true in the Egress rule. IF set, we setup a cluster with SSL context. IF we use the use_tls: true, we can even get rid of the HTTPS port, and basically ask users to configure standard HTTP, and then set the use_tls: true (makes things much more readable and avoids ambiguity).

[vadimeisenbergibm]

@rshriram agreed regarding use_tls: true. With SNI support in Envoy envoyproxy/envoy#1843 we could support the real HTTPS.

[rshriram]

Can I assign this issue to you ?

[vadimeisenbergibm]

@rshriram Sure, please assign the issue to me.

[vadimeisenbergibm]

@rshriram I wonder what will be gained by supporting MongoDB and Redis protocols explicitly, instead of plain TCP?

[rshriram]

More metrics and control :). If we are supporting TCP, might as well support the other non HTTP protocols as well.

[kyessenov]

Issue moved to istio/istio #1476 via ZenHub