megaease/easemesh

Supporting mTLS in EaseMesh

benja-wu opened this issue · 5 comments

benja-wu commented

Background

  • As a mesh product, security between micro-services is essential in production-ready requirements.
  • Popular mesh products, e.g., Istio, Linkerd, OSM[0], use mTLS to secure micro-service communications.
  • mTLS[1] is used for bi-directional security between two services where the TLS protocol is applied in both directions.[2]

Requirements

  1. Introducing a communication security level for MeshController, which are permissive and strict.
  2. Enhancing controller plane for assigning and updating certificates periodically for every micro-services inside EaseMehs at the strict level.
  3. Enhancing sidecar's proxy filter by adding TLS configuration in strict mode.
  4. Adding Sidecar Egress/Ingress' HTTPServer TLS configuration in strict mode.
  5. Adding Mesh IngressController for watching its cert in strict mode.

Design

  1. MeshController Spec
kind: MeshController
...
secret:                           // newly added section 
    mtlsMode:  permissive         // "strict" is the enabling mTLS
    caProvider: self              // "self" means we will sign/refresh roo/application cert/key by EaseMesh itsef
                                  //      consider supporting outer CA such as `Valt`
    rootCerTTLl:  87600h  // ttl for  root cert/key 
    appCertTTLl:  48h     // ttl for  certificates for one service
  1. Adding a certificates structure for every mesh service, it contains the HTTP server's cert and key for Ingress/Egress
serviceName: order
issueTime:  "2021-09-14T07:37:06Z"
ttl:  48h 
certBase64: xxxxx=== 
keyBase64: 339999===

And storing it into /mesh/service-mtls/spec/%s // + servicename layout.

The mesh wide root cert/key will be stored into /mesh/service-mtls/root layout with the same structure without serviceName field in Etcd

  1. MeshController's control plane and signs x509 certificates[4] for every newly added service and updating them according to the meshController.secret.certRefreshInterval.

  2. Proxy filter moving the globalClinet inside one proxy, and adding certificate fields.

kind: proxy
name: one-proxy
...
certBase64: xxxxx=== 
keyBase64: 339999===
rootCertBase64: y666====
...
  1. Add CertManager and CertProvider modules in MeshMaster. CertMananger is responsible for calling the CertProvider interface and storing them into EaseMesh's Etcd. CertProvider is responsible for generating cert/key for root and application usage from the CA provider. Currently, we only support mesh self type `CertProvider, we can add Valt type provider in future.
         // Certificate is one cert for mesh service or root CA.
	Certificate struct {
		ServiceName string `yaml:"servieName" jsonschema:"omitempty"`
		CertBase64  string `yaml:"CertBase64" jsonschema:"required"`
		KeyBase64   string `yaml:"KeyBase64" jsonschema:"required"`
		TTL         string `yaml:"ttl" jsonschema:"required,format=duration"`
		IssueTime   string `yaml:"issueTime" jsonschema:"required,format=timerfc3339"`
	}

      	// CertProvider is the interface declaring the methods for the Certificate provider, such as
	// easemesh-self-sign, Valt, and so on.
	CertProvider interface {
		// SignAppCertAndKey signs a cert, key pair for one service's instance
		SignAppCertAndKey(serviceName string, host, ip string, ttl time.Duration) (cert *spec.Certificate, err error)

		// SignRootCertAndKey signs a cert, key pair for root
		SignRootCertAndKey(time.Duration) (cert *spec.Certificate, err error)

		// GetAppCertAndKey gets cert and key for one service's instance
		GetAppCertAndKey(serviceName, host, ip string) (cert *spec.Certificate, err error)

		// GetRootCertAndKey gets root ca cert and key
		GetRootCertAndKey() (cert *spec.Certificate, err error)

		// ReleaseAppCertAndKey releases one service instance's cert and key
		ReleaseAppCertAndKey(serviceName, host, ip string) error

		// ReleaseRootCertAndKey releases root CA cert and key
		ReleaseRootCertAndKey() error

		// SetRootCertAndKey sets existing app cert
		SetAppCertAndKey(serviceName, host, ip string, cert *spec.Certificate) error

		// SetRootCertAndKey sets exists root cert into provider
		SetRootCertAndKey(cert *spec.Certificate) error
	}
  1. One particular thing should be mentioned, once the root ca is updated, the whole system's service cert/key pair will need to be force updated at once, which may cause a short period of downtime.

Related modification

  1. HTTPServer
  • As for Easegress' HTTPServer, we had already supporting HTTPS, but for mTLS, it needs to enable tls.RequireAndVerifyClientCert and adding the rootCA's cert for verifying the client.
kind: httpserver
name: demo
...

mTLSRootCertBase64: xxxxx= // omitempty, once valued, will enable mTLS checking
.....

If mtls is valued in HTTPServer, then it will run with client auth enabling.

        // if mTLS configuration is provided, should enable tls.ClientAuth and
	// add the root cert
	if len(spec.MTLSRootCertBase64) != 0 {
		rootCertPem, _ := base64.StdEncoding.DecodeString(spec.MTLSRootCertBase64)
		certPool := x509.NewCertPool()
		certPool.AppendCertsFromPEM(rootCertPem)

		tlsConf.ClientAuth = tls.RequireAndVerifyClientCert
		tlsConf.ClientCAs = certPool
	}
  1. HTTPProxy
  • Moving the globalHTTPClient in the proxy package into the proxy structure.
  • Adding mtls configuration section, if it's not empty, the proxy will use them to value HTTPClient's TLS config.
kind: httpproxy
name: demo-proxy
....
mtls:
    certBase64:  xxxx=
    keyBase64:  yyyy=
    rootCertBase64: zzzz=
....

References

  1. https://github.com/openservicemesh/osm/blob/main/DESIGN.md
  2. https://en.wikipedia.org/wiki/Mutual_authentication#mTLS
  3. https://kofo.dev/how-to-mtls-in-golang
  4. https://medium.com/@shaneutt/create-sign-x509-certificates-in-golang-8ac4ae49f903
  5. https://venilnoronha.io/a-step-by-step-guide-to-mtls-in-go
  6. https://github.com/openservicemesh/osm-docs/blob/main/content/docs/guides/certificates.md
zhao-kun commented

My suggestion is to refer to other mesh products. No matter Istio or OSM, an extra component certificate-manager (Citadel in the istio, Valt in osm) be introduced, but I don't see it in your design. What do you think about it? A certificate-manager is at least responsible for providing a common trust root to allow sidecar to validate and authenticate each other.

benja-wu commented

My suggestion is to refer to other mesh products. No matter Istio or OSM, an extra component certificate-manager (Citadel in the istio, Valt in osm) be introduced, but I don't see it in your design. What do you think about it? A certificate-manager is at least responsible for providing a common trust root to allow sidecar to validate and authenticate each other.

  • The rootCA can be signed in the control plane's MeshController.
  • OSM also supports an internal cert-manager, which is Tresor implemented by Golang. So I assign this function to MechContoller's master role too.
xxx7xxxx commented

The name of model and refreshInterval is too general for mTLS. Please make them specific such as securityLevel and certRefreshInterval, or move them under section security.

And do we support both automatically generating certs by the control plane(only refresh this by control plane), and manually config from users?

benja-wu commented

@xxx7xxxx

  • About the name, I will update it.
  • About the certs, among popular mesh products, they can introduce certs from outside products, such as Valt. So I am working on figuring out the OSM's design. We should also support outer certs providers here. Let me update this design later.
benja-wu commented

Close after merged.