aws/aws-cloud-map-mcs-controller-for-k8s

K8s controller implementing Multi-Cluster Services API based on AWS Cloud Map.

AWS Cloud Map MCS Controller for K8s

Introduction

The AWS Cloud Map Multi-cluster Service Discovery Controller for Kubernetes (K8s) implements the Kubernetes KEP-1645: Multi-Cluster Services API and KEP-2149: ClusterId for ClusterSet identification, which allows services to communicate across multiple clusters. The implementation relies on AWS Cloud Map for enabling cross-cluster service discovery. We have detailed step-by-step setup guide!

⚠ NOTE: The current version is in Alpha phase, and NOT intended for production use. The support will be limited to critical bug fixes.

Checkout the Graduation Criteria for moving the project to the next phase.

Installation

Perform the following installation steps on each participating cluster.

• For multi-cluster service discovery and consumption, the controller should be installed on a minimum of 2 EKS clusters.
• Participating clusters should be provisioned into a single AWS account, within a single AWS region.

Dependencies

Network

The AWS Cloud Map MCS Controller for K8s provides service discovery and communication across multiple clusters, therefore implementations depend on end-end network connectivity between workloads provisioned within each participating cluster.

• In deployment scenarios where participating clusters are provisioned into separate VPCs, connectivity will depend on correctly configured VPC Peering, inter-VPC routing, and Security Group configuration. The VPC Reachability Analyzer can be used to test and validate end-end connectivity between worker nodes within each cluster.
• Undefined behavior may occur if controllers are deployed without the required network connectivity between clusters.

Configure CoreDNS

Install the CoreDNS multicluster plugin into each participating cluster. The multicluster plugin enables CoreDNS to lifecycle manage DNS records for ServiceImport objects.

To install the plugin, run the following commands.

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/coredns-clusterrole.yaml"
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/coredns-configmap.yaml"
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/coredns-deployment.yaml"

Install Controller

To install the latest release of the controller, run the following commands.

NOTE: AWS region environment variable can be optionaly set like export AWS_REGION=us-west-2 Otherwise the controller will infer region in the order AWS_REGION environment variable, ~/.aws/config file, then EC2 metadata (for EKS environment)

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release"

📌 See Releases section for details on how to install other versions.

The controller must have sufficient IAM permissions to perform required Cloud Map operations. Grant IAM access rights AWSCloudMapFullAccess to the controller Service Account to enable the controller to manage Cloud Map resources.

Usage

Configure cluster.clusterset.k8s.io and clusterset.k8s.io

cluster.clusterset.k8s.io is a unique identifier for the cluster.

clusterset.k8s.io is an identifier that relates to the ClusterSet in which the cluster belongs.

apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
  name: cluster.clusterset.k8s.io
spec:
  value: [Your Cluster identifier]
---
apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
  name: clusterset.k8s.io
spec:
  value: [Your ClusterSet identifier]

Example:

apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
  name: cluster.clusterset.k8s.io
spec:
  value: my-first-cluster
---
apiVersion: about.k8s.io/v1alpha1
kind: ClusterProperty
metadata:
  name: clusterset.k8s.io
spec:
  value: my-clusterset

Export services

Then assuming you already have a Service installed, apply a ServiceExport yaml to the cluster in which you want to export a service. This can be done for each service you want to export.

kind: ServiceExport
apiVersion: multicluster.x-k8s.io/v1alpha1
metadata:
  namespace: [Your service namespace here]
  name: [Your service name]

Example: This will export a service with name my-amazing-service in namespace hello

kind: ServiceExport
apiVersion: multicluster.x-k8s.io/v1alpha1
metadata:
  namespace: hello
  name: my-amazing-service

See the samples directory for a set of example yaml files to set up a service and export it. To apply the sample files run the following commands.

kubectl create namespace example
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/example-deployment.yaml
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/example-service.yaml
kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/samples/example-serviceexport.yaml

Import services

In your other cluster, the controller will automatically sync services registered in AWS Cloud Map by applying the appropriate ServiceImport. To list them all, run the following command.

kubectl get ServiceImport -A

Releases

AWS Cloud Map MCS Controller for K8s adheres to the SemVer specification. Each release updates the major version tag (eg. vX), a major/minor version tag (eg. vX.Y) and a major/minor/patch version tag (eg. vX.Y.Z). To see a full list of all releases, refer to our Github releases page.

NOTE: AWS region environment variable can be optionally set like export AWS_REGION=us-west-2 Otherwise controller will infer region in the order AWS_REGION environment variable, ~/.aws/config file, then EC2 metadata (for EKS environment)

The following command format is used to install from a particular release.

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release[?ref=*git version tag*]"

Run the following command to install the latest release.

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release"

The following example will install release v0.1.0.

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release?ref=v0.1.0"

We also maintain a latest tag, which is updated to stay in line with the main branch. We do not recommend installing this on any production cluster, as any new major versions updated on the main branch will introduce breaking changes.

To install from latest tag run the following command.

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_latest"

Graduation Criteria

Alpha -> Beta Graduation

• Implement the resiliency milestone.

Beta -> GA Graduation

• Scalability/performance testing.
• Implement the observability and deployment milestone.
• KEP-1645: Multi-Cluster Services API and KEP-2149: ClusterId for ClusterSet identification are Beta or GA.

Slack community

We have an open Slack community where users may get support with integration, discuss controller functionality and provide input on our feature roadmap. https://awsappmesh.slack.com/#k8s-mcs-controller Join the channel with this invite.

Contributing

aws-cloud-map-mcs-controller-for-k8s is an open source project. See CONTRIBUTING for details.

License

This project is distributed under the Apache License, Version 2.0, see LICENSE and NOTICE for more information.