Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet.
Clusternet (Cluster Internet) is an open source add-on that helps you manage thousands of millions of Kubernetes clusters as easily as visiting the Internet. No matter the clusters are running on public cloud, private cloud, hybrid cloud, or at the edge, Clusternet lets you manage/visit them all as if they were running locally. This also help eliminate the need to juggle different management tools for each cluster.
Clusternet can also help deploy and coordinate applications to multiple clusters from a single set of APIs in a hosting cluster.
Clusternet will help setup network tunnels in a configurable way, when your clusters are running in a VPC network, at the edge, or behind a firewall.
Clusternet also provides a Kubernetes-styled API, where you can continue using the Kubernetes way, such as KubeConfig, to visit a certain Managed Kubernetes cluster, or a Kubernetes service.
Clusternet is multiple platforms supported now, including linux/amd64, linux/arm64, linux/ppc64le, linux/s390x
, linux/386 and linux/arm;
Clusternet is light-weighted that consists of two components, clusternet-agent and clusternet-hub.
clusternet-agent is responsible for
- auto-registering current cluster to a parent cluster as a child cluster, which is also been called
ManagedCluster; - reporting heartbeats of current cluster, including Kubernetes version, running platform,
healthz/readyz/livezstatus, etc; - setting up a websocket connection that provides full-duplex communication channels over a single TCP connection to parent cluster;
clusternet-hub is responsible for
- approving cluster registration requests and creating dedicated resources, such as namespaces, serviceaccounts and RBAC rules, for each child cluster;
- serving as an aggregated apiserver (AA), which is used to serve as a websocket server that maintain multiple active websocket connections from child clusters;
- providing Kubernstes-styled API to redirect/proxy/upgrade requests to each child cluster;
- coordinating and deploying applications to multiple clusters from a single set of APIs;
📌 📌 Note:
Since
clusternet-hubis running as an AA, please make sure that parent apiserver could visit theclusternet-hubservice.
For every Kubernetes cluster that wants to be managed, we call it child cluster. The cluster where child clusters are registerring to, we call it parent cluster.
clusternet-agent runs in child cluster, while clusternet-hub runs in parent cluster.
ClusterRegistrationRequestis an object thatclusternet-agentcreates in parent cluster for child cluster registration.ManagedClusteris an object thatclusternet-hubcreates in parent cluster after approvingClusterRegistrationRequest.HelmChartis an object contains a helm chart configuration.Subscriptiondefines the resources that subscribers want to install into clusters. For every matched cluster, a correspondingBaseobject will be created in its dedicated namespace.Clusternetprovides a two-stage priority based override strategy.LocalizationandGlobalizationwill define the overrides with priority, where lower numbers are considered lower priority.Localizationis namespace-scoped resource, whileGlobalizationis cluster-scoped. Refer to Deploying Applications to Multiple Clusters on how to use these.Baseobjects will be rendered toDescriptionobjects withGlobalizationandLocalizationsettings applied.Descriptionis the final resources to be deployed into target child clusters.
If you want to get participated and become a contributor to Clusternet, please don't hesitate to refer to our CONTRIBUTING document for details.
A developer guide is ready to help you
- build binaries for all platforms, such as
darwin/amd64,linux/amd64,linux/arm64, etc; - build docker images for multiple platforms, such as
linux/amd64,linux/arm64, etc;
You need to deploy clusternet-agent and clusternet-hub in child cluster and parent cluster respectively.
$ kubectl apply -f deploy/hubNext, you need to create a token for cluster registration, which will be used later by
clusternet-agent. Either a bootstrap token or a service account token is okay.
-
If bootstrapping authentication is supported, i.e.
--enable-bootstrap-token-auth=trueis explicitly set in the kube-apiserver running in parent cluster,$ # this will create a bootstrap token 07401b.f395accd246ae52d $ kubectl apply -f manifests/samples/cluster_bootstrap_token.yaml -
If bootstrapping authentication is not supported by the kube-apiserver in parent cluster (like k3s) , i.e.
--enable-bootstrap-token-auth=false(which defaults to befalse), please use serviceaccount token instead.$ # this will create a serviceaccount token $ kubectl apply -f manifests/samples/cluster_serviceaccount_token.yaml $ kubectl get secret -n clusternet-system -o=jsonpath='{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=="cluster-bootstrap-use")].data.token}' | base64 --decode; echo HERE WILL OUTPUTS A LONG STRING. PLEASE REMEMBER THIS.
clusternet-agent runs in child cluster and helps register self-cluster to parent cluster.
clusternet-agent could be configured with below three kinds of SyncMode (configured by flag --cluster-sync-mode),
Pushmeans that all the resource changes in the parent cluster will be synchronized, pushed and applied to child clusters byclusternet-hubautomatically.Pullmeansclusternet-agentwill watch, synchronize and apply all the resource changes from the parent cluster to child cluster.Dualcombines bothPushandPullmode. This mode is strongly recommended, which is usually used together with feature gateAppPusher.
Feature gate AppPusher works on agent side, which is introduced mainly for below two reasons,
-
SyncModeis not suggested getting changed after registration, which may bring in inconsistent settings and behaviors. That's whyDualmode is strong recommended. WhenDualmode is set, feature gateAppPusherprovides a way to help switchPushmode toPullmode without really changing flag--cluster-sync-mode, and vice versa. -
For security concerns, such as child cluster security risks, etc.
When a child cluster has disabled feature gate
AppPusher, the parent cluster won't deploy any applications to it, even if SyncModePushorDualis set. At this time, this child cluster is working likePullmode.Resources to be deployed are represented as
Description, you can run your own controllers as well to watch changes ofDescriptionobjects, then distribute and deploy resources.
Upon deploying clusternet-agent, a secret that contains token for cluster registration should be created firstly.
$ # create namespace clusternet-system if not created
$ kubectl create ns clusternet-system
$ # here we use the token created above
$ PARENTURL=https://192.168.10.10 REGTOKEN=07401b.f395accd246ae52d envsubst < ./deploy/templates/clusternet_agent_secret.yaml | kubectl apply -f -📌 📌 Note:
If you're creating service account token above, please replace
07401b.f395accd246ae52dwith above long string token that outputs.
The PARENTURL above is the apiserver address of the parent cluster that you want to register to, the https scheme
must be specified and it is the only one supported at the moment. If the apiserver is not listening on the standard
https port (:443), please specify the port number in the URL to ensure the agent connects to the right endpoint, for
instance, https://192.168.10.10:6443.
$ # before deploying, you could update the SyncMode if needed
$ kubectl apply -f deploy/agent$ # clsrr is an alias for ClusterRegistrationRequest
$ kubectl get clsrr
NAME CLUSTER ID STATUS AGE
clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 dc91021d-2361-4f6d-a404-7c33b9e01118 Approved 3d6h
$ kubectl get clsrr clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 -o yaml
apiVersion: clusters.clusternet.io/v1beta1
kind: ClusterRegistrationRequest
metadata:
labels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusters.clusternet.io/cluster-name: clusternet-cluster-dzqkw
clusters.clusternet.io/registered-by: clusternet-agent
name: clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118
spec:
clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
clusterName: clusternet-cluster-dzqkw
clusterType: EdgeCluster
status:
caCertificate: REDACTED
dedicatedNamespace: clusternet-dhxfs
managedClusterName: clusternet-cluster-dzqkw
result: Approved
token: REDACTEDAfter ClusterRegistrationRequest is approved, the status will be updated with corresponding credentials that can be
used to access parent cluster if needed. Those credentials have been set with scoped RBAC rules, see blow two rules for
details.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
clusternet.io/autoupdate: "true"
labels:
clusters.clusternet.io/bootstrapping: rbac-defaults
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusternet.io/created-by: clusternet-hub
name: clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118
rules:
- apiGroups:
- clusters.clusternet.io
resources:
- clusterregistrationrequests
verbs:
- create
- get
- apiGroups:
- proxies.clusternet.io
resourceNames:
- dc91021d-2361-4f6d-a404-7c33b9e01118
resources:
- sockets
verbs:
- '*'and
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
clusternet.io/autoupdate: "true"
labels:
clusters.clusternet.io/bootstrapping: rbac-defaults
clusternet.io/created-by: clusternet-hub
name: clusternet-managedcluster-role
namespace: clusternet-dhxfs
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'$ # mcls is an alias for ManagedCluster
$ # kubectl get mcls -A
$ # or append "-o wide" to display extra columns
$ kubectl get mcls -A -o wide
NAMESPACE NAME CLUSTER ID CLUSTER TYPE SYNC MODE KUBERNETES READYZ AGE
clusternet-dhxfs clusternet-cluster-dzqkw dc91021d-2361-4f6d-a404-7c33b9e01118 EdgeCluster Dual v1.19.10 true 7d23h
$ kubectl get mcls -n clusternet-dhxfs clusternet-cluster-dzqkw -o yaml
apiVersion: clusters.clusternet.io/v1beta1
kind: ManagedCluster
metadata:
labels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusters.clusternet.io/cluster-name: clusternet-cluster-dzqkw
clusternet.io/created-by: clusternet-agent
name: clusternet-cluster-dzqkw
namespace: clusternet-dhxfs
spec:
clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
clusterType: EdgeCluster
syncMode: Dual
status:
apiserverURL: http://10.0.0.10:8080
appPusher: true
healthz: true
k8sVersion: v1.19.10
lastObservedTime: "2021-06-30T08:55:14Z"
livez: true
platform: linux/amd64
readyz: trueThe status of ManagedCluster is updated by clusternet-agent every 3 minutes for default, which can be configured by
flag --cluster-status-update-frequency.
Clusternet supports visiting all your managed clusters with RBAC.
There is one prerequisite here, that is kube-apiserver should allow anonymous requests. The
flag --anonymous-auth is set to be true by default. So you can just ignore this unless this flag is set to false
explicitly.
Below is a simple snippet to show how to list namespaces in a child cluster with curl.
$ # Here the token is base64 decoded and from your child cluster.
$ CHILDCLUSTERTOKEN="TOKEN-BASE64-DECODED-IN-YOUR-CHILD-CLUSTER"
$ # specify the child cluster id
$ CHILDCLUSTERID="dc91021d-2361-4f6d-a404-7c33b9e01118"
$ # The Parent Cluster APIServer Address
$ APISERVER="https://10.0.0.10:6443"
$ curl -k -XGET -H "Accept: application/json" -H "Impersonate-User: clusternet" -H "Impersonate-Extra-Clusternet-Token: ${CHILDCLUSTERTOKEN}" -H "Authorization: Basic system:anonymous" "${APISERVER}/apis/proxies.clusternet.io/v1alpha1/sockets/${CHILDCLUSTERID}/proxy/direct/api/v1/namespaces"You need to follow below 2 steps to construct a valid kubeconfig to access a child cluster.
Append /apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/https/<SERVER-URL>
or /apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/direct at the end of original parent cluster
server address
CLUSTER-IDis a UUID for your child cluster, which is auto-populated byclusternet-agent, such as dc91021d-2361-4f6d-a404-7c33b9e01118. You could get this UUID from objectsClusterRegistrationRequest,ManagedCluster, etc. Also this UUID is labeled with keyclusters.clusternet.io/cluster-id.
SERVER-URLis the apiserver address of your child cluster, it could belocalhost,127.0.0.1and etc, only ifclusternet-agentcould access.
You can follow below commands to help modify above changes.
$ # suppose your parent cluster kubeconfig locates at /home/demo/.kube/config.parent
$ kubectl config view --kubeconfig=/home/demo/.kube/config.parent --minify=true --raw=true > ./config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
$
$ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://10.0.0.10:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
$
$ # suppose your child cluster running at https://demo1.cluster.net
$ kubectl config set-cluster `kubectl config get-clusters | grep -v NAME` \
--server=https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/https/demo1.cluster.net
$ # or just use the direct proxy path
$ kubectl config set-cluster `kubectl config get-clusters | grep -v NAME` \
--server=https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct📌 📌 Note:
Clusternet supports both http and https scheme.
If you want to use scheme
httpto demonstrate how it works, i.e./apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/http/<SERVER-URL>, you can simply run a local proxy in your child cluster, for example,$ kubectl proxy --address='10.212.0.7' --accept-hosts='^*$'Please replace
10.212.0.7with your real local IP address.Then follow above url modification as well.
Then update user entry with credentials from child clusters
🙈 🙈 Note:
Clusternet-hubdoes not care about those credentials at all, passing them directly to child clusters.
Here the tokens can be bootstrap tokens, ServiceAccount tokens , etc.
Please follow below modifications.
$ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
$ # below is what we modified in above step 1
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
$
$ # modify user part to below
$ vim config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
...
user:
username: system:anonymous
as: clusternet
as-user-extra:
clusternet-token:
- BASE64-DECODED-PLEASE-CHANGE-MEPlease replace BASE64-DECODED-PLEASE-CHANGE-ME to a token that valid from child cluster. Please notice the
tokens replaced here should be base64 decoded.
Please follow below modifications.
$ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
$ # below is what we modified in above step 1
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
$
$ # modify user part to below
$ vim config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118
...
user:
username: system:anonymous
as: clusternet
as-user-extra:
clusternet-certificate:
- CLIENT-CERTIFICATE-DATE-BASE64-ENCODED-PLEASE-CHANGE-ME
clusternet-privatekey:
- CLIENT-KEY-DATE-PLEASE-BASE64-ENCODED-CHANGE-MEPlease replace CLIENT-CERTIFICATE-DATE-BASE64-ENCODED-PLEASE-CHANGE-ME
and CLIENT-KEY-DATE-PLEASE-BASE64-ENCODED-CHANGE-ME with certficate and private key from child cluster. Please
notice the tokens replaced here should be base64 encoded.
Clusternet has provided two ways to help interact with Clusternet.
- kubectl plugin kubectl-clusternet
- use client-go to interact with Clusternet
Clusternet supports deploying applications to multiple clusters from a single set of APIs in a hosting cluster.
📌 📌 Note:
Feature gate
Deployershould be enabled byclusternet-hub.
First, let's see an exmaple application. Below Subscription "app-demo" defines the target child clusters to be
distributed to, and the resources to be deployed with.
# examples/applications/subscription.yaml
apiVersion: apps.clusternet.io/v1alpha1
kind: Subscription
metadata:
name: app-demo
namespace: default
spec:
subscribers: # defines the clusters to be distributed to
- clusterAffinity:
matchLabels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118 # PLEASE UPDATE THIS CLUSTER-ID TO YOURS!!!
feeds: # defines all the resources to be deployed with
- apiVersion: apps.clusternet.io/v1alpha1
kind: HelmChart
name: mysql
namespace: default
- apiVersion: v1
kind: Namespace
name: foo
- apiVersion: apps/v1
kind: Service
name: my-nginx-svc
namespace: foo
- apiVersion: apps/v1
kind: Deployment
name: my-nginx
namespace: fooBefore applying this Subscription, please
modify examples/applications/subscription.yaml
with your clusterID.
Clusternet also provides a two-stage priority based override strategy. You can define
namespace-scoped Localization and cluster-scoped Globalization with priorities (ranging from 0 to 1000, default to
be 500), where lower numbers are considered lower priority. These Globalization(s) and Localization(s) will be applied
by order from lower priority to higher. That means override values in lower Globalization will be overridden by those
in higher Globalization. Globalization(s) come first and then Localization(s).
💫 💫 For example,
Globalization (priority: 100) -> Globalization (priority: 600) -> Localization (priority: 100) -> Localization (priority 500)
Meanwhile, below override policies are supported,
ApplyNowwill apply overrides for matched objects immediately, including those are already populated.- Default override policy
ApplyLaterwill only apply override for matched objects on next updates (including updates onSubscription,HelmChart, etc) or new created objects.
Before applying these Localization(s), please
modify examples/applications/localization.yaml
with your ManagedCluster namespace, such as clusternet-5l82l.
After installing kubectl plugin kubectl-clusternet, you could run below commands to distribute this application to child clusters.
$ kubectl clusternet apply -f examples/applications/
helmchart.apps.clusternet.io/mysql created
namespace/foo created
deployment.apps/my-nginx created
service/my-nginx-svc created
subscription.apps.clusternet.io/app-demo created
$ # or
$ # kubectl-clusternet apply -f examples/applications/Then you can view the resources just created,
$ # list Subscription
$ kubectl clusternet get subs -A
NAMESPACE NAME AGE
default app-demo 6m4s
$ kubectl clusternet get chart
NAME CHART VERSION REPO STATUS AGE
mysql mysql 8.6.2 https://charts.bitnami.com/bitnami Found 71s
$ kubectl clusternet get ns
NAME CREATED AT
foo 2021-08-07T08:50:55Z
$ kubectl clusternet get svc -n foo
NAME CREATED AT
my-nginx-svc 2021-08-07T08:50:57Z
$ kubectl clusternet get deploy -n foo
NAME CREATED AT
my-nginx 2021-08-07T08:50:56ZClusternet will help deploy and coordinate applications to multiple clusters. You can check the status by following
commands,
$ kubectl clusternet get mcls -A
NAMESPACE NAME CLUSTER ID SYNC MODE KUBERNETES READYZ AGE
clusternet-5l82l clusternet-cluster-hx455 dc91021d-2361-4f6d-a404-7c33b9e01118 Dual v1.21.0 true 5d22h
$ # list Descriptions
$ kubectl clusternet get desc -A
NAMESPACE NAME DEPLOYER STATUS AGE
clusternet-5l82l app-demo-generic Generic Success 2m55s
clusternet-5l82l app-demo-helm Helm Success 2m55s
$ kubectl describe desc -n clusternet-5l82l app-demo-generic
...
Status:
Phase: Success
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyDeployed 2m55s clusternet-hub Description clusternet-5l82l/app-demo-generic is deployed successfully
$ # list Helm Release
$ # hr is an alias for HelmRelease
$ kubectl clusternet get hr -n clusternet-5l82l
NAME CHART VERSION REPO STATUS AGE
helm-demo-mysql mysql 8.6.2 https://charts.bitnami.com/bitnami deployed 2m55sYou can also verify the installation with Helm command line in your child cluster,
$ helm ls -n abc
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helm-demo-mysql abc 1 2021-07-06 14:34:44.188938 +0800 CST deployed mysql-8.6.2 8.0.25📌 📌 Note:
Admission webhooks could be configured in parent cluster, but please make sure that dry-run mode is supported in these webhooks. At the same time, a webhook must explicitly indicate that it will not have side-effects when running with
dryRun. That issideEffectsmust be set toNoneorNoneOnDryRun.While, these webhooks could be configured per child cluster without above limitations as well.