⛵ Build Amazon EKS using Terraform, GitHub Actions and GitOps

... managed by Flux/ArgoCD and serviced with Renovate ...

GitHub repository: https://github.com/ruzickap/k8s-tf-eks-gitops
Web Pages: https://ruzickap.github.io/k8s-tf-eks-gitops

📖 Overview

✨ Cluster setup

🎨 Cluster components

cilium: For cluster networking.
ingress-nginx: Provides ingress cluster services.
SOPS: Encrypts secrets which is safe to store - even to a public repository.
external-dns: Creates DNS entries in Cloud Provider's DNS service.
cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.

📂 Repository structure

flux tree kustomization flux-system --compact

Output:

Kustomization/flux-system/flux-system
├── Kustomization/flux-system/cluster-apps
│   ├── HelmRelease/metrics-server/metrics-server
│   ├── HelmRelease/polaris/polaris
│   ├── Kustomization/flux-system/cert-manager
│   │   └── HelmRelease/cert-manager/cert-manager
│   ├── Kustomization/flux-system/cert-manager-certificate
│   ├── Kustomization/flux-system/cert-manager-clusterissuer
│   ├── Kustomization/flux-system/cert-manager-crds
│   ├── Kustomization/flux-system/cluster-autoscaler
│   │   └── HelmRelease/cluster-autoscaler/cluster-autoscaler
│   ├── Kustomization/flux-system/crossplane
│   │   └── HelmRelease/crossplane-system/crossplane
│   ├── Kustomization/flux-system/crossplane-provider
│   ├── Kustomization/flux-system/crossplane-providerconfig
│   ├── Kustomization/flux-system/dex
│   │   └── HelmRelease/dex/dex
│   ├── Kustomization/flux-system/external-dns
│   │   └── HelmRelease/external-dns/external-dns
│   ├── Kustomization/flux-system/ingress-nginx
│   │   └── HelmRelease/ingress-nginx/ingress-nginx
│   ├── Kustomization/flux-system/kube-prometheus-stack
│   │   └── HelmRelease/kube-prometheus-stack/kube-prometheus-stack
│   ├── Kustomization/flux-system/kubernetes-dashboard
│   │   └── HelmRelease/kubernetes-dashboard/kubernetes-dashboard
│   ├── Kustomization/flux-system/oauth2-proxy
│   │   └── HelmRelease/oauth2-proxy/oauth2-proxy
│   └── Kustomization/flux-system/podinfo
│       └── HelmRelease/podinfo/podinfo
├── Kustomization/flux-system/cluster-apps-secrets
├── Kustomization/flux-system/sources
│   ├── HelmRepository/flux-system/autoscaler
│   ├── HelmRepository/flux-system/bitnami
│   ├── HelmRepository/flux-system/crossplane
│   ├── HelmRepository/flux-system/dex
│   ├── HelmRepository/flux-system/fairwinds-stable
│   ├── HelmRepository/flux-system/ingress-nginx
│   ├── HelmRepository/flux-system/jetstack
│   ├── HelmRepository/flux-system/kubernetes-dashboard
│   ├── HelmRepository/flux-system/metrics-server
│   ├── HelmRepository/flux-system/oauth2-proxy
│   ├── HelmRepository/flux-system/podinfo
│   └── HelmRepository/flux-system/prometheus-community
└── GitRepository/flux-system/flux-system

🤖 Automate all the things

GitHub Actions for checking code formatting
Renovate Renovate GitHub action keeps my application charts and container images up-to-date

🕸️ Secrets

There are several secrets:

cluster-apps-vars-terraform-secret - used for providing Terraform variables to Flux/Kustomizations: eks.tf
cluster-apps-secrets - secrets specific to cluster: cluster-apps-secrets.yaml
cluster-apps-group-secrets - secrets specific to cluster group: cluster-apps-secrets.yaml

🤷‍♂️ Notes

Describe the directory structure
Check emails form policy-reporter
Put all HelmRepository objects to flux-system instead of "namespaces" to be able to share them
Check snapshots (cnpg/velero) + KMS keys (if they are being deleted)
kubernetes-dashboard - auto login not working

🤝 Thanks

A lot of inspiration for my cluster came from the people that have shared their clusters over at k8s-at-home and many other "GitHub" repositories...

Requirements for K8s cluster management

Not complete...

Run change on only 3% of the clusters, then another 3%, ...
Changes applied to group of clusters
Easily add / remove application from specific cluster / cluster group
One source (file) for "variables/secrets"
Move cluster from one cluster to another... ?
Different TF code for different clusters / cluster groups

ruzickap/k8s-tf-eks-gitops