/k8s-hard-way

My take on Kubernetes the hard way and way further

Primary LanguageHCL

Failed Containership

k8s the hard way - terraform

Disclaimer: this is - by no means - production grade material.

⚠️ All private keys/encryption tokens and other secrets are stored in the terraform state & on disk; some are even displayed on screen. Furthermore there are certain in-cluster security issues:

  • K8s Pods reach the aws metadataservice
  • IAM Roles set on ec2 instances/user instead of IRSA
  • Dex idP clientSecrets are visible

The project evolved to a playground for all of my k8s/aws tech related curiosity.

This is an aws based k8s-the-hard-way setup (inspired by Kelsey Hightower), solely deployed with terraform and hacky shell scripts, triggered by cloud-init. No kops or kubeadm.

While initially deployed on Rhel8 with SELinux, it's now based on flatcar and going through a major terraform refactoring.

Roadmap

  • Clustersetup
    • CoreDNS
    • CNI Networking with weave calico
    • Static pods
    • Move away from shell-scripts to ignition
      https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/configuration.md
    • Switch from rhel8 to flatcar /talon/bottlerocket
    • binary checksum verification
    • K8s bootstrap tokens
      Kubelets are provisioned on the fly with bootstrap tokens and the csr-approver
    • kube-bench with a reasonable score
    • Move the bastion LB for the k8s api to aws_lb
      The Bastion Host has a nginx loadbalancing the k8s-api; This should be replaced by an aws network lb
    • etcd autodiscovery (etcd in autoscalingroups)
      Right now dns is used for autodiscovery; but the etcd are not part of a scaling group
    • Instance access with aws SSM instead of ssh
      SSH access is not possible as the instances are configured to use the aws systemsmanager
  • Clusteraddons
  • IdentityManagement
    • [] Dex as idP with Github Backend for all login related Toosl (kubectl, argoCD)
    • Implement IRSA for aws acces
  • Autoscaling
    • ...
    • scaling of nodes dependend on load -> karpenter
    • spot instances
  • Application
    • ArgoCD
  • Github
    • Dependabot
    • Terracost Integration for PRs
  • Ungrouped
    • ImageScanWebook
    • Block aws metadata access from cluster
    • crossplane vs aws-controllers-k8s
    • kyverno vs gatekeeper/opa vs kubevious
    • cluster backup -> velero
    • refactor terraform in module groups
    • https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/
    • logging: cloudwatch
    • check encryption: etcd; ebs; s3
    • kubecost
    • Update to 1.24
    • Terraform state to s3 bucket

Implementation

Hosts

Rhel8 Images are used for control- & workernodes with SELinux enabled.

The current setup is using Flatcar as immutable OS.

Network

The controller & worker nodes are evenly distributed among all availabe AZs:

  • AZ 1a CIDR: 10.10.16.0/20
  • AZ 1b CIDR: 10.10.32.0/20
  • AZ 1c CIDR: 10.10.48.0/20
  • Service CIDR: 10.32.0.0/24
  • Cluster CIDR: 10.200.0.0/16
  • Cluster DNS: 10.32.0.53

dex

intro in dex... installing kubectl login plugin...

open ToDos

  • view only clusterrolebinding

Usage

References

Gerneral Reference

Kubernetes TLS

Kubernetes Bootstrapping

Etcd Autodiscovery

Userdata/Ignition