/toolbox

A toolbox of utilities that can be used in a cloud-native environment and Devops Toolchains

MIT LicenseMIT

Cloud Native Toolbox

A Toolbox of utilities that can be used in a cloud-native environment. Mostly focused on the Devops Toolchain and how various CNCF projects can be leveraged to address potential solutions at each step (or across steps)

Quickstart

  • Opt: Clone Repo
  • Click Gitpod ready-to-code

User Story:

  • As a DevOps Engineer, quickly bring up a Kubernetes CLI/GUI app that can be run from anywhere (vscode-server with a kubectl/kustomize/helm baked in)
  • As a DevOps Engineer, I need a quick "reference" for common configurations. Should be installed relatively quickly - leveraging either the operator framework, Helm Charts, or kustomize/kubectl as necessary.
    • Quick debugging
      • Spin up a variety of debugging pods (network tools, eBPL based toosl, etc)
    • Common references for essential utilities in most K8S clusters (like BKPR, but...modern-ish) (or also like Digital Ocean's Starter Kit
      • Operator Lifecycle Manager - only selectivily for complex reosureces
      • Monitoring Stack
        • Promtheus Operator (Metrics) (include alerts)
        • Loki (Plain YAML or Helm?)
        • Grafana (Metrics/Logs)
        • Pixie Operator (Tracing/Network)
      • Ingress Stack
        • Nginx Ingress Controller
        • Cert-Manager
        • Oauth2 Proxy
        • External DNS
      • Security Stack
        • Sealed Secretes - Feels the most native k8s way to do secrets without being tied to a specific backend (such as with the External Secrets operator)
        • Security Operator (All in One): Aqua's Starboard
          • Trivy Scanner for COntiner Scanning
          • Configuration audit via built-in polcies (rego written)
          • Node Configuration audit via kube-bench
          • Automated Pen Testing for kube-hunter
        • Individual Tools
          • Runtime Security: Falco
          • Configuration: Kubebench, kubehunter
          • Live Container Scanning: Kube-scan
      • Scaling (HPA, VPA, Cluster Auto-Scaler)
      • GitOps - FluxCD. Prefer Flux's handing of Helm charts versus ArgoCD (the other major competitor). Although Argo might make doing kustomize patches after a helm chart application a bit easier...
        • Although honestly, if a Helm chart isn't htat flexible, just leverage helm template to render the raw yaml, and massage with kustomize.
      • Backup/Restore
        • Should MOSTLY be in the gitops repo, but a nice second layer in some circumstances
        • Useful for Sealed Secrets, as well as anything in PVs
        • Velero for the most part
  • As a DevOps Engineer, need a platform to quickly test out new configuration changes:
    • Spin up clusters easily on a variety of platform (Managed K8S or k3s on IaaS )

Overall Thoughts/Notes

  • Bootstrap Methods (Note: ways to "bootstrap" a cluster that has a web interface for management)
    • local machine (vscode devcontainer optimized) - leverages k3d/k3s locally
    • GitHub actions directly - Initialize
    • GitHub actions through act
    • Initial cluster: fairly minimal:
      • Web Interface
      • External Secrets
      • Flux?
      • tf-controller that can spin up additional resources
      • Cluster-API?
  • Pathways
    • kubectl direct application
      • GitPod environment
    • k8s cluster for primary bootstrap
    • FluxCD for bootstrap deployment
    • clusterapi for k8s cluster deployemnts (either manually applied, or via FluxCD)
    • terraform controller https://github.com/weaveworks/tf-controller - For other
    • opt: local git repo for flux for air-gapped - gitea. Or maybe I assume GitHub itself is available
  • Targets
    • k8s debugging
    • k8s basics ... Something like bitnami production runtime, but lighter weight
    • k8s example apps. Apps in separate repos, but implementation here

Directory Layout

  • .github: Workflows to build/test all below
  • docker: Useful Docker containers and templates
  • packer: For any custom image buildes necessary
  • terraform: Primary IaC tool
  • k8s:
    • manifests: Individual k8s "applications. Should be "kustomize" applicable applicatons. Helm charts should be exploded with "helm template", but with a Makefile that can be used to regenerate if necessary
    • clusters: Example "clusters". Kustomize of apps above
  • scripts: Collection of random scripts

Tools

Central Components

  • Docker - Allow for easily bundling and extracting resources to single use containers
  • 12 Factor App - Try to ensure all toolbox components are designed with a 12 factor mentality in mind. Sometimes difficult in deep debugging situation, btu still an admirable goal.
  • Kubernetes Patterns EBook (PDF) - Great overview of Kubernetes primatives and how to apply then to a variety of problems.

Kubernetes Core Components/Tools

Debugging Tools

  • Netshoot: General purpose docker container useful for a variety of network and system debugging
  • kubectl debug: Plugin to allow debug containers to be spun up in same namespace as pod you wish to debug.

Practice Environments

Always good to have an example "app" to practice deployments and debugging.