Compliance Oriented Kubernetes Setup for AWS.
Kubespot is an open source terraform module that attempts to create a complete compliance-oriented Kubernetes setup on AWS, Google Cloud and Azure. These add additional security such as additional system logs, file system monitoring, hard disk encryption and access control. Further, we setup the managed Redis and SQL on each of the Cloud providers with limited access to the Kubernetes cluster so things are further locked down. All of this should lead to setting up a HIPAA / PCI / SOC2 being made straightforward and repeatable.
This covers how we setup your infrastructure on AWS, Google Cloud and Azure. These are the three Cloud Providers that we currently support to run Kubernetes. Further, we use the managed service provided by each of the Cloud Providers. This document covers everything related to how infrastructure is setup within each Cloud, how we create an isolated environment for Compliance and the commonalities between them.
brew install kubectl kubernetes-helm awscli terraform
If the infrastructure is using the opsZero infrastructure as code template then you access the resources like the following:
Add your IAM credentials in ~/.aws/credentials
.
[profile_name]
aws_access_key_id=<>key>
aws_secret_access_key=<secret_key>
region=us-west-2
cd environments/<nameofenv>
make kubeconfig
export KUBECONFIG=./kubeconfig # add to a .zshrc
kubectl get pods
Kubespot uses Karpenter as the default autoscaler. To configure the autoscaler we need to create a file like the one below and run:
kubectl apply -f karpenter.yml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
consolidation:
enabled: true # If set to true the nodes will minimize to fit the pods
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["t", "c", "m"]
- key: "kubernetes.io/arch"
operator: In
values: ["amd64"]
- key: "karpenter.k8s.aws/instance-cpu"
operator: In
values: ["1", "2", "4", "8", "16"]
- key: "karpenter.k8s.aws/instance-hypervisor"
operator: In
values: ["nitro"]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
limits:
resources:
cpu: 200
provider:
securityGroupSelector:
Name: <cluster-name>-node
subnetSelector:
Name: <cluster-name>-private
tags:
karpenter.sh/discovery: <cluster-name>
ttlSecondsUntilExpired: 86400 # How long to keep the node before cycling
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
opsZero provides support for our modules including:
- Email support
- Zoom Calls
- Implementation Guidance
Name | Version |
---|---|
aws | ~> 4.50.0 |
aws.virginia | ~> 4.50.0 |
helm | 2.8.0 |
http | n/a |
kubernetes | >= 2.0 |
null | n/a |
tls | n/a |
Name | Description | Type | Default | Required |
---|---|---|---|---|
alb_controller_version | The chart version of the ALB controller helm chart | string |
"1.4.4" |
no |
aws_load_balancer_controller_enabled | Enable ALB controller by default | bool |
true |
no |
aws_profile | AWS profile to use | string |
n/a | yes |
cidr_block | The CIDR block used by the VPC | string |
"10.2.0.0/16" |
no |
cidr_block_private_subnet | The CIDR block used by the private subnet | list |
[ |
no |
cidr_block_public_subnet | The CIDR block used by the private subnet | list |
[ |
no |
cluster_logging | List of the desired control plane logging to enable | list |
[ |
no |
cluster_private_access | Whether the Amazon EKS private API server endpoint is enabled | bool |
false |
no |
cluster_public_access | Whether the Amazon EKS private API server endpoint is enabled | bool |
true |
no |
cluster_public_access_cidrs | List of CIDR blocks. Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled | list |
[ |
no |
cluster_version | Desired Kubernetes master version | string |
"1.23" |
no |
csi_secrets_store_enabled | Specify whether the CSI driver is enabled on the EKS cluster | bool |
false |
no |
csi_secrets_store_version | The version of the CSI store helm chart | string |
"1.3.0" |
no |
efs_enabled | Specify whether the EFS is enabled on the EKS cluster | bool |
false |
no |
eips | List of Elastic IPs | list |
[] |
no |
enable_egress_only_internet_gateway | Create an egress-only Internet gateway for your VPC0 | bool |
false |
no |
enable_ipv6 | Enable an Amazon-provided IPv6 CIDR block with a /56 prefix length for the VPC | bool |
false |
no |
enable_nat | Whether the NAT gateway is enabled | bool |
true |
no |
enabled_metrics_asg | A list of metrics to collect | list |
[ |
no |
environment_name | Name of the environment to create AWS resources | string |
n/a | yes |
fargate_selector | Terraform object to create the EKS fargate profiles | map |
{ |
no |
govcloud | Set if the environment is govcloud | bool |
false |
no |
iam_users | List of IAM users | list |
[] |
no |
karpenter_enabled | Specify whether the karpenter is enabled | bool |
false |
no |
karpenter_version | The version of the karpenter helm chart | string |
"v0.23.0" |
no |
legacy_subnet | Specify how the subnets should be created | bool |
true |
no |
metrics_server_version | The version of the metric server helm chart | string |
"3.8.3" |
no |
monitoring_role_arn | The ARN for the IAM role that permits RDS to send enhanced monitoring metrics to CloudWatch Logs | string |
"" |
no |
node_group_cpu_threshold | The value of the CPU threshold | string |
"70" |
no |
node_groups | Terraform object to create the EKS node groups | map |
{} |
no |
node_role_policies | A list of The ARN of the policies you want to attach | list |
[] |
no |
nodes_blue_desired_capacity | The number of Amazon EC2 instances that should be running in the group | number |
0 |
no |
nodes_blue_instance_type | The instance type to use for the instance | string |
"t3.micro" |
no |
nodes_blue_max_instance_lifetime | The maximum amount of time, in seconds, that an instance can be in service | number |
604800 |
no |
nodes_blue_max_size | The maximum size of the Auto Scaling Group | number |
0 |
no |
nodes_blue_min_size | The minimum size of the Auto Scaling Group | number |
0 |
no |
nodes_blue_root_device_size | Size of the volume in gibibytes (GiB) | string |
"20" |
no |
nodes_blue_spot_price | The maximum price to use for reserving spot instances. | any |
null |
no |
nodes_blue_subnet_ids | A list of subnet IDs to launch resources in | list |
[] |
no |
nodes_green_desired_capacity | The number of Amazon EC2 instances that should be running in the group | number |
0 |
no |
nodes_green_instance_type | The instance type to use for the instance | string |
"t3.micro" |
no |
nodes_green_max_instance_lifetime | The maximum amount of time, in seconds, that an instance can be in service | number |
604800 |
no |
nodes_green_max_size | The maximum size of the Auto Scaling Group | number |
0 |
no |
nodes_green_min_size | The minimum size of the Auto Scaling Group | number |
0 |
no |
nodes_green_root_device_size | Size of the volume in gibibytes (GiB) | string |
"20" |
no |
nodes_green_spot_price | The maximum price to use for reserving spot instances. | any |
null |
no |
nodes_green_subnet_ids | A list of subnet IDs to launch resources in | list |
[] |
no |
nodes_in_public_subnet | INSECURE! Only use this if you want to avoid paying for the NAT. Also set enable_nat to false | bool |
false |
no |
redis_enabled | Whether the redis cluster is enabled | bool |
false |
no |
redis_engine_version | Version number of the cache engine to be used for the cache clusters in this replication group | string |
"7.0" |
no |
redis_node_type | Instance class of the redis cluster to be used | string |
"cache.t4g.micro" |
no |
redis_num_nodes | Number of nodes for redis | number |
1 |
no |
repos | n/a | list |
[] |
no |
sql_cluster_enabled | Whether the sql cluster is enabled | bool |
false |
no |
sql_database_name | The name of the database to create when the DB instance is created | string |
"" |
no |
sql_encrypted | Specify whether the DB instance is encrypted | bool |
true |
no |
sql_engine | The name of the database engine to be used for this DB cluster | string |
"aurora-postgresql" |
no |
sql_engine_mode | The database engine mode | string |
"provisioned" |
no |
sql_engine_version | The engine version to use | string |
"14.3" |
no |
sql_identifier | The name of the database | string |
"" |
no |
sql_instance_allocated_storage | The allocated storage in gibibytes | number |
20 |
no |
sql_instance_class | The instance type of the RDS instance. | string |
"db.t4g.micro" |
no |
sql_instance_enabled | Whether the sql instance is enabled | bool |
false |
no |
sql_instance_engine | The database engine to use | string |
"postgres" |
no |
sql_instance_max_allocated_storage | the upper limit to which Amazon RDS can automatically scale the storage of the DB instance | number |
200 |
no |
sql_master_password | Password for the master DB user | string |
"" |
no |
sql_master_username | Username for the master DB user | string |
"" |
no |
sql_node_count | The number of instances to be used for this DB cluster | number |
0 |
no |
sql_parameter_group_name | Name of the DB parameter group to associate | string |
"" |
no |
sql_rds_multi_az | Specify if the RDS instance is enabled multi-AZ | bool |
false |
no |
sql_serverless_max | The maximum capacity for the DB cluster | number |
2 |
no |
sql_serverless_min | The maximum capacity for the DB cluster | number |
2 |
no |
sql_serverless_seconds_until_auto_pause | The time, in seconds, before the DB cluster in serverless mode is paused | number |
300 |
no |
sql_skip_final_snapshot | Determines whether a final DB snapshot is created before the DB instance is deleted. | bool |
true |
no |
sql_subnet_group_include_public | Include public subnets as part of the clusters subnet configuration. | bool |
false |
no |
sso_roles | Terraform object of the IAM roles | map |
{ |
no |
tags | Terraform map to create custom tags for the AWS resources | map |
{} |
no |
vpc_flow_logs_enabled | Specify whether the vpc flow log is enabled | bool |
false |
no |
zones | AZs for the subnets | list |
[ |
no |
Name | Description |
---|---|
eks_cluster | n/a |
eks_cluster_token | n/a |
nat_gateway_ids | n/a |
node_role | n/a |
node_security_group_id | n/a |
private_route_table | n/a |
private_subnet_ids | n/a |
public_route_table | n/a |
public_subnet_ids | n/a |
redis_elasticache_subnet_group_name | n/a |
vpc_id | n/a |