This repository contains the source for the DFDS Crossplane configuration package. Using it will provide a set of composite cloud resources (AWS) with sane defaults defined.
Installs an AWS RDS Instance (currently postgres only) with a security group allowing public access into the default VPC.
The following are required in order to use the configuration package
- A Kubernetes cluster
- Crossplane installed through Helm
- An AWS account with an IAM key and secret that has sufficient permissions to deploy resources
- Crossplane CLI installed on local machine
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --version <version>
- Execute
docker pull dfdsdk/dfds-infra && kubectl crossplane install configuration dfdsdk/dfds-infra
to get the latest version, or use a release tag to install a specific version:
kubectl crossplane install configuration dfdsdk/dfds-infra <tag>
You can find the list of tags at https://hub.docker.com/repository/docker/dfdsdk/dfds-infra
- Execute
kubectl crossplane update configuration dfdsdk-dfds-infra [release-number]
- Execute
kubectl delete configuration.pkg dfdsdk-dfds-infra
For each namespace in which you wish to provision resources you will need to install a ProviderConfig which follows the convention of
namespace name suffixed with -aws
. See examples/awsproviderconfig.yaml
which installs a ProviderConfig for the namespace my-namespace
, resulting in the ProviderConfig name of my-namespace-aws
.
You will need to generate a kubernetes secret containing your credentials that the ProviderConfig references. This can be done by placing credentials in a file, i.e creds.conf
in the format:
[default]
aws_access_key_id = ACCESSKEYHERE
aws_secret_access_key = SECRETKEYHERE
And running the command kubectl create secret generic aws-creds -n my-namespace --from-file=creds=./creds.conf
Set environment variables to use with commands below
export AWS_REGION=region-here
export CLUSTER_NAME=clustername
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
export IAM_ROLE_NAME=name-to-give-iam-role
SERVICE_ACCOUNT_NAMESPACE=namespace-ie-crossplane-system-or-upbound-system
The first requirement is that you enable IAM roles for service accounts in your cluster by Creating an IAM OIDC provider for your cluster
eksctl utils associate-iam-oidc-provider --cluster "${CLUSTER_NAME}" --region "${AWS_REGION}" --approve
Our next step is to install Crossplane
kubectl create namespace $SERVICE_ACCOUNT_NAMESPACE
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm install crossplane --namespace $SERVICE_ACCOUNT_NAMESPACE crossplane-stable/crossplane
Then install the AWS provider referencing a ControllerConfig containing an annotation
cat > provider.yaml <<EOF
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
name: aws-config
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::$AWS_ACCOUNT_ID:role/$IAM_ROLE_NAME
spec:
podSecurityContext:
fsGroup: 2000
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws
spec:
package: crossplane/provider-aws:v0.22.0
controllerConfigRef:
name: aws-config
EOF
kubectl apply -f provider.yaml
Then we need to create an IAM role for our service account in our EKS cluster's AWS account
kubectl get serviceaccounts -n $SERVICE_ACCOUNT_NAMESPACE
SERVICE_ACCOUNT_NAME=provider-aws-[id from aws service account name here]
Option 1: Using EKSCTL
#eksctl create iamserviceaccount --cluster "${CLUSTER_NAME}" --region "${AWS_REGION}" --name="$SERVICE_ACCOUNT_NAME" --namespace="$SERVICE_ACCOUNT_NAMESPACE" --role-name="$IAM_ROLE_NAME" --role-only --attach-policy-arn="arn:aws:iam::aws:policy/AdministratorAccess" --approve --override-existing-serviceaccounts
Note: Using EKSCTL will use StringEquals in the Trust Relationship and use the exact name of the aws-provider service account. This would mean it needs editing with each upgrade of the AWS provider so it is preferrable to use the AWS CLI method below instead
Option 2: Using AWS CLI
read -r -d '' TRUST_RELATIONSHIP <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
"${OIDC_PROVIDER}:sub": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:provider-aws-*"
}
}
}
]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json
aws iam create-role --role-name "${IAM_ROLE_NAME}" --assume-role-policy-document file://trust.json --description "IAM role for provider-aws"
aws iam attach-role-policy --role-name "${IAM_ROLE_NAME}" --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
Then create a ProviderConfig to provision resources in our account
cat > provider-config.yaml <<EOF
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: aws-provider
spec:
assumeRoleARN: "arn:aws:iam::$AWS_ACCOUNT_ID:role/$IAM_ROLE_NAME"
credentials:
source: InjectedIdentity
EOF
kubectl apply -f provider-config.yaml
To deploy resources into another AWS account, we need to do the following:
Create a role that trusts the role from our EKS AWS account:
MY_OTHER_AWS_ACCOUNT_IAM_ROLE_NAME=provider-aws
MY_OTHER_AWS_ACCOUNT_ID=111111111
read -r -d '' TRUST_RELATIONSHIP_OTHER_AWS_ACCOUNT <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$AWS_ACCOUNT_ID:role/$IAM_ROLE_NAME"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
echo "${TRUST_RELATIONSHIP_OTHER_AWS_ACCOUNT}" > trust_other.json
aws iam create-role --role-name "${MY_OTHER_AWS_ACCOUNT_IAM_ROLE_NAME}" --assume-role-policy-document file://trust_other.json --description "IAM role for $IAM_ROLE_NAME in $AWS_ACCOUNT_ID"
Attach AdministratorAccess permissions to the role:
aws iam attach-role-policy --role-name "${MY_OTHER_AWS_ACCOUNT_IAM_ROLE_NAME}" --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
Create a new providerconfig set to assume that role. i.e:
cat > provider-config-other.yaml <<EOF
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: aws-provider-$MY_OTHER_AWS_ACCOUNT_ID
spec:
assumeRoleARN: "arn:aws:iam::$MY_OTHER_AWS_ACCOUNT_ID:role/$MY_OTHER_AWS_ACCOUNT_IAM_ROLE_NAME"
credentials:
source: InjectedIdentity
EOF
kubectl apply -f provider-config-other.yaml
When creating a resource for this capability, reference this providerconfig. i.e:
apiVersion: efs.aws.crossplane.io/v1alpha1
kind: FileSystem
metadata:
name: remote-efs-example
spec:
forProvider:
region: eu-west-1
providerConfigRef:
name: aws-provider-$MY_OTHER_AWS_ACCOUNT_ID
A single Kubernetes ProviderConfig will need to be installed in the Kubernetes cluster. See examples/kubernetesproviderconfig.yaml
We use a dynamic claim to provision an RDS Postgres database instance. Running kubectl apply -f examples/sqldatabases/claim-dynamic-nonproduction.yaml
will deploy a database with non-production defaults set.
To view the status of the install, execute kubectl get claim
or kubectl describe claim dynamic-claim-sql-nonprod-example
for more information. You should be able to view the raw RDS Instance using kubectl get rdsinstance
.
Provisioning of a database can take up to 10 minutes.
You can update your manifest to change properties of the database. Try adding storageSize: 30
to the parameters in the manifest and running an apply.
Simply run kubectl delete -f examples/sqldatabases/claim-dynamic-nonproduction.yaml
to delete the deployed database or run kubectl delete claim dynamic-claim-sql-nonprod-example
The folder structure of this repository is as follows:
- build
- Contains pipelines and scripts used to build and publish the configuration package
- examples
- Contains examples of ProviderConfig files and composite resources in the relevant subfolders
- package
- Contains the composite resource definition manifest files in the relevant versioned subfolders (I.e package\containerregistries\v1) with the package configuration at the top level
- package\aws
- Contains RBAC composites of AWS resources to work in multi-tenancy cluster. See the README in this folder
New resources should go into a versioned sub-folder inside packages and an example of the resource should go into a sub-folder of the same name under examples.
There is a script at build/build.sh
that you can provide parameter values for (docker repo name, tag) and execute in order to build and publish a version of this configuration package.
Release images are created by pipeline whenever a new release is created in Github
Extracting info about the resources and debugging information from the claim
> kubectl get claim -n my-namespace
awsbucketv1.storage.xplane.dfds.cloud/awsbucketdfds
> kubectl describe awsbucketv1.storage.xplane.dfds.cloud/awsbucketdfds -n my-namespace
Extract the name of the bucket from Status.CreatedResources section. Then get additional information from Status.AtProvider section like ARN
> kubectl describe bucket awsbucketdfds-fcvd6
For addtional debugging, extract composite resource that is refrenced by the claim in the Spec.ResourceRef.Kind and Spec.ResourceRef.Name so you can describe the composite resource that is created by the claim which controls to the managed resources
> kubectl describe XAWSBucketV1 awsbucketdfds-fcvd6