/secret-agent

Generate random Kubernetes secrets and optionally store them in a Cloud Secret Manager

Primary LanguageGoApache License 2.0Apache-2.0

secret-agent - Secret generator and manager for k8s

PkgGoDev Go Report Card License

Secret agent logo a go gopher with sunglasses and hawaiian style shirt

The secret-agent is a Kubernetes operator that generates the secrets required by the ForgeRock® Identity Platform. The secrets are stored in-cluster as Kubernetes secrets and can also be stored in a cloud secret manager.

Installation

To install the latest secret-agent release in a Kubernetes environment, run:

kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/latest/download/secret-agent.yaml

Specific versions of the operator can be installed by running:

SA_VERSION=v0.1.0 kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/download/${SA_VERSION}/secret-agent.yaml

Configuration

Once the operator is installed, new secrets can be generated by providing a secret agent configuration (SAC) object. The SAC is a custom kubernetes object monitored by the secret-agent operator. All the secrets’ specifications are defined through the SAC.

For more information on how to create a SAC, see the Secret Agent Configuration Schema and/or the Examples sections.

Once the SAC file has been created, it can be pushed to the cluster as with any other resource.

For example:

kubectl create -f config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml

It is important to note that the Kubernetes secrets produced by the secret-agent will be placed in the same namespace as the SAC. If similar secrets are desired in multiple namespaces, one SAC would be required per namespace.

Usage

Enabling Cloud Backup

The secret-agent can be configured to back up all the generated secrets in a cloud provider's secret manager solution. When this feature is enabled, secrets stored in the secret managers are considered the source of truth.

If a cloud provider has been configured, the operator will attempt to load the secret data from that cloud provider. If the secret is found in the cloud provider's secret manager, the operator will use the found data as the Kubernetes secret data. The operator will only generate new secrets if no secret data is found in the cloud provider.

The secret-agent supports the following cloud providers:

  • Google Secret Manager
  • AWS Secrets Manager
  • Azure Key Vault

It is possible to run the secret-agent without setting up a cloud provider. This is useful when debugging or testing applications. To disable cloud provider support, set spec.appConfig.secretsManager to “none”. This is only possible if spec.appConfig.createKubernetesObjects is set to true.

In addition, it is possible to configure the secret-agent to store secrets in the secret manager without creating local Kubernetes secrets. This is useful if your applications can access the cloud secret manager directly and the secret-agent is only used to generate such secrets. To achieve this, set spec.appConfig.createKubernetesObjects to false. Do note that spec.appConfig.secretsManager cannot be set to "none".

In order to fetch and store secrets in the AWS Secrets Manager, the user must provide credentials with the necessary permissions.

Set up Cloud Backup With AWS Secret Manager

The secret-agent expects credentials to be discoverable via standard AWS mechanisms. These credentials can be provided in a number of ways, for example:

  • Environment Variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
  • Shared Credentials file: ~/.aws/credentials
  • Shared Configuration file: (~/.aws/config
  • EC2 Instance Metadata (preferred): Obtains credentials from 169.254.169.254

Refer to AWS documentation for instructions on how to obtain credentials and grant necessary permissions to access the AWS Secrets Manager. The secret-agent needs to access read/write secrets. This can be achieved by allowing access to the arn:aws:iam::aws:policy/SecretsManagerReadWrite AWS managed policy.

Even though the recommended way to obtain credentials is to use the EC2 Instance Metadata service, it is possible to provide custom credentials via a Kubernetes secret. The secret reference is provided in the SAC in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the user is expected to publish the cloud credentials' secret in the same namespace as the operator. This target namespace can be changed by changing the runtime argument --cloud-secrets-namespace=[NS_NAME] located in the operator's manifest. If this argument is omitted completely, the namespace will default to the namespace of each SAC.

Once these credentials are posted to a Kubernetes secret, the next step is to configure the AWS Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets AWS Secret Manager in us-east-1:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: AWS
    awsRegion: us-east-1

optional for AWS: The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  AWS_ACCESS_KEY_ID: QU....[base64 encoded key].....GY=
  AWS_SECRET_ACCESS_KEY: cRB.....[base64 encoded access key].......BB==

Note: The maximum secret size supported by AWS is 65Kb For more information, see AWS documentation.

Set up Cloud Backup With GCP Secret Manager

The secret-agent expects credentials to be discoverable via standard GCP mechanisms. These credentials can be provided in a number of ways, including:

Please refer to the GCP Documentation for instructions on how to create a service account with the necessary permissions to access the GCP Secrets Manager. The secret-agent needs access to read/write secrets. This can be achieved by assigning the Secret Manager Admin role to the service account provided.

Workload identity

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE. For more information on how to enable workload identity see GCP Documentation.

In general, the user creates a Google Cloud service account with the proper role attached and enables workload identity in their GKE cluster. The Kubernetes services account is already created for you during the secret-agent deployment.

Run the following commands to enable workload identity for the secret-agent deployment:

PROJECTID=myproject #GCP project ID
GSA_NAME=mygcpserviceaccount #GCP service account name
# Create the GCP IAM policy binding
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${PROJECTID}.svc.id.goog[secret-agent-system/secret-agent-manager-service-account]" ${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com
# Annotate the Kubernetes service account
kubectl -n secret-agent-system annotate serviceaccounts secret-agent-manager-service-account iam.gke.io/gcp-service-account=${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com

Note: in order to use workload identity, no spec.appConfig.credentialsSecretName should be provided. If credentials are provided, secret-agent will use the provided credentials instead.

Credentials file

The credentials are provided to the operator using a kubernetes secret under the GOOGLE_CREDENTIALS_JSON data key. The name of this secret is provided in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the operator will look for a secret with the provided name in the operator's own namespace. The user can specify a different namespace by setting the argument --cloud-secrets-namespace=[NS_NAME]. If this argument is omitted, the operator's default behavior is to fetch the credentials from the same namespace as the SAC.

The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  GOOGLE_CREDENTIALS_JSON: .....[base64 encoded service account json].....
Configure the GCP Secret Manager

Once the necessary credentials are provided to secret-agent using workload identity or a credentials file, the next step is to configure the GCP Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets GCP Secret Manager for the example-project-id project:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** skip if using workload identity **]
    secretsManager: GCP
    gcpProjectID: example-project-id

Set up Cloud Backup With Azure Key Vault

note: Azure's API response time on Key Vault is long and will delay the creation of secrets. It might be beneficial to deploy a SAC before long before deploying an application if use Azure Key Vault

The secret-agent uses credentials which are available using two different methods: Azure Managed Identities (recommended for Azure deployemnts) or explicit credentials. Explicit credentials are configured in a secret referenced in the SAC spec spec.appConfig.credentialsSecretName. Example Azure Configuration for a SAC:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: Azure
    azureVaultName: secret-agent-vault

If no secret is provided in credentialsSecretName, the operator's Azure client will attempt to authenticate using managed identities. For more information, see Azure's documentation. This is the recommended configuration for deployments in Azure's AKS.

Otherwise, the credentials may be explicitly set in the credentialsSecretName secret. The service principle associated with the keys will need the role Key Vault Secrets Officer when using an RBAC policy based Key Vault.

Example credentials secret:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
data:
  # AZURE_TENANT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_SECRET: # OPTIONAL: Update if using Azure Key Vault

Note: The maximum secret size supported by Azure is 25Kb For more information, see Azure documentation.

Importing your own secrets

In addition to generating secrets, the secret-agent allow users to import their own secrets. This is especially useful for things like certificates and certificate authorities that can be referenced by other secrets in the SAC. For example, a user can import their organization's CA and use it to sign certificates generated by the secret-agent.

All that is required is to provide a Kubernetes secret with the same name and same key names as described in the SAC. It is important to note that if the cloud backup feature is enabled, the secret to be imported must be provided using the cloud manager's secret manager. The secret-agent will ignore local secrets if cloud backup is enabled.

Naming Convention For Cloud Backups

There is a naming convention used by secret-agent to store and read secrets from the cloud secret managers. In general, the names follow the format:

$prefix-$namespace-$secretName-$keyName [If secretsManagerPrefix is provided]
$namespace-$secretName-$keyName [If no prefix is provided]

Due to cloud provider limitations, all /, . and _ characters in secret names are replaced by - when accessing the cloud secret managers.

For example, consider the following secret agent configuration:

---
apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: forgerock-sac
  namespace: dev
spec:
  appConfig:
    secretsManagerPrefix: "devCluster"
    awsRegion: us-east-1
    secretsManager: AWS
  secrets:
  - name: ds-passwords
    keys:
      - name: dirmanager.pw
        type: password

The secret generated by the this SAC would be stored as devCluster-dev-ds-passwords-dirmanager-pw in the AWS Secret Manager. The same name would apply to other cloud providers.

Some key types require more than one secret to be backed up. Such key types require a public and private components stored separately. This is the case for key types: ca, keypair, ssh and keytool. These secrets use the same naming convention described above and append a suffix to the main name as constructed previously.

Key Type Name
ca $NAME-pem
$NAME-private-pem
keypair $NAME-pem
$NAME-private-pem
ssh $NAME
$NAME-pub
keytool $NAME
$NAME-storepass
$NAME-keypass

In the preceding table, $NAME follows the convention at the top of this section.

Examples

We provide a sample SAC that exercises all features of the secret-agent. See the samples folder.

Secret Agent Configuration Schema

The following tables list the configurable parameters of the secret agent configuration (SAC) and their default values.

App Config

Parameter Description Default
spec.appConfig.createKubernetesObjects Create Kubernetes secrets for each generated secret. Can't be set to false if spec.appConfig.secretsManager is set to "none" true
spec.appConfig.secretTimeout Set the timeout in seconds for generating each individual secret 40
spec.appConfig.secretsManager Select the cloud provider to target. If "none", secrets will not be backed up in any cloud secret manager. Can't be set to "none" if spec.appConfig.createKubernetesObjects is false none
spec.appConfig.secretsManagerPrefix Prefix added to the name of the secrets stored in the cloud secret manager. ""
spec.appConfig.credentialsSecretName Name of the Kubernetes secret containing the credentials to access the cloud provider. ""
spec.appConfig.gcpProjectID When using GCP as the secret mgr, specify the project ID. ""
spec.appConfig.awsRegion When using AWS as the secret mgr, specify the region. ""
spec.appConfig.azureVaultName When using Azure as the secret mgr, specify the vault name. ""
spec.secrets List of Kubernetes secrets to create. See Secret Config. []

Secret Config

Parameter Description Default
name Name of the Kubernetes secret to generate. ""
keys List of the specs of each key in the Kubernetes secret. See Key Config. []

Key Config

Parameter Description Default
name Name of the key in the Kubernetes secret. ""
type Type of key to generate. Available values: ca;literal;password;ssh;keyPair;truststore;keytool. ""
spec.value Used when key type is literal. Specify the value of the password. ""
spec.isBase64 Used when key type is literal. If true, interpret the value to be used for the secret as a base64 encoded string. false
spec.length Used when key type is password. Specify the length of the password to generate. 32
spec.useBinaryCharacters Used when key type is password. If true, use the full byte range for each character, not just the ASCII range. false
spec.algorithm Used when key type is keyPair. Specify the algorithm used to generate the keyPair. ""
spec.sans Used when key type is keyPair. Specify alternate DNS names used by the certificate. ""
spec.selfSigned Used when key type is keyPair. If true, generate a self signed certificate. false
spec.signedWithPath Used when key type is keyPair. Specify the path to the CA in the SAC secretname/keyname. ""
spec.duration Used when key type is is ca or keyPair. Specify the valid duration of the certificate. If a negative duration is specified (-72h) the certificate is generated with an expiry date in the past. 3650d
spec.distinguishedName.country Used when key type is ca or keyPair. Specify the country name. ""
spec.distinguishedName.organization Used when key type is ca or keyPair. Specify the organization name. ""
spec.distinguishedName.organizationUnit Used when key type is ca or keyPair. Specify the organizationUnit name. ""
spec.distinguishedName.locality Used when key type is ca or keyPair. Specify the locality name. ""
spec.distinguishedName.province Used when key type is ca or keyPair. Specify the province name. ""
spec.distinguishedName.streetAddress Used when key type is ca or keyPair. Specify the streetAddress. ""
spec.distinguishedName.postalCode Used when key type is ca or keyPair. Specify the postalCode. ""
spec.distinguishedName.serialNumber Used when key type is ca or keyPair. Specify the serialNumber. ""
spec.distinguishedName.commonName Used when key type is ca or keyPair. Specify the commonName for the certificate. ""
spec.truststoreImportPaths Used when key type is truststore. List of paths of certificates in the form secretname/keyname that will be imported into the truststore. ""
spec.storeType Used when key type is keytool. Specify the keystore type. Available values: pkcs12;jceks;jks. ""
spec.storePassPath Used when key type is keytool. Specify the path to the secret in the SAC to use as the keystore password in the form secretname/keyname. ""
spec.keyPassPath Used when key type is keytool. Specify the path to the secret in the SAC to use as the key password in the form secretname/keyname. ""
spec.keytoolAliases Used when key type is keytool. Specify the aliases to include in the keystore. See Keytool Aliases Config. []

Keytool Aliases Config

Parameter Description Default
name Name of the alias in the keystore. ""
cmd keytool command used to create the alias in the keystore. Supported cmds: genkeypair;genseckey;importcert;importpassword;importkeystore. ""
args Args passed to the keytool command provided in cmd. ""
sourcePath Used when the keystore cmd is importcert, importpassword or importkeystore. Specify the path to the secret in the SAC to import into the alias in the form secretname/keyname. Note: importcert only imports the public key. importkeystore must be used to import a key pair. ""
isKeyPair If importing a keypair using importkeystore, must be set to true. false

Runtime Arguments

Argument Description Default
--metrics-addr The address the metric endpoint binds to. Set to 0 to disable metrics. ":8080"
--health-addr The address the healthz/readyz endpoint binds to. ":8081"
--enable-leader-election Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. "false"
--cert-dir Directory where to store/read the webhook certs. "/tmp/k8s-webhook-server/serving-certs"
--cloud-secrets-namespace Namespace where the cloud credentials secrets are located. Defaults to the SAC namespace. SAC's metadata.namespace
--debug Enable debug logs. "false"

Running Tests

Tests can be run using make tests. Some of the tests exercise parts of the code that uses keytool, and kubebuildertools. Those must be installed locally in order to run the tests. Another option is to use docker:

  • Ensure you're Kubernetes context is pointing to a test cluster, such as minikube, then
    • docker build -t gcr.io/forgerock-io/secret-agent-testing:latest -f --target=tester .
    • docker run -it --rm -v ${PWD}:/root/go/src/github.com/ForgeRock/secret-agent gcr.io/forgerock-io/secret-agent-testing:latest
    • go test ./...