HashiCorp Vault Service Broker

This repository provides an implementation of the open service broker API for HashiCorp's Vault. The service broker connects to an existing Vault cluster and can be used by multiple tenants within Cloud Foundry.

Getting Started

The HashiCorp Vault Service Broker does not run a Vault server for you. There is an assumption that the Vault cluster is already setup and configured. This Vault server does not need to be running under Cloud Foundry, OpenShift, Kubernetes, etc, but it must be accessible from within those environments or wherever the broker is deployed.

These getting started instructions assume the Vault server's address and token are specified as the following environment variables:

$ export VAULT_ADDR="https://vault.company.internal/"
$ export VAULT_TOKEN="abcdef-134255-..."

Additionally the broker is configured to use basic authentication. The variables will be:

$ export AUTH_USERNAME="vault"
$ export AUTH_PASSWORD="broker-secret-password"

Deploying the Broker

The first step is deploying the broker. This broker can run anywhere including Cloud Foundry, Kubernetes, Heroku, HashiCorp Nomad, or your local laptop. This example shows running the broker under Cloud Foundry.

First, create a space in which to run the broker:

$ cf create-space vault-broker

Switch over to that space:

$ cf target -s vault-broker

Deploy the vault-broker by cloning this repository:

$ git clone https://github.com/hashicorp/vault-service-broker
$ cd cf-vault-broker

And push to Cloud Foundry:

$ cf push --random-route --no-start

The --random-route flag is optional, but it allows us to easily run more than one Vault broker if needed, instead of relying on "predictable names".
The --no-start flag is important because we have not supplied the required environment variables to our application yet. It will fail to start now.

To configure the broker, provide the following environment variables:

$ cf set-env vault-broker VAULT_ADDR "$VAULT_ADDR"
$ cf set-env vault-broker VAULT_TOKEN "$VAULT_TOKEN"
$ cf set-env vault-broker SECURITY_USER_NAME "$AUTH_USERNAME"
$ cf set-env vault-broker SECURITY_USER_PASSWORD "$AUTH_PASSWORD"

Now that it's configured, start the broker:

$ cf start vault-broker

To verify the HashiCorp Vault broker is running, execute:

$ cf apps

name           requested state   instances   memory   disk   urls
vault-broker   started           1/1         256M     512M   vault-broker-torpedolike-reexploration.local.pcfdev.io

Grab the URL and save it in a variable or copy it to your clipboard - we will need this later.

export BROKER_URL=$(cf app vault-broker | grep -E -w 'urls:|routes:' | awk '{print $2}')

NOTE: Different versions of Cloud Foundry display this information differently. If the result of the pipeline above is empty, try running cf app vault-broker and look at the output. It is possible that the key has changed again and you'll need to grep for that instead of urls: or routes:.

Again, there is no requirement that our broker run under Cloud Foundry - this could be a URL pointing to any service that hosts the broker, which is just a Golang HTTP server.

To verify the broker is working as expected, query its catalog:

$ curl -s "${AUTH_USERNAME}:${AUTH_PASSWORD}@${BROKER_URL}/v2/catalog"

The result will be JSON that includes the list of plans for the broker:

{
  "services": [
    {
      "id": "0654695e-0760-a1d4-1cad-5dd87b75ed99",
      "name": "hashicorp-vault",
      "description": "HashiCorp Vault Service Broker",
      "bindable": true,
      "tags": [""],
      "plan_updateable": false,
      "plans": [
        {
          "id": "0654695e-0760-a1d4-1cad-5dd87b75ed99.shared",
          "name": "shared",
          "description": "Secure access to Vault's storage and transit backends",
          "free": true
        }
      ]
    }
  ]
}

The HashiCorp Vault Service Broker is now running under Cloud Foundry and ready to receive requests.

Register the Vault Broker

Before it can bind to services, the broker must be registered with Cloud Foundry. Remember that there is no requirement that the broker be running under Cloud Foundry, so we will need to provide the broker registration service the BROKER_URL from above.

To register the broker, an application developer first creates a new space where they will request the broker access:

$ cf create-space example
$ cf target -s example

Next, register the broker in this space:

$ cf create-service-broker vault-broker "${AUTH_USERNAME}" "${AUTH_PASSWORD}" "https://${BROKER_URL}" --space-scoped

Note: This will allow developers in the current space to access the broker. If other developers in another space want to access it, they will need to create their own instance. This is a good starting workflow, but more complex setups should investigate allowing access to plans globally or per-organization.

Note: Here we are scoping the broker to a space. In larger installations, it may be desirable to run a centralized instance of the broker that is accessible to all spaces in the organization. For more information on this deployment pattern, please see the notes in the standard broker section.

To verify the command worked, query the marketplace. You should see the Vault broker with a plan of 'default' in addition to any other services you may have access to:

$ cf marketplace
service           plans             description
hashicorp-vault   shared            HashiCorp Vault Service Broker
# ...

Create a service instance and bind an app

After registering the service in the marketplace, it is now possible to create a service instance and bind to it.

First, create a service instance using the default plan. For this example we will name the service instance 'my-vault':

$ cf create-service hashicorp-vault shared my-vault

With a service instance in place, you are ready to bind an app. Suppose we have an app called 'my-app'.

$ cf bind-service my-app my-vault

You will need to restage the app to pick up the environment changes.

$ cf restage my-app

When the app starts back up, its VCAP_SERVICES environment variable will contain an entry for the my-vault service. You can confirm this by checking the app's environment variables:

$ cf env my-app

The VCAP_SERVICES environment variable will have a section similar to the following:

{
  "hashicorp-vault": [
    {
      "name": "my-vault",
      "plan": "shared",
      "label": "hashicorp-vault",
      "credentials": {
        "address": "https://vault.company.internal:8200/",
        "auth": {
          "accessor": "171a13e0-cd51-b4ae-4b29-81321600ceb2",
          "token": "5142df0f-cc39-a899-5e74-dd357c5e1152"
        },
        "backends": {
          "generic": "cf/8bcae1a7-e1b8-4c9a-a0f4-ee1e538ebfbe/secret",
          "transit": "cf/8bcae1a7-e1b8-4c9a-a0f4-ee1e538ebfbe/transit"
        },
        "backends_shared": {
          "organization": "cf/bc58452e-b7d1-46fe-bf74-5335e70708ad/secret",
          "space": "cf/072bfc67-1beb-4dd1-beb7-f7b05f00fc1a/secret"
        }
      }
    }
  ]
}

The keys of the credentials section are as follows:

address - address to the Vault server to make requests against
auth.accessor - token accessor which can be used for logging
auth.token - token to supply with requests to Vault
backends.generic - namespace in Vault where this token has full CRUD access to the static secret storage ("generic") backend
backends.transit - namespace in Vault where this token has full access to the transit ("encryption as a service") backend
backends_shared.organization - namespace in Vault where this token has read-only access to organization-wide data; all instances have read-only access to this path, so it can be used to share information across the organization.
backends_shared.space - namespace in Vault where this token has read-write access to space-wide data; all instances have read-write access to this path, so it can be used to share information across the space.

Internals

Architecture and Assumptions

To ease in setup and administration, the HashiCorp Vault Service Broker makes a few assumptions about the Vault setup including:

The Vault server is already running and is accessible by the broker.
The Vault server may be used by other applications (it is not exclusively tied to Cloud Foundry).
All instances of an application will share a token. This goes against the recommended Vault usage, but this is a limitation of the Cloud Foundry service broker model.
Any Vault operations performed outside of Cloud Foundry will require users to rebind their instances.

When a new service instance is provisioned using the broker, it will mount the following paths:

Mount the generic backend at /cf/<organization_id>/secret/
Mount the generic backend at /cf/<space_id>/secret/
Mount the generic backend at /cf/<instance_id>/secret/
Mount the transit backend at /cf/<instance_id>/transit/

The mount operation is idempotent, so service instances in the same organization or space will not re-create the mount. These mount points will be returned to the application in the secret data, so there is no need to "guess" or interpolate these strings.

The read-only organization mount allows for sharing secrets organization wide, and the read-write mount permits sharing secrets between applications in the same Cloud Foundry Space. The transit backend is mounted to provide "encryption-as-a-service" on a per-application level.

After mounting is complete, the broker generates a custom policy specific for this instance which grants the following:

Read-only access to "cf/<organization_id>/*"
Read-write access to "cf/<space_id>/*"
Full access to "cf/<instance_id>/*"

This policy is named "cf-<instance_id>" and can be further customized outside of Cloud Foundry by a Vault administrator.

Next the broker creates a new token role. This role creates a periodic token with the above policy attached. This does not create the token yet, just the role for generating the token.

When a service instance is bound to an application, the broker performs the following operations:

Create a new token against the previous "cf-<instance_id>" role.
Start a background process to renew this token
Generate and returning the binding credentials (see above for the schema)

It is important to note that all instances of a Cloud Foundry application will share the same vault_token. This is not the recommended pattern for using Vault, but it is an existing limitation of the service broker model.

Unbinding and Deleting

When unbinding from a service or deleting the service broker entirely, the broker deletes an instance-specific data. For safety, the broker does not delete an space or organization-specific mounts, even if there are no remaining service brokers using it.

Broker Vault Token Permissions

The Cloud Foundry Vault Broker requires a VAULT_TOKEN to operate. This token should have elevated permissions in Vault, as it will be responsible for generating new mounts and committing data to its internal data structure.

Here is a sample policy to assign to this token:

# Manage internal state under "/broker", but since this token is going to
# generate children, it needs full management of the "/cf/*" space
path "/cf/" {
  capabilities = ["list"]
}

path "/cf/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# List all mounts
path "sys/mounts" {
  capabilities = ["read", "list"]
}

# Create mounts under the "/cf/" prefix
path "sys/mounts/cf/*" {
  capabilities = ["create", "update", "delete"]
}

# Create policies with the "cf-*" prefix
path "sys/policy/cf-*" {
  capabilities = ["create", "update", "delete"]
}

# Create token role
path "/auth/token/roles/cf-*" {
  capabilities = ["create", "update", "delete"]
}

# Create tokens from role
path "/auth/token/create/cf-*" {
  capabilities = ["create", "update"]
}

# Revoke tokens by accessor
path "/auth/token/revoke-accessor" {
  capabilities = ["create", "update"]
}

Additionally, this token should be a periodic token. The Cloud Foundry Vault Broker will renew this periodic token automatically.

Authenticate as a root token or user with sudo privilege in Vault (this is required to create a periodic token):

$ vault auth <token>

Create the policy specific for the broker:

$ vault write sys/policy/cf-broker rules=@cf-broker.hcl

Create a periodic token

$ vault token-create -period="30m" -orphan -policy=cf-broker

Grab the value for "token" and store it somewhere safe for now - you will need this when configuring the HashiCorp Vault Service Broker.

Service Broker Configuration

The service broker is designed to be configured using environment variables. It currently recognizes the following.

SERVICE_DESCRIPTION (default: "HashiCorp Vault Service Broker") - description of the service to show in the marketplace
SERVICE_ID (default: "0654695e-0760-a1d4-1cad-5dd87b75ed99") - UUID of the broker
SERVICE_NAME (default: "hashicorp-vault") - name of the service to show in the marketplace
SERVICE_TAGS (default: none) - comma-separated list of tags for the service
PLAN_NAME (default: "shared") - the name of the plan in the marketplace
PLAN_DESCRIPTION (default: "Secure access to Vault's storage and transit backends") - description of the plan in the marketplace
PORT (default: "8000") - port to bind and listen on as the server (broker)
VAULT_ADDR (default: "https://127.0.0.1:8200") - address to the Vault server
VAULT_ADVERTISE_ADDR (default: "$VAULT_ADDR") - address to advertise to clients as Vault's address. This defaults to the value supplied for VAULT_ADDR, but can be overridden. This is most useful when the broker communicates to Vault on a local subnet, but clients communicate through a public subnet.
VAULT_RENEW (default: true) - enable renewal of the token provided to Vault. The token given to Vault is assumed to be a periodic token, and the broker will automatically renew it to prevent it from expiring. If an out-of-band process is managing the renewal, disable this by setting it to "false".
VAULT_TOKEN (default: none) - token to authenticate the broker to Vault. This token should have permission to mount and unmount backends, read, list, and delete paths, and create tokens with role permissions. Please see the Vault Token Permissions section for more information on the requirements for this token.
SECURITY_USER_NAME - (default: none) - username for basic auth
SECURITY_USER_PASSWORD - (default: none) - password for basic auth

Granting Access to Other Paths

The service broker has an opinionated setup of policies and mounts to provide a simplified user experience for getting started which matches the organizational model of Cloud Foundry. However an application may require access to existing data or backends.

Once the broker creates the policy for a service id cf-<instance_id> that policy may be modified by a user with permissions in Vault to add additional capabilities. The default policy can be discovered by reading it:

$ vault read -field=rules sys/policy/cf-<instance_id>
# ...

Append any additional rules to the end.

Global Standard Broker

The default configuration and examples above use a "space scoped" broker. For larger installations, it may be desirable to run a centralized instance of the broker that is accessible to all spaces and applications in the system. This is generally called a "standard broker" in Cloud Foundry terminology. To deploy a global broker, an admin must add, publish, and permit access as follows.

First, create the broker. Note the lack of --space-scoped as compared to the previous commands.

$ cf create-service-broker vault-broker "${AUTH_USERNAME}" "${AUTH_PASSWORD}" "https://${BROKER_URL}"

To verify the broker was created:

$ cf service-access
# ...

broker: vault-broker
   service           plan     access   orgs
   hashicorp-vault   shared   none

Notice the access is listed as "none". This broker will not appear in the marketplace until activated by an admin. To activate, run:

$ cf enable-service-access hashicorp-vault

It is possible to restrict the access to particular organizations or plans.

$ cf enable-service-access hashicorp-vault -o my-org

Verify the plan is enabled:

$ cf service-access
# ...

broker: vault-broker
   service           plan     access   orgs
   hashicorp-vault   shared   all

And check in the marketplace

$ cf marketplace
service           plans             description
hashicorp-vault   shared            HashiCorp Vault Service Broker
# ...

Now all spaces have access to this centralized broker!

Contributing

Clone the repo
Make changes on a branch
Test changes
Submit a Pull Request to GitHub

v6/vault-service-broker