consul-template pkiCert does not return CA_Chain data causing issues during intermediary rotation

[astundzia]

Hashicorp issue

This is for Hashicorp ticket #153442. This fixes the issues we are seeing as described in the hashicorp ticket.

Consul Template version

latest main, based off of v0.39.0

Configuration

pid_file = "/var/run/vault-agent-pki.pid"

auto_auth {
  method "approle" {
    config = {
      role_id_file_path = "/etc/vault/role_id"
      secret_id_file_path = "/etc/vault/secret_id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/etc/vault-token"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address = "127.0.0.1:8100"
  tls_disable = true
}

vault {
  address = "https://192.168.0.106:8200"
}

template {
  destination = "/etc/certs/certs.pem"
  wait {
    min = "5s"
    max = "30s"
  }
  contents = <<EOH
          {{- with pkiCert "pki/astundzia/issuing_ca/issue/genctl-consoleproxy-tls" "common_name=console-proxy.vpc.cloud.ibm.local" "format=pem" -}}
          {{ .Cert }}
          {{ .Key  }}
          {{ .Key | writeToFile "/etc/certs/client.key" "" "" "0644" }}
          {{ .Cert | writeToFile "/etc/certs/client.crt" "" "" "0644" }}
          {{- end }}
          {{- with secret "pki/astundzia/issuing_ca/cert/ca_chain" -}}
          {{- .Data.ca_chain }}
          {{- .Data.ca_chain | writeToFile "/etc/certs/client.crt" "" "" "0644" "append,newline" }}
          {{ end }}
EOH
}

Command

Running vault as vault-agent

Expected behavior & bug

When using vault-agent which uses consul-template, I expect the entire CAChain (the data from the ca_chain API) to be returned when I use with pkiCert.

The data returned from with pkiCert does not contain the full CA Chain, only the issuer (via .CA). This means we need to use {{- with secret "pki/astundzia/issuing_ca/cert/ca_chain" -}} to fetch the remaining bits of the ca_chain.

The problem is that when we rotate the intermediary CA, it causes {{- with secret "pki/astundzia/issuing_ca/cert/ca_chain" -}} to return a updated CA Chain containing the new issuer/intermediary CA.

However, the data in .Certificate hasnt yet expired. At that moment, vault-agent puts forward a client.crt that contains the old leaf cert + new intermedaries. This is invalid/malformed, the leaf cert isnt issued from the intermediary and doesnt validate.

If we were able to get the entire CA_Chain via with pkiCert, we could do two things:

1. When the issuer rotates, we do not incorrectly rotate the intermedaries. This fixes our main issue, which causes malformed certificates to appear in our applications.
2. We can reduce the overall number of requests made to /issuing_ca/cert/ca_chain, since the data doesnt change until the certificate expires and rotates.

The Vault API returns this data, certificate, issuing_ca, ca_chain, key, yet with pkiCert only returns certificate , key and issuing_ca. The fix here is to parse the ca_chain and build up a CAChain object that we can use within our templates. This makes the with pkiCert match the functionality provided from the existing vault API.

Steps to reproduce

1. Stand up a vault server
2. Set up a intermediary CA, and a issuing CA
3. Use vault-agent to fetch the certificate using with pkiCert. Notice the ca_chain data is not contained. Fetch the rest of the ca_chain with {{- with secret "pki/..../issuing_ca/cert/ca_chain" -}}. The certificate generated at this point is correct.
4. Rotate the issuer/intermediary CA
5. Interrupt/restart vault-agent, causing the template to rerender. At this point notice with pkiCert returns the previously used certificate.
6. Fetch the rest of the ca_chain using {{- with secret "pki/..../issuing_ca/cert/ca_chain" -}}, notice this data returns a issuer/intermediary CA that has different Authority Key Identifier's, causing the certificate that is generated to fail validation.

Note, I have this fixed in my PR, please review the PR before closing this.

References

• Hashicorp ticket #153442.