Can't initialize a new Vault with vault-init container
pixelfields opened this issue · 3 comments
After deploying a new stateful set to kubernetes cluster, is expected that vault-init container will initialize the Vault. But I got an error here.
Looks like there is a wrong payload send to the vault API, just because return code from the API call is 400 Bad Request, and the Vault remains uninitialized.
vault-init log output
2019/01/30 12:27:05 Starting the vault-init service...
2019/01/30 12:27:05 Head http://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused
2019/01/30 12:27:15 Vault is not initialized. Initializing and unsealing...
2019/01/30 12:27:15 init: non 200 status code: 400
2019/01/30 12:27:16 storage: object doesn't exist
2019/01/30 12:27:16 Next check in 10s
2019/01/30 12:27:26 Vault is not initialized. Initializing and unsealing...
2019/01/30 12:27:26 init: non 200 status code: 400
2019/01/30 12:27:26 storage: object doesn't exist
2019/01/30 12:27:26 Next check in 10s
2019/01/30 12:27:36 Vault is not initialized. Initializing and unsealing...
2019/01/30 12:27:36 init: non 200 status code: 400
2019/01/30 12:27:36 storage: object doesn't exist
2019/01/30 12:27:36 Next check in 10s
2019/01/30 12:27:46 Vault is not initialized. Initializing and unsealing...
2019/01/30 12:27:46 init: non 200 status code: 400
2019/01/30 12:27:46 storage: object doesn't exist
2019/01/30 12:27:46 Next check in 10s
vault log output
==> Vault server configuration:
GCP KMS Crypto Key: vault-init
GCP KMS Key Ring: vault
GCP KMS Project: vault-sandbox-xxxx
GCP KMS Region: europe-west1
Seal Type: gcpckms
Api Address: https://xx.xx.xx.xx
Cgo: disabled
Cluster Address: https://10.160.1.9:8201
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Listener 2: tcp (addr: "10.160.1.9:8200", cluster address: "10.160.1.9:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: (not set)
Mlock: supported: true, enabled: true
Storage: gcs (HA available)
Version: Vault v1.0.1
Version Sha: [redacted]
2019-01-30T12:27:07.865Z [WARN] core: stored unseal key(s) supported but none found
==> Vault server started! Log data will stream in below:
2019-01-30T12:27:15.974Z [WARN] core: stored keys supported on init, forcing shares/threshold to 1
2019-01-30T12:27:26.225Z [WARN] core: stored keys supported on init, forcing shares/threshold to 1
On the other hand, I'm able to initialize the Vault with simple curl command from a side container running in the same pod, so I guess the issue is not networking here:
root@vault-0:/# curl -i --header "Content-Type: application/json" --request PUT --data '{"secret_shares": 1, "secret_threshold": 1, "stored_shares": 1, "recovery_shares": 1, "recovery_threshold": 1}' http://127.0.0.1:8200/v1/sys/init
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 30 Jan 2019 12:34:46 GMT
Content-Length: 228
{"keys":[],"keys_base64":[],"recovery_keys":["xxxx"],"recovery_keys_base64":["xxxx"],"root_token":"xxx"}
My stateful set configuration is pretty similar as yours, only one ubuntu container is added for the vault init testing:
containers:
- name: vault-init
image: sethvargo/vault-init
imagePullPolicy: IfNotPresent
env:
- name: GCS_BUCKET_NAME
value: "${gcs_bucket_name}"
- name: KMS_KEY_ID
value: "projects/${project}/locations/${kms_region}/keyRings/${kms_key_ring}/cryptoKeys/${kms_crypto_key}"
- name: VAULT_ADDR
value: "http://127.0.0.1:8200"
- name: VAULT_SECRET_SHARES
value: "1"
- name: VAULT_SECRET_THRESHOLD
value: "1"
- name: vault-test
image: ubuntu
imagePullPolicy: IfNotPresent
command: ["/bin/bash", "-ec", "while :; do echo '.'; sleep 5 ; done"]
- name: vault
image: "${vault_image}"
imagePullPolicy: IfNotPresent
args: ["server"]
securityContext:
capabilities:
add: ["IPC_LOCK"]
ports:
- containerPort: 8200
name: vault-port
protocol: TCP
- containerPort: 8201
name: cluster-port
protocol: TCP
resources:
requests:
cpu: "500m"
memory: "256Mi"
volumeMounts:
- mountPath: /etc/vault/tls
name: vault-tls
env:
- name: VAULT_ADDR
value: "http://127.0.0.1:8200"
- name: POD_IP_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: VAULT_LOCAL_CONFIG
value: |
api_addr = "https://${load_balancer_ip}"
cluster_addr = "https://$(POD_IP_ADDR):8201"
log_level = "warn"
ui = true
seal "gcpckms" {
project = "${project}"
region = "${kms_region}"
key_ring = "${kms_key_ring}"
crypto_key = "${kms_crypto_key}"
}
storage "gcs" {
bucket = "${gcs_bucket_name}"
ha_enabled = "true"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = "true"
}
listener "tcp" {
address = "$(POD_IP_ADDR):8200"
tls_cert_file = "/etc/vault/tls/vault.crt"
tls_key_file = "/etc/vault/tls/vault.key"
tls_disable_client_certs = true
}
What versions of those containers are you using? Please add the latest versions of the vault-init and vault container and change IfNotPresent to Always and see if the issue persists. I think you're getting an old version of vault-init.
Thank you! So looks like it helps.
2019/01/30 20:13:58 Starting the vault-init service...
2019/01/30 20:13:58 Head http://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused
2019/01/30 20:14:08 Vault is not initialized.
2019/01/30 20:14:08 Initializing...
2019/01/30 20:14:13 Encrypting unseal keys and the root token...
2019/01/30 20:14:13 Unseal keys written to gs://xxx-vault-storage/unseal-keys.json.enc
2019/01/30 20:14:13 Root token written to gs://xxx-vault-storage/root-token.enc
2019/01/30 20:14:13 Initialization complete.
2019/01/30 20:14:14 Next check in 10s
Just realized that there is a little bit mess with tags here:
https://hub.docker.com/r/sethvargo/vault-init/tags
The latest tag has the last update 6 months ago, but in fact the latest build is tagged with 1.0.0, 2 months ago, so you have to define image like image: sethvargo/vault-init:1.0.0, not image: sethvargo/vault-init:latest. Maybe it should be corrected somehow.
Sorry about that - fixed.