can't see secrets?
Justin-DynamicD opened this issue · 11 comments
Trying the new 2.3.0 as a container, and connecting to vault 0.8.3 (also in a container). Both containers are mounted as host, and vault itself is using a consul backend. Now in the vault.json, I am using a non-default service name, as the default name "vault" is taken already in consul, so we had to grab something different and use "site-vault".
Everything seems to work fine ... except actually seeing secrets. "Secret backends" sits completely blank, even when using the root token. I think I see some open cases about "-" causing issues, so I think ti may be that, but I figured I'd open a case in case it's unique.
I should add that yes: I can vault write and vault read just fine.
Hey @Justin-DynamicD, I wonder if what @tallpauley merged in recently was related to this #189 ? Can you try running :latest and see if that fixes it? If not, does :2.2.0 work?
When you say service name, do you mean the mount path?
Nevermind, you are using the consul namespace "site-vault" to register in consul as the root namespace, interesting. Vault-UI doesn't actually interface with how Vault is configured and makes calls directly to Vault's API, so if its available via the Vault CLI, it should be available in Vault-UI as well. Hmm 🤔 . Yea, could you try :2.2.0 and see if this is just regression with the new release?
It's certainly one that doesn't make sense to me either. the non-standard vault name is the result of the dev team already having a container named "vault" floating around, so we had to come-up with something unique. It's been good so far, though honestly the entire stack is in it's infancy so that could change.
I tried both latest and 2.2.0, no difference. Auth backends work fine (can browse, add, remove). Haven't tried changing the vault version, but I guess that's the only variable left?
If you're using 0.8.3, then it may not matter as I don't believe there are any rest API changes since 0.6.x. It would be helpful to know if you're getting any errors in your browser's console logs OR if there are any failed network calls. I believe all modern browsers have these tools, but I know for sure that Chrome does.
Are you able to pull up any of these? Otherwise, I could try setting up a local vault with a non-standard vault service name through a Consul backend hopefully tonight, but if not, then later this week.
So lets try to get a little more clarity on other variables:
-
We run 3 "hashi-stack" servers (for lack of a better term). Each server has docker and runs 4 containers all bound net=host: consul, vault, nomad, and vault-ui.
-
Because we needed 3 consul boxes for raft quorum, we just added vault to use the consul backend and setup multiple vault boxes in ha. (1 active and 2 standby, mirroring the consul deployment)
-
Each vault-ui container simply does a http://localhost:8200 lookup to find it's respective vault as the relationship is 1:1. In the future I may switch this out but for now this model works well.
OK so debug time:
I do see an error when I log into a server using the developer tools for chrome:
GET http://:8000/v1/sys/health?vaultaddr=http:%2F%2Flocalhost:8200 429 (too many requests) web-bundle.js:20
This is consistent across all nodes.
AHA!
You are getting a load-balancing error. Vault uses error code 429 differently from most other server return code implementations. Vault uses 429 as:
429 - Default return code for health status of standby nodes, indicating a warning.
https://www.vaultproject.io/api/index.html#429
Interesting problem to solve, and there are many ways to solve it, but you need to control the Vault server that Vault-UI hits. It needs to hit the leader. Now, there are configuration values in Vault that set it up such that requests sent to standby nodes are forwarded over to the leader. This configuration needs to be put in place. Do you have this already?
You can read more about it here, if you don't have it setup already: https://www.vaultproject.io/docs/concepts/ha.html
@Justin-DynamicD, can you confirm or deny if this is the issue you are seeing?
Just got in the office this morning, I'll go through both the immediate tweak (make sure vault only looks at the active server) as well as read a bit on the ha flags on vault itself to make sure everything is "proper". Will edit this post with results.
OK so the error message changed a bit ... so I think I'm inching closer.
Changes made:
- In the vault container, I made sure the env variables "VAULT_ADDR" and "VAULT_REDIRECT_ADDR" existed as per documentation linked.
- In Vault-UI, I set the vault backend to "active.lv-vault.service.consul". Thanks to consul this always returns the active server.
When logging into vault-ui now, the "connect to" in the status bar up-top changes. If vault is active on the same host as vault-ui, it will return "http://localhost:8200". if the two are on separate boxes it returns "http://active.lv-vault.service.consul:8200". I'm assuming this is fine behavior.
I don't get any error messages anymore in the GUI (with one exception not relevant here)... however I still cannot expand the secrets backend :(. It just quietly returns nothing.
@Justin-DynamicD, this issue may have to do with the change to the generic into kv. Vault-UI had generic hard coded, so if running Vault 0.8.3 and the backend was kv by default, you would have run into issues as I verified myself.
Give djenriquez/vault-ui:latest a shot, that has the fix merged in. Let me know what you see.
Hey @Justin-DynamicD, I can confirm that, with 0.8.3 of Vault, secret backends will use kv as the backend by default, which did not work until the fix in #197. You should be good to go with the latest version. Please verify and close the issue if all is good.
Thank you very much for the followup. yes, I can confirm switching to latest (and flushing out the old images from disk) has indeed solved my problem. Thanks everyone for all the help!!!