sethvargo/vault-secrets-gen

Code: 500 on generating password via vault-secrets-gen

typlo-s opened this issue · 6 comments

typlo-s commented

I am getting code 500 on generating passwords via the latest vault (go-plugin).

root@uat-tekesvault001:/etc/vault.d/plugins# vault write gen/password length=36 symbols=0
Error writing data to gen/password: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/gen/password
Code: 500. Errors:

  • 1 error occurred:
    • internal error

Note: The vault server is using a self-signed certificate.

Logs:

Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:17:30.431+0530 [ERROR] secrets.secrets-gen.secrets-gen_29db1cb9.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr fo>
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:17:30.432+0530 [ERROR] rollback: error rolling back: path=gen/
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: error=
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: | Unrecognized remote plugin message:
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: |
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: | This usually means that the plugin is either invalid or simply
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]: | needs to be recompiled to support the latest protocol.
Oct 20 12:17:30 uat-tekesvault001 vault[3421430]:
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:18:30.362+0530 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h2027eb13bfdb55aeeda926e639259645e563d1c3b5404439bbc118ec68213>
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:18:30.434+0530 [ERROR] secrets.secrets-gen.secrets-gen_29db1cb9.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr fo>
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:18:30.435+0530 [ERROR] rollback: error rolling back: path=gen/
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: error=
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: | Unrecognized remote plugin message:
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: |
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: | This usually means that the plugin is either invalid or simply
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]: | needs to be recompiled to support the latest protocol.
Oct 20 12:18:30 uat-tekesvault001 vault[3421430]:
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: 2021-10-20T12:18:57.568+0530 [ERROR] secrets.secrets-gen.secrets-gen_29db1cb9.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr fo>
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: 2021-10-20T12:18:57.569+0530 [ERROR] core: failed to run existence check:
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: error=
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: | Unrecognized remote plugin message:
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: |
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: | This usually means that the plugin is either invalid or simply
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]: | needs to be recompiled to support the latest protocol.
Oct 20 12:18:57 uat-tekesvault001 vault[3421430]:
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:19:30.367+0530 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/hfad1ca76ee4e3f52bd13651c89e7f509a2c43fe4f0b082a83623d1e2ac44d>
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:19:30.429+0530 [ERROR] secrets.secrets-gen.secrets-gen_29db1cb9.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr fo>
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:19:30.430+0530 [ERROR] rollback: error rolling back: path=gen/
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: error=
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: | Unrecognized remote plugin message:
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: |
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: | This usually means that the plugin is either invalid or simply
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]: | needs to be recompiled to support the latest protocol.
Oct 20 12:19:30 uat-tekesvault001 vault[3421430]:
Oct 20 12:19:57 uat-tekesvault001 vault[3421430]: 2021-10-20T12:19:57.502+0530 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/hcb309e1c8cd4e6e398796b6abe866762de4a786ac94d16ee3de98c712022d>
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:20:30.361+0530 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/hd65448c87790003e4d51ac0ec5165f856d05361d8cc4c383b55468be10211>
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:20:30.426+0530 [ERROR] secrets.secrets-gen.secrets-gen_29db1cb9.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr fo>
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: 2021-10-20T12:20:30.427+0530 [ERROR] rollback: error rolling back: path=gen/
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: error=
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: | Unrecognized remote plugin message:
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: |
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: | This usually means that the plugin is either invalid or simply
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]: | needs to be recompiled to support the latest protocol.
Oct 20 12:20:30 uat-tekesvault001 vault[3421430]:

sethvargo commented

Looks like a duplicate of #30. Can you try the steps there?

typlo-s commented

Tried the resolution steps in #30 but still failed in the latest release tag. Note: I have already tried to access the URL externally.

The certificate is valid.

root@uat-tekesvault001:/home/XXXX# curl -v https://uat-tekesvault001.XXXX:8200/ui/
* Uses proxy env variable no_proxy == '.XXXX'
*   Trying 10.57.17.48:8200...
* TCP_NODELAY set
* Connected to uat-tekesvault001.XXXX (10.57.17.48) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=XXXX.com
*  start date: Oct 21 10:14:22 2021 GMT
*  expire date: Sep 21 10:14:22 2023 GMT
*  subjectAltName: host "uat-tekesvault001.XXXX" matched cert's "*.XXXX"
*  issuer: C=IN; ST=KA; L=BLR; O=XXXX; OU=XXXX; CN=XXXXCA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555813695860)
> GET /ui/ HTTP/2
> Host: uat-tekesvault001.XXXX:8200
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< accept-ranges: bytes
< cache-control: no-store
< content-security-policy: default-src 'none'; connect-src 'self'; img-src 'self' data:; script-src 'self'; style-src 'unsafe-inline' 'self'; form-action 'none'; frame-ancestors 'none'; font-src 'self'
< content-type: text/html; charset=utf-8
< last-modified: Thu, 21 Oct 2021 10:35:46 GMT
< service-worker-allowed: /
< vary: Accept-Encoding
< content-length: 5094
< date: Thu, 21 Oct 2021 10:45:15 GMT
<
<!DOCTYPE html lang="en">

Logs:

root@uat-tekesvault001# vault write generate/password length=36 symbols=0
Error writing data to generate/password: Error making API request.

URL: PUT https://XXXXXX:8200/v1/generate/password
Code: 500. Errors:

* 1 error occurred:
	* internal error


Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   error=
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | Unrecognized remote plugin message:
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   |
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | This usually means that the plugin is either invalid or simply
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | needs to be recompiled to support the latest protocol.
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:10:56.148+0530 [ERROR] rollback: error rolling back: path=generate/
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   error=
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | Unrecognized remote plugin message:
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   |
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | This usually means that the plugin is either invalid or simply
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:   | needs to be recompiled to support the latest protocol.
Oct 21 16:10:56 uat-tekesvault001 vault[3547968]:
Oct 21 16:11:22 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:22.610+0530 [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h5a7d91250beb09d963ae957a32bc9912f9a61b748ec608e8cb0185b185561453
Oct 21 16:11:27 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:27.104+0530 [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h9aba985fc7d0fc749d421f219327ef8fad88e61cf582e19543a01364e5a1ab81
Oct 21 16:11:39 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:39.045+0530 [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h103ddb59f751f47c419b48e2234576353dd5b0570692d6417a5734455e1d054f
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.097+0530 [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h1f16fcfcef0f5ea2e1ed5c52ec003f159474aaf76bbd552c141ace20f4d032f1
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.099+0530 [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h1c2cbe40f86aa0ad792649029f882cc00074025a5e5146bc9af5cf1c3d96d420
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.151+0530 [ERROR] secrets.secrets-gen.secrets-gen_778cb63d.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr found" timestamp=202>
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.152+0530 [ERROR] rollback: error rolling back: path=generate/
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   error=
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | Unrecognized remote plugin message:
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   |
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | This usually means that the plugin is either invalid or simply
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | needs to be recompiled to support the latest protocol.
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.155+0530 [ERROR] secrets.secrets-gen.secrets-gen_6cb4c383.secrets-gen.vault-secrets-gen: plugin tls init: error="no vault api_addr found" timestamp=202>
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]: 2021-10-21T16:11:56.156+0530 [ERROR] rollback: error rolling back: path=gen/
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   error=
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | Unrecognized remote plugin message:
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   |
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | This usually means that the plugin is either invalid or simply
Oct 21 16:11:56 uat-tekesvault001 vault[3547968]:   | needs to be recompiled to support the latest protocol.
sethvargo commented

What version of Vault are you running? In general, these issues are extremely hard to debug because each environment is different.

typlo-s commented

Hi Seth,

Using the below version of vault (on Ubuntu 20.04 focal) - precompiled by Hashicorp:

root@uat-tekesvault001:/home/user# vault version
Vault v1.8.2 (aca76f63357041a43b49f3e8c11d67358496959f)
root@uat-tekesvault001:/home/user# vault plugin list | grep -i secret
secrets-gen

Plugin version: Latest release tag (0.1.3) and tried compiling myself as well, but getting the same internal server error.

Steps to reproduce:
Create a CA
Sign a certificate using that CA
Use this certificate and key in the vault config
Update system's CA chain by placing the ca.crt into /usr/local/share/ca-cerrtificates and run update-ca-certificates
Try generating a password/passphrase

https://uat-tekesvault001.XXXX:8200/v1/generate/password
{
"errors": [
"1 error occurred:\n\t* internal error\n\n"
]
}

Before you ask, the certificate is valid and showing up as green in Chrome, with the correct SAN and extensions.

typlo-s commented

Update:

I was able to resolve it by defining api_addr inside the vault.hcl file.
While vault only uses the api_addr for clustering (HA), could you help answer why the plugin requires this?

/etc/vault.d/vault.hcl
...
api_addr = "https://uat-tekesvault001.XXXX:8200"
...

github-actions commented

This issue has been automatically locked since there has not been any
recent activity after it was closed. Please open a new issue for
related bugs.