KV Engine: Recursively list keys

Question

KV Engine: Recursively list keys

vfauth opened this issue 6 years ago · 47 comments

Is your feature request related to a problem? Please describe.
With a KV engine, if I want to list all keys in the directory /foo/, it only returns keys directly under /foo/
For example, if I have the following keys:

/foo/some_key
/foo/bar/some_other_key

A LIST operation on /foo/ returns some_key and bar/, while I would like to have some_key and bar/some_other_key

Describe the solution you'd like
Add a parameter to either recursively return ALL keys in the provided path.

Describe alternatives you've considered
Another way to do it would be to add a parameter specifying the depth up to which look recursively for keys.

Answer 1 · 2018-11-27T16:20:01.000Z

This is a very useful feature, especially for the HTTP API

Answer 2 · 2019-07-05T19:07:48.000Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine
./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

Answer 3 · 2019-09-26T21:29:28.000Z

Also interested in this kind of feature.

Answer 4 · 2020-07-22T21:02:12.000Z

For anyone ending up here, I created a small cli to perform recursive kv read/list operations while we wait for the native solution.
Not very tested yet, I will be fixing bugs as they show up.

Answer 5 · 2021-05-28T12:50:32.000Z

@agaudreault-jive, thanks!
Btw, local readonly path="$1" doesn't work as you might expect. use local -r path="$1" https://stackoverflow.com/a/45409823

Answer 6 · 2021-06-24T20:48:17.000Z

Issues that are not reproducible and/or have not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. If a feature request is being closed, it means that it is not on the product roadmap. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them.

Please refer to our contributing guidelines for details on issue lifecycle.

Answer 7 · 2021-08-24T14:04:08.000Z

Would love this feature

Answer 8 · 2021-08-25T11:09:09.000Z

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

var secretListPath []string

func isNil(v interface{}) bool {
	return v == nil || (reflect.ValueOf(v).Kind() == reflect.Ptr && reflect.ValueOf(v).IsNil())
}

// ListSecret returns a list of secrets from Vault
func ListSecret(vaultCli *vault.Client, path string) (*vault.Secret, error) {
	secret, err := vaultCli.Logical().List(path)
	if err != nil {
		log.Println("Couldn't list from the Vault.")
	}

	if isNil(secret) {
		log.Printf("Couldn't list %s from the Vault.", path)
	}
	return secret, err
}

// RecursiveListSecret returns a list of secrets paths from Vault
func RecursiveListSecret(vaultCli *vault.Client, path string) []string {
	secretList, err := ListSecret(vaultCli, path)
	if err == nil && secretList != nil {
		for _, secret := range secretList.Data["keys"].([]interface{}) {
			if strings.HasSuffix(secret.(string), "/") {
				RecursiveListSecret(vaultCli, path+secret.(string))
			} else {
				secretListPath = append([]string{strings.Replace(path, "metadata", "data", -1) + secret.(string)}, secretListPath...)
			}
		}
	}
	return secretListPath
}

Answer 9 · 2021-08-25T11:23:24.000Z

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

@rnsc Just in case you didn't notice my comment above and it can help, I wrote a small go based cli for this (#5275 (comment))

Answer 10 · 2021-11-05T20:23:12.000Z

Adding my vote for this feature please.

Answer 11 · 2022-02-18T01:21:29.000Z

Ditto!
Essentially looking for some kind of vault-grep to find out any secret/engine/etc. that has some key, value, or pathname that matches a desired string.

Answer 12 · 2022-05-04T11:02:27.000Z

One more vote for search

Answer 13 · 2022-08-16T22:58:02.000Z

please implement this. It has been 4 years!

Answer 14 · 2022-08-23T13:58:37.000Z

This is a must have, listing all secrets now is a major issue.

Answer 15 · 2022-08-26T22:18:14.000Z

if 103 people upvoted this feature request, maybe it's about time to start adding this feature 😸
For instance, Consul lists all the keys recursively, and Redis does it by default (using: KEYS *).
And whilst the solution from @kir4h works pretty well, it would be nice to expose this feature through the API, and libraries for any language can make use of this functionality.

Answer 16 · 2022-09-23T08:50:07.000Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

Answer 17 · 2022-11-14T18:13:55.000Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!
./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

This script doesn't work if you have spaces in the path name. Even with passing the argument with double quotes.

Answer 18 · 2022-11-14T22:29:35.000Z

I replicated that script in python

import os
import subprocess
import json

#
# Note: Before running, you must authenticate via the vault cli or the commands will fail.
#

def traverse (inputPath):
    path = inputPath

    result = subprocess.getoutput('vault kv list -format=json -address="'+vaultAddr+'" "'+inputPath+'"')
    jResult = json.loads(result)

    for secret in jResult:
        if "/" in secret:
            print(inputPath+secret)
            traverse(inputPath+secret)
        elif "/" not in jResult:
            print(inputPath+secret)

parentPath = "Secret Engine/" #change to secret engine name that you are want to scan
vaultAddr = "https://vault.whatever" #change to vault address

vaults = subprocess.getoutput('vault kv list -format=json -address="'+vaultAddr+'" "'+parentPath+'"')
vault = json.loads(vaults)

for secret in vault:
    print(parentPath+secret)
    if "/" in secret:
        traverse(parentPath+secret)

Answer 19 · 2022-11-19T16:56:38.000Z

Had the same problem and ended up writing my own little tool which helps me listing KV secrets recursively in various useful formats:

https://github.com/FalcoSuessgott/vkv

Answer 20 · 2022-12-21T20:58:48.000Z

Adding my vote for this feature please.

Answer 21 · 2023-02-13T16:45:49.000Z

me too, seems Hashiguys doesnt want to

Answer 22 · 2023-03-15T15:36:56.000Z

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?
What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?
What if I have LIST or READ access to a secret within a path but not the parents within that path?
Thanks!

Answer 23 · 2023-03-15T16:02:24.000Z

Good news! For your question, I would say:

404, to give no information about the key existence
404, to give no information about the key existence
200, because the access to this specific key has been allowed

Answer 24 · 2023-03-15T16:24:15.000Z

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

What if I have LIST or READ access to a secret within a path but not the parents within that path?
Thanks!

I agree with vfauth's thinking, give does not exist or permission denied, best to not let someone assume a secret or path exists if they don't have permission.
Same as above
Display the secret as the root path, or don't display any path information at all.

Answer 25 · 2023-03-15T16:38:40.000Z

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:

What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

What if I have LIST or READ access to a secret within a path but not the parents within that path?

Thanks!

For me it should silently not return anything that it cannot list or read in the sub tree.
Now of course if the given path cannot even be listed, a 404 would be good. If LIST is ok but no secrets nor any sub path can be read a 404 or a 201 empty response would be appropriate imho.

If you want to recurse from a top level path and you only have read access to a secret from two levels down I'd argue returning a 201 would be the correct behaviour. As it wouldn't show up in the UI either would it?

That's just how I would implement this for me though with the predicate that people know that recursively iterating over a path that you don't have the appropriate ACL for your token set to do is a user error and that given how this works we cannot hold the hands of the user too much without implementing insane logic.

Answer 26 · 2023-03-18T16:05:51.000Z

Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:
* What should the behavior be if I don't have READ access to any given secret or path? Should it be included or redacted? What would you expect to see?

* What if I have LIST access to a given folder but not subfolder? Again, what would you expect to see?

* What if I have LIST or READ access to a secret within a path but not the parents within that path?
  Thanks!

I think I disagree with some of the suggestions above, mainly because the topic of this issue is about recursively LISTing keys, not READing the secret data at those keys, right?

keys that you can LIST but can't READ should absolutely present in the recursive list output.
a simple label/indicator for keys that you don't have READ access to could be a nice enhancement, but imo the purpose of the recursive list command isn't to precisely enumerate all of your access permissions, but simply get an easy overview of the path/key structure.
for a sub-path/key where a user lacks LIST permissions I wouldn't expect anything under that sub-path to be present in the output, full stop.
even if the user (strangely) has other permissions on the sub-path/key like READ or WRITE, the focus here is on recursively listing keys, and if LIST is restricted, it's seemingly not something the admins want the user to be able to enumerate.
from a user-friendliness standpoint, I would like to see the output inform the user that the unlistable sub-path/key exists but the user lacks LIST permission to enumerate the path further.
this is a great question, and imo the answer is an error should be promptly returned by vault
a recursive list logically can't start at a root the user doesn't have LIST access to, so attempting to do so at a path where LIST isn't permitted should return a nice simple error informing the user they don't have LIST permissions at that path.

Answer 27 · 2023-04-14T21:14:07.000Z

@finnstech This is great news. Is there any low/high confidence guess about when such functionality woudl be released? This quarter, this year?

Answer 28 · 2023-07-08T18:30:27.000Z

Is there any progress on this issue? I'm keen on using this feature.

Answer 29 · 2023-07-13T18:03:30.000Z

Also interested in this feature

Answer 30 · 2023-07-14T09:11:23.000Z

Same here.
We had to super awkwardly implement width-first-search with limited depth in Terraform to be able to retrieve all the paths of secrets of a sub-tree.

Answer 31 · 2023-08-09T10:58:06.000Z

very interested in this feature

Answer 32 · 2023-08-16T10:22:26.000Z

Any updates on this? This would be a highly beneficial feature for our bank-vaults project.

Answer 33 · 2023-08-17T07:02:50.000Z

How can you not have search yet?

Answer 34 · 2023-08-17T15:47:28.000Z

How can you not have search yet?

@oyvhvi exactly!

At the moment we are implementing our own vault-cli with all workarounds due to this missing feature.

Answer 35 · 2023-11-01T20:26:38.000Z

Just adding another vote for this. I know search in Vault has been a highly contentious topic for literally years at this point, but even some basic search functionality in the KV engine would be incredibly beneficial. As a mitigation for all the performance concerns that are always brought up, how about making it an option one could enable or disable on a given KV engine?

Answer 36 · 2023-11-03T13:09:15.000Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

I built on your code, I had a use case to export all the KV secrets for a given path and save it locally in a secrets.json file. I refactored it to export the secrets rather than list them. My Vault was sitting behind Cloudflare Zero trust, I adjusted the script to optionally pass CF_TOKEN to by pass Cloudflare as well, thought people might find this helpful for their use case:

#!/usr/bin/env bash

# vault-kv-export.sh

set -eo pipefail

readonly ARB_TEMP_SECRETS_FILE="arbitrary_temp_secrets.json"
readonly TEMP_SECRETS_FILE="temp_secrets.json"
readonly SECRETS_FILE="secrets.json"

log() {
    local log_type="$1"
    local message="$2"
    local timestamp
    timestamp=$(date +"%Y-%m-%d %H:%M:%S")
    echo "[$log_type] [$timestamp] $message"
}

log_info() {
    log "INFO" "$1"
}

log_error() {
    log "ERROR" "$1"
    exit 1
}

traverse() {
    local path="$1"
    local result

    local headers=()
    if [[ -n "${CF_TOKEN}" ]]; then
        headers+=("-header" "cf-access-token=${CF_TOKEN}")
    fi

    result=$(vault kv list -format=json "${headers[@]}" "${path}" 2>&1) || log_error "Failed to list secrets: ${result}"

    while IFS= read -r secret; do
        if [[ "${secret}" == */ ]]; then
            traverse "${path}${secret}"
        else
            local secret_data
            secret_data=$(vault kv get -format=json "${headers[@]}" "${path}${secret}" | jq -r '.data') || log_error "Failed to get secret data: ${secret_data}"

            if [[ "${secret_data}" != "null" ]]; then
                echo "{\"path\":\"${path}${secret}\",\"value\":{\"data\":${secret_data}}}," >>"${ARB_TEMP_SECRETS_FILE}"
            fi
        fi
    done < <(echo "${result}" | jq -r '.[]')
}

main() {
    log_info "Starting secrets retrieval process."

    [[ -f "${ARB_TEMP_SECRETS_FILE}" ]] && rm -f "${ARB_TEMP_SECRETS_FILE}"
    [[ -f "${TEMP_SECRETS_FILE}" ]] && rm -f "${TEMP_SECRETS_FILE}"
    [[ -f "${SECRETS_FILE}" ]] && rm -f "${SECRETS_FILE}"

    if [[ -n "${CF_TOKEN}" ]]; then
        log_info "CF_TOKEN detected."
    else
        log_info "CF_TOKEN not provided. Headers will not be attached to Vault requests."
    fi

    local vaults
    if [[ "$1" ]]; then
        vaults=("${1%"/"}/")
        log_info "Retrieving all secrets under ${vaults[*]}.."
    else
        local headers=()
        if [[ -n "${CF_TOKEN}" ]]; then
            headers+=("-header" "cf-access-token=${CF_TOKEN}")
        fi
        log_info "No secret engine provided. Retrieving all secrets.."
        result=$(vault secrets list -format=json "${headers[@]}" 2>&1) || log_error "Failed to list secrets engines: ${result}"
        mapfile -t vaults < <(echo "${result}" | jq -r 'to_entries[] | select(.value.type=="kv") | .key')
    fi

    for vault in "${vaults[@]}"; do
        traverse "${vault}"
    done

    echo "[" >"${TEMP_SECRETS_FILE}"
    sed '$s/,$//' "${ARB_TEMP_SECRETS_FILE}" >>"${TEMP_SECRETS_FILE}"
    echo "]" >>"${TEMP_SECRETS_FILE}"

    jq . "${TEMP_SECRETS_FILE}" >"${SECRETS_FILE}"
    rm "${ARB_TEMP_SECRETS_FILE}" "${TEMP_SECRETS_FILE}"

    log_info "Secrets retrieval completed and saved to ${SECRETS_FILE}"
}

[[ "$0" == "${BASH_SOURCE[0]}" ]] && main "$@"

a sample output:

$ ./vault-kv-export.sh secret2/confluent_cloud

[INFO] [2023-11-03 15:06:15] Starting secrets retrieval process.
[INFO] [2023-11-03 15:06:15] CF_TOKEN detected.
[INFO] [2023-11-03 15:06:15] Retrieving all secrets under secret2/confluent_cloud/..
[INFO] [2023-11-03 15:06:27] Secrets retrieval completed and saved to secrets.json

reading secrets.json

$ cat secrets.json

[
  {
    "path": "secret2/confluent_cloud/global-creds",
    "value": {
      "data": {
        "data": {
          "confluent_cloud_token": "REDACTED"
        },
        "metadata": {
          "created_time": "2023-11-02T13:14:09.616943342Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
  {
    "path": "secret2/confluent_cloud/service/credentials",
    "value": {
      "data": {
        "data": {
          "api_key": "REDACTED",
          "api_secret": "REDACTED"
        },
        "metadata": {
          "created_time": "2023-11-02T13:14:10.336792836Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
...

Hope it helps

Answer 37 · 2023-11-15T09:54:43.000Z

I am adding my vote, is there any news about when we can have it ? or at least what are the best alternatives for the moment?

Answer 38 · 2023-11-16T15:14:33.000Z

would also like to see this implemented

Answer 39 · 2023-12-06T10:20:57.000Z

It would be great to get an update on this. The last official comment on this was March. Secret cycling without this is unbelievably frustrating, especially for complex systems you weren't around for at the beginning (e.g. is this value still used somewhere?).

Answer 40 · 2023-12-08T20:07:21.000Z

Adding my vote - this would be super helpful for our use case!

Answer 41 · 2023-12-09T19:42:29.000Z

Hi everyone,

I managed to implement a server-side recursive secret search feature by adding a new API endpoint to the KV engine.
This was done by modifying the existing KV secret plugin source code, which means that Vault needs to be recompiled locally.

You can check my work here if you're interested for carrying further testing: https://github.com/kosmos-education/vault-plugin-secrets-kv

As I mentioned in the repository, please note that this is a highly experimental feature which could potentially lead to stability issues or vulnerabilities.
In my case, the plugin was tested against a Vault instance with 6000+ secrets stored

Answer 42 · 2023-12-15T09:17:07.000Z

Any progress or developments regarding this? This feature would provide significant benefits.

Answer 43 · 2023-12-30T17:23:50.000Z

I know this feature request is focused on CLI/API, but are there any plans to provide a tree ui for KV mounts on top of this upcoming work? I think that would make the UI much more usable for ops that need to deal on a daily basis with a lot of static secrets.

Also interested in knowing what the community thinks about this. Cheers.

Answer 44 · 2024-01-04T08:51:24.000Z

Using vault-kv-search and fzf creates a nice way to browse through kv secrets

vault-kv-search --search=path kv -r . --json | jq -r .path | uniq | fzf --preview 'vault kv get --format=yaml ${} | faq -f yaml .data

Answer 45 · 2024-02-18T16:24:00.000Z

@adrianlzt What's the faq command? (And you're missing a closing quote)

Answer 46 · 2024-02-18T18:07:50.000Z

faq is like jq but with form input/output formats (yaml between them).

I'm finally using (as the first get of all keys in my environment is quite slow):

vault read -format json sys/internal/ui/mounts/ | faq -r '.data.secret | keys | .[]' | grep -vw -e sys -e cubbyhole -e identity | xargs -n 1 vaku folder list -p > $temp_vault_path_file
cat $temp_vault_path_file | fzf --preview 'vault kv get -format=yaml ${} | faq -f yaml .data.data' -q "$1 $2 $3"

Answer 47 · 2024-09-23T11:32:49.000Z

6 years and no basic implementation of this feature?