List operations does not work in external plugins

[LeSuisse]

Environment:

• Vault Version: Vault v0.9.3 ('5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d')
• Operating System/Architecture: Linux

Vault Config File:

plugin_directory = "/path/to/my/plugin/directory"

Startup Log Output:

==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
               Log Level: trace
                   Mlock: supported: true, enabled: false
        Redirect Address: http://127.0.0.1:8200
                 Storage: inmem
                 Version: Vault v0.9.3
             Version Sha: 5acd6a21d5a69ab49d0f7c0bf540123a9b2c696d

Expected Behavior:
It is possible to do a LIST request on a path that support it.

Actual Behavior:
Any LIST request fail with an internal server error and you got these logs from Vault:

2018/01/30 23:43:53.168510 [TRACE] plugin.HandleRequest: path=kv/ status=started type=mock-plugin transport=gRPC
2018/01/30 23:43:53.169698 [TRACE] plugin.HandleRequest: path=kv/ status=finished type=mock-plugin transport=gRPC err=<nil> took=1.185434ms
2018/01/30 23:44:01.599923 [TRACE] rollback: attempting rollback: path=auth/token/
2018/01/30 23:44:01.599933 [TRACE] rollback: attempting rollback: path=identity/
2018/01/30 23:44:01.599938 [TRACE] rollback: attempting rollback: path=secret/
2018/01/30 23:44:01.599940 [TRACE] rollback: attempting rollback: path=sys/
2018/01/30 23:44:01.599958 [TRACE] rollback: attempting rollback: path=cubbyhole/
2018/01/30 23:44:01.599945 [TRACE] rollback: attempting rollback: path=mock/
2018/01/30 23:44:01.600441 [TRACE] plugin.HandleRequest: path= status=started type=mock-plugin transport=gRPC
2018/01/30 23:44:01.601123 [TRACE] plugin.HandleRequest: path= status=finished type=mock-plugin transport=gRPC err=unsupported operation took=680.425µs

The issue seems to have been introduced with Vault 0.9.2/0.9.3.

Steps to Reproduce:
The issue can be reproduced with the mock-plugin.

1. Build the mock-plugin
2. Add the mock-plugin to the plugin catalog
3. Mount the mock-plugin
4. Add a value into the kv store
5. Do a LIST request on mock/kv

[everesio]

I have the same issue. In github.com/hashicorp/vault/logical/response_util.go RespondErrorCommon
[]string is expected but got []interface {}

Plugin code:

func (b *GsAuthBackend) pathRoleList(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) {
	roles, err := req.Storage.List(ctx, "role/")
	if err != nil {
		return nil, err
	}
	resp := logical.ListResponse(roles)

	keysRaw, _ := resp.Data["keys"]
	b.Logger().Info(fmt.Sprintf("pathRoleList: type=%v keysRaw=%v ", reflect.TypeOf(keysRaw), keysRaw))
	return resp, nil
}

2018/01/31 00:20:31.264548 [INFO ] plugin.vault-gs-plugin: pathRoleList: type=[]string keysRaw=[testrole]

Vault

keys, ok := keysRaw.([]string)
if !ok {
	return http.StatusInternalServerError, nil
}

2018/01/31 00:20:31.264712 [INFO ] github.com/hashicorp/vault/logical/response_util.go RespondErrorCommon: type=[]interface {} keysRaw=[testrole]

[briankassouf]

This is a regression with the gRPC transport type. Working on a fix for this, but in the meantime you can force your plugin to fall back to netRPC transport by unsetting the VAULT_VERSION environment variable. This will cause the plugin to fail the version check that is required to choose gRPC as the transport. In go this can be done with os.Unsetenv("VAULT_VERSION"). This should be done before plugin.Serve is called. Here's the mock plugin's main function as an example:

func main() {
	apiClientMeta := &pluginutil.APIClientMeta{}
	flags := apiClientMeta.FlagSet()
	flags.Parse(os.Args[1:]) // Ignore command, strictly parse flags

	tlsConfig := apiClientMeta.GetTLSConfig()
	tlsProviderFunc := pluginutil.VaultPluginTLSProvider(tlsConfig)

	factoryFunc := mock.FactoryType(logical.TypeLogical)

	// Fail the version check, falling back to netRPC
	os.Unsetenv("VAULT_VERSION")

	err := plugin.Serve(&plugin.ServeOpts{
		BackendFactoryFunc: factoryFunc,
		TLSProviderFunc:    tlsProviderFunc,
	})
	if err != nil {
		log.Println(err)
		os.Exit(1)
	}
}

[briankassouf]

Just merged the fix for this, along with some tests to ensure we are hitting this code path. Thank you for reporting this and doing some investigation.

We also bumped the minimum version required for gRPC to be selected as the transport to 0.9.4, so Go plugins compiled with the patch (91dffed) for this should fall back to netRPC on vault servers < 0.9.4. Once version 0.9.4 comes out these gRPC issues will be resolved and gRPC transport will be selected as the default.

@LeSuisse @everesio and @cypherhat (I saw you reacted to the comment above) Once your dependencies are updated it should be safe to remove the os.Unsetenv("VAULT_VERSION") call.

[cypherhat]

I added the os.Unsetenv("VAULT_VERSION") call and removed it a few hours later. Ain't OSS grand? My plugin is happy.