accumulo-client properties like "ssl.truststore.password" do not get found inside a CredentialProvider if one is provided
gwynlionhart opened this issue · 4 comments
Describe the bug
When you setup a CredentialProvider JCEKS file with an alias "ssl.truststore.password" for use in the accumulo-client, then Accumulo will not be able to find and retrieve that value from the JCEKS. I got a bit lost, but I think what was happening was that the ClientConfConverter class was only ever looking for the server property "rpc.javax.net.ssl.trustStorePassword" and not the client-specific value. So long as I put "rpc.javax.net.ssl.trustStorePassword" as an alias into the JCEKs file I didn't have an issue.
To Reproduce
Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):
- Setup an accumulo-client.properties with basic info but importantly ssl enabled, a credential provider specified, and a ssl.truststore.path
#Whatever you have for instance.name and instance.zookeepers in accumulo.properties
instance.name=instance
instance.zookeepers=zookeeper:2181
#When defined, Accumulo will try to retrieve matching properties out of the jceks
general.security.credential.provider.paths=jceks://file/path/to/accumulo/conf/accumulo.jceks
auth.principal=accumulo
auth.type=password
auth.token=password
#SSL Configuration
ssl.enabled=true
ssl.truststore.path=/path/to/accumulo/conf/ssl/truststore.jks
- Launch Accumulo itself with SSL enabled, then try to connect to Accumulo shell with the above accumulo-client.properties
- See error
[root@accumulo scripts]# accumulo shell --config-file accumulo-client.properties
2024-05-23T13:27:21,382 [clients.ClientServiceThriftClient] WARN : Failed to find an available server in the list of servers: [ssl:accumulo:9997 (120000)]
With java debugging enabled you will see that it is giving an empty password to the truststore. And if you debug on line 172 of ClientConfConverter.java and click play a few times to see the properties it's looking for, you will find that it is only searching the for properties like "rpc.javax.net.ssl.trustStorePassword"
Expected behavior
Entering Client properties should work in the CredentialProvider lookup in the same way that we are able to use the Client properties in the accumulo-client.properties file. I'm unsure if that's feasible if the list of properties supported by Credentialprovider is defined by Hadoop and not Accumulo.
In which case, another (easier) solution would be to update the list of properties in the documentation to include a caveat about CredentialProviders and how the property name conversion thing means you need to put the server property into the JCEKS. For example, in the Description column of "ssl.truststore.password" you could include "Note: if protecting this property in a CredentialProvider specified in 'general.security.credential.provider.paths', then you must use the property "rpc.javax.net.ssl.trustStorePassword" inside of the CredentialProvider."
Semi-related when it comes to properties documentation, but I think there are a few server properties you do need to specify in accumulo-client.properties that aren't documented on the client properties page. For example, if Accumulo is using TLSv1.3, then the accumulo-client.properties file has to specify "rpc.ssl.client.protocol=TLSv1.3" though that property is only listed in the server properties page, otherwise you get infinite
"2024-05-23T13:34:54,194 [transport.TIOStreamTransport] WARN : Error closing output stream.
java.net.SocketException: Connection or outbound has closed
" stack traces. You have to put the property in both server and client property files if the value isn't the default TLSv1.2, or else "accumulo-cluster stop" will throw a similar error. Oh, and currently the client properties don't mention that general.security.credential.provider.paths is a supported property either, though I'm very grateful that it is supported still.
You are correct that the Hadoop CredentialProvider mechanism is not used for the SSL properties.
However, I am not sure about implementing that feature. I'm not very familiar with all the CredentialProvider mechanisms, but the one that I've interacted with (jceks file used for tests that I recently updated in #4580) isn't that secure. It is just a Java keystore that doesn't have a passphrase.. which is not much better than a plaintext file. Perhaps CredentialProvider has implementations that are more secure, or there's a way to securely pass a passphrase for the jceks file itself, but I'm not familiar with those methods at the moment.
By comparison, the whole point of these SSL properties is to allow you to store the relevant SSL/TLS certificates into a keystore. In other words, it seems to serve a similar purpose. It sounds like you're wanting to store the passphrase for the keystore holding the certs in a separate keystore holding the passphrase. But then, is that second keystore going to be encrypted? If so, where are you going to store the passphrase for that second keystore? Wouldn't you just have to pass that to CredentialProvider (assuming it even supports that, which as I said above, I'm not sure about)?
I'm sure I have room to learn some more about how the CredentialProvider works, but without more information, it currently seems to me that it's turtles all the way down and I don't see much point in adding code to support adding another layer. If there is a way to securely unlock an encrypted keystore without a keystore passphrase needing to be stored insecurely, perhaps the JSSE system properties (which these client properties are intended to mirror) support something more naturally, without using Hadoop's CredentialProvider in an insecure way? I'm not sure the answer, but it seems more worth pursuing that than it does adding an extra insecure layer.
My main issue was that some SSL properties are handled by the CredentialProvider. If you put the value for the property "rpc.javax.net.ssl.trustStorePassword" or "rpc.javax.net.ssl.keyStorePassword" into a Credential Provider, and you have the property general.security.credential.provider.paths set to point to that credentialprovider store (JCEKS in my example), then Accumulo will retrieve the password out of the JCEKS rather than expecting the property to be in the properties file.
In contrast (and where I thought an issue might be) is that the property "ssl.truststore.password" on the client side does not get treated in the same way. If you put that into the JCEKS, the Accumulo code doesn't find it, so it then looks in the properties file (where it also doesn't find it), then fails b/c it can't open the truststore.
The quickest workaround (which I did) is that even on the client side I put "rpc.javax.net.ssl.trustStorePassword" as the alias for the truststore password entry in the JCEKS. That property works even on the client side because in that ClientConfConverter it ends up converting "ssl.truststore.password" into "rpc.javax.net.ssl.trustStorePassword" anyway. But I originally thought it may be a bug that the property wasn't being found like the server side property was found.
But on the topic of security of the JCEKS: absolutely agree, not a long-term solution. I will be looking for a secure way to store the password for the user/root user/truststore/keystore in some other manner that doesn't amount to a plain text representation on the file system. Maybe implementing something off the CredentialProvider interface to grab passwords in some other secure manner
@gwynlionhart Which version of Accumulo are you using? And did this work in a previous version?
I previously was using Accumulo 1.10.2. At the time my accumulo.conf had the same-named properties as my server's accumulo-site.xml, so the JCEKS files for each had the same alias names for both server and client JCEKS files (the client had fewer properties of course, not needing the root user's password for example).
When I upgraded to 2.1.2, I moved the client side from accumulo.conf to accumulo-client.properties and tried updating the property names in line with what the docs defined. Though in my original post aboveI did note that some properties are omitted from the documentation that ended up required for me, but that's a different issue. I also eventually stopped using accumulo-client.properties due to #4573 and the issue with CredentialProviderTokens, but will go back to the properties file when 2.1.3 releases with that fix.
So when I updated to using the accumulo-client.properties, I moved to using ssl.truststore.password and updated my JCEKS keystore to have that alias, but then I encountered issues arising from the Accumulo client not finding the truststore password during initialization. When I reverted the JCEKS file's alias for that entry back to "rpc.javax.net.ssl.trustStorePassword", then the client initialized correctly