Add Azure Storage / Azure Key Vault extensibility to DataProtection
GrabYourPitchforks opened this issue · 53 comments
These solve two different but related problems. The DataProtection stack requires that all machines in the environment are able to point to the same key repository so that they can share key information. Out-of-box the DataProtection stack has support for the file system (including UNC paths) and the Windows registry. By adding support for Azure Storage, applications running within distributed environments would be able to use that as an alternative repository.
Once all machines agree on a key repository, they're faced with the problem of key protection at rest, since we generally don't want the keys sitting around in plaintext in the repository. The DataProtection stack has built-in support for using Windows DPAPI, CNG DPAPI, or an X.509 certificate. Adding support for Azure Key Vault would offer a mechanism for easing the burden of secret management in a distributed environment, and it would complement support for Azure Storage.
Not sure if this would be better suited as a sample application, but throwing the idea out there.
I don't think this will fit in V1. We should probably write a sample around it when we document these things better.
👍 I think this is somewhat critical given that so much is moving to Azure right now. A very common scenario will be using Antiforgery with forms in web farm apps across Azure VM's, which creates and validates tokens with the data protection system. Azure Files might cut it for a network share (?) but the docs state that Core CLR cannot use the X.509 certificate bits to secure the keys on the share (if I'm reading that correctly).
Even if it were very manual at this point, a sample for hacking up an Azure Key Vault-to-Data Protection IKeyManager or XmlKeyManager implementation would be very helpful in the interim. AFAIK there is no sample anywhere of how this is done, correct?
@guardrex There's a workaround for doing certificate-based encryption if you're on Core CLR. However, it requires that you be on Windows 8.1 / Windows Server 2012 R2. See https://docs.asp.net/en/latest/security/data-protection/implementation/key-encryption-at-rest.html#certificate-based-encryption-with-windows-dpapi-ng.
You're correct in that there's currently no sample for using Azure Key Vault.
@GrabYourPitchforks Thanks ... I'll give that a shot. I'm going to try this with Azure File Storage.
@GrabYourPitchforks This isn't entirely surprising: I setup an Azure File Storage share with a key directory and confirmed the share was working on the VM, then used this ...
configure.PersistKeysToFileSystem(new DirectoryInfo(@"\\<my-storage-account-name>.file.core.windows.net\dataprotection\keys\"));
... and it looks like its choking on the age-old problem of permissions. With .NET/IIS working on ApplicationPoolIdentity or NetworkService, it doesn't seem to have permission to access the share, so navigating to a view with Antiforgery configured, it fails with ...
IOException: The parameter is incorrect
System.IO.Win32FileSystem.CreateDirectory(String fullPath)
Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.<GetAllElementsCore>d__15.MoveNext()
System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.GetAllElements()
Microsoft.AspNet.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNet.DataProtection.KeyManagement.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow)
Microsoft.AspNet.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
Yeah, you have these flashbacks to the good 'ole days of impersonation! Yuck. I'll investigate making the share available to the app further; but given your familiarity with the packages and implementation, do you have any tips?
Let's say I go with a custom IXmlRepository and use Azure Table Storage. Something like this:
In ConfigureServices(...) of Startup.cs
services.AddDataProtection();
var serviceDescriptor = new ServiceDescriptor(typeof(IXmlRepository), typeof(ICustomXmlRepository), ServiceLifetime.Singleton);
services.Add(serviceDescriptor);XmlRepository.cs [The FetchAll() and Insert() are methods for grabbing and saving the keys. This app will have its own Azure Table for the keys, and multiple instances of the app will be hitting the table.]
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Xml.Linq;
using Microsoft.AspNet.DataProtection.Repositories;
using ShorehamPublic.Models;
namespace MyApp.Repositories
{
public class ICustomXmlRepository : IXmlRepository
{
public IReadOnlyCollection<XElement> GetAllElements()
{
IList<XElement> keyList = new List<XElement>();
IEnumerable<DataProtectionKey> keyEntities = DataProtectionKey.FetchAll().Result;
foreach (var key in keyEntities)
{
keyList.Add(XElement.Parse(key.Data));
}
IReadOnlyCollection<XElement> readElements = new ReadOnlyCollection<XElement>(keyList);
return readElements;
}
public async void StoreElement(XElement element, string friendlyName)
{
await DataProtectionKey.Insert(element.ToString());
}
}
}- This seems to work ok in light testing, but what would the gotchas be for having instances of the app using this approach?
- If this is the only app (with multiple instances) using a table of keys, I don't need to supply an application name in configuration, correct?
- I noted
<!-- Warning: the key below is in an unencrypted form. -->above the key value. What's the best way to handle this? Can I encrypt with out-of-the-box methods/configuration, or should I encrypt and decrypt the wholeXElementin my table storage model methods? - What about removing the table entities? I've created the entities with a timestamp
RowKey. I presume I should clear the old keys on my own ... keys older than 90 days can be deleted from the table? - Did I make a royal code smell anywhere? ... E.g., Did I register the custom
IXmlRepositorycorrectly? Does the implementation of the custom repository look ok?
/cc @blowdart
I reworked the existing Azure extensions for DataProtection I've been working on and migrated the project to https://github.com/GrabYourPitchforks/DataProtection.Azure/tree/dev. Feel free to give that a shot if you're looking for something quick and dirty. Maybe one day it can be contributed back into the main project. :)
In particular, the system ends up being more reliable if you use blobs instead of tables. Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.
To your other questions: if there's only one application pointed at a repository, you don't need to worry about the app name. You also don't need to worry about cleaning up old keys. The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.
You can use the existing key encryption methods like certs. The system is designed such that repositories and key encryption methods are separate concerns and can be mixed-and-matched freely.
I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.
@GrabYourPitchforks That's great. Thanks.
Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.
That table storage version I did seems to work, but I didn't stress test it yet. I feel you though on the "traversing entities," which is why I envision a single table per app for this and used a common PartitionKey for all entities in the table (i.e., to pull back all entities in one bite from a single edge server). I'm also taking the entire Xml and just plopping it into a property of the entity. I'm not trying to parse it and not dealing with key-value pairs. I'll do a stress test on it and see how it holds up.
The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.
This is very interesting! The only use I have for this for the moment is for Antiforgery. If I recall correctly, the Antiforgery header/cookie system is setup for 90-day expiration out-of-the-box (?); so if I were only using Data Protection for Antiforgery, I guess I could kill old keys at 90-days for sure ... but that would be a strange case where someone opens a webform and leaves it open for as long as 89 days and finally POST's back to the server hoping for an Antiforgery validation to work. In any case, it would be easy for me to drop old keys, since my RowKey is actually a timestamp and easy to filter/delete on periodically.
I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.
Yeah, that's going to be excellent. That will be the gold standard for Azure-hosted apps IMHO. I'll give your DataProtection.Azure a shot, and I'll keep an eye out for Azure Storage + Azure Key Vault.
One ? though ... about the <!-- Warning: the key below is in an unencrypted form. --> note above the key value in the Xml. What's the best way to handle this if I want it encrypted? Is there something in-the-box, or should I just encrypt/decrypt the whole Xml payload just before dropping into storage and just before handing back to Data Protection?
Call any of the ProtectKeysWith* APIs, such as ProtectKeysWithCertificate, during the configuration routine. You can mix and match ProtectKeysWith* and PersistKeysTo* calls. The system is designed such the data repository doesn't know anything about how keys are encrypted at rest, and the key protection mechanism doesn't know anything about how the key files are stored.
@GrabYourPitchforks Good grief. Sorry about that ... we just touched on that above. Thanks for your help ... I'm good now.
@muratg We should revisit this for RC2. People are asking about it more and more.
👍 for Azure Table Storage as one of the offerings.
I like @GrabYourPitchforks' extension that uses blob storage. We should consider getting it in our branch. Putting this in RC2 for now to consider in RC2.
@muratg I know ... it's a "file"-type structure/format. I was just saying if multiple options could exist, it would be nice to have a choice.
@muratg Oh, yes, of course. I really like Azure Key Vault. Would love to use that for this.
👍 on Azure Key Vault.
Is there any progress update on this topic?
Ah, so this is why my users have to log in again every time I slotswap... Any way for Azure Websites to just support this out of the box across slots, or at least sticky to the slot?
We just tried to set this up on a custom VM (running Azure Service Fabric) and an Azure file storage share, but dealing with the permissions is just a huge PITA so I gave up. I will try to use @GrabYourPitchforks 's azure blob storage provider now.
I really think you should provide at least one official cluster-aware storage provider for RTM - preferable based on some Azure stuff to have a good Azure story.
If you don't provide something here, I can already see the countless hours wasted by people trying to get their UNC share permissions working :-(
I upgraded @GrabYourPitchforks 's code to RC2 in case anyone needs it: GrabYourPitchforks/DataProtection.Azure#1
Based on our discussion with @blowdart and @DamianEdwards, we'll initially go with a blob storage backed implementation.
If you don't provide something here, I can already see the countless hours wasted by people trying to get their UNC share permissions working :-(
wastedHours += 10
Took me a while to understand why my users need to login every time I swap slots on Azure...
I decided to wait for the official solution, but it just seems to take forever, so I'm curious, currently, my application doesn't have too much sensitive data, Is there a way to just specify a fixed encryption key?
And later once there's an official solution for Azure to change and use that?
@guardrex maybe you are aware of such a trick?
Yeah. I'm doing that now in a few apps with Azure Table Storage as the key repository, but I haven't published it. I'm at the 💪 gym 💪 right now, but if we can talk on Slack about it in a couple of hours, I'll show you the code I'm using.
@guardrex I knew you'll have something dirty for me... 😄
I'm looking for something even more secure than Azure Table Storage:
services.AddDataProtection()
.UseThisBloodyKeyForever("Qwe123"));
Note the capital Q for extra protection!
You could do whatever you like with the key ring, as these are really separate issues.
I'm over at Slack now now, can you PM me over there? If you don't have an account, go through http://tattoocoder.com/aspnet-slack-sign-up/
There is a blob storage provider coming.
But we won't ever be providing a UseAFixedKeyBecauseILikeBeingInsecure option :D
But we won't ever be providing a UseAFixedKeyBecauseILikeBeingInsecure option :D
BOOH!!! 🙈😜
I provided it to him privately. I don't plan on publishing it (or even widely sharing it). I suggested putting the key in Azure Key Vault. @gdoron told me he plans to use the official Azure bits when you release it.
Key Vault isn't really an option for us. KeyVault expects to control the keys entirely. Data Protection creates and rotates. The only way we could use KeyVault is to put them as secrets, not keys, and then your app has to have permissions to create new secrets and granting that violates least privilege,
Playing devil's advocate: how is blob storage different in that case? You also have to provide write access for the application if it needs to rotate. And I'm pretty sure most people will use the big storage key for this, which is extremely overprivileged IMO - especially if people also use the storage for other stuff etc.
With key vault, at least you can control access via Azure AD. I don't see an issue with using secrets for this. There are also already other things like certificates etc that need to be stored as secrets in order to use them with Azure ARM templates etc.
Or do you see other issues with key vault security?
I'd hope people would use a separate blob store for keys :)
Plus KeyVault doesn't support saying "Give me all versions of this secret" which is also problematic for us, as we want to say "Give me all keys" and get them, without having to use a paging interface.
I'm not sure how blob storage is inherently more secure than providing a string, especially since the string could come from somewhere secure. For example a slot setting. I do see how a hard coded literal key would be less secure, but it won't always be a literal.
For storage anyway you need to have some way to get an access key to the machine so that it can go talk to blob storage (or keyvault) to get the secret. It's basically using a secret to get a secret. So why not just have the original secret be the one that's used?
I realize that using slot settings may not scale to a ton of secrets, but for simple apps with just a handful of secrets, why add the extra layer?
Well, after almost an hour of trying to implement the UseThisBloodyKeyForeverSecurityIsOverratedBro, I realized you made it impossible 😄 because I still need to implement a repository to store the keys for when I change slots on my azure web app.
I hope the official solution will be shipped soon, it's really missing.
I'll just not change slots until it'll be shipped 😢
Data Protection creates and rotates.
I do key rotation using the DP bits. I use a master key and replace the key encryption and key decryption bits to use my master key.
services.AddSingleton<IXmlRepository, CustomXmlRepository>();
services.AddSingleton<IXmlEncryptor, CustomXmlEncryptor>();
services.AddSingleton<IXmlDecryptor, CustomXmlDecryptor>();This avoids me having to use certs on servers. Replacing the repository with Azure Table Storage prevents me having to use a network share. I'm happy with what I have and will keep an 👀 on development to see if/when I can switch over to MS bits.
(and are hopefully working on)
@blowdart From my experience that's not obviously granted, I couple of months ago emailed Scott Gu for constantly getting an (unstyled IIS) 404 on Azure registration!, he forwarded the email to several PMs and they emailed me, it will be fixed in several weeks...
@blowdart As an aside tho: If you only register replacements for IXmlEncryptor and IXmlDecryptor ... not the repository IXmlRepository ... Why does it throw a .....
Apparently, they must all be replaced together.
Just to be sure, even the AzureBlobStorageProvider that is being worked on won't work when I change slots unless Azure will fix that stuff, right?
Or did I get it wrong?
I'd hope people would use a separate blob store for keys :)
That's very optimistic thinking. :) I'd use Atwood's JavaScript law and say "Everything that can be stored in one storage account, will eventually be stored in one Storage account". 😄
Plus KeyVault doesn't support saying "Give me all versions of this secret" which is also problematic for us, as we want to say "Give me all keys" and get them, without having to use a paging interface.
True, working with secrets is not very straight forward unfortunately.
However, one huge advantage of key vault is, that you can use certificate based authentication. This somehow removes the issue of "providing a secret to get a secret" as mentioned by @dfederm .
Seems like one secret can have up to 25kb so maybe you can combine/compress stuff?
@cwe1ss cert-based auth is still a secret, it's just a different way of delivering the secret. And unless I'm mistaken, I don't think Azure Websites supports installing certs on the machines anyway.
One thing I might have missed with the slot setting thing though is that if the data protection stuff also rotates the key itself then it wouldn't work since slot settings are read-only from the app (unless you do some shenanigans like call the management APIs from the app itself, but that requires extra secrets as well anyway). So storage/keyvault works since it allows the app to not only get the secret but also set it when rotating it.
Of course, but it's using a different medium and no one can commit it to Source control (at least you must be pretty stupid to do so 😄). But handling certificates with expirations etc is also quite some work unfortunately.
I'm mostly using Service Fabric which allows to install certificates. But as far as I know you can upload certificates to Azure Web Apps - I think you must also set some magic appSettings key to actually be able to use them though.
Blob storage will work between slots. And Web Apps do support certs.
@gdoron The slots thing azure knows about (and are hopefully working on)
@blowdart FYI I was told by an Escalation Engineer from the ASFS program that if it will be implemented it probably won't happen in the near future.
He explained to me that WebApp should be agnostic to the application and platform it's being used for, so WebApp shouldn't handle ASP.NET Core differently.
He suggested I won't wait for the fix and use the code from the PR: #163
I think it's reasonable and makes sense.
Yup. That'll will become part of the next core release.
As for the WebApp being agnostic, well I'd disagree, but it's Azure's choice.
I have just spent a few hours debugging a System.OperationCanceledException: The operation was canceled. which turned out to be caused by this:
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
An unhandled exception has occurred: The antiforgery token could not be decrypted.
System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {9725081b-7caf-4642-ae55-93cf9c871c36} was not found in the key ring.
I now understand that this is happening because I'm deploying to a separate staging slot and then swapping to production to avoid downtime. I though the keys stored in /ASP.NET/DataProtection-Keys/ were synchronized between all sites, but this is obviously not the case for slots.
Will this be handled for me by the Web App platform in the future? If so, when?
What is the current recommended approach to handle this issue? I've seen a blob storage implementation here, but I would feel more comfortable if there was an "official" implementation I could use.
@henningst There's an official implementation for Azure Blob Storage. If you would like an Azure Table Storage implementation, you can hack your own key repository, but that's not recommended by MS.
For the KeyVault extensibility we're waiting on some KeyVault SDK changes. Removing this from 2.0.0-preview2 for now to bring it back to triage.