Partition Key is random
rgrajan opened this issue · 6 comments
The partition key is important for Azure Table Storage. Currently it stores the ticks from the timestamp and it will affect the table service storage and high burden during the scaling. Refer https://azure.microsoft.com/en-in/documentation/articles/storage-table-design-guide/
You should have a property exposed from the client where we can pass the partition key. Please check the feasibility.
Can you point me to which part of the given link says using the partition scheme we are using will result in high burden during scaling?
Based on my understanding, having a small number of entities per partition actually improves scalability since it gives more flexibility to the table storage service to distribute load. The main disadvantage is you can't use Entity Group Transactions (not something we need in this case, except when batching, but in that instance a consistent partition key is used for the batch).
The design we chose was based on the design Microsoft themselves use(d - it's been a while since I looked?) for diagnostics storage.
Take a look at #1 / #7 / #9 which discuss an implementation for customisation of row key. It makes sense to allow for customisation of partition key as well if we are allowing row key to be customised.
I, for one, agree with @rgrajan that using a random partition key is not ideal. We'd like to use the source context, which is what we've been using on projects with NLog and which allows for faster queries when you know the source context you're looking for.
Azure Table documentation expands a lot on the subject of choice of partition key. Any single choice is certain to raise discussions. It really makes sense to allow customization of the partition key.
Moreover, @robdmoore, I also agree that the feature of allowing customization of the partition key goes hand-in-hand with allowing customization of the row key, because if you'll use a partition key with few possible values you'll probably need to use the ticks on the row key instead.
WriteTo.AzureTableStorage() has IKeyGenerator keyGenerator parameter.
IKeyGenerator has two methods:
string GeneratePartitionKey(LogEvent logEvent);
string GenerateRowKey(LogEvent logEvent, string suffix = null);
The default implementation is in https://github.com/serilog/serilog-sinks-azuretablestorage/blob/dev/src/Serilog.Sinks.AzureTableStorage/Sinks/KeyGenerator/DefaultKeyGenerator.cs
As @alexswi pointed out, there is already a way to choose your own PartitionKey and RowKey, which is to implement the interface IKeyGenerator and set the implementation on the configuration.
For more details, see this stack overflow answer.
I believe this issue should be closed.
obsolete