yokawasa/fluent-plugin-azure-loganalytics

Check fields/body size not to exceed data limits of Azure Monitor Data Collection API

Closed this issue · 6 comments

yokawasa commented

Enhancement

There are data limits of Azure Monitor Data Collection API. An enhancement is to check fields/body size not to exceed data limits of Azure Monitor Data Collection API.

Data Limits

There are some constraints around the data posted to the Azure Monitor Data collection API.

Maximum of 30 MB per post to Azure Monitor Data Collector API. This is a size limit for a single post. If the data from a single post that exceeds 30 MB, you should split the data up to smaller sized chunks and send them concurrently.
Maximum of 32 KB limit for field values. If the field value is greater than 32 KB, the data will be truncated.
Recommended maximum number of fields for a given type is 50. This is a practical limit from a usability and search experience perspective.
A table in a Log Analytics workspace only supports up to 500 columns (referred to as a field in this article).
The maximum number of characters for the column name is 500.

relevant issues

marcosflobo commented

@yokawasa I see this issue closed, does it mean that this feature is released in version 0.6.0? 😃

yokawasa commented

@marcosflobo
Thanks for the comment. The release is not for this feature.
I'm releasing new version for this feature shortly.

yokawasa commented

@marcosflobo
I've just published fluent-plugin-azure-loganalytics-0.7.0 that change base azure-loganalytics-datacollector-api to ">= 0.5.0". azure-log-analytics-data-collector-0.5.0 added check body size not to exceed data limit of 30MB in Azure Monitor Data Collection API

I didn't add each fields size check as I thought it may cause some performance issue if the num of field is too large. Instead, I added some description on data limits on README.md.

  • ❤️2
yokawasa commented

@marcosflobo I'm closing this issue
Please open an issue if you have any other suggestions!
Thanks again for your suggestion.

marcosflobo commented

Thank you very much @yokawasa for your work for the open-source community!! We'll upgrade our version in production very soon thanks to you!

  • 😄1
xrstf commented

Are there plans to support

A table in a Log Analytics workspace only supports up to 500 columns (referred to as a field in this article).

out of the box or should users use the record_transformer to remove unwanted/overhanging fields themselves?