splunk receiving too much data from remote write

[cbrdy]

@lukemonahan, We observed that splunk received 350G+ of data for one day via remote write but when I checked the disk size of prometheus instance, it's not even 100G for that day. How is this possible ?

Thanks,
Charan

[lukemonahan]

Splunk will be measuring license usage according to the cost model for metrics.

Unlike event data, metrics data counts against a license at a fixed 150 bytes per metric event

https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/AboutSplunksLicenseUsageReportView#Identify_metrics_data_in_your_license_usage_report

So it is likely you transmitted some ~2.5 billion metrics to Splunk to cause this license usage.

Prometheus disk usage will be for compressed data, as compared to the raw data ingested into Splunk anyway, so this will not be directly comparable.

[cbrdy]

thanks for the reply. After a painful research, I found out last night that splunk licensing model is eating a bulk of out daily limit :(