Feature Request - Native custom check scheduler

[neilharris123]

This is a feature requests for the implementation of a feature to allow scheduling the execution of custom metric checks via the splunk-otel-collector.

For example, with the Datadog agent (which in many ways is the Datadog equivalent of the splunk-otel-collector), it comes with a built in cron-like ability to execute custom checks on a schedule. So for example, you create a python scripts to retrieve/generate a metric value, and the agent sends a JSON payload containing metric name and value to the REST API based on the frequency you decide.

With the splunk-otel-collector, at the moment, we are executing our custom metric python script via the hosts native cron. Cron trigger the script, the script generates the metric and builds a payload, sends the payload to the splunk-otel-collector http exporter (http://127.0.0.1:4318/v1/metrics), which in turn send it to Splunk Observability. This works ok, but it would be nicer to be able to schedule this via the splunk-otel-collector, rather than relying on the systems cron.

[atoulme]

Thank you for entering this issue.

Would this exclusively call out to a REST API? How often are you looking to poll?

[neilharris123]

I am currently only looking to call a REST API, however I was thinking of a feature which would be more generic, allowing users to execute any checks/scripts that generate metrics in the correct format, so not necessarily only for polling endpoints.
Looking to poll every minute, but finer checks would also be good.

[atoulme]

I have filed https://ideas.splunk.com/ideas/SFXIMMID-I-486 to follow up. Let's continue the conversation in Splunk Ideas if you don't mind.