Azure/azure-functions-kafka-extension

need control in setting the commit offset value (i.e manual commit) with Kafka extension instead of auto commit

thilakbe opened this issue · 2 comments

thilakbe commented

At high load, we see duplicate events consumption issue when we use Function app with Kafka trigger. Hence, we are looking for a feature where we could set the offset commit value manually instead of auto commit which is managed by Azure functions platform.

PSanetra commented

We need to set commit offset manually for a different reason as we have a use case where we need to cache events for a specific time and flush the events if a duration or memory threshold is reached. We want to commit after flushing and not when the events where received.

marcelmedina commented

+1

This should be a standard capability available with Azure Functions.

I've seen many integrations where teams decided to leverage their own worker services due to the lack of this capability in Azure Functions.