Azure-Samples/azure-spring-boot-samples

[QUERY] How to set the visibility timeout and checkpoint mode on StorageQueueTemplate?

soumabrata-chakraborty opened this issue · 6 comments

soumabrata-chakraborty commented

Query/Question
Earlier up to version 2.14.0 of the spring integration library for Azure Storage Queue, the StorageQueueMessageSource used to take a StorageQueueOperation object. This was replaced by StorageQueueTemplate in version 4.0.0
The earlier StorageQueueOperation object allowed us to set the checkpoint mode and visibility timeout
These setters are missing in StorageQueueTemplate object
How do I set these values when working with StorageQueueTemplate?
Also the StorageQueueMessageSource seems to always all the underlying StorageQueueTemplate with a visibilityTimeout that is set to null!

Why is this not a Bug or a feature Request?
At this point this is just a question. Depending on the response, this may be followed up with a feature request.

Setup (please complete the following information if applicable):

Information Checklist

  • Query Added
  • Setup information Added
yiliuTo commented

Hi @soumabrata-chakraborty thanks for reaching us. There is the breaking changes about the visibility timeout and checkpoint mode for StorageQueueTemplate compared with StorageQueueOperation.
For visibility timeout, now we don't provide any API to set a default visibility timeout for the template object, while when using the StorageQueueTemplate directly you can call the StorageQueueTemplate#receiveAsync to specify the timeout value. while for using StorageQueueMessageSource, then we don't support to specify it.
For checkpoint mode, we removed this conception in StorageQueueTemplate, and developers have to manually checkpoint messages now. This is because for the previous RECORD mode, messages will be checkpointed before they have been truly consumed by the customer handling code.

soumabrata-chakraborty commented

Hi @yiliuTo - thanks for response. This breaking changes introduces missing functionality for usage with spring integration.
I understand the change for the Checkpoint mode being MANUAL. This was the recommended approach for spring integration earlier too.
However, not being able to specify visibility timeout in StorageQueueMessageSource is an issue.
I am using StorageQueueMessageSource with @InboundChannelAdapter in Spring Integration.
I have multiple instances of my application deployed in Production, and each instance listens to the same queue.
It takes my integration flow about 3 minutes to process a message.
If the StorageQueueMessageSource does not support visibility timeout - then it becomes 30 seconds default I think -- so I cannot prevent different instances of my spring integration app from picking up the same message from the queue.
The only option left to me would be to extend StorageQueueMessageSource and introduce a configurable visibility timeout in it - but perhaps its better to change the azure project code and add this functionality with a default of 30 seconds if not configured by user.
If you think this feature makes sense - I am happy to contribute this back to the azure project.

yiliuTo commented

Hi @soumabrata-chakraborty your request fully makes sense to us. And the reason why we decided to remove the option of visibility timeout from StorageQueueTemplate is that, we think this property is related with each message instead of the whole StorageQueueTemplate, thus we don't think StorageQueueTemplate should hold a member of visibilty timeout.
Also your contribution is always welcome, and we would like to know whether you have other ideas rather than adding a member for StorageQueueTemplate?

soumabrata-chakraborty commented

@yiliuTo IMO the member is to be added to StorageQueueMessageSource and not to StorageQueueTemplate
Created Azure/azure-sdk-for-java#29566 for this. Will raise a PR shortly.

Yes it should be a per request parameter as far as StorageQueueTemplate is concerned, however it fits it with being a initialization set up for StorageQueueMessageSource
No changes are proposed or needed in StorageQueueTemplate

The reason why visibilityTimeout can be a member of StorageQueueMessageSource is that it is a one time bean source configuration and it makes sense that my spring integration message source / Inbound Channel Adapter is configured with a certain pre set visibility timeout depending on the specific queue which is being polled since messages from a particular type of queue would typically need a particular amount of time to process in an app

Of course if you are polling a different queue, then you will have an all together different StorageQueueMessageSource bean with its own corresponding visibility timeout.

yiliuTo commented

Hi @soumabrata-chakraborty the pr has been merged and you can look forward to our next release which is expected to be published in this month.