Empty virtual queues are not deleted
nacho-pkz opened this issue · 6 comments
We have implemented a maven library, in order to manage all the Amazon services, and a requestor-responder pattern implemented, where the requestor, is a Grails application, and responder is a Spring-boot application. Both applications, use the same maven library to communicate with Amazon (S3, SQS, Personalize,...).
On each requestor and responder restart, the number of temporary queues increase, and exist a lot of pending messages:
Reading the documentation, I saw that empty temporary queues must to be deleted, after 5 minutes idle. In order to test it, I purged all the messages on temporary queues, and after 30 minutos, those queues are not deleted:
What is the problem? We are using amazon-sqs-java-temporary-queues-client version 1.2.1.
Hi,
This depends on your configurations, there's a few things to keep in mind:
- Idle Queue Retention Period:
withIdleQueueRetentionPeriodSeconds(long period), sets the retention period of the virtual queue even if idle. - Idle Queue Sweeping Period:
withIdleQueueSweepingPeriod(int period, TimeUnit timeUnit), sets the sweeping period to check for all queues starting with for idleness and deletes them. - Queue Heartbeat Interval:
withQueueHeartbeatInterval(long heartbeatIntervalSeconds), sets the interval for the heartbeating mechanism to keep the queue alive.
When creating a new Requester Client, the default value for the idle queue sweeping period is of 5 minutes. This means that every 5 minutes it will check for all queues beginning with the internal queue prefix for idleness. It checks 2 things:
- Is the queue Idle: Checks whether (current timestamp - the last heart beat timestamp) is greater than the idle queue retention period
- Is the queue empty
If both those cases are true, then the idle queue will get deleted.
Note: A Host queue will not get deleted by the idle queue sweeper
By default, the heart beating mechanism will tag the queue every 5 seconds, the idle queue retention period is set to 5 minutes and the idle queue sweeping period is set to 5 minutes. This means that the queue will never get deleted even if it is idle, since the heart beat is keeping it alive.
sqsRequester = AmazonSQSRequesterClientBuilder.standard()
.withAmazonSQS(sqs)
.withIdleQueueSweepingPeriod(60, TimeUnit.SECONDS)
.withQueueHeartbeatInterval(420)
.build();
In this example, we set the idle queue sweeping period to every minute, meaning it will check for idle queues every minute. The heart beat is set to every 7 minutes and by default the idle queue retention period is set to 5 minutes. Therefore on the 6th minute, if a virtual queue has been idle, it will get deleted by the idle queue sweeper. On send and receive message calls, the heart beat timestamp is updated, allowing the virtual queue to stay alive.
Hi @adam-aws , what do you mean? I've got the configuration by default (heartbeat=5 sc, idleQueueRetention=5 min, idleQueueSweeping=5 min). With the configuration by default, all the temporary queues will never be idle? Which is the recommended configuration?
The number of Temporary queues continues to increase, and on application client, we have a lot of log messages:
[ERROR] 2021-06-25 13:38:58,378 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2243) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2210) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2199) at com.amazonaws.services.sqs.AmazonSQSClient.executeTagQueue(AmazonSQSClient.java:2087) at com.amazonaws.services.sqs.AmazonSQSClient.tagQueue(AmazonSQSClient.java:2058) at com.amazonaws.services.sqs.AmazonSQSClient.tagQueue(AmazonSQSClient.java:2099) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.heartbeatToQueue(AmazonSQSIdleQueueDeletingClient.java:281) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.heartbeatToQueueIfNecessary(AmazonSQSIdleQueueDeletingClient.java:297) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.receiveMessage(AmazonSQSIdleQueueDeletingClient.java:381) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.lambda$receiveMessage$5(AmazonSQSVirtualQueuesClient.java:204) at java.util.Optional.orElseGet(Optional.java:267) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.receiveMessage(AmazonSQSVirtualQueuesClient.java:204) at com.amazonaws.services.sqs.util.SQSMessageConsumer.poll(SQSMessageConsumer.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 23 more [ERROR] 2021-06-25 13:38:58,420 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2243) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2210) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2199) at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1637) at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1607) at com.amazonaws.services.sqs.util.AbstractAmazonSQSClientWrapper.receiveMessage(AbstractAmazonSQSClientWrapper.java:153) at com.amazonaws.services.sqs.AmazonSQSIdleQueueDeletingClient.receiveMessage(AmazonSQSIdleQueueDeletingClient.java:382) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.lambda$receiveMessage$5(AmazonSQSVirtualQueuesClient.java:204) at java.util.Optional.orElseGet(Optional.java:267) at com.amazonaws.services.sqs.AmazonSQSVirtualQueuesClient.receiveMessage(AmazonSQSVirtualQueuesClient.java:204) at com.amazonaws.services.sqs.util.AbstractAmazonSQSClientWrapper.receiveMessage(AbstractAmazonSQSClientWrapper.java:153) at com.amazonaws.services.sqs.util.SQSMessageConsumer.poll(SQSMessageConsumer.java:118) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 22 more
Can you try and update the configurations so that it fits your needs? So increasing the heartbeat interval to not have it do it every 5 seconds, since the sweeper runs every 5mins it wont get deleted.
Also, a queue is only “idle” and automatically deleted if it is empty and has no traffic, including empty receives.
@adam-aws , can you say me which is the recommended configuration? The default configuration makes SQS temporary queues to not work? If I increase the heartbeat from 5 seconds, what is the good value? any value greater than idle queue sweeper?
And what about the exception that appears on my last comment?
[ERROR] 2021-06-25 13:38:58,378 SQSQueueUtils - Unexpected exception com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at
Yes correct, so you can configure the Idle Queue Retention Period if you want to make sure the queue stays alive for a certain time, even if idle (by default 5min). Then based on your needs, you can set the idle queue sweepers value (by default 5min) if you want the sweeper to check quicker or slower.
You can change the heartbeat to be longer than the sweeper, otherwise, the queue will never be considered idle and therefore never deleted.
Concerning the errors, seems like your application is using too many threads which is exceeding the AWS SDK limit.