getting 429 while sending bulk request
ChiragGangwani opened this issue · 1 comments
ChiragGangwani commented
PGSync version: 2.5.1
Postgres version: 12
Elasticsearch version: latest
Redis version: latest
Python version: 3.10
Problem Description:
While updating 9000 row in db some time it post to open search but some time it through an exception of bulk request.
we have small instance.
these parameter are set
ELASTICSEARCH_CHUNK_SIZE=1000
ELASTICSEARCH_MAX_CHUNK_BYTES=3242880
ELASTICSEARCH_MAX_RETRIES=14
ELASTICSEARCH_QUEUE_SIZE=1
ELASTICSEARCH_STREAMING_BULK=True
ELASTICSEARCH_THREAD_COUNT=1
ELASTICSEARCH_TIMEOUT=320
QUERY_CHUNK_SIZE=500
Error Message (if any):
ERROR:pgsync.search_client: Exception TransportError(429, '429 Too Many Requests /_bulk')
data1-pgsync-1 | Traceback (most recent call last):
nsupegemini commented
Looks like ELASTICSEARCH_TIMEOUT is not used in bulk update
pgsync/pgsync/search_client.py
Line 188 in 9511670
@toluaina Could we please add timeout field to bulk update?