toluaina/pgsync

getting 429 while sending bulk request

ChiragGangwani opened this issue · 1 comments

PGSync version: 2.5.1

Postgres version: 12

Elasticsearch version: latest

Redis version: latest

Python version: 3.10

Problem Description:
While updating 9000 row in db some time it post to open search but some time it through an exception of bulk request.

we have small instance.
these parameter are set

ELASTICSEARCH_CHUNK_SIZE=1000
ELASTICSEARCH_MAX_CHUNK_BYTES=3242880
ELASTICSEARCH_MAX_RETRIES=14
ELASTICSEARCH_QUEUE_SIZE=1
ELASTICSEARCH_STREAMING_BULK=True
ELASTICSEARCH_THREAD_COUNT=1
ELASTICSEARCH_TIMEOUT=320
QUERY_CHUNK_SIZE=500

Error Message (if any):

ERROR:pgsync.search_client: Exception TransportError(429, '429 Too Many Requests /_bulk')
data1-pgsync-1  | Traceback (most recent call last):

Looks like ELASTICSEARCH_TIMEOUT is not used in bulk update

for _ in self.parallel_bulk(

@toluaina Could we please add timeout field to bulk update?