Same task executed multiple time
nehaMittal18Git opened this issue · 0 comments
Describe the bug
same task executed multiple time. and poll count shows as 2. as per analysis 2 client pod pick same task at same time from client queue and execute. please help us to know how we can avoid this issue.
Details
Conductor version: 3.2.0
Persistence implementation: Dynomite
Queue implementation:
Lock: Redis
Workflow definition:
Task definition:
Event handler definition:
To Reproduce
Steps to reproduce the behavior:
during performance testing we observer that in 4 workflow same task executed 2 times.
Please let me know which logs should I enabled to analyze the issue
Expected behavior
same task should not executed twice
Additional context
also memory reequipment increase at conductor from 4G to 8G after redis lock enabled
config
conductor.indexing.enabled=true
conductor.redis.hosts=conductor-dyno-redis-service:8102:us-east-1b
conductor.redis.maxConnsPerHost=31
conductor.redis.workflowNamespacePrefix=conductor
conductor.redis.queueNamespacePrefix=conductor_queues
conductor.app.asyncIndexingEnabled=true
conductor.app.eventQueuePollInterval=100
conductor.app.eventQueuePollCount=10
conductor.app.eventQueueLongPollTimeout=1000
conductor.app.eventSchedulerPollThreadCount=8
queues.dynomite.threads=20
conductor.workflowRepairServiceEnabled=true
conductor.redis.queuesNonQuorumPort=22122
conductor.workflow.elasticsearch.instanceType=external
conductor.elasticsearch.url=http://conductor-es-service:9200
conductor.elasticsearch.indexName=conductor-es
conductor.elasticsearch.version=6
conductor.elasticsearch.async.dao.worker.queue.size=500
conductor.elasticsearch.async.dao.max.pool.size=20
conductor.elasticsearch.username=elastic
conductor.elasticsearch.password=P@ssw0rd
conductor.redis.availabilityZone=us-east-1b
conductor.workflow-status-listener.type=archive
conductor.workflow-status-listener.archival.ttlDuration=600
conductor.workflow-status-listener.archival.delaySeconds=60
conductor.app.workflowInputPayloadSizeThreshold=20MB
conductor.app.workflowOutputPayloadSizeThreshold=20MB
conductor.app.taskInputPayloadSizeThreshold=20MB
conductor.app.taskOutputPayloadSizeThreshold=20MB
conductor.workflow-execution-lock.type=redis
conductor.app.workflow-execution-lock-enabled=true
conductor.redis-lock.serverType=single
conductor.redis-lock.serverAddress=redis://conductor-dyno-redis-service:22122
conductor.redis-lock.namespace=conductor-locking