Netflix/conductor

Same task executed multiple time

nehaMittal18Git opened this issue · 0 comments

Describe the bug
same task executed multiple time. and poll count shows as 2. as per analysis 2 client pod pick same task at same time from client queue and execute. please help us to know how we can avoid this issue.
Details
Conductor version: 3.2.0
Persistence implementation: Dynomite
Queue implementation:
Lock: Redis
Workflow definition:
Task definition:
Event handler definition:

To Reproduce
Steps to reproduce the behavior:
during performance testing we observer that in 4 workflow same task executed 2 times.
Please let me know which logs should I enabled to analyze the issue

Expected behavior
same task should not executed twice

Screenshots
image

Additional context
also memory reequipment increase at conductor from 4G to 8G after redis lock enabled

config
conductor.indexing.enabled=true
conductor.redis.hosts=conductor-dyno-redis-service:8102:us-east-1b
conductor.redis.maxConnsPerHost=31

conductor.redis.workflowNamespacePrefix=conductor

conductor.redis.queueNamespacePrefix=conductor_queues

conductor.app.asyncIndexingEnabled=true
conductor.app.eventQueuePollInterval=100
conductor.app.eventQueuePollCount=10
conductor.app.eventQueueLongPollTimeout=1000
conductor.app.eventSchedulerPollThreadCount=8
queues.dynomite.threads=20

conductor.workflowRepairServiceEnabled=true


conductor.redis.queuesNonQuorumPort=22122

conductor.workflow.elasticsearch.instanceType=external

conductor.elasticsearch.url=http://conductor-es-service:9200

conductor.elasticsearch.indexName=conductor-es
conductor.elasticsearch.version=6
conductor.elasticsearch.async.dao.worker.queue.size=500
conductor.elasticsearch.async.dao.max.pool.size=20
conductor.elasticsearch.username=elastic
conductor.elasticsearch.password=P@ssw0rd

conductor.redis.availabilityZone=us-east-1b

conductor.workflow-status-listener.type=archive
conductor.workflow-status-listener.archival.ttlDuration=600
conductor.workflow-status-listener.archival.delaySeconds=60

conductor.app.workflowInputPayloadSizeThreshold=20MB
conductor.app.workflowOutputPayloadSizeThreshold=20MB
conductor.app.taskInputPayloadSizeThreshold=20MB
conductor.app.taskOutputPayloadSizeThreshold=20MB

conductor.workflow-execution-lock.type=redis
conductor.app.workflow-execution-lock-enabled=true

conductor.redis-lock.serverType=single
conductor.redis-lock.serverAddress=redis://conductor-dyno-redis-service:22122

conductor.redis-lock.namespace=conductor-locking