Task is marked as "TIME_OUT" after an interval much larger than the time specified in "responseTimeoutSeconds"

[caubechankiu]

Describe the bug
Task is marked as "TIME_OUT" after an interval much larger than the time specified in "responseTimeoutSeconds"

Details
Conductor version: 3.18.8
Persistence implementation: MySQL
Workflow definition:

{
  "createTime": 1693020291627,
  "updateTime": 1693020540567,
  "accessPolicy": {},
  "name": "simple_workflow",
  "description": "A Simple Workflow",
  "version": 1,
  "tasks": [
    {
      "name": "simple_task",
      "taskReferenceName": "simple_task",
      "inputParameters": {
        "hellomsg": "${workflow.input.hellomsg}"
      },
      "type": "SIMPLE",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false
    }
  ],
  "inputParameters": [
    "hellomsg"
  ],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "example@email.com",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

Task definition:

{
  "name": "simple_task",
  "description": "A Simple Task",
  "retryCount": 3,
  "timeoutSeconds": 60,
  "inputKeys": [
    "hellomsg"
  ],
  "outputKeys": [],
  "timeoutPolicy": "TIME_OUT_WF",
  "retryLogic": "FIXED",
  "retryDelaySeconds": 10,
  "responseTimeoutSeconds": 10,
  "inputTemplate": {},
  "rateLimitPerFrequency": 0,
  "rateLimitFrequencyInSeconds": 1,
  "ownerEmail": "example@email.com",
  "backoffScaleFactor": 1
}

To Reproduce
Steps to reproduce the behavior:

1. Call api to start new workflow

curl -X 'POST' \
  'http://localhost:8080/api/workflow/simple_workflow?priority=0' \
  -H 'accept: text/plain' \
  -H 'Content-Type: application/json' \
  -d '{
  "hellomsg": "World"
}'

response

56f869dc-617a-469c-bbe7-635e1fd823b6

2. Call api to poll task

curl -X 'GET' \
  'http://localhost:8080/api/tasks/poll/simple_task' \
  -H 'accept: */*'

response

{
  "taskType": "simple_task",
  "status": "IN_PROGRESS",
  "inputData": {
    "hellomsg": "World"
  },
  "referenceTaskName": "simple_task",
  "retryCount": 0,
  "seq": 1,
  "pollCount": 1,
  "taskDefName": "simple_task",
  "scheduledTime": 1693025902375,
  "startTime": 1693026064953,
  "endTime": 0,
  "updateTime": 1693026064954,
  "startDelayInSeconds": 0,
  "retried": false,
  "executed": false,
  "callbackFromWorker": true,
  "responseTimeoutSeconds": 10,
  "workflowInstanceId": "56f869dc-617a-469c-bbe7-635e1fd823b6",
  "workflowType": "simple_workflow",
  "taskId": "91646b0c-1751-427a-bd7e-f17e87784a35",
  "callbackAfterSeconds": 0,
  "outputData": {},
  "workflowTask": {
    "name": "simple_task",
    "taskReferenceName": "simple_task",
    "inputParameters": {
      "hellomsg": "${workflow.input.hellomsg}"
    },
    "type": "SIMPLE",
    "startDelay": 0,
    "optional": false,
    "taskDefinition": {
      "createTime": 1693025612726,
      "createdBy": "",
      "accessPolicy": {},
      "name": "simple_task",
      "description": "A Simple Task",
      "retryCount": 3,
      "timeoutSeconds": 60,
      "inputKeys": [
        "hellomsg"
      ],
      "outputKeys": [],
      "timeoutPolicy": "TIME_OUT_WF",
      "retryLogic": "FIXED",
      "retryDelaySeconds": 10,
      "responseTimeoutSeconds": 10,
      "inputTemplate": {},
      "rateLimitPerFrequency": 0,
      "rateLimitFrequencyInSeconds": 1,
      "ownerEmail": "example@email.com",
      "backoffScaleFactor": 1
    },
    "asyncComplete": false
  },
  "rateLimitPerFrequency": 0,
  "rateLimitFrequencyInSeconds": 1,
  "workflowPriority": 0,
  "iteration": 0,
  "subworkflowChanged": false,
  "taskDefinition": {
    "createTime": 1693025612726,
    "createdBy": "",
    "accessPolicy": {},
    "name": "simple_task",
    "description": "A Simple Task",
    "retryCount": 3,
    "timeoutSeconds": 60,
    "inputKeys": [
      "hellomsg"
    ],
    "outputKeys": [],
    "timeoutPolicy": "TIME_OUT_WF",
    "retryLogic": "FIXED",
    "retryDelaySeconds": 10,
    "responseTimeoutSeconds": 10,
    "inputTemplate": {},
    "rateLimitPerFrequency": 0,
    "rateLimitFrequencyInSeconds": 1,
    "ownerEmail": "example@email.com",
    "backoffScaleFactor": 1
  },
  "queueWaitTime": 162578,
  "loopOverTask": false
}

3. Go to http://localhost:5000/execution/56f869dc-617a-469c-bbe7-635e1fd823b6?task=(id%3A%2791646b0c-1751-427a-bd7e-f17e87784a35%27)
4. See result

Expected behavior
The duration of the task will be around 10 seconds as specified in "responseTimeoutSeconds"

Actual Behavior
The duration of the task is 24 seconds.
I have tried several times but the result is still the same, the duration of the task is around 20-30 seconds

Screenshots

1. The task result

2. The task definition

[prajwalbandak]

Hey @caubechankiu , will love to work on this issue, Can you assign this one to me?