telstra/open-kilda

Add kafka chunking for flow validation process.

IvanChupin opened this issue · 1 comments

We have a problem during validating some flow, there might be the following errors:

Could not validate flow: Timeout for waiting response on command CommandMessage{timestamp=1717430534700, correlation_id=47fde2e0-94fc-4d48-9f8d-67ab1d55f517 : 91bf3693-79da-49d5-a951-858cf4fa020a : akupko@mirantis.com_4d9034df-6449-4bb3-bfe5-f97c66c4213e, cookie=null, destination=null, payload=DumpRulesForFlowHsRequest(switchId=****)}",
  "error-description": "Error in SpeakerWorker"
}

Also there is an error in the logs:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 1297950 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

While investigating the problem we found out that during flow validation the Floodlight modules tries to send the too big entities with the rules toward the kilda floodlight-topology via the kafka topic. As result kafka client logs the errors about too large messages.
In order to address this issue we agreed to split entities before sending it via the kafka topic.