taverntesting/tavern

Multiple MQTT Responses to Single Request - Support out of order messages

RFRIEDM-Trimble opened this issue · 7 comments

As referenced in #385, which is currently implmented in the feature-2.0 branch, the MQTT messages can arrive out of order, as MQTT provides no order guarantee. While the test infrastructure supports a YAML spec that allow a list of response objects, the test only passes if the messages are received in the same order as the test response expects, which may or may not happen.

The scope of this issue would be to

  • Design any modifications to the YAML spec to specify if order is required or not to pass
  • Implement necessary modifications to the logic, such as starting a thread to accumulate responses, to support out-of-order
  • Create a system test that can check out of order responses
  • Verify the system test passes with out of order responses
  • Link necessary MQTT spec documentation in support of "order guarantee" in MQTT

I neglected to realise that could be the case with MQTT, it probably just needs to start a separated thread for each response and collect it at the end.

Originally posted by @michaelboulton in #385 (comment)

I did look at this before (See #789) and the issue was that I'd made it so that responses were handled separately by the core tavern code, when it should have been handled by the plugin code itself. I have fixed that now in the basic case. There were a few different ways of doing it, but the way done in that PR seemed like it made the most sense to be able to extend it to then be able to handle messages with different QoS arriving in specific orders

That has been pushed as 2.0.0alpha9 (This should probably move out of 'alpha' now...)

That has been pushed as 2.0.0alpha9 (This should probably move out of 'alpha' now...)

Just tested it out, it appears to work to handle out of order messages, however if a test fails, the output is still assuming one message and one response per topic. I don't know how to debug it other than go in the broker and enable verbose logging or set up another subscriber to view all messages.

For example, in a test, say I send message A, and expect B, C on topic T, but get A,B,D instead
If it doesn't ever receive C, then it follows this format with only one response.

----------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------
Expected: {'topic': 'T', 'json': C, 'timeout': 10, 'qos': 0}
Response: b'{D}'

I would expect the Resonse to list out ALL the messages it received on that topics it's listening on for the test. Like so:

Expected: {'topic': 'T', 'json': C, 'timeout': 10, 'qos': 0}
Responses: b'{A}', b'{B}', b'{D}'

I think part of the issue is is being too greedy in getting the responses, ie it will sometimes ignore extra messages it receives once it has received the ones .

Also I don't know if I'm misremembering something, but what is the output you've given from? I don't think Tavern prints like that, it prints

E   tavern._core.exceptions.TestFailError: Test 'step 1 - ping/pong' failed:
E   - Expected '3ty4' on topic '/device/123/status/response' but no such message received
E   - Expected 'sdf' on topic '/device/123/status/response' but no such message received

Member

I edited the json output to remove proprietary info, but the actual log message is the same. I'll take the action to create minimum reproducible bug in the system integration tests (likely next week) to isolate this issue.

I believe this should now be fixed in 2.0.0alpha17

Going to close this as it should be fixed in 2.0