Wrong order of results

<iq id="b74c8082-54a4-4770-b5ef-fa5658665f10:sendIQ" to="mops@mydomain.local" type="set" from="mops@mydomain.local/jsxc-b69642ac"><query xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019"><x xmlns="jabber:x:data" type="submit"><field type="hidden" var="FORM_TYPE"><value>urn:xmpp:mam:2</value></field><field var="with"><value>admin@mydomain.local</value></field></x><set xmlns="http://jabber.org/protocol/rsm"><max>20</max><before/></set></query></iq>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3263"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:57.372Z"/><message xmlns="jabber:client" id="131f9520-ac13-4cff-b65e-baa8d28c7b90" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>F</body><origin-id xmlns="urn:xmpp:sid:0" id="131f9520-ac13-4cff-b65e-baa8d28c7b90"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3262"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:55.759Z"/><message xmlns="jabber:client" id="97b11150-d4d8-4ed3-b5c0-955954d3f91f" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>E</body><origin-id xmlns="urn:xmpp:sid:0" id="97b11150-d4d8-4ed3-b5c0-955954d3f91f"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3261"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:54.186Z"/><message xmlns="jabber:client" id="b133b07c-9c8e-4320-99a2-fadff47dd281" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>D</body><origin-id xmlns="urn:xmpp:sid:0" id="b133b07c-9c8e-4320-99a2-fadff47dd281"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3260"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:52.670Z"/><message xmlns="jabber:client" id="49b15b84-b6de-4285-b100-01bff702f0ab" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>C</body><origin-id xmlns="urn:xmpp:sid:0" id="49b15b84-b6de-4285-b100-01bff702f0ab"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3259"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:51.217Z"/><message xmlns="jabber:client" id="2e9ba264-4f01-4de8-830f-f676b33c7746" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>B</body><origin-id xmlns="urn:xmpp:sid:0" id="2e9ba264-4f01-4de8-830f-f676b33c7746"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<message to="mops@mydomain.local/jsxc-b69642ac"><result xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019" id="3258"><forwarded xmlns="urn:xmpp:forward:0"><delay xmlns="urn:xmpp:delay" stamp="2021-01-13T20:11:50.371Z"/><message xmlns="jabber:client" id="7ac40b3d-d209-461f-ac02-341414e9d52c" to="admin@mydomain.local" type="chat" from="mops@mydomain.local/jsxc-99752c3c"><body>A</body><origin-id xmlns="urn:xmpp:sid:0" id="7ac40b3d-d209-461f-ac02-341414e9d52c"></origin-id><request xmlns="urn:xmpp:receipts"></request><active xmlns="http://jabber.org/protocol/chatstates"></active><markable xmlns="urn:xmpp:chat-markers:0"></markable></message></forwarded></result></message>

<iq type="result" id="b74c8082-54a4-4770-b5ef-fa5658665f10:sendIQ" from="mops@mydomain.local" to="mops@mydomain.local/jsxc-b69642ac"><fin xmlns="urn:xmpp:mam:2" queryid="e01b5704-e2dd-488b-a598-2579c81a1019"><set xmlns="http://jabber.org/protocol/rsm"><first>3263</first><last>3258</last><count>6</count></set></fin></iq>

i noticed the id (3263) in result iq the in "first" is higher than the id (3258) in "last"

<set xmlns="http://jabber.org/protocol/rsm" > <first>3263</first><last>3258</last><count>6</count></set>

I confirm the problem.

Does this affect anything client side?
Which version of Monitoring is this using? Which version of Openfire?

RSM doesn't require sequential ordering of IDs, just IDs that Openfire (or in this case Monitoring) treats as unique.
I suppose it's worth confirming - are these results actually in the wrong order? The messages were indeed A -> F, not F -> A? :)

RSM doesn't require sequential ordering of IDs, just IDs that Openfire (or in this case Monitoring) treats as unique.
I suppose it's worth confirming - are these results actually in the wrong order? The messages were indeed A -> F, not F -> A? :)

Xep 313 demands saving messages in same order as they were written in chat!
Furthermore we should not rely on clients implementation to catch wrong orders of messages...
OF 4.6.1 and MAM 2.2.

XEP 0313:

3.1 Order of messages
Order within the archive MUST be preserved, where the order of messages is the same as the order that the client originally received them (or would have received them if online). Throughout this document the term 'chronological order' refers to this order, however implementors should take care not to rely on timestamps alone for ordering messages, as multiple messages may share the same timestamp.

Does this affect anything client side?
Which version of Monitoring is this using? Which version of Openfire?

Jsxc has trouble to request further messages because jsxc uses the id of first tag to know where to start next request as it is described in
xep 313 5. Archive metadata
When planning a query, a client may wish to learn the current state of the archive. This includes information about the first/last entries in the archive.

I'm not suggesting ordering is ignored. I'm saying RSM allows labelling of messages however you fancy. IDs don't need to be sequential numbers - they can be randomised or words, or any implementation at all, so long as the uniqueness and permanence is guaranteed.

As such, seeing an apparently out of order RSM result IQ is not a guarantee of having out of order results.

Those results do appear out of order, however.

I'm not suggesting ordering is ignored. I'm saying RSM allows labelling of messages however you fancy. IDs don't need to be sequential numbers - they can be randomised or words, or any implementation at all, so long as the uniqueness and permanence is guaranteed.

As such, seeing an apparently out of order RSM result IQ is not a guarantee of having out of order results.

Those results do appear out of order, however.

Sorry, i know that theese ids dont have to be numbers... i mean the result sets order... in the log above you see "last" message is on top and first at the end... but clients will normally process messages from top to end as they will arrive

My suggestion: check last tags content with first message id and do the same with content of first tag with the id of the last message ... if they are both the same the list is reversed and should be reversed before sending to the client...

OF 4.6.1 and MAM 2.2

Right, I've worked out how this is happening.

The query contained <before/>.

This triggers backwards paging:

openfire-monitoring-plugin/src/java/com/reucon/openfire/plugin/archive/xep0059/XmppResultSet.java

Lines 40 to 43 in 64053dd

    
           // If 'before' is set it's an indication that backwards paging is desired, even if it's empty. 
        
           if ( after == null && setElement.element( "before" ) != null ) { 
        
               pagingBackwards = true; 
        
           }

Which determines the order of the SQL result:

openfire-monitoring-plugin/src/java/com/reucon/openfire/plugin/archive/impl/PaginatedMessageDatabaseQuery.java

Line 276 in 64053dd

sql += " ORDER BY sentDate " + (isPagingBackwards ? "DESC" : "ASC");

I've found the commit that introduced this new behaviour, and I'm looking into how to keep the good things that brought, but also restore the old behaviour.

👍

	// If 'before' is set it's an indication that backwards paging is desired, even if it's empty.
	if ( after == null && setElement.element( "before" ) != null ) {
	pagingBackwards = true;
	}