tomolimo/mailanalyzer

Wrong behaviour?

Closed this issue · 5 comments

andrepetinga commented

Hello!
I have a question regarding the default behaviour of the plugin. You wrote that:

"It creates a new table in the DB with the purpose of storing email guid (generated by email servers) in order to be able (if possible) to match emails in mailgate which have been sent using 'CC' and 'Reply to all'. It cannot keeps track of forwarded emails and replies to them."

But, I think the plugin is only looking for the original email and matching the subject.
The test we made:

  • Send an email from my Gmail to my institucional email and to another coworker, using BCC.
    image

  • Each one of us got the email in Outlook. Then I send the received email, using "Forward" to our support email
    image

  • Note that the subject was not changed, the only thing was Outlook adding the "FW:".

  • Other coworker did the same, just writing other text on the body and of course, with is email signature.

  • GLPI created my ticket (I send it first) and then the other email did not create a new ticket, it was added as a followup to my ticket.
    image

  • In addition to this, the other coworker did exactly the same thing again, but this time changing the subject. And in this case, a new ticket was created.

In our understanding, this is a wrong behaviour, because the original email was sent to multiple subjects and then each one forward it to a support email, not using CC or Reply to all, because support email was not in the original email.
So it seems that the plugin is actually keeping track of Forwarded emails.

Can you confirm this?

This is important to us, because we work on a School and some departments send bulk emails to several students with same subject, and if any has a question, they can reply to the support email to make the question. The problem ocurred when we discovered that, in one of these communications, one student replied and a ticket was created. When other students replied, those replies did not create new tickets, followups were added to the original ticket :(

Sorry for all the text...

tomolimo commented

Hello @andrepetinga

To understand the behavior of the mailanalyzer plugin, you first have to know that email clients and email servers are not behaving the same way. One client may behave one way, and another client may behave another way.

Then, to explain how is behaving mailanalyzer:

  1. It's getting some information from the email headers, information that may come from email clients or email servers: the so called "Thread-Index" and the "Message-ID". If you send an email to GLPI, you'll see these two info in the DB:
    image
    For example, on this screencopy, you may see on the rows with id 5 and 1 are from a Thread-Index header param, and the values with the < and > are from a Message-Id header param. You may find both in emails, you may find only the Message-Id (one more time, it depends on the clients and servers). In this example, there were both info for ticket 368042.
  2. For the replies or the forward, how are they detected? Usually, the email clients will add the original email identifier in the header of the reply, and in the header of the forward. They are adding the original Thread-Index or/and the original Message-Id in the header of the new email. Then when MailAnalyzer plugin will receive one of these emails, it will check if the Thread-Index or the Message-Id are already existing in the DB, and if yes it will add the emails as followups into the existing ticket.
  3. But sometimes the email clients will not send back the references (in case of replies) or they will create new references
  4. And the Thread-Index will be re-computed at any time by the email clients if the original subject is changed! Not recomputed by the Re: (or Fwd:), but recomputed if the user added a char to the end of the subject (for example), and then when the MailAnalyzer will receive this email it will not be able to add it to an existing ticket, as the references are not matching an existing ticket.
  5. Usually, when you forward an email, the references to the original email are lost. But we may imagine that some email clients are keeping them in the forwarded emails. If they are kept, then MailAnalyzer will match the emails to an existing ticket. If not, then it will allow te creation of a new ticket.
  6. MailAnalyzer is NOT using the subject of the emails in order to match them to existing tickets. It is ONLY using the internal Thread-Index and/or Message-Id.
  7. You may check the headers of the received emails to see if the the Thread-Index is present or not (and if this Thread-Index is the same than the original) from the replies of the forwarded emails.

I hope this clarify the MailAnalyzer behavior, if not don't hesitate to ask me.

Thank you,
Regards
Tomolimo

andrepetinga commented

If I understood you correctly, then looking at this:
image

We can see a ticket with 1 "Thread-Index" (line) 1 and 3 "Message-ID" (lines 2, 3 and 4).

This is the message I've sent:

image

And this is my coworker message:

image

So you are saving the Message-ID from each message and, in adition, the "In-Reply-To" reference. That is why we have only 2 emails sent, but 3 lines inserted. Correct?

But looking at this messages, the "Thread-Index" is different! So, how are they getting stuck on the same ticket?

Other thing, the "Thread-Index" in the Database is "a13fa0b95325ce4cbbb86ccc45f2613f" which is not matching with anyone of the id's from the email header.

In the email I received (sent from Gmail to our corporate email), I found the "Message-ID" which is the same as the "In-Reply-To" found on the 2 emails sent to GLPI. But I cannot find "Thread-Index" on this message.
Also opening the sent email in Gmail, I can only find the "Message-ID", not the "Thread-Index".

tomolimo commented

Hello @andrepetinga

So you are saving the Message-ID from each message and, in adition, the "In-Reply-To" reference. That is why we have only 2 emails sent, but 3 lines inserted. Correct?

Yes

But looking at this messages, the "Thread-Index" is different! So, how are they getting stuck on the same ticket?

Other thing, the "Thread-Index" in the Database is "a13fa0b95325ce4cbbb86ccc45f2613f" which is not matching with anyone of the id's from the email header.

Here is the explanation: the Thread-Index is a complex computation (you may check here the complete decoding of the Thread-Index: http://msdn.microsoft.com/en-us/library/ee202481%28v=exchg.80%29.aspx), and we store in the DB not really the Thread-Index, but a part of it named PtypGuid (GUID (16 bytes): A PtypGuid type ([MS-OXCDATA] section 2.11.1) that is generated for each new conversation thread), and this GUID is computed with this formula: $messages_id[] = bin2hex(substr(base64_decode($message->threadindex), 6, 16 )); in the Thread-Index you not only the thread index of the original message, but also an index of the current message and some other info.

Try this PHP code:

$thread_index = "AQHW6NdDoT+guVMlzky7uGzMRfJhP6oj3OEQ";
echo bin2hex(substr(base64_decode($thread_index), 6, 16 ));

echo "\r\n";

$thread_index = "AQHW6NdD1jfUP1rHmEC/CFoLI/Wd7aoj3OAw";
echo bin2hex(substr(base64_decode($thread_index), 6, 16 ));

Here are the results:

a13fa0b95325ce4cbbb86ccc45f2613f 
d637d43f5ac79840bf085a0b23f59ded

a13fa0b95325ce4cbbb86ccc45f2613f -> same GUID than in DB
d637d43f5ac79840bf085a0b23f59ded -> GUIDs are differents, but in the email you can find the same Message-Id in the "References" field than the first email. so it means that these emails are matching the same tickets.

In the email I received (sent from Gmail to our corporate email), I found the "Message-ID" which is the same as the "In-Reply-To" found on the 2 emails sent to GLPI. But I cannot find "Thread-Index" on this message.

As explained, email clients are managing these indexes like they want...

Thank you,
Regards,
Tomolimo

andrepetinga commented

Ok, thanks for the answers.

tomolimo commented

you're welcome 😃