jstedfast/MailKit

Server disconnect when using MoveTo Trash folder on Gmail

RafaTeo opened this issue · 11 comments

RafaTeo commented

Describe the bug
When attempting to move an email from the Inbox to the Trash in Gmail set to English(US) throws ImapProtocolException. This issue does not appear when the language is set to English(UK) or other languages.

Platform (please complete the following information):

  • OS: Windows
  • .NET Runtime: .Net Framework 4.8
  • .NET Framework: .Net Framework 4.8
  • MailKit Version: 4.7.1.1

Exception
"The IMAP server has unexpectedly disconnected."
at MailKit.Net.Imap.ImapStream.ReadAhead(Int32 atleast, CancellationToken cancellationToken)
at MailKit.Net.Imap.ImapStream.ReadToken(String specials, CancellationToken cancellationToken)
at MailKit.Net.Imap.ImapStream.ReadToken(CancellationToken cancellationToken)
at MailKit.Net.Imap.ImapEngine.ReadToken(CancellationToken cancellationToken)
at MailKit.Net.Imap.ImapCommand.Step()
at MailKit.Net.Imap.ImapEngine.Iterate()
at MailKit.Net.Imap.ImapEngine.Run(ImapCommand ic)
at MailKit.Net.Imap.ImapFolder.MoveTo(IList`1 uids, IMailFolder destination, CancellationToken cancellationToken)
at MailKit.MailFolder.MoveTo(UniqueId uid, IMailFolder destination, CancellationToken cancellationToken)

Expected behavior
Move email from Inbox to Trash folder.

Code Snippets

    internal class Program
    {
        const string EMAIL = "<your gmail here>";
        const string PASSWORD = "<your app password here>";
        const string TITLE = "Email Subject To Be Moved to Trash";
        static NetworkCredential credentials = new NetworkCredential(EMAIL, PASSWORD);
        static void Main(string[] args)
        {
            Send();
            var query = MailKit.Search.SearchQuery.SubjectContains(TITLE);
            using (var client = new ImapClient())
            {
                client.Connect("imap.gmail.com", 993, SecureSocketOptions.SslOnConnect);
                client.Authenticate(credentials);
                client.Inbox.Open(FolderAccess.ReadWrite);
                client.Inbox.Check();
                var trashFolder = client.GetFolder(MailKit.SpecialFolder.Trash);
                var uids = client.Inbox.Search(query);
                foreach (var uid in uids)
                   client.Inbox.MoveTo(uid, trashFolder);                
            }            
        }

        static void Send()
        {
            var mail = new MailboxAddress(EMAIL, EMAIL);
            var msg = new MimeMessage();
            msg.To.Add(mail);
            msg.From.Add(mail);
            msg.Subject = TITLE;
            msg.Body = (new BodyBuilder() { TextBody = EMAIL + Environment.NewLine + "DELETE-ME" }).ToMessageBody();
            using (var smtp = new MailKit.Net.Smtp.SmtpClient())
            {
                smtp.Connect("smtp.gmail.com", 465, SecureSocketOptions.SslOnConnect);
                smtp.Authenticate(credentials);
                smtp.Send(msg);
            }
        }
    }

Protocol Logs
Connected to imaps://imap.gmail.com:993/
S: * OK Gimap ready for requests from 186.195.238.180 e40mb58663833oow
C: A00000000 CAPABILITY
S: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER
S: A00000000 OK Thats all she wrote! e40mb58663833oow
C: A00000001 AUTHENTICATE PLAIN ********
S: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
S: A00000001 OK *****@gmail.com authenticated (Success)
C: A00000002 NAMESPACE
S: * NAMESPACE (("" "/")) NIL NIL
S: A00000002 OK Success
C: A00000003 LIST "" "INBOX" RETURN (SUBSCRIBED CHILDREN)
S: * LIST (\HasNoChildren \Subscribed) "/" "INBOX"
S: A00000003 OK Success
C: A00000004 LIST (SPECIAL-USE) "" "" RETURN (SUBSCRIBED CHILDREN)
S: * LIST (\All \HasNoChildren \Subscribed) "/" "[Gmail]/All Mail"
S: * LIST (\Drafts \HasNoChildren \Subscribed) "/" "[Gmail]/Drafts"
S: * LIST (\HasNoChildren \Sent \Subscribed) "/" "[Gmail]/Sent Mail"
S: * LIST (\HasNoChildren \Junk \Subscribed) "/" "[Gmail]/Spam"
S: * LIST (\Flagged \HasNoChildren \Subscribed) "/" "[Gmail]/Starred"
S: * LIST (\HasNoChildren \Subscribed \Trash) "/" "[Gmail]/Trash"
S: A00000004 OK Success
C: A00000005 LIST "" "[Gmail]" RETURN (SUBSCRIBED CHILDREN)
S: * LIST (\HasChildren \NonExistent \Subscribed) "/" "[Gmail]"
S: A00000005 OK Success
C: A00000006 SELECT INBOX (CONDSTORE)
S: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
S: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing *)] Flags permitted.
S: * OK [UIDVALIDITY 594757023] UIDs valid.
S: * 1174 EXISTS
S: * 0 RECENT
S: * OK [UIDNEXT 46898] Predicted next UID.
S: * OK [HIGHESTMODSEQ 39782651]
S: A00000006 OK [READ-WRITE] INBOX selected. (Success)
C: A00000007 CHECK
S: A00000007 OK Success
C: A00000008 UID SEARCH RETURN (ALL) SUBJECT "ABCDEFGHIJKLMN_- 0123456789"
S: * ESEARCH (TAG "A00000008") UID ALL 46897
S: A00000008 OK SEARCH completed (Success)
C: A00000009 UID MOVE 46897 "[Gmail]/Trash"

jstedfast commented

Based on the stacktrace and the log, it looks like MailKit's ImapClient was waiting for a response from the server but the server dropped the connection for some reason.

GMail is an odd IMAP server because its Trash folder is really a "virtual folder" (funny story, but the first commercial email client I worked on back in the early 2000's pioneered the concept of virtual folders (as far as I'm aware) and it was pretty cool).

Anyway, the way that GMail's folders work is that the only real folder is the "All Mail" folder, all of the rest of the folders are virtual, meaning that they query the "All Mail" folder for messages that have a label matching the name of the virtual folder ("Inbox" for INBOX, "[Gmail]/Trash" for the Trash folder, etc).

The point I'm making is that another way of "moving" messages into the Trash folder is to mark them as deleted:

foreach (var uid in uids)
    client.Inbox.AddFlags(uid, MessageFlags.Deleted, true);

I'm going to assume that you intend to do more processing than just delete a message, but if not, you can be far more efficient using the following 1 line of code to delete all of the search results[1]:

client.Inbox.AddFlags(uids, MessageFlags.Deleted, true);

Depending on your GMail settings (you can configure the IMAP behavior in GMail's web interface), you might also need to call the following method to remove deleted messages from the Inbox:

client.Inbox.Expunge(uids);

If you haven't changed any IMAP settings in GMail, then I think Deleting them is all you need to do.

Hope that helps.

Notes:

  1. Any API that can operate on a single UID can also operate on a set of UIDs (except GetMessage() because messages are so heavyweight that I did not add such an API in order to discourage consuming huge amounts of memory).
RafaTeo commented

Adding the deleted flag does not move the email to the trash folder, it deletes the email. The strange thing is that it is possible to move it to the trash folder if the email is configured with the English (UK) language, in which case the folder appears with the name "[Gmail]/bin".

jstedfast commented
  1. Open GMail's web UI in your browser.
  2. Click on the Gear icon in the upper right corner.
  3. Click "See All Settings".
  4. Click "Forwarding and IMAP/POP"
  5. In the "IMAP Access" section, set the following options:
  • Enable IMAP

When I mark a message in IMAP as deleted:

  • Auto-Expunge on - Immediately update the server. (default)
  • Auto-Expunge off - Wait for the client to update the server.

When a message is marked as deleted and expunged from the last visible IMAP folder:

  • Archive the message (default)
  • Move the message to the Trash
  • Immediately delete the message forever

If you set these option values, it will behave the closest to real IMAP.

RafaTeo commented

Is there a way to never delete the email?
I want my application to always just send the email to the Trash "folder" without relying on the Gmail configuration. Actually deleting the email is too dangerous.

jstedfast commented

Well, I would have suggested using MoveTo(), haha, but apparently that doesn't work! 😢

tobyb123 commented

Using the AddFlags() fixed our issues.

I have a scheduled task running which runs once an hour to clear out some emails we get through. Which has worked well for quite a while now and this was using the MoveTo() method.

But a few weeks ago we randomly just started getting errors sent to our dashboard saying the IMAP server unexpectedly disconnected.

We had a backup on the fail to use the AddFlags() function which is working fine.

Nothing had changed our end so I wonder if Gmail has altered something on their end.

RafaTeo commented

The problem with AddFlags is that it relies on the Gmail configuration. It might expunge the message or just send it to the trash folder. I want it to always send to the trash folder.

Using the AddFlags() fixed our issues.

I have a scheduled task running which runs once an hour to clear out some emails we get through. Which has worked well for quite a while now and this was using the MoveTo() method.

But a few weeks ago we randomly just started getting errors sent to our dashboard saying the IMAP server unexpectedly disconnected.

We had a backup on the fail to use the AddFlags() function which is working fine.

Nothing had changed our end so I wonder if Gmail has altered something on their end.

The problem with AddFlags() is that it relies on the Gmail configuration. It might expunge the message or just send it to the trash folder. I want it to always send to the trash folder.

tobyb123 commented

Did you manage to find a solution to get MoveTo() to work?

I don't know why our code just stopped working. For a long time we have had no issues then all of a sudden it was generating errors.

Maybe there is something you can change in the gmail settings but I couldn't see much as to why.

jstedfast commented

I don't know why our code just stopped working. For a long time we have had no issues then all of a sudden it was generating errors.

Just to be clear, when it went from working to not working, did you update MailKit at all? Or did the server just stop working?

I can't think of any MailKit changes to the IMAP code that would affect this in any recent releases, but I figured I'd verify.

If this is indeed a bug in GMail, hopefully Google fixes their MOVE bug sooner rather than later.

Surely people must be hitting this using other clients as well.

RafaTeo commented

I noticed that if you change the language to English(UK), or any other, it works.
There might be something related to the "trash" word.

tobyb123 commented

@jstedfast I am using version 3.6.0. No updates were made to MailKit prior to the errors coming or to the program which was running off the scheduled task. I think this is due to a change Gmail has made on their end.

I am just verifying that changing the language to English(UK) works. Just waiting for more emails to come through to test with.

I'll let you know if I find anything.

  • 👍1