bamthomas/aioimaplib

fetch error

cyberlis opened this issue · 13 comments

cyberlis commented

I want fetch email with this parts '(UID FLAGS INTERNALDATE)'
status, data = await self.imap_client.fetch('3', '(UID FLAGS INTERNALDATE)')
I get wrong response Response(result='OK', lines=['64', 'Success'])

I think this heppens because of
fetch_message_with_literal_data_re = re.compile(rb'\* [0-9]+ FETCH [\w \\\[\]\(\)]+ \{(?P<size>\d+)\}\r\n')

I don't have size field in my response
b'* 64 FETCH (UID 328 INTERNALDATE "14-Oct-2012 17:49:28 +0000" FLAGS (\\Seen))\r\nCGOJ66 OK Success\r\n'

bamthomas commented

Hi @cyberlis. thank you for your report. Your hypothesis seems the good one. I'm going to have a look.

bamthomas commented

humm it has matched
message_data_without_literal_re = re.compile(r'[0-9]+ ((FETCH)|(EXPUNGE))([\w \(\)]+)?')

And then later :
text, command = '64 FETCH (UID 328 INTERNALDATE "14-Oct-2012 17:49:28 +0000" FLAGS (\\Seen))'.split()[0:2]

Then text=64 command=FETCH
We have not the end of the line.
Then on the next line you have the 'Success' that's what you can see in your list (two lines)

Is it ok if the line is added ?
So you would have :
Response(result='OK', lines=['64 FETCH (UID 328 INTERNALDATE "14-Oct-2012 17:49:28 +0000" FLAGS (\\Seen))', 'Success'])

bamthomas commented

Could you test the master version ?
If it is ok for you, I will upload a new release.

cyberlis commented

ok. this fixed this case
status, data = await self.imap_client.fetch('3', '(UID FLAGS INTERNALDATE)')

but i have another case

data = await self.imap_client.fetch(
            '1',
            '(UID FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)])'
        )

I am getting this response

<class 'aioimaplib.aioimaplib.Response'> Response(result='OK', lines=['1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}', 'Success'])

BODY Header fields are missing

Debug output

2017-03-24 17:37:55,450 DEBUG [aioimaplib:244] Sending : b'IBLO3 FETCH 1 (UID FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)])\r\n'
2017-03-24 17:37:55,609 DEBUG [aioimaplib:188] Received : b'* 1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}\r\nFrom: =?koi8-r?B?7MnTz9fJyyD8xMnL?= <edbi@yandex.ru>\r\nTo: "ckyberlis@gmail.com" <ckyberlis@gmail.com>\r\nSubject: =?koi8-r?B?687Jx8E=?=\r\nMessage-Id: <1759921342902669@web6g.yandex.ru>\r\nDate: Sun, 22 Jul 2012 00:31:09 +0400\r\n\r\n)\r\n'
2017-03-24 17:37:55,610 INFO [aioimaplib:237] unknown data received From: =?koi8-r?B?7MnTz9fJyyD8xMnL?= <edbi@yandex.ru>
2017-03-24 17:37:55,610 INFO [aioimaplib:237] unknown data received To: "ckyberlis@gmail.com" <ckyberlis@gmail.com>
2017-03-24 17:37:55,610 INFO [aioimaplib:237] unknown data received Subject: =?koi8-r?B?687Jx8E=?=
2017-03-24 17:37:55,610 INFO [aioimaplib:237] unknown data received Message-Id: <1759921342902669@web6g.yandex.ru>
2017-03-24 17:37:55,610 INFO [aioimaplib:237] unknown data received Date: Sun, 22 Jul 2012 00:31:09 +0400
2017-03-24 17:37:55,611 INFO [aioimaplib:237] unknown data received )
2017-03-24 17:37:55,611 DEBUG [aioimaplib:188] Received : b'IBLO3 OK Success\r\n'

For my test i am using gmail server.

And Case with email range

data = await self.imap_client.fetch(
            '1:3',
            '(UID FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)])'
        )

Response:

<class 'aioimaplib.aioimaplib.Response'> Response(result='OK', lines=['1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}', '2 FETCH (UID 218 INTERNALDATE "30-Jul-2012 22:30:22 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {295}', '3 FETCH (UID 233 INTERNALDATE "15-Aug-2012 19:50:03 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {302}', 'Success'])

And debug output:

2017-03-24 17:43:16,209 DEBUG [aioimaplib:244] Sending : b'OALF3 FETCH 1:3 (UID FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)])\r\n'
2017-03-24 17:43:16,387 DEBUG [aioimaplib:188] Received : b'* 1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}\r\nFrom: =?koi8-r?B?7MnTz9fJyyD8xMnL?= <edbi@yandex.ru>\r\nTo: "ckyberlis@gmail.com" <ckyberlis@gmail.com>\r\nSubject: =?koi8-r?B?687Jx8E=?=\r\nMessage-Id: <1759921342902669@web6g.yandex.ru>\r\nDate: Sun, 22 Jul 2012 00:31:09 +0400\r\n\r\n)\r\n* 2 FETCH (UID 218 INTERNALDATE "30-Jul-2012 22:30:22 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC RE'
2017-03-24 17:43:16,387 INFO [aioimaplib:237] unknown data received From: =?koi8-r?B?7MnTz9fJyyD8xMnL?= <edbi@yandex.ru>
2017-03-24 17:43:16,387 INFO [aioimaplib:237] unknown data received To: "ckyberlis@gmail.com" <ckyberlis@gmail.com>
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Subject: =?koi8-r?B?687Jx8E=?=
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Message-Id: <1759921342902669@web6g.yandex.ru>
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Date: Sun, 22 Jul 2012 00:31:09 +0400
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received )
2017-03-24 17:43:16,388 DEBUG [aioimaplib:197] Incomplete line, storing partial : b'* 2 FETCH (UID 218 INTERNALDATE "30-Jul-2012 22:30:22 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC RE'
2017-03-24 17:43:16,388 DEBUG [aioimaplib:188] Received : b'PLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {295}\r\nTo: ckyberlis@gmail.com\r\nSubject: =?utf-8?Q?=D0=A0=D0=B5=D0=B3=D0=B8=D1=81=D1=82=D1=80=D0=B0?=\r\n =?utf-8?Q?=D1=86=D0=B8=D1=8F_=D0=B2_=D0=B8=D0=B3=D1=80=D0=B5?= Lineage 2\r\nMessage-ID: <1343687420.50170afc667fe@asima.ws>\r\nDate: Tue, 31 Jul 2012 02:30:20 +0400\r\nFrom: "asima.ws" <info@asima.ws>\r\n\r\n)\r\n* 3 FETCH (UID 233 INTERNALDATE "15-Aug-2012 19:50:03 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {302}\r\nDate: Wed, 15 Aug 2012 23:50:03 +0400\r\nMessage-Id: <E1T1jbP-0001Z6-Ah@mrelay5.hh.ru>\r\nSubject: =?UTF-8?B?0JTQvtGA0L7Qs9GDINC90LXQvtCx0Ys=?=\r\n =?UTF-8?B?0YfQvdGL0Lwg0YDQtdC30Y7QvNC1IQ==?=\r\nFrom: HeadHunter <info@site.hh.ru>\r\nTo: =?UTF-8?B?0JTQtdC90LjRgSDQm9C40YHQvtCy0LjQug==?= <ckyberlis@gmail.com>\r\n\r\n)\r\n'
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received To: ckyberlis@gmail.com
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Subject: =?utf-8?Q?=D0=A0=D0=B5=D0=B3=D0=B8=D1=81=D1=82=D1=80=D0=B0?=
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received  =?utf-8?Q?=D1=86=D0=B8=D1=8F_=D0=B2_=D0=B8=D0=B3=D1=80=D0=B5?= Lineage 2
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Message-ID: <1343687420.50170afc667fe@asima.ws>
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received Date: Tue, 31 Jul 2012 02:30:20 +0400
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received From: "asima.ws" <info@asima.ws>
2017-03-24 17:43:16,388 INFO [aioimaplib:237] unknown data received )
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received Date: Wed, 15 Aug 2012 23:50:03 +0400
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received Message-Id: <E1T1jbP-0001Z6-Ah@mrelay5.hh.ru>
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received Subject: =?UTF-8?B?0JTQvtGA0L7Qs9GDINC90LXQvtCx0Ys=?=
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received  =?UTF-8?B?0YfQvdGL0Lwg0YDQtdC30Y7QvNC1IQ==?=
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received From: HeadHunter <info@site.hh.ru>
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received To: =?UTF-8?B?0JTQtdC90LjRgSDQm9C40YHQvtCy0LjQug==?= <ckyberlis@gmail.com>
2017-03-24 17:43:16,389 INFO [aioimaplib:237] unknown data received )
2017-03-24 17:43:16,389 DEBUG [aioimaplib:188] Received : b'OALF3 OK Success\r\n'
bamthomas commented

Ok what I guess here without looking at the code is that the fetch_message_with_literal_data_re does not match

1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}

I'm going to check

bamthomas commented

Does the a932a8b commit fix your issue ?

cyberlis commented

My result

<class 'aioimaplib.aioimaplib.Response'> Response(result='OK', lines=[b'From: =?koi8-r?B?7MnTz9fJyyD8xMnL?= <edbi@yandex.ru>\r\nTo: "ckyberlis@gmail.com" <ckyberlis@gmail.com>\r\nSubject: =?koi8-r?B?687Jx8E=?=\r\nMessage-Id: <1759921342902669@web6g.yandex.ru>\r\nDate: Sun, 22 Jul 2012 00:31:09 +0400\r\n\r\n', b'To: ckyberlis@gmail.com\r\nSubject: =?utf-8?Q?=D0=A0=D0=B5=D0=B3=D0=B8=D1=81=D1=82=D1=80=D0=B0?=\r\n =?utf-8?Q?=D1=86=D0=B8=D1=8F_=D0=B2_=D0=B8=D0=B3=D1=80=D0=B5?= Lineage 2\r\nMessage-ID: <1343687420.50170afc667fe@asima.ws>\r\nDate: Tue, 31 Jul 2012 02:30:20 +0400\r\nFrom: "asima.ws" <info@asima.ws>\r\n\r\n', b'Date: Wed, 15 Aug 2012 23:50:03 +0400\r\nMessage-Id: <E1T1jbP-0001Z6-Ah@mrelay5.hh.ru>\r\nSubject: =?UTF-8?B?0JTQvtGA0L7Qs9GDINC90LXQvtCx0Ys=?=\r\n =?UTF-8?B?0YfQvdGL0Lwg0YDQtdC30Y7QvNC1IQ==?=\r\nFrom: HeadHunter <info@site.hh.ru>\r\nTo: =?UTF-8?B?0JTQtdC90LjRgSDQm9C40YHQvtCy0LjQug==?= <ckyberlis@gmail.com>\r\n\r\n', 'Success'])

Now we have body but we lost first parts

* 1 FETCH (UID 211 INTERNALDATE "21-Jul-2012 20:31:55 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {224}\

 2 FETCH (UID 218 INTERNALDATE "30-Jul-2012 22:30:22 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {295}\r\n

* 3 FETCH (UID 233 INTERNALDATE "15-Aug-2012 19:50:03 +0000" FLAGS (\\Seen) BODY[HEADER.FIELDS (FROM TO CC BCC REPLY-TO DATE SUBJECT MESSAGE-ID IN-REPLY-TO REFERENCES)] {302}\r\n

We need this parts too

bamthomas commented

We did not "loose" it as it has never been implemented like this. I will see if it would be difficult to add the response part

cyberlis commented

Sorry for "loose". I didn't know that it has never been implemented like this

bamthomas commented

Is it ok with the response line ?

cyberlis commented

yes

bamthomas commented

Cool !
Thank you for your issue and help to improve the lib.
As an explanation for the fetch with BODY, I found it more convenient to have the message bodies directly at index 0. But I understand the need for, for example UID that is not in the body of the message.

It will break the API because now the messages bodies will be between 1 and n not the full list, but we can see the first line as response from the server so it makes sense.

I'm closing this issue and make a new release.

bamthomas commented

See release 0.5.20