[Python-ideas] New imaplib implementation for Python 3.2+ standard library

[Maxim Khitrov]

On Tue, Jul 26, 2011 at 10:42 AM, Michael Foord <fuzzyman at gmail.com> wrote:
> On 25 July 2011 23:39, Maxim Khitrov <max at mxcrypt.com> wrote:
>> On Mon, Jul 25, 2011 at 4:27 PM, Michael Foord <fuzzyman at gmail.com> wrote:
>> > On 25 July 2011 20:38, Maxim Khitrov <max at mxcrypt.com> wrote:
>> >> On Mon, Jul 25, 2011 at 3:21 PM, Michael Foord <fuzzyman at gmail.com>
>> >> wrote:
>> >> > There is an existing, well tested and widely used, replaced for
>> >> > imaplib
>> >> > that
>> >> > I would suggest should be the first for consideration in replacing
>> >> > imaplib:
>> >> >
>> >> > http://imapclient.freshfoo.com/
>> >> >
>> >> > (Sorry.)
>> >> >
>> >> > All the best,
>> >> >
>> >> > Michael Foord
>> >>
>> >> I have it beat at the "Python 3 support is in the works" feature ;)
>> >> Mine doesn't handle 2.x though.
>> >>
>> >> In any case, I would not have been able to use IMAPClient for my
>> >> project, because the requirements were for no dependencies outside of
>> >> Python 3.2.
>> >>
>> >> Do you know if the developers of IMAPClient considered getting it into
>> >> the standard library? My goal wasn't just to have another IMAP
>> >> implementation, but something better available as part of Python.
>> >>
>> >
>> >
>> > I don't think Menno Smitts would object to adding Python 3 support or
>> > adding
>> > IMAPClient to the standard library. His goal was to create something
>> > useful
>> > to overcome what he saw (and evidently you agree) as irreparable
>> > brokenness
>> > in parts of imaplib.
>> >
>> > My point is that if there is an existing widely-used and battle-tested
>> > alternative, we would be wise to look at that first.
>>
>> Agreed. However, I just took a look through IMAPClient source code and
>> have to correct you on your original assertion. IMAPClient is not a
>> replacement for imaplib, but a wrapper around it.
>>
>> The author went down the same road that I originally started out on.
>> He added a parser, a UTF-7 codec, and changed the overall interface to
>> be more user-friendly. All good improvements, but at the core,
>> IMAPClient relies entirely on imaplib, and thus inherits most of its
>> design flaws.
>>
>
> Well I'm sure Menno would be interested if you could actually demonstrate
> those limitations rather than merely assert that. :-)
>
> (As he implemented to overcome problems with imaplib and that is why people
> use it.)
>
> Not that you're necessarily wrong, but I'm skeptical.

Most of the limitations are outlined in my README file. I'm not sure
what kind of a demonstration would convince you. The problem is that
if your applications do not require things like data compression,
asynchronous/parallel command execution, or multiple literals in
commands, you will not agree that there is anything wrong with the
existing tools. It's very much a matter of personal experience.

One of the things I'm trying to address with my library is strict
adherence to the current version of the IMAP4 protocol. The other is
performance; hence the implementation of extensions such as SASL-IR,
IDLE, non-synchronizing literals, multiappend, and compression.

On the performance side, if you have an application that's trying do
some sort of processing of a 6 GB mailbox with 700,000 messages in it,
executing a separate FETCH command for each message will take you a
week to finish. If you try to be clever and FETCH 1000 messages at a
time, for example, you'll quickly run into a few problems:

1. The data for all 1000 messages is buffered in memory before you
ever to get to see anything.

2. If the server fails to retrieve one of those messages, which
actually happens rather frequently, at least on Gmail servers, you
lose the data for the other 999. You'll get a 'NO' response from
imaplib or an exception from IMAPClient, and an explanation of what
went wrong instead of any data.

3. If you requested data for messages 1:1000 and the server decides to
send you a FLAGS update for message 9999, imaplib will return that
response as part of the data for the original request.

All are interface design problems, which are inherited by IMAPClient.

- Max

P.S. It is not my intention to discourage the use of IMAPClient in any
way. Its existence is a good thing for 99% of the users, because it
does address a number of key imaplib issues with just the response
parser and a UTF-7 codec. My point is that there are real-world use
cases out there that cannot be handled by imaplib or IMAPClient, and
for those, I'm offering my library as a more general solution that
should satisfy the remaining 1% :)