append in IMAP4 from imaplib very slow
sp at abusix.org
Thu Jul 26 11:25:40 CEST 2012
On 07/26/2012 09:49 AM, Simon Pirschel wrote:
> On 07/26/2012 05:21 AM, Tim Chase wrote:
>> On 07/25/12 12:47, Simon Pirschel wrote:
>>> I'm currently experimenting with IMAP using Python 2.7.3 and
>>> IMAP4 from imaplib. I noticed the performance to be very bad. I
>>> read 5000 files from a directory and append them to an IMAP
>>> INBOX. The hole procedure of reading and appending is taking
>>> about 210 seconds.
>>> I set up the exact same code in Perl to check if there is a
>>> general IMAP server configuration issue, since CPU and I/O isn't
>>> the problem. The same amount of data on the same IMAP server is
>>> done after 7.9 seconds using Perl.
>>> The difference is huge and I tried to narrow the issue down by
>>> profiling the Python code. The profile results are, 206 seconds
>>> are spent in calling socket.recv.
>> While I don't know the ins and outs of the Perl code, it looks like
>> the Python imaplib.py is doing some pretty straight-forward
>> send/receive work. However, IMAP4 does support both pipelining and
>> multi-append extensions, so if the Perl code takes advantage of the
>> features if the server offers them, it might be reaping significant
>> gains over the naïve Python code.
>> Can you get a debugging dump of the commands being sent each way?
>> (particularly if you see a bunch of APPEND commands before the reply
>> acknowledgement, as detailed at )
>> http://www.ietf.org/rfc/rfc3502.txt on page 4
> I took a strace of all network related sys calls from Python and Perl.
> It seems both don't do multi appends and waiting for the OK after
> appending one mail. But the Python version is getting a completely
> different response.
> Python: "* 204 EXISTS\r\n* 1 RECENT\r\nNBGC3 OK [APPENDUID 1343283946
> 204] Append completed.\r\n"
> Perl: "2 OK [APPENDUID 1343283946 255] Append completed.\r\n"
> The Python lib seems to trigger a EXISTS and RECENT lookup on every
> append, which could be the performance difference. I couldn't figure
> out why, yet!
> Python strace: http://aboutsimon.com/py.strace
> Perl strace: http://aboutsimon.com/pl.strace
> Strings are shortened to 40 chars.
Ok, forget about the EXISTS and RECENT response. The server will
response this way if you selected a mailbox, which I did in the Python
code but not in the Perl code. I disabled selecting the mailbox in
Python and there is no difference in the runtime. So, this isn't the issue.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list