[Python-bugs-list] [ python-Bugs-507713 ] mem leak in imaplib
noreply@sourceforge.net
noreply@sourceforge.net
Thu, 21 Feb 2002 10:02:45 -0800
Bugs item #507713, was opened at 2002-01-23 13:28
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=507713&group_id=5470
Category: Python Library
Group: Python 2.1.1
Status: Open
Resolution: None
Priority: 5
Submitted By: Scott Blomquist (scottdb)
Assigned to: Piers Lauder (pierslauder)
Summary: mem leak in imaplib
Initial Comment:
When run in a multithreaded environment, the imaplib
will leak memory if not run with the -O option.
Long running, multithreaded programs that we have
which use the imaplib will run fine for a undefined
period of time, then suddenly start to grow in size
until they take as much mem as the system will give to
them. Once they start to grow, they continue to grow
at a pretty consistent rate.
Specifically:
If the -O option is not used, in the _log method
starting on line 1024 in the imaplib class, the
imaplib keeps the last 10 commands that are sent.
def _log(line):
# Keep log of last `_cmd_log_len' interactions for
debugging.
if len(_cmd_log) == _cmd_log_len:
del _cmd_log[0]
_cmd_log.append((time.time(), line))
Unfortunately, in a multithreaded environment,
eventually the len of the list will become larger than
the _cmd_log_len, and since the test is for equality,
rather than greater-than-equal-to, once the len of the
_cmd_log gets larger than _cmd_log_len, nothing will
ever be removed from the _cmd_log, and the list will
grow without bound.
We added the following to test this hypothesis, we
created a basic test which creates 40 threads. These
threads sit in while 1 loops and create an imaplib and
then issue the logout command. We also added the
following debug to the method above:
if len(_cmd_log) > 10:
print 'command log len is:', len(_cmd_log)
We started the test, which ran fine, without leaking,
for about 10 minutes, and without printing anything
out. Somewhere around ten minutes, the process started
to grow in size rapidly, and at the same time, the
debug started printing out, and the size of the
_cmd_log list did indeed grow very large, very fast.
We repeated the test and the same symptoms occured,
this time after only 5 minutes.
----------------------------------------------------------------------
>Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2002-02-21 10:02
Message:
Logged In: YES
user_id=3066
No objection here, but don't hold back on my account!
----------------------------------------------------------------------
Comment By: Piers Lauder (pierslauder)
Date: 2002-02-17 16:16
Message:
Logged In: YES
user_id=196212
On further consideration, I think that the command logging
should be done with a circular buffer implemented using a
dictionary - it has two advantages: it's immune to thread
contentions, and it's twice as fast as a truncating list.
I'm also convinced these routines should be per IMAP4
instance - ie: per socket - so i've moved them into the class.
I've attached a diff with the current CVS
If noone disagrees, I'll make these changes.
----------------------------------------------------------------------
Comment By: Piers Lauder (pierslauder)
Date: 2002-02-16 02:39
Message:
Logged In: YES
user_id=196212
I aggree that the line:
if len(_cmd_log) == _cmd_log_len:
should be changed, though I favour the form:
while len(_cmd_log) >= _cmd_log_len:
del _cmd_log[0]
rather than the version suggested in the patch:
if len(_cmd_log) > _cmd_log_len:
del _cmd_log[:-_cmd_log_len]
However, if imaplib is gpoing to be used by multiple
threads, perhaps the best solution is to move these
debugging routines entirely into the IMAP4 class, so
that the logs are per-connection, rather than global?
----------------------------------------------------------------------
Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2002-02-14 22:09
Message:
Logged In: YES
user_id=3066
It looks like the problem still exists in Python 2.1.2, 2.2,
and CVS.
I've attached a patch that I think solves this problem, but
this isn't easy for me to test. Please check this.
Assigning to Piers Lauder since he knows more about this
module than I do.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=507713&group_id=5470