[Python-bugs-list] [ python-Bugs-227699 ] Memory leaks using imp.load_module

[noreply@sourceforge.net]

Bugs item #227699, was opened at 2001-01-05 12:30
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=227699&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Charles G Waldman (cgw)
Assigned to: Barry Warsaw (bwarsaw)
Summary: Memory leaks using imp.load_module

Initial Comment:
#!/usr/bin/env python

import os
import sys
import imp

print os.getpid()

sys.path.append("/tmp")

count = 0

while (count<1000):
    modname = "testmod%s" % count
    count = count + 1

    filename = '/tmp/'  + modname + '.py'
    f  = open(filename, 'w')
    for x in range(10000):
        f.write('x%s=%s\n'%(x,x))
    f.close()
    
    modinfo = imp.find_module(modname)
    print 'loading', modname
    m = apply(imp.load_module, ('testmod',) + modinfo)
    
## run "watch -n 1 ps up <pid>
## where <pid> is the pid printed out by this program
    


----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-18 17:42

Message:
Logged In: YES 
user_id=6380

Using the PYTHONDUMPREFS=1 environment variable and the
debug version of Python, I don't see this leaking objects
except that all the file names ('/tmp/testmod0.py' etc.)
seem to be saved as interned strings, which eternalizes them
of course. Hm, I wonder why the filename is interned? (I ran
it twice, with different counts (100 and 1000), and 1000
instead of 10,000 variables, saving the PYTHONDUMPREFS
output to two different files; then diffed the output
files.)

So if it's leaking beyond this, it's not in the form of
objects.

And note that Insure++ (or any leak detection tool worth the
name) doesn't consider interned strings leaks -- they are
reachable via a global variable.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-18 16:28

Message:
Logged In: YES 
user_id=6380

Barry, note that he's cheating with the module name passed
to load_module(): the module name is always set to
'testmod', so he's in fact overwriting the module on each
load. (This acts the same way as reload(), so it's not
creating new module objects -- it's loading the code in the
same namespace over and over.)

I see the process size grow very quickly to 15M, probably on
account of the huge parse tree for those 10,000 assignment
statements (much less for the 10,000 actual variables
created). After that it gradually grows -- after 100 loads
it's up to 27 M. 

Maybe Insure doesn't see this because the data is actually
kept alive somewhere?

----------------------------------------------------------------------

Comment By: Barry Warsaw (bwarsaw)
Date: 2001-10-18 15:30

Message:
Logged In: YES 
user_id=12800

I'm having a hard time tracking down the leak.  I don't
believe it's at the C layer since preliminary Insure runs
don't indicate any leaked blocks (there are plenty of inuse
blocks, but those generally don't count).

I will note that if you comment out the imp.load_module call
the memory usage remains constant.

Loading each module though is going to naturally consume
more memory, right?  And those loaded modules aren't ever
going to get freed, at least without mucking about in
sys.modules.  The memory consumption with ps is pretty
meager, so I suspect we don't have a valid leak here, but
just normal memory consumption due to importation of 10k
modules.

I'm running a more detailed insure run right now just to be
sure, but I'm skeptical that there's really a bug here.

----------------------------------------------------------------------

Comment By: Barry Warsaw (bwarsaw)
Date: 2001-02-23 11:53

Message:
I can verify that memory grows with this process.  I will run it under insure to get more information.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-01-18 19:43

Message:
Barry...?!?!

----------------------------------------------------------------------

Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2001-01-07 15:32

Message:
Assigned to Barry since he's the memory leak expert (& has the right tools).

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=227699&group_id=5470