[Python-bugs-list] [Bug #129810] Memore leak in pickle and cPickle

[noreply@sourceforge.net]

Bug #129810, was updated on 2001-Jan-23 07:28
Here is a current snapshot of the bug.

Project: Python
Category: Python Library
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: vlk
Assigned to : bwarsaw
Summary: Memore leak in pickle and cPickle

Details: # When Pickler.object is used for dump typles into file and
Unpickler for 
# load from files. A loaded object are not garbage collected.
# When function dump(object,file) is used Unpickler works fine.
# Problem is in pickle and cPickle module
# tested on Python 2.0 Red Hat Linux 6.2

import cPickle			
#import pickle			 
import gc

f=open("xxx","w")
pic=cPickle.Pickler(f,1)	# ERROR
#pic=pickle.Pickler(f,1)	# ERROR
for i in range(100):
	#cPickle.dump(([long(i),long(i),long(i),long(i)],i),f)	
		# this is OK
	#pickle.dump(([long(i),long(i),long(i),long(i)],i),f)	
		# this is OK
	pic.dump(([long(i),long(i),long(i),long(i)],i))		
		# Memory leak

f.close()
gc.set_debug(gc.DEBUG_STATS)
f=open("xxx","r")
u=cPickle.Unpickler(f)
try:
	while 1:
		gc.collect()
		print u.load()
except EOFError:
	pass
f.close()

Follow-Ups:

Date: 2001-Feb-23 10:59
By: bwarsaw

Comment:
# When Pickler.object is used for dump typles into file and Unpickler for
load
# from files. A loaded object are not garbage collected.  When function
# dump(object,file) is used Unpickler works fine.  Problem is in pickle
and
# cPickle module tested on Python 2.0 Red Hat Linux 6.2

import gc 

def main():
    fp = open('/tmp/xxx', 'w') 
    pic = pickle.Pickler(fp, 1)                   # ERROR 

    for i in range(10000):
        pickle.dump(([long(i), long(i), long(i), long(i)], i), fp) 
        # this is OK 
        pic.dump(([long(i), long(i), long(i), long(i)], i))
        # Memory leak 

    fp.close() 
    gc.set_debug(gc.DEBUG_STATS) 

    fp = open('/tmp/xxx')

    upic = pickle.Unpickler(fp)
    try: 
        while 1: 
            gc.collect() 
            print upic.load() 
    except EOFError: 
        pass

    fp.close()


if __name__ == '__main__':
    import sys
    if len(sys.argv) > 1:
        import cPickle
        pickle = cPickle
    else:
        import pickle

    main()

-------------------------------------------------------

Date: 2001-Feb-23 10:58
By: bwarsaw

Comment:
When cranking up the number of objects placed in the pickle to 10000, I do
see some memory growth when unpickling as clocked by top.  The cPickle
growth is much smaller than the pickle growth, which already appears fairly
minimal.

I will investigate further with Insure++.

I don't see how the problem could be related to __del__ since the only
thing we're dumping and loading are builtin objects (tuples, lists, longs,
and ints).
-------------------------------------------------------

Date: 2001-Jan-23 19:28
By: gvanrossum

Comment:
Barry, can you look into this?  I would first see if this is really
reproducable without using Insure++; somehow it looks a bit fishy.  Could
also be fixed in 2.1 because now modules participate in gc.  Or could have
to do with a __del__?  Also, I doubt the claim that this is a leak with
both pickle and cPickle.

-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=129810&group_id=5470