[ python-Bugs-876108 ] surprising memory usage with generators

[SourceForge.net]

Bugs item #876108, was opened at 2004-01-13 14:54
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=876108&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Kjetil Jacobsen (kjetilja)
Assigned to: Nobody/Anonymous (nobody)
Summary: surprising memory usage with generators

Initial Comment:
When running the attached script on Linux with integer
arguments 14 through 19, then the output is something
like this:

1000 iterations, VmSize: 6432 kB   VmData: 1168 kB
2000 iterations, VmSize: 6692 kB   VmData: 1428 kB
..

I.e. additional ~256 kb memory is allocated after 2k 
iterations.  This is rather weird.

If the script is run with other arguments, say 140 
through 190, then the output is something like this
(which is what one should expect):

1000 iterations, VmSize: 6432 kB   VmData: 1168 kB
2000 iterations, VmSize: 6432 kB   VmData: 1168 kB
3000 iterations, VmSize: 6432 kB   VmData: 1168 kB

I.e. no memory increase.  The reason why this is
somewhat disturbing is that we used generators in the
same way as this in a larger project and found the
interpreter to be continously leaking.

To stop the memory leak problem we used an alternate
scheme (the one in the doit_noleak) where the generator
is called only in a list comprehension.

Running with a --with-pydebug enabled interpreter
reveals that the additional memory is due to an extra
pymalloc arena.  Note that the behaviour is the same
for python 2.2.X without pymalloc enabled so this does
not seem to be a pymalloc specific issue.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=876108&group_id=5470