[Patches] [ python-Patches-1444398 ] Make itertools.tee participate in GC

[SourceForge.net]

Patches item #1444398, was opened at 2006-03-06 21:33
Message generated for change (Comment added) made by twouters
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1444398&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Modules
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Thomas Wouters (twouters)
Assigned to: Thomas Wouters (twouters)
Summary: Make itertools.tee participate in GC

Initial Comment:
A small patch to make itertools.tee objects participate
in GC; solves the memoryleak in test_generators (and
any other use of tee objects in cycles.)


----------------------------------------------------------------------

>Comment By: Thomas Wouters (twouters)
Date: 2006-03-20 22:34

Message:
Logged In: YES 
user_id=34209

Hmm. An alternative fix would be to make itertools.tee only
partially participate in GC. That is, not have it traverse
over all the stored items, but just the iterator. It would
prevent the (potentially rather heavy) cost of going over
all cached items, but still leaves a potential for
unfindable cycles.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2006-03-19 09:02

Message:
Logged In: YES 
user_id=80475

Okay, go ahead and add GC to itertools.tee().  The 
test_generators examples make it clear that there are 
valid use cases for feeding a teeobject back into itself.

Tim, the original rationale did not have to do with 
millions of tee objects; rather, it posited that a tee 
could contain many objects of some other type and that 
traversing it during GC was simply a waste of time.

Thomas, the patch needs work, but I don't currently have 
time to go through it.  Do your best to model after the 
other tools which have gc.  That includes setting the 
tp_flags slot, doing an untrack before dealloc starts, 
replacing PyObject_Del and PyObject_New with their GC 
counterparts.  See section 2.1.3 in Extending and 
Embedding for the details and examples.  Offhand, I think 
it may also need a tp_clear entry but I'm not sure.

If you can work out a good patch, go ahead an apply it for 
the alpha 1 release.  If not, assign back to me and I'll 
get to it when I can (possibly for the second alpha).


 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-03-07 01:52

Message:
Logged In: YES 
user_id=31435

If there are likely to be millions of some type of object,
then avoiding gc for that type on grounds of reducing memory
use may be arguable (or, as in the cases of tuples or lists,
may not be arguable regardless).  Else "if it points to a
PyObject, it might be in a cycle, so gc-enable it" generally
rules.

I'll note in passing that the m235 and fib generators
weren't _intended_ to stress Python in any way, but have
been extremely effective at doing so since generators first
went in.  Think of them as a pair of canaries in the coal
mine :-)

----------------------------------------------------------------------

Comment By: Thomas Wouters (twouters)
Date: 2006-03-07 01:12

Message:
Logged In: YES 
user_id=34209

Well, I'm fine with removing the scope-enclosed generator
versions of _m235() and fib() from test_generators, if
that's what you're implying. :> However, I think it's
slightly more suitable to just have tee participate in GC.


----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2006-03-07 01:05

Message:
Logged In: YES 
user_id=80475

I will take a look at the patch soon.  IIRC, there was a 
concious decision to not have tee engage in GC because no 
normal use cases created cycles (they seem to arise only 
when intentionally creating a cycle for test code).



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1444398&group_id=5470