Brett C. wrote:

Been rather quite around here lately so I just wanted to do a quick check to see what the status is on PEPs 342 and 343. I noticed Nick's PEP is still not up. Probably too busy with that fix for genexps in the AST branch, huh, Nick? =)

Something like that. . . still, I finally got around to fixing the formatting in the text file and sending it back to David :)

Guido, you need something hashed out from us at this point, or do you have this all settled in your head and are just waiting for time to lock it down in the PEP? Or should the PEPs be changed from draft to final and an implementation (which I am *not* volunteering for =) is now needed?

There's a generator finalisation PEP to come from Raymond, and Guido has made some comments on that front that probably need to be folded into PEP 343. And implementations will probably have to wait for the AST branch anyway. . . Off-to-fix-the-generator-symbol-table-entries'ly, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

Nick Coghlan wrote:

Nick Coghlan wrote:

...

Brett C. wrote:

...

I noticed Nick's PEP is still not up. Probably too busy with that fix for genexps in the AST branch, huh, Nick? =)

Something like that. . . still, I finally got around to fixing the formatting in the text file and sending it back to David :)

Add to that AST patches for the genexp scoping, lambda nested args and creation of distinct code objects for lambdas defined on different lines. I guess I finally got over the Python overdose resulting from trying to keep up with the PEP 340 discussion :)

Wow! Thanks, Nick! Now I am the slacker. =) To give you and everyone else a general timeline on the AST stuff, I am coming up on the last week of school for me. After this upcoming week I have a week to pack, graduation weekend, and then I start my internship in the Bay Area. My hope is to put in at least an hour into Python every other night after work (although, knowing me, that hour will morph into three hours at least =). I am planning to split this time between the AST branch, reworking the dev docs at python.org/dev, and finally writing the Python 3000 exceptions reorg PEP. I am hoping to making some decent headway on the branch this summer before I leave for UBC. For those of you who want to keep track of the progress of the AST branch, there is a running tracker item, bug #1191458 (http://www.python.org/sf/1191458), that lists the currently failing tests. Once that bug report is closed then the AST is semantically complete. I do, though, want to go through and clean up the syntax to match PEP 7 specs before it gets merged into the mainline once it is semantically working. -Brett

At 11:12 AM 5/31/2005 -0700, Guido van Rossum wrote:

uncertainty of the possibility to call close() from the destructor may have slowed that down.

If you're talking about the bit about __del__ not working when hanging off a cycle, my apologies for creating that confusion, I misunderstood Tim's post. If you're talking about the circular reference scenario involving exceptions, then I think I've already given a tentative explanation why such a cycle will not include the generator-iterator itself (only its frame) as long as the generator-iterator clears its frame's f_back when the frame is not in use. It would probably be a good idea to verify this, but I think the basic idea is sound. The only scenario in which a generator-iterator would be uncollectable is if you took some action outside the iterator to pass it into itself, or if you stored the iterator in a global variable. However, even in the latter case, the phase of Python shutdown that clears module contents would break the cycle and cause the generator to finalize, albeit a bit late and perhaps in a broken way. Still, storage of generator-iterators in module globals is a very unlikely scenario, since it's more common to loop over such iterators than it is to store them in any kind of variable.

If Raymond would rather defer to me, I can give it a shot in a revised version of PEP 343,

Thanks, that would be great. The decimal conversation used up a lot of my available cycles. Raymond

[Guido]

...

Yes, the generator does clear its f_back when it's suspended.

[Phillip]

I realize this won't fix all your worries; I just want to rule out this one *particular* form of cycle as a possibility; i.e., to show that mere reference to a generator-iterator in a frame does not create a cycle:

callerframe ---------> traceback2 | ^ | | | | | | | +----------------+ | v v geniter -> genframe -> traceback1 ^ | | | +----------+

As you can see, the geniter itself doesn't have a reference to its calling frame, so as soon as the highest-level traceback object is released, the cycle collector should release the upper cycle, allowing the geniter to complete, and releasing the lower cycle.

The scenario assumes, by the way, that the traceback object referenced by a frame includes a pointer to that same frame, which I'm not sure is the case. I was under the impression that the current frame is only added to the traceback when the frame is exited, in which case the two cycles shown above wouldn't even exist; each traceback would be pointing to the *next* frame down, and there would be no cycles at all. It seems to me that this would almost have to be the design, since tracebacks existed before cyclic GC did.

Alas, your assumption is valid; this would indeed cause a cycle, much to the despair of early Python programmers. There used to be a whole body of literature about the best way to avoid this (never save a traceback, or if you do, clear it when you're done with it before exiting the frame). When you raise and immediately catch an exception, there is a single traceback object that references the current frame (the frame where it was raised *and* caught). So if you store sys.exc_info() in a local, you have a cycle already: try: raise Exception except: x = sys.exc_info()[2] # save the traceback Now we have the following cycle (with slightyl more detail than your diagram): tb_frame frame <------------- traceback | ^ | | v 'x' | f_locals ------------------+ BTW, note the repercussions this has for Ping's PEP 344 -- because the Exception instance references the traceback in that proposal, all code that catches an exception into a variable creates a cycle, like this: try: raise Exception except Exception, err: pass This would creates the following cycle: tb_frame frame <------------- traceback | ^ | | __traceback__ v | f_locals ---------------> err The good news in all this is that none of these objects has a __del__ method in the proposal; only the 'geniter' object would have one, and getting it involved in a cycle does seem like rather unlikely. I hereby declare my worries unwarranted and will happily add language to the revamped PEP 343 that a geniter object should have a tp_del slot and a corresponding __del__ attribute. This further complicates has_finalizer() in gcmodule.c, to the point where the latter might have to be turned into an internal slot.

...

Sure. But I still have some reservations, since cycles can pop up in the strangest of places (especially when tracebacks are involved -- tracebacks have been causing problems due to cycles and keeping variables alive almost since Python's inception). I posted about this a while ago, but don't recall seeing a response that took my fear away.

Well, I can't prove that it's not possible to create such cycles, certainly. But maybe we should finally get rid of the deprecated sys.exc_type/value/traceback variables, so that they can't root any cycles?

I sort of doubt that these are the main source of live cycles. After all, they are reset whenever a frame is popped (grep the sources for reset_exc_info). -- --Guido van Rossum (home page: http://www.python.org/~guido/)