
I've been thinking about this a lot, but haven't made much progess. Here's a brain dump. I've been thinking about integrating PEP 325 (Resource-Release Support for Generators) into the for-loop code, so that you could replace the_lock.acquire() try: BODY finally: the_lock.release() with for dummy in synchronized(the_lock): BODY or perhaps even (making "for VAR" optional in the for-loop syntax) with in synchronized(the_lock): BODY Then synchronized() could be written cleanly as follows: def synchronized(lock): lock.acquire() try: yield None finally: lock.release() But then every for-loop would have to contain an extra try-finally clause; the translation of for VAR in EXPR: BODY would become __it = iter(EXPR) try: while True: try: VAR = __it.next() except StopIteration: break BODY finally: if hasattr(__it, "close"): __it.close() which I don't particularly like: most for-loops DON'T need this, since they don't use a generator but some other form of iterator, or even if they use a generator, not all generators have a try/finally loop. But the bytecode compiler can't know that, so it will always have to generate this code. It also changes the semantics of using a generator in a for-loop slightly: if you break out of the for-loop before the generator is exhausted you will still get the close() call. It's also a bit funny to see this approach used with the only other use case for try/finally we've looked at, which requires passing a variable into the block: the "with_file" use case. We now can write with_file as a nice and clean generator: def with_file(filename): f = open(filename) try: yield f finally: f.close() but the use looks very odd because it is syntactically a for-loop but there's only one iteration: for f in with_file("/etc/passwd"): for line in f: print line[:line.find(":")] Seeing this example makes me cringe -- why two nested for loops to loop over the lines of one file??? So I think that this is probably not the right thing to pursue, and we might be better off with something along the lines of PEP 310. The authors of PEP 310 agree; under Open Issues they wrote: There are some simiralities in concept between 'with ...' blocks and generators, which have led to proposals that for loops could implement the with block functionality[3]. While neat on some levels, we think that for loops should stick to being loops. (Footnote [3] references the tread that originated PEP 325.) Perhaps the most important lesson we've learned in this thread is that the 'with' keyword proposed in PEP 310 is redundant -- the syntax could just be [VAR '=']* EXPR ':' BODY IOW the regular assignment / expression statement gets an optional colon-plus-suite at the end. So now let's assume we accept PEP 310 with this change. Does this leave any use cases for anonymous blocks uncovered? Ruby's each() pattern is covered by generators; personally I prefer Python's for var in seq: ... over Ruby's much-touted seq.each() {|var| ...} The try/finally use case is covered by PEP 310. (If you want to combine this with a for-loop in a single operation, you'll need PEP 325.) The use cases where the block actually returns a value are probably callbacks for things like sort() or map(); I have to admit that I'd rather keep lambda for these (and use named functions for longer blocks) than introduce an anonymous block syntax that can return values! I also note that if you *already* have a comparison function, Ruby's Array sort method doesn't let you pass it in as a function argument; you have to give it a block that calls the comparison function, because blocks are not the same as callables (and I'm not sure that Ruby even *has* callables -- everything seems to be a block). My tentative conclusion remains: Python doesn't need Ruby blocks. Brian Sabbey ought to come up with more examples rather than arguments why his preferred syntax and semantics are best. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
[SNIP - using 'for' syntax to delineate the block and resource]
So I think that this is probably not the right thing to pursue,
I totally agree with your reasoning on this.
Sure, but is the redundancy *that* bad? You should be able to pick up visually that something is an anonymous block from the indentation but I don't know how obvious it would be. Probably, in the end, this minimal syntax would be fine, but it just seems almost too plain in terms of screaming at me that something special is going on there (the '=' in an odd place just quite cut if for me for my meaning of "special").
I think I agree with Samuele that it would be more pertinent to put all of this effort into trying to come up with some way to handle cleanup in a generator. -Brett

[Brett]
I think I agree with Samuele that it would be more pertinent to put all of this effort into trying to come up with some way to handle cleanup in a generator.
I.e. PEP 325. But (as I explained, and you agree) that still doesn't render PEP 310 unnecessary, because abusing the for-loop for implied cleanup semantics is ugly and expensive, and would change generator semantics; and it bugs me that the finally clause's reachability depends on the destructor executing. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
yes, PEP325 would work in combination with PEP310, whether a combined thing (which cannot be the current for as dicussed) is desirable is a different issue: these anyway f = file(...): for line in f: ... vs. it = gen(): for val in it: ... would be analogous in a PEP310+325 world.

Guido van Rossum <gvanrossum@gmail.com> wrote:
Yes and no. PEP 325 offers a method to generators that handles cleanup if necessary and calls it close(). Obviously calling it close is a mistake. Actually, calling it anything is a mistake, and trying to combine try/finally handling in generators with __exit__/close (inside or outside of generators) is also a mistake. Start by saying, "If a non-finalized generator is garbage collected, it will be finalized." Whether this be by an exception or forcing a return, so be it. If this were to happen, we have generator finalization handled by the garbage collector, and don't need to translate /any/ for loop. As long as the garbage collection requirement is documented, we are covered (yay!). What about ... i.__enter__() try: ... finally: i.__exit__() ... types of things? Well, you seem to have offered a syntax ... [VAR '=']* EXPR: BODY ... which seems to translate into ... [VAR = ] __var = EXPR try: BODY finally: __var.__exit__() ... or something like that. Great! We've got a syntax for resource allocation/freeing outside of generators, and a non-syntax for resource allocation/freeing inside of generators. - Josiah

On Apr 21, 2005, at 8:59 PM, Josiah Carlson wrote:
Well, for the CPython implementation, couldn't you get away with using garbage collection to do everything? Maybe I'm missing something.. import weakref class ResourceHandle(object): def __init__(self, acquire, release): acquire() # if I understand correctly, this is safer than __del__ self.ref = weakref.ref(self, lambda o:release()) class FakeLock(object): def acquire(self): print "acquired" def release(self): print "released" def with_lock(lock): r = ResourceHandle(lock.acquire, lock.release) yield None del r
I could imagine someone complaining about generators that are never used missing out on the acquire/release. That could be solved with a trivial rewrite: def with_lock(lock): def _with_lock(r): yield None del r return _with_lock(ResourceHandle(lock.acquire, lock.release))
Of course, this just exaggerates Guido's "it bugs me that the finally clause's reachability depends on the destructor executing".. but it does work, in CPython. It seems to me that this pattern would be painless enough to use without a syntax change... -bob

Bob Ippolito wrote:
[SNIP] Well, if you are missing something then so am I since your suggestion is basically correct. The only issue is that people will want more immediate execution of the cleanup code which gc cannot guarantee. That's why the ability to call a method with the PEP 325 approach gets rid of that worry. -Brett

On Apr 22, 2005, at 12:28 AM, Brett C. wrote:
Well in CPython, if you are never assigning the generator to any local or global, then you should be guaranteed that it gets cleaned up at the right time unless it's alive in a traceback somewhere (maybe you WANT it to be!) or some insane trace hook keeps too many references to frames around.. It seems *reasonably* certain that for reasonable uses this solution WILL clean it up optimistically. -bob

Guido van Rossum wrote:
Right, I'm not saying PEP 310 shouldn't also be considered. It just seems like we are beginning to pile a lot on this discussion by bringing in PEP 310 and PEP 325 in at the same time since, as pointed out, there is no guarantee that anything will be called in a generator and thus making PEP 310 work in generators does not seem guaranteed to solve that problem (although I might have missed something; just started really following the thread today). At this point anonymous blocks just don't seem to be happening, at least not like in Ruby. Fine, I didn't want them anyway. Now we are trying to simplify resource cleanup and handling. What I am trying to say is that generators differ just enough as to possibly warrant a separate discussion from all of this other resource handling "stuff". So I am advocating a more focused generator discussion since resource handling in generators is much more difficult than the general case in non-generator situations. I mean obviously in the general case all of this is handled already in Python today with try/finally. But with generators you have to jump through some extra hoops to get similar support (passing in anything that needs to be cleaned up, hoping that garbage collection will eventually handle things, etc.).
and it bugs me that the finally clause's reachability depends on the destructor executing.
Yeah, I don't like it either. I would rather see something like: def gen(): FILE = open("stuff.txt", 'rU') for line in FILE: yield line cleanup: FILE.close() and have whatever is in the 'cleanup' block be either accessible from a method in the generator or have it become the equivalent of a __del__ for the generator, or maybe even both (which would remove contention that whatever needs to be cleaned up is done too late thanks to gc not guaranteeing immediate cleanup). This way you get the guaranteed cleanup regardless and you don't have to worry about creating everything outside of the generator, passing it in, and then handling cleanup in a try/finally that contains the next() calls to the generator (or any other contortion you might have to go through). Anyway, my random Python suggestion for the day. -Brett

On Thu, 21 Apr 2005, Guido van Rossum wrote:
It seems to me that, in general, Python likes to use keywords for statements and operators for expressions. Maybe the reason lambda looks like such a wart is that it uses a keyword in the middle of an expression. It also uses the colon *not* to introduce an indented suite, which is a strange thing to the Pythonic eye. This suggests that an operator might fit better. A possible operator for lambda might be ->. sort(items, key=x -> x.lower()) Anyway, just a thought. -- ?!ng

Ka-Ping Yee wrote:
It seems to me that, in general, Python likes to use keywords for statements and operators for expressions.
Probably worth noting that 'for', 'in' and 'if' in generator expressions and list comprehensions blur this distinction somewhat... Steve -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

On Thu, Apr 21, 2005, Guido van Rossum wrote:
Yes, it could. The question then becomes whether it should. Because it's easy to indent Python code when you're not using a block (consider function calls with lots of args), my opinion is that like the "optional" colon after ``for`` and ``if``, the resource block *should* have a keyword. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code -- not in reams of trivial code that bores the reader to death." --GvR

I do not know that I have ever needed 'anonymous blocks', and I have therefore not followed this discussion in detail, but I appreciate Python's beauty and want to see it maintained. So I have three comments and yet-another syntax proposal that I do not remember seeing (but could have missed). 1. Python's integration of for loops, iterators, and generators are, to me, a gem of program language design that distinguishes Python from other languages I have used. Using them to not iterate but to do something else may be cute, but in a perverted sort of way. I would rather have 'something else' done some other way. 2. General-purpose passable block objects with parameters look a lot like general-purpose anonymous functions ('full lambdas'). I bet they would be used a such if at all possible. This seems to me like the wrong direction. 3. The specific use-cases for Python not handled better by current syntax seem to be rather specialized: resource management around a block. So I cautiously propose: with <resource type> <specific resource>: <suite> with the exact semantics dependent on <resource type>. In particular: with lock somelock: codeblock could abbreviate and mean somelock.acquire() try: codeblock finally: somelock.release() (Guido's example). with file somefile: codeblock might translate to (the bytecode equivalent of) if isinstance(somefile, basestring?): somefile = open(somefile,defaults) codeblock somefile.close The compound keywords could be 'underscored' but I presume they could be parsed as is, much like 'not in'. Terry J. Reedy

Guido> or perhaps even (making "for VAR" optional in the for-loop syntax) Guido> with Guido> in synchronized(the_lock): Guido> BODY This could be a new statement, so the problematic issue of implicit try/finally in every for statement wouldn't be necessary. That complication would only be needed for the above form. (Of course, if you've dispensed with this I am very likely missing something fundamental.) Skip

Skip Montanaro wrote:
s/in/with/ to get PEP 310. A parallel which has been bugging me is the existence of the iterator protocol (__iter__, next()) which you can implement manually if you want, and the existence of generators, which provide a nice clean way of writing iterators as functions. I'm wondering if something similar can't be found for the __enter__/__exit__ resource protocol. Guido's recent screed crystallised the idea of writing resources as two-part generators: def my_resource(): print "Hi!" # Do entrance code yield None # Go on with the contents of the 'with' block print "Bye!" # Do exit code Giving the internal generator object an enter method that calls self.next() (expecting None to be returned), and an exit method that does the same (but expects StopIteration to be raised) should suffice to make this possible with a PEP 310 style syntax. Interestingly, with this approach, "for dummy in my_resource()" would still wrap the block of code in the entrance/exit code (because my_resource *is* a generator), but it wouldn't get the try/finally semantics. An alternative would be to replace the 'yield None' with a 'break' or 'continue', and create an object which supports the resource protocol and NOT the iterator protocol. Something like: def my_resource(): print "Hi!" # Do entrance code continue # Go on with the contents of the 'with' block print "Bye!" # Do exit code (This is currently a SyntaxError, so it isn't ambiguous in any way) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
That's a very interesting suggestion. I've been lurking, thinking about a way to use something like PEP 310 to help manage database transactions. Here is some typical code that changes something under transaction control: begin_transaction() try: changestuff() changemorestuff() except: abort_transaction() raise else: commit_transaction() There's a lot of boilerplate code there. Using your suggestion, I could write that something like this: def transaction(): begin_transaction() try: continue except: abort_transaction() raise else: commit_transaction() with transaction(): changestuff() changemorestuff() Shane

Shane Hathaway wrote:
For that to work, the behaviour would need to differ slightly from what I envisioned (which was that the 'continue' would be behaviourally equivalent to a 'yield None'). Alternatively, something equivalent to the above could be written as: def transaction(): begin_transaction() continue ex = sys.exc_info() if ex[0] is not None: abort_transaction(): else: commit_transaction(): Note that you could do this with a normal resource, too: class transaction(object): def __enter__(): begin_transaction() def __exit__(): ex = sys.exc_info() if ex[0] is not None: abort_transaction(): else: commit_transaction(): Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
Oh, it is ambiguous, as soon as you insert a for/while statement in your resource function and want to call continue in there. Other than that, it's very neat. Maybe "yield" alone (which is always a SyntaxError) could be used. Reinhold -- Mail address is perfectly valid!

On 4/21/05, Guido van Rossum <gvanrossum@gmail.com> wrote:
How is this different from: def synchronized(lock): def synch_fn(block): lock.acquire() try: block() finally: lock.release() return synch_fn @synchronized def foo(): BLOCK True, it's non-obvious that foo is being immediately executed, but regardless I like the way synchronized is defined, and doesn't use yield (which in my opinion is a non-obvious solution)

After reading a lot of contributions (though perhaps not all -- this thread seems to bifurcate every time someone has a new idea :-) I'm back to liking yield for the PEP 310 use case. I think maybe it was Doug Landauer's post mentioning Beta, plus scanning some more examples of using yield in Ruby. Jim Jewett's post on defmacro also helped, as did Nick Coghlan's post explaining why he prefers 'with' for PEP 310 and a bare expression for the 'with' feature from Pascal (and other languages :-). It seems that the same argument that explains why generators are so good for defining iterators, also applies to the PEP 310 use case: it's just much more natural to write def with_file(filename): f = open(filename) try: yield f finally: f.close() than having to write a class with __entry__ and __exit__ and __except__ methods (I've lost track of the exact proposal at this point). At the same time, having to use it as follows: for f in with_file(filename): for line in f: print process(line) is really ugly, so we need new syntax, which also helps with keeping 'for' semantically backwards compatible. So let's use 'with', and then the using code becomes again this: with f = with_file(filename): for line in f: print process(line) Now let me propose a strawman for the translation of the latter into existing semantics. Let's take the generic case: with VAR = EXPR: BODY This would translate to the following code: it = EXPR err = None while True: try: if err is None: VAR = it.next() else: VAR = it.next_ex(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" if not hasattr(it, "next_ex"): raise (The variables 'it' and 'err' are not user-visible variables, they are internal to the translation.) This looks slightly awkward because of backward compatibility; what I really want is just this: it = EXPR err = None while True: try: VAR = it.next(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" pass but for backwards compatibility with the existing argument-less next() API I'm introducing a new iterator API next_ex() which takes an exception argument. If that argument is None, it should behave just like next(). Otherwise, if the iterator is a generator, this will raised that exception in the generator's frame (at the point of the suspended yield). If the iterator is something else, the something else is free to do whatever it likes; if it doesn't want to do anything, it can just re-raise the exception. Also note that, unlike the for-loop translation, this does *not* invoke iter() on the result of EXPR; that's debatable but given that the most common use case should not be an alternate looping syntax (even though it *is* technically a loop) but a more general "macro statement expansion", I think we can expect EXPR to produce a value that is already an iterator (rather than merely an interable). Finally, I think it would be cool if the generator could trap occurrences of break, continue and return occurring in BODY. We could introduce a new class of exceptions for these, named ControlFlow, and (only in the body of a with statement), break would raise BreakFlow, continue would raise ContinueFlow, and return EXPR would raise ReturnFlow(EXPR) (EXPR defaulting to None of course). So a block could return a value to the generator using a return statement; the generator can catch this by catching ReturnFlow. (Syntactic sugar could be "VAR = yield ..." like in Ruby.) With a little extra magic we could also get the behavior that if the generator doesn't handle ControlFlow exceptions but re-raises them, they would affect the code containing the with statement; this means that the generator can decide whether return, break and continue are handled locally or passed through to the containing block. Note that EXPR doesn't have to return a generator; it could be any object that implements next() and next_ex(). (We could also require next_ex() or even next() with an argument; perhaps this is better.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Might this be a good time to introduce __next__ (having the same signature and semantics as your proposed next_ex) and builtin next(obj, exception=None)? def next(obj, exception=None): if hasattr(obj, '__next__'): return obj.__next__(exception) if exception is not None: return obj.next(exception) # Will raise an appropriate exception return obj.next() Tim Delaney

On 4/25/05, Tim Delaney <tcdelaney@optusnet.com.au> wrote:
Hmm, it took me a while to get this, but what you're ssaying is that if you modify Guido's "what I really want" solution to use VAR = next(it, exc) then this builtin next makes "API v2" stuff using __next__ work while remaining backward compatible with old-style "API v1" stuff using 0-arg next() (as long as old-style stuff isn't used in a context where an exception gets passed back in). I'd suggest that the new builtin have a "magic" name (__next__ being the obvious one :-)) to make it clear that it's an internal implementation detail. Paul. PS The first person to replace builtin __next__ in order to implement a "next hook" of some sort, gets shot :-)

Paul Moore wrote:
Yes, but it could also be used (almost) anywhere an explicit obj.next() is used. it = iter(seq) while True: print next(it) for loops would also change to use builtin next() rather than calling it.next() directly.
There aren't many builtins that have magic names, and I don't think this should be one of them - it has obvious uses other than as an implementation detail.
PS The first person to replace builtin __next__ in order to implement a "next hook" of some sort, gets shot :-)
Damn! There goes the use case ;) Tim Delaney

Tim Delaney wrote:
I think there's some confusion here. As I understood the suggestion, __next__ would be the Python name of the method corresponding to the tp_next typeslot, analogously with __len__, __iter__, etc. There would be a builtin function next(obj) which would invoke obj.__next__(), for use by Python code. For loops wouldn't use it, though; they would continue to call the tp_next typeslot directly.
I think he meant next(), not __next__. And it wouldn't work anyway, since as I mentioned above, C code would bypass next() and call the typeslot directly. I'm +1 on moving towards __next__, BTW. IMO, that's the WISHBDITFP. :-) -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Guido van Rossum wrote: [SNIP]
Can I suggest the name next_exc() instead? Everything in the sys module uses "exc" as the abbreviation for "exception". I realize you might be suggesting using the "ex" as the suffix because of the use of that as the suffix in the C API for an extended API, but that usage is not prominent in the stdlib. Also, would this change in Python 3000 so that both next_ex() and next() are merged into a single method? As for an opinion of the need of 'with', I am on the fence, leaning towards liking it. To make sure I am understanding the use case, it is to help encapsulate typical resource management with proper cleanup in another function instead of having to constantly pasting in boilerplate into your code, right? So the hope is to be able to create factory functions, typically implemented as a generator, that encapsulate the obtaining, temporary lending out, and cleanup of a resource? Is there some other use that I am totally missing that is obvious?
Honestly, I am not very comfortable with this magical meaning of 'break', 'continue', and 'return' in a 'with' block. I realize 'return' already has special meaning in an generator, but I don't think that is really needed either. It leads to this odd dichotomy where a non-exception-related statement directly triggers an exception in other code. It seems like code doing something behind my back; "remember, it looks like a 'continue', but it really is a method call with a specific exception instance. Surprise!" Personally, what I would rather see, is to have next_ex(), for a generator, check if the argument is a subclass of Exception. If it is, raise it as such. If not, have the 'yield' statement return the passed-in argument. This use of it would make sense for using the next_ex() name. Then again I guess having exceptions triggering a method call instead of hitting an 'except' statement is already kind of "surprise" semantics anyway. =) Still, I would like to minimize the surprises that we could spring. And before anyone decries the fact that this might confuse a newbie (which seems to happen with every advanced feature ever dreamed up), remember this will not be meant for a newbie but for someone who has experience in Python and iterators at the minimum, and hopefully with generators. Not exactly meant for someone for which raw_input() still holds a "wow" factor for. =)
Yes, that requirement would be good. Will make sure people don't try to use an iterator with the 'with' statement that has not been designed properly for use within the 'with'. And the precedence of requiring an API is set by 'for' since it needs to be an iterable or define __getitem__() as it is. -Brett

"Brett C." <bac@OCF.Berkeley.EDU> wrote in message news:426C54BF.2010906@ocf.berkeley.edu...
I have accepted the fact that Python has become a two-level language: basic Python for expressing algorithms + advanced features (metaclasses, decorators, CPython-specific introspection and hacks, and now possibly 'with' or whatever) for solving software engineering issues. Perhaps there should correspondingly be two tutorials. Terry J. Reedy

Brett C. wrote:
This is dangerously close to the "you don't need to know about it if you're not going to use it" argument, which is widely recognised as false. Newbies might not need to know all the details of the implementation, but they will need to know enough about the semantics of with-statements to understand what they're doing when they come across them in other people's code. Which leads me to another concern. How are we going to explain the externally visible semantics of a with-statement in a way that's easy to grok, without mentioning any details of the implementation? You can explain a for-loop pretty well by saying something like "It executes the body once for each item from the sequence", without having to mention anything about iterators, generators, next() methods, etc. etc. How the items are produced is completely irrelevant to the concept of the for-loop. But what is the equivalent level of description of the with-statement going to say? "It executes the body with... ???" And a related question: What are we going to call the functions designed for with-statements, and the objects they return? Calling them generators and iterators (even though they are) doesn't seem right, because they're being used for a purpose very different from generating and iterating. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Greg Ewing wrote:
I am not saying it is totally to be ignored by people staring at Python code, but we don't need to necessarily spell out the intricacies.
It executes the body, calling next() on the argument name on each time through until the iteration stops.
I like "managers" since they are basically managing resources most of the time for the user. -Brett

On Mon, 25 Apr 2005, Brett C. wrote:
It executes the body, calling next() on the argument name on each time through until the iteration stops.
There's a little more to it than that. But on the whole I do support the goal of finding a simple, short description of what this construct is intended to do. If it can be described accurately in a sentence or two, that's a good sign that the semantics are sufficiently clear and simple.
I like "managers" since they are basically managing resources most of the time for the user.
No, please let's not call them that. "Manager" is a very common word to describe all kinds of classes in object-oriented designs, and it is so generic as to hardly mean anything. (Sorry, i don't have a better alternative at the moment.) -- ?!ng

Brett C. wrote:
It executes the body, calling next() on the argument name on each time through until the iteration stops.
But that's no good, because (1) it mentions next(), which should be an implementation detail, and (2) it talks about iteration, when most of the time the high-level intent has nothing to do with iteration. In other words, this is too low a level of explanation. Greg

Guido van Rossum wrote: [snip illustration of how generators (and other iterators) can be modified to be used in with-blocks]
I'm sure I could get used to it, but my intuition for with f = with_file(filename): for line in f: print process(line) is that the f = with_file(filename) executes only once. That is, as you said, I don't expect this to be a looping syntax. Of course, as long as the generators (or other objects) here yield only one value (like with_file does), then the with-block will execute only once. But because the implementation lets you make the with-block loop if you want, it makes be nervous... I guess it would be helpful to see example where the looping with-block is useful. So far, I think all the examples I've seen have been like with_file, which only executes the block once. Of course, the loop allows you to do anything that you would normally do in a for-loop, but my feeling is that this is probably better done by composing a with-block that executes the block only once with a normal Python for-loop. I'd almost like to see the with-block translated into something like it = EXPR try: VAR = it.next() except StopIteration: raise WithNotStartedException err = None try: BODY except Exception, err: # Pretend "except Exception:" == "except:" pass try: it.next_ex(err) except StopIteration: pass else: raise WithNotEndedException where there is no looping at all, and the iterator is expected to yield exactly one item and then terminate. Of course this looks a lot like: it = EXPR VAR = it.__enter__() err = None try: BODY except Exception, err: # Pretend "except Exception:" == "except:" pass it.__exit__(err) So maybe I'm just still stuck on the enter/exit semantics. ;-) STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

At 09:12 PM 4/24/05 -0600, Steven Bethard wrote:
I guess it would be helpful to see example where the looping with-block is useful.
Automatically retry an operation a set number of times before hard failure: with auto_retry(times=3): do_something_that_might_fail() Process each row of a database query, skipping and logging those that cause a processing error: with x,y,z = log_errors(db_query()): do_something(x,y,z) You'll notice, by the way, that some of these "runtime macros" may be stackable in the expression. I'm somewhat curious what happens to yields in the body of the macro block, but I assume they'll just do what would normally occur. Somehow it seems strange, though, to be yielding to something other than the enclosing 'with' object. In any case, I'm personally more excited about the part where this means we get to build co-routines with less magic. The 'with' statement itself is of interest mainly for acquisition/release and atomic/rollback scenarios, but being able to do retries or skip items that cause errors is often handy. Sometimes you have a list of things (such as event callbacks) where you need to call all of them, even if one handler fails, but you can't afford to silence the errors either. Code that deals with that scenario well is a bitch to write, and a looping 'with' would make it a bit easier to write once and reuse many.

On 4/24/05, Phillip J. Eby <pje@telecommunity.com> wrote:
Thanks for the examples! If I understand your point here right, the examples that can't be easily rewritten by composing a single-execution with-block with a for-loop are examples where the number of iterations of the for-loop depends on the error handling of the with-block. Could you rewrite these with PEP 288 as something like: gen = auto_retry(times=3) for _ in gen: try: do_something_that_might_fail() except Exception, err: # Pretend "except Exception:" == "except:" gen.throw(err) gen = log_errors(db_query()) for x,y,z in gen: try: do_something(x,y,z) except Exception, err: # Pretend "except Exception:" == "except:" gen.throw(err) Obviously, the code is cleaner using the looping with-block. I'm just trying to make sure I understand your examples right. So assuming we had looping with-blocks, what would be the benefit of using a for-loop instead? Just efficiency? Or is there something that a for-loop could do that a with-block couldn't? STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Phillip J. Eby wrote:
These are also possible by combining a normal for loop with a non-looping with (but otherwise using Guido's exception injection semantics): def auto_retry(attempts): success = [False] failures = [0] except = [None] def block(): try: yield None except: failures[0] += 1 else: success[0] = True while not success[0] and failures[0] < attempts: yield block() if not success[0]: raise Exception # You'd actually propagate the last inner failure for attempt in auto_retry(3): with attempt: do_something_that_might_fail() The non-looping version of with seems to give the best of both worlds - multipart operation can be handled by multiple with statements, and repeated use of the same suite can be handled by nesting the with block inside iteration over an appropriate generator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

At 04:57 PM 4/24/05 -0700, Guido van Rossum wrote:
[uncontrolled drooling, followed by much rejoicing] If this were available to generators in general, you could untwist Twisted. I'm basically simulating this sort of exception/value passing in peak.events to do exactly that, except I have to do: yield somethingBlocking(); result=events.resume() where events.resume() magically receives a value or exception from outside the generator and either returns or raises it. If next()-with-argument and next_ex() are available normally on generators, this would allow you to simulate co-routines without the events.resume() magic; the above would simply read: result = yield somethingBlocking() The rest of the peak.events coroutine simulation would remain around to manage the generator stack and scheduling, but the syntax would be cleaner and the operation of it entirely unmagical.

Guido van Rossum wrote:
or with with_file(filename) as f: ... ? (assignment inside block-opening constructs aren't used in Python today, as far as I can tell...)
slightly weird, but useful enough to be cool. (maybe "return value" is enough, though. the others may be slightly too weird... or should that return perhaps be a "continue value"? you're going back to the top of loop, after all). </F>

"Fredrik Lundh" <fredrik@pythonware.com> wrote in message news:d4i6hg$q88$1@sea.gmane.org...
with <target> as <value>: would parallel the for-statement header and read smoother to me. for <target> as <value>: would not need new keyword, but would require close reading to distinguish 'as' from 'in'. Terry J. Reedy

[ Simon Percivall ]:
I do not have strong feelings about this issue, but for completeness sake... Mixing both suggestions: from <target> as <value>: <BODY> That resembles an import statement which some may consider good (syntax/keyword reuse) or very bad (confusion?, value focus). cheers, Senra -- Rodrigo Senra -- MSc Computer Engineer rodsenra(at)gpr.com.br GPr Sistemas Ltda http://www.gpr.com.br/ Personal Blog http://rodsenra.blogspot.com/

I have just noticed that this whole notion is fairly similar to the "local" statement in ML, the syntax for which looks like this: local <declarations> in <declarations> end The idea is that the first declarations, whatever they are, are processed without putting their names into the surrounding scope, then the second declarations are processed *with* putting their names into the surrounding scope. For example: local fun add(x:int, y:int) = x+y in fun succ(x) = add(x, 1) end This defines succ in the surrounding scope, but not add. So in Python terms, I think this would be local: <suite> in: <suite> or, for example: local: <target> = value in: blah blah blah

Guido van Rossum wrote:
Indeed - the transaction example is very easy to write this way: def transaction(): begin_transaction() try: yield None except: abort_transaction() raise else: commit_transaction()
Not supporting iterables makes it harder to write a class which is inherently usable in a with block, though. The natural way to make iterable classes is to use 'yield' in the definition of __iter__ - if iter() is not called, then that trick can't be used.
Perhaps 'continue' could be used to pass a value into the iterator, rather than 'return'? (I believe this has been suggested previously in the context of for loops) This would permit 'return' to continue to mean breaking out of the containing function (as for other loops).
So, "VAR = yield x" would expand to something like: try: yield x except ReturnFlow, ex: VAR = ReturnFlow.value ?
That seems a little bit _too_ magical - it would be nice if break and continue were defined to be local, and return to be non-local, as for the existing loop constructs. For other non-local control flow, application specific exceptions will still be available. Regardless, the ControlFlow exceptions do seem like a very practical way of handling the underlying implementation.
With this restriction (i.e. requiring next_ex, next_exc, or Terry's suggested __next__), then the backward's compatible version would be simply your desired semantics, plus an attribute check to exclude old-style iterators: it = EXPR if not hasattr(it, "__next__"): raise TypeError("'with' block requires 2nd gen iterator API support") err = None while True: try: VAR = it.next(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" pass The generator objects created by using yield would supply the new API, so would be usable immediately inside such 'with' blocks. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Terry Reedy wrote:
If you're defining it by means of a generator, you don't need a class at all -- just make the whole thing a generator function. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

"Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote in message news:426DC75A.1010005@canterbury.ac.nz...
Terry Reedy wrote:
The part you quoted was by Nick Coghlan, not me, as indicated by the >> (now >>>) instead of > (which would now be >>) in front of the lines.
Not supporting iterables makes it harder to write a class which is ...

Guido> At the same time, having to use it as follows: Guido> for f in with_file(filename): Guido> for line in f: Guido> print process(line) Guido> is really ugly, so we need new syntax, which also helps with Guido> keeping 'for' semantically backwards compatible. So let's use Guido> 'with', and then the using code becomes again this: Guido> with f = with_file(filename): Guido> for line in f: Guido> print process(line) How about deferring major new syntax changes until Py3K when the grammar and semantic options might be more numerous? Given the constraints of backwards compatibility, adding more syntax or shoehorning new semantics into what's an increasingly crowded space seems to always result in an unsatisfying compromise. Guido> Now let me propose a strawman for the translation of the latter Guido> into existing semantics. Let's take the generic case: Guido> with VAR = EXPR: Guido> BODY What about a multi-variable case? Will you have to introduce a new level of indentation for each 'with' var? Skip

Guido van Rossum wrote:
I like the general shape of this, but I have one or two reservations about the details. 1) We're going to have to think carefully about the naming of functions designed for use with this statement. If 'with' is going to be in there as a keyword, then it really shouldn't be part of the function name as well. Instead of with f = with_file(pathname): ... I would rather see something like with f = opened(pathname): ... This sort of convention (using a past participle as a function name) would work for some other cases as well: with some_data.locked(): ... with some_resource.allocated(): ... On the negative side, not having anything like 'with' in the function name means that the fact the function is designed for use in a with-statement could be somewhat non-obvious. Since there's not going to be much other use for such a function, this is a bad thing. It could also lead people into subtle usage traps such as with f = open(pathname): ... which would fail in a somewhat obscure way. So maybe the 'with' keyword should be dropped (again!) in favour of with_opened(pathname) as f: ... 2) I'm not sure about the '='. It makes it look rather deceptively like an ordinary assignment, and I'm sure many people are going to wonder what the difference is between with f = opened(pathname): do_stuff_to(f) and simply f = opened(pathname) do_stuff_to(f) or even just unconsciously read the first as the second without noticing that anything special is going on. Especially if they're coming from a language like Pascal which has a much less magical form of with-statement. So maybe it would be better to make it look more different: with opened(pathname) as f: ... * It seems to me that this same exception-handling mechanism would be just as useful in a regular for-loop, and that, once it becomes possible to put 'yield' in a try-statement, people are going to *expect* it to work in for-loops as well. Guido has expressed concern about imposing extra overhead on all for-loops. But would the extra overhead really be all that noticeable? For-loops already put a block on the block stack, so the necessary processing could be incorporated into the code for unwinding a for-block during an exception, and little if anything would need to change in the absence of an exception. However, if for-loops also gain this functionality, we end up with the rather embarrassing situation that there is *no difference* in semantics between a for-loop and a with-statement! This could be "fixed" by making the with-statement not loop, as has been suggested. That was my initial thought as well, but having thought more deeply, I'm starting to think that Guido was right in the first place, and that a with-statement should be capable of looping. I'll elaborate in another post.
This is a very elegant idea, but I'm seriously worried by the possibility that a return statement could do something other than return from the function it's written in, especially if for-loops also gain this functionality. Intercepting break and continue isn't so bad, since they're already associated with the loop they're in, but return has always been an unconditional get-me-out-of-this-function. I'd feel uncomfortable if this were no longer true. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

[Greg Ewing]
I like the general shape of this, but I have one or two reservations about the details.
That summarizes the feedback so far pretty well. I think we're on to something. And I'm not too proud to say that Ruby has led the way here to some extent (even if Python's implementation would be fundamentally different, since it's based on generators, which has some different possibilities and precludes some Ruby patterns).
Of course. I only used 'with_opened' because it's been the running example in this thread.
Or how about with synchronized(some_resource): ...
This seems a pretty mild problem; one could argue that every function is only useful in a context where its return type makes sense, and we seem to be getting along just fine with naming conventions (or just plain clear naming).
Ouch. That one hurts. (I was going to say "but f doesn't have a next() method" when I realized it *does*. :-) It is *almost* equivalent to for f in open(pathname): ... except if the "..." block raises an exception. Fortunately your proposal to use 'as' makes this mistake less likely.
But that doesn't look so great for the case where there's no variable to be assigned to -- I wasn't totally clear about it, but I meant the syntax to be with [VAR =] EXPR: BLOCK where VAR would have the same syntax as the left hand side of an assignment (or the variable in a for-statement).
Right.
Fredrik said this too, and as long as we're going to add 'with' as a new keyword, we might as well promote 'as' to become a real keyword. So then the syntax would become with EXPR [as VAR]: BLOCK I don't see a particular need for assignment to multiple VARs (but VAR can of course be a tuple of identifiers).
(You can already put a yield inside a try-except, just not inside a try-finally.)
Probably.
There would still be the difference that a for-loop invokes iter() and a with-block doesn't. Also, for-loops that don't exhaust the iterator leave it available for later use. I believe there are even examples of this pattern, where one for-loop searches the iterable for some kind of marker value and the next for-loop iterates over the remaining items. For example: f = open(messagefile) # Process message headers for line in f: if not line.strip(): break if line[0].isspace(): addcontinuation(line) else: addheader(line) # Process message body for line in f: addbody(line)
So perhaps the short description of a with-statement that we give to newbies could be the following: """ The statement: for VAR in EXPR: BLOCK does the same thing as: with iter(EXPR) as VAR: # Note the iter() call BLOCK except that: - you can leave out the "as VAR" part from the with-statement; - they work differently when an exception happens inside BLOCK; - break and continue don't always work the same way. The only time you should write a with-statement is when the documentation for the function you are calling says you should. """
But they wouldn't!
Me too. Let me explain the use cases that led me to throwing that in (I ws running out of time and didn't properly explain it) and then let me propose an alternative. This is a bit long, but important! *First*, in the non-looping use cases (like acquiring and releasing a lock), a return-statement should definitely be allowed when the with-statement is contained in a function. There's lots of code like this out there: def search(self, eligible, default=None): self.lock.acquire() try: for item in self.elements: if eligible(item): return item # no eligible iems return default finally: self.lock.release() and this translates quite nicely to a with-statement: def search(self, eligible, default=None): with synchronized(self.lock): for item in self.elements: if eligible(item): return item # no eligible iems return default *Second*, it might make sense if break and continue would be handled the same way; here's an example: def alt_search(self): for item in self.elements: with synchronized(item): if item.abandoned(): continue if item.eligible(): break else: item = self.default_item return item.post_process() (I realize the case for continue isn't as strong as that for break, but I think we have to support both if we support one.) *Third*, if there is a try-finally block around a yield in the generator, the finally clause absolutely must be executed when control leaves the body of the with-statement, whether it is through return, break, or continue. This pretty much means these have to be turned into some kind of exception. So the first example would first be transformed into this: def search(self, eligible, default=None): try: with synchronized(self.lock): for item in self.elements: if eligible(item): raise ReturnFlow(item) # was "return item" # no eligible iems raise ReturnFlow(default) # was "return default" except ReturnFlow, exc: return exc.value before applying the transformation of the with-statement, which I won't repeat here (look it up in my previous long post in this thread). (BTW I do agree that it should use __next__(), not next_ex().) I'm assuming the following definition of the ReturnFlow exception: class ReturnFlow(Exception): def __init__(self, value=None): self.value = value The translation of break into raise BreakFlow() and continue into rase ContinueFlow() is now obvious. (BTW ReturnFlow etc. aren't great names. Suggestions?) *Fourth*, and this is what makes Greg and me uncomfortable at the same time as making Phillip and other event-handling folks drool: from the previous three points it follows that an iterator may *intercept* any or all of ReturnFlow, BreakFlow and ContinueFlow, and use them to implement whatever cool or confusing magic they want. For example, a generator can decide that for the purposes of break and continue, the with-statement that calls it is a loop, and give them the usual semantics (or the opposite, if you're into that sort of thing :-). Or a generator can receive a value from the block via a return statement. Notes: - I think there's a better word than Flow, but I'll keep using it until we find something better. - This is not limited to generators -- the with-statement uses an arbitrary "new-style" iterator (something with a __next__() method taking an optional exception argument). - The new __next__() API can also (nay, *must*, to make all this work reliably) be used to define exception and cleanup semantics for generators, thereby rendering obsolete PEP 325 and the second half of PEP 288. When a generator is GC'ed (whether by reference counting or by the cyclical garbage collector), its __next__() method is called with a BreakFlow exception instance as argument (or perhaps some other special exception created for the purpose). If the generator catches the exception and yields another value, too bad -- I consider that broken behavior. (The alternative would be to keep calling __next__(BreakFlow()) until it doesn't return a value, but that feels uncomfortable in a finalization context.) - Inside a with-statement, user code raising a Flow exception acts the same as the corresponding statement. This is slightly unfortunate, because it might lead one to assume that the same is true for example in a for-loop or while-loop, but I don't want to make that change. I don't think it's a big problem. Given that 1, 2 and 3 combined make 4 inevitable, I think we might as well give in, and *always* syntactically accept return, break and continue in a with-statement, whether or not it is contained in a loop or function. When the iterator does not handle the Flow exceptions, and there is no outer context in which the statement is valid, the Flow exception is turned into an IllegalFlow exception, which is the run-time equivalent of SyntaxError: 'return' outside function (or 'break' outside loop, etc.). Now there's one more twist, which you may or may not like. Presumably (barring obfuscations or bugs) the handling of BreakFlow and ContinueFlow by an iterator (or generator) is consistent for all uses of that particular iterator. For example synchronized(lock) and transactional(db) do not behave as loops, and forever() does. Ditto for handling ReturnFlow. This is why I've been thinking of leaving out the 'with' keyword: in your mind, these calls would become new statement types, even though the compiler sees them all the same: synchronized(lock): BLOCK transactional(db): BLOCK forever(): BLOCK opening(filename) as f: BLOCK It does require the authors of such iterators to pick good names, and it doesn't look as good when the iterator is a method of some object: self.elements[0].locker.synchronized(): BLOCK You proposed this too (and I even commented on it, ages ago in this same endless message :-) and while I'm still on the fence, at least I now have a better motivational argument (i.e., that each iterator becomes a new statement type in your mind). One last thing: if we need a special name for iterators and generators designed for use in a with-statement, how about calling them with-iterators and with-generators. The non-looping kind can be called resource management iterators / generators. I think whatever term we come up with should not be a totally new term but a combination of iterator or generator with some prefix, and it should work both for iterators and for generators. That's all I can muster right now (I should've been in bed hours ago) but I'm feeling pretty good about this. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Five random thoughts: 1. So if break and continue are allowed in with statements only when there is an enclosing loop, it would be a inconsistency; consider for item in seq: with gen(): continue when the generator gen catches the ContinueFlow and does with it what it wants. It is then slightly unfair not to allow with x: continue Anyway, I would consider both counterintuitive. So what about making ReturnFlow, BreakFlow and ContinueFlow "private" exceptions that cannot be caught in user code and instead introducing a new statement that allows passing data to the generator? 2. In process of handling this, would it be reasonable to (re)introduce a combined try-except-finally statement with defined syntax (all except before finally) and behavior (finally is always executed)? 5. What about the intended usage of 'with' as in Visual B.. NO, NO, NOT THE WHIP! (not that you couldn't emulate this with a clever "generator": def short(x): yield x with short(my.long["object"]reference()) as _: _.spam = _.ham = _.eggs() yours, Reinhold -- Mail address is perfectly valid!

Guido van Rossum wrote:
Well, my point still stands. People are going to write try-finally around their yields and expect the natural thing to happen when their generator is used in a for-loop.
Hmmm. But are these big enough differences to justify having a whole new control structure? Whither TOOWTDI?
Surely you jest. Any newbie reading this is going to think he hasn't a hope in hell of ever understanding what is going on here, and give up on Python in disgust.
Let me explain the use cases that led me to throwing that in
Yes, I can see that it's going to be necessary to treat return as an exception, and accept the possibility that it will be abused. I'd still much prefer people refrain from abusing it that way, though. Using "return" to spell "send value back to yield statement" would be extremely obfuscatory.
(BTW ReturnFlow etc. aren't great names. Suggestions?)
I'd suggest just calling them Break, Continue and Return.
Hey, I like that last one! Well done!
Except that if it's no longer a "with" statement, this doesn't make so much sense... Greg

[Guido]
(You can already put a yield inside a try-except, just not inside a try-finally.)
[Greg]
Well, the new finalization semantics should take care of that when their generator is finalized -- its __next__() will be called with some exception. But as long you hang on to the generator, it will not be finalized, which is distinctly different from the desired with-statement semantics.
Indeed, but apart from declaring that henceforth the with-statement (by whatever name) is the recommended looping construct and a for-statement is just a backwards compatibility macro, I just don't see how we can implement the necessary immediate cleanup semantics of a with-statement. In order to serve as a resource cleanup statement it *must* have stronger cleanup guarantees than the for-statement can give (if only for backwards compatibility reasons).
And surely you exaggerate. How about this then: The with-statement is similar to the for-loop. Until you've learned about the differences in detail, the only time you should write a with-statement is when the documentation for the function you are calling says you should.
That depends on where you're coming from. To Ruby users it will look completely natural because that's what Ruby uses. (In fact it'll be a while before they appreciate the deep differences between yield in Python and in Ruby.) But I accept that in Python we might want to use a different keyword to pass a value to the generator. I think using 'continue' should work; continue with a value has no precedent in Python, and continue without a value happens to have exactly the right semantics anyway.
Too close to break, continue and return IMO.
Then of course we'll call it after whatever the new statement is going to be called. If we end up calling it the foible-statement, they will be foible-iterators and foible-generators. Anyway, I think I'll need to start writing a PEP. I'll ask the PEP editor for a number. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

I've written a PEP about this topic. It's PEP 340: Anonymous Block Statements (http://python.org/peps/pep-0340.html). Some highlights: - temporarily sidestepping the syntax by proposing 'block' instead of 'with' - __next__() argument simplified to StopIteration or ContinueIteration instance - use "continue EXPR" to pass a value to the generator - generator exception handling explained -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
I'm still trying to build a case for a non-looping block statement, but the proposed enhancements to generators look great. Any further suggestions I make regarding a PEP 310 style block statement will account for those generator changes. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Guido van Rossum wrote:
This looks pretty cool. Some observations: 1. It looks to me like a bare return or a return with an EXPR3 that happens to evaluate to None inside a block simply exits the block, rather than exiting a surrounding function. Did I miss something, or is this a bug? 2. I assume it would be a hack to try to use block statements to implement something like interfaces or classes, because doing so would require significant local-variable manipulation. I'm guessing that either implementing interfaces (or implementing a class statement in which the class was created before execution of a suite) is not a use case for this PEP. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton <jim@zope.com> wrote in news:426F7A8F.8090109@zope.com:
No, the return sets a flag and raises StopIteration which should make the iterator also raise StopIteration at which point the real return happens. If the iterator fails to re-raise the StopIteration exception (the spec only says it should, not that it must) I think the return would be ignored but a subsquent exception would then get converted into a return value. I think the flag needs reset to avoid this case. Also, I wonder whether other exceptions from next() shouldn't be handled a bit differently. If BLOCK1 throws an exception, and this causes the iterator to also throw an exception then one exception will be lost. I think it would be better to propogate the original exception rather than the second exception. So something like (added lines to handle both of the above): itr = EXPR1 exc = arg = None ret = False while True: try: VAR1 = next(itr, arg) except StopIteration: if exc is not None: if ret: return exc else: raise exc # XXX See below break + except: + if ret or exc is None: + raise + raise exc # XXX See below + ret = False try: exc = arg = None BLOCK1 except Exception, exc: arg = StopIteration()

Duncan Booth wrote:
Only if exc is not None The only return in the pseudocode is inside "if exc is not None". Is there another return that's not shown? ;) I agree that we leave the block, but it doesn't look like we leave the surrounding scope. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

I feel like we're quietly, delicately tiptoeing toward continuations...
No way we aren't. We're not really adding anything to the existing generator machinery (the exception/value passing is a trivial modification) and that is only capable of 80% of coroutines (but it's the 80% you need most :-). As long as I am BDFL Python is unlikely to get continuations -- my head explodes each time someone tries to explain them to me. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Thanks! This was a bug in the PEP due to a last-minute change in how I wanted to handle return; I've fixed it as you show (also renaming 'exc' to 'var' since it doesn't always hold an exception). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Good catch. I've fixed this in the PEP.
I don't think so. It's similar to this case: try: raise Foo except: raise Bar Here, Foo is also lost. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

[Jim Fulton]
I would like to get back to the discussion about interfaces and signature type declarations at some point, and a syntax dedicated to declaring interfaces is high on my wish list. In the mean time, if you need interfaces today, I think using metaclasses would be easier than using a block-statement (if it were even possible using the latter without passing locals() to the generator). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

At 12:30 AM 4/27/05 -0700, Guido van Rossum wrote:
Very nice. It's not clear from the text, btw, if normal exceptions can be passed into __next__, and if so, whether they can include a traceback. If they *can*, then generators can also be considered co-routines now, in which case it might make sense to call blocks "coroutine blocks", because they're basically a way to interleave a block of code with the execution of a specified coroutine.

[Phillip Eby]
The PEP is clear on this: __next__() only takes Iteration instances, i.e., StopIteration and ContinueIteration. (But see below.) I'm not sure what the relevance of including a stack trace would be, and why that feature would be necessary to call them coroutines. But... Maybe it would be nice if generators could also be used to implement exception handling patterns, rather than just resource release patterns. IOW, maybe this should work: def safeLoop(seq): for var in seq: try: yield var except Exception, err: print "ignored", var, ":", err.__class__.__name__ block safeLoop([10, 5, 0, 20]) as x: print 1.0/x This should print 0.1 0.2 ignored 0 : ZeroDivisionError 0.02 I've been thinking of alternative signatures for the __next__() method to handle this. We have the following use cases: 1. plain old next() 2. passing a value from continue EXPR 3. forcing a break due to a break statement 4. forcing a break due to a return statement 5. passing an exception EXC Cases 3 and 4 are really the same; I don't think the generator needs to know the difference between a break and a return statement. And these can be mapped to case 5 with EXC being StopIteration(). Now the simplest API would be this: if the argument to __next__() is an exception instance (let's say we're talking Python 3000, where all exceptions are subclasses of Exception), it is raised when yield resumes; otherwise it is the return value from yield (may be None). This is somewhat unsatisfactory because it means that you can't pass an exception instance as a value. I don't know how much of a problem this will be in practice; I could see it causing unpleasant surprises when someone designs an API around this that takes an arbitrary object, when someone tries to pass an exception instance. Fixing such a thing could be expensive (you'd have to change the API to pass the object wrapped in a list or something). An alternative that solves this would be to give __next__() a second argument, which is a bool that should be true when the first argument is an exception that should be raised. What do people think? I'll add this to the PEP as an alternative for now. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

At 01:27 PM 4/27/05 -0700, Guido van Rossum wrote:
Well, you need that feature in order to retain traceback information when you're simulating threads with a stack of generators. Although you can't return from a generator inside a nested generator, you can simulate this by keeping a stack of generators and having a wrapper that passes control between generators, such that: def somegen(): result = yield othergen() causes the wrapper to push othergen() on the generator stack and execute it. If othergen() raises an error, the wrapper resumes somegen() and passes in the error. If you can only specify the value but not the traceback, you lose the information about where the error occurred in othergen(). So, the feature is necessary for anything other than "simple" (i.e. single-frame) coroutines, at least if you want to retain any possibility of debugging. :)
Yes, it would be nice. Also, you may have just come up with an even better word for what these things should be called... patterns. Perhaps they could be called "pattern blocks" or "patterned blocks". Pattern sounds so much more hip and politically correct than "macro" or even "code block". :)
I think it'd be simpler just to have two methods, conceptually "resume(value=None)" and "error(value,tb=None)", whatever the actual method names are.

[Guido]
I'm not sure what the relevance of including a stack trace would be, and why that feature would be necessary to call them coroutines.
[Phillip]
OK. I think you must be describing continuations there, because my brain just exploded. :-) In Python 3000 I want to make the traceback a standard attribute of Exception instances; would that suffice? I really don't want to pass the whole (type, value, traceback) triple that currently represents an exception through __next__().
Yes, but the word has a much loftier meaning. I could get used to template blocks though (template being a specific pattern, and this whole thing being a non-OO version of the Template Method Pattern from the GoF book).
Part of me likes this suggestion, but part of me worries that it complicates the iterator API too much. Your resume() would be __next__(), but that means your error() would become __error__(). This is more along the lines of PEP 288 and PEP 325 (and even PEP 310), but we have a twist here in that it is totally acceptable (see my example) for __error__() to return the next value or raise StopIteration. IOW the return behavior of __error__() is the same as that of __next__(). Fredrik, what does your intuition tell you? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

At 02:50 PM 4/27/05 -0700, Guido van Rossum wrote:
Probably my attempt at a *brief* explanation backfired. No, they're not continuations or anything nearly that complicated. I'm "just" simulating threads using generators that yield a nested generator when they need to do something that might block waiting for I/O. The pseudothread object pushes the yielded generator-iterator and resumes it. If that generator-iterator raises an error, the pseudothread catches it, pops the previous generator-iterator, and passes the error into it, traceback and all. The net result is that as long as you use a "yield expression" for any function/method call that might do blocking I/O, and those functions or methods are written as generators, you get the benefits of Twisted (async I/O without threading headaches) without having to "twist" your code into the callback-registration patterns of Twisted. And, by passing in errors with tracebacks, the normal process of exception call-stack unwinding combined with pseudothread stack popping results in a traceback that looks just as if you had called the functions or methods normally, rather than via the pseudothreading mechanism. Without that, you would only get the error context of 'async_readline()', because the traceback wouldn't be able to show who *called* async_readline.
In Python 3000 I want to make the traceback a standard attribute of Exception instances; would that suffice?
If you're planning to make 'raise' reraise it, such that 'raise exc' is equivalent to 'raise type(exc), exc, exc.traceback'. Is that what you mean? (i.e., just making it easier to pass the darn things around) If so, then I could probably do what I need as long as there exist no error types whose instances disallow setting a 'traceback' attribute on them after the fact. Of course, if Exception provides a slot (or dictionary) for this, then it shouldn't be a problem. Of course, it seems to me that you also have the problem of adding to the traceback when the same error is reraised... All in all it seems more complex than just allowing an exception and a traceback to be passed.
The point of passing it in is so that the traceback can be preserved without special action in the body of generators the exception is passing through. I could be wrong, but it seems to me you need this even for PEP 340, if you're going to support error management templates, and want tracebacks to include the line in the block where the error originated. Just reraising the error inside the generator doesn't seem like it would be enough.
I was thinking that maybe these would be a "coroutine API" or "generator API" instead. That is, something not usable except with generator-iterators and with *new* objects written to conform to it. I don't really see a lot of value in making template blocks work with existing iterators. For that matter, I don't see a lot of value in hand-writing new objects with resume/error, instead of just using a generator. So, I guess I'm thinking you'd have something like tp_block_resume and tp_block_error type slots, and generators' tp_iter_next would just be the same as tp_block_resume(None). But maybe this is the part you're thinking is complicated. :)

[Phillip]
OK, I sort of get it, at a very high-level, although I still feel this is wildly out of my league. I guess I should try it first. ;-)
Right, this would be a standard part of the Exception base class, just like in Java.
Of course, it seems to me that you also have the problem of adding to the traceback when the same error is reraised...
I think when it is re-raised, no traceback entry should be added; the place that re-raises it should not show up in the traceback, only the place that raised it in the first place. To me that's the essence of re-raising (and I think that's how it works when you use raise without arguments).
All in all it seems more complex than just allowing an exception and a traceback to be passed.
Making the traceback a standard attribute of the exception sounds simpler; having to keep track of two separate arguments that are as closely related as an exception and the corresponding traceback is more complex IMO. The only reason why it isn't done that way in current Python is that it couldn't be done that way back when exceptions were strings.
*** I have to think about this more... ***
(You mean existing non-generator iterators, right? existing *generators* will work just fine -- the exception will pass right through them and that's exactly the right default semantics. Existing non-generator iterators are indeed a different case, and this is actually an argument for having a separate API: if the __error__() method doesn't exist, the exception is just re-raised rather than bothering the iterator. OK, I think I'm sold.
For that matter, I don't see a lot of value in hand-writing new objects with resume/error, instead of just using a generator.
Not a lot, but I expect that there may be a few, like an optimized version of lock synchronization.
No, this is where I feel right at home. ;-) I hadn't thought much about the C-level slots yet, but this is a reasonable proposal. Time to update the PEP; I'm pretty much settled on these semantics now... -- --Guido van Rossum (home page: http://www.python.org/~guido/)

At 03:58 PM 4/27/05 -0700, Guido van Rossum wrote:
It's not unlike David Mertz' articles on implementing coroutines and multitasking using generators, except that I'm adding more "debugging sugar", if you will, by making the tracebacks look normal. It's just that the *how* requires me to pass the traceback into the generator. At the moment, I accomplish that by doing a 3-argument raise inside of 'events.resume()', but it would be really nice to be able to get rid of 'events.resume()' in a future version of Python.
I think maybe I misspoke. I mean adding to the traceback *so* that when the same error is reraised, the intervening frames are included, rather than lost. In other words, IIRC, the traceback chain is normally increased by one entry for each frame the exception escapes. However, if you start hiding that inside of the exception instance, you'll have to modify it instead of just modifying the threadstate. Does that make sense, or am I missing something?
My point was mainly that we can err on the side of caller convenience rather than callee convenience, if there are fewer implementations. So, e.g. multiple methods aren't a big deal if it makes the 'block' implementation simpler, if only generators and a handful of special template objects are going need to implement the block API.
Note that it also doesn't require a 'next()' builtin, or a next vs. __next__ distinction, if you don't try to overload iteration and templating. The fact that a generator can be used for templating, doesn't have to imply that any iterator should be usable as a template, or that the iteration protocol is involved in any way. You could just have __resume__/__error__ matching the tp_block_* slots. This also has the benefit of making the delineation between template blocks and for loops more concrete. For example, this: block open("filename") as f: ... could be an immediate TypeError (due to the lack of a __resume__) instead of biting you later on in the block when you try to do something with f, or because the block is repeating for each line of the file, etc.

[Phillip]
I'm not familiar with Mertz' articles and frankly I still fear it's head-explosive material. ;-)
Adding to the traceback chain already in the exception object is totally kosher, if that's where the traceback is kept.
Well, the way my translation is currently written, writing next(itr, arg, exc) is a lot more convenient for the caller than having to write # if exc is True, arg is an exception; otherwise arg is a value if exc: err = getattr(itr, "__error__", None) if err is not None: VAR1 = err(arg) else: raise arg else: VAR1 = next(itr, arg) but since this will actually be code generated by the bytecode compiler, I think callee convenience is more important. And the ability to default __error__ to raise the exception makes a lot of sense. And we could wrap all this inside the next() built-in -- even if the actual object should have separate __next__() and __error__() methods, the user-facing built-in next() function might take an extra flag to indicate that the argument is an exception, and to handle it appropriate (like shown above).
I'm not convinced of that, especially since all *generators* will automatically be usable as templates, whether or not they were intended as such. And why *shouldn't* you be allowed to use a block for looping, if you like the exit behavior (guaranteeing that the iterator is exhausted when you leave the block in any way)? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

At 05:19 PM 4/27/05 -0700, Guido van Rossum wrote:
It doesn't guarantee that, does it? (Re-reads PEP.) Aha, for *generators* it does, because it says passing StopIteration in, stops execution of the generator. But it doesn't say anything about whether iterators in general are allowed to be resumed afterward, just that they should not yield a value in response to the __next__, IIUC. As currently written, it sounds like existing non-generator iterators would not be forced to an exhausted state. As for the generator-vs-template distinction, I'd almost say that argues in favor of requiring some small extra distinction to make a generator template-safe, rather than in favor of making all iterators template-promiscuous, as it were. Perhaps a '@block_template' decorator on the generator? This would have the advantage of documenting the fact that the generator was written with that purpose in mind. It seems to me that using a template block to loop over a normal iterator is a TOOWTDI violation, but perhaps you're seeing something deeper here...?

Phillip J. Eby wrote:
I wonder if something can be done like what was done for (dare I say it?) "old-style" iterators: "The intention of the protocol is that once an iterator's next() method raises StopIteration, it will continue to do so on subsequent calls. Implementations that do not obey this property are deemed broken. (This constraint was added in Python 2.3; in Python 2.2, various iterators are broken according to this rule.)"[1] This would mean that if next(itr, ...) raised StopIteration, then next(itr, ...) should continue to raise StopIteration on subsequent calls. I don't know how this is done in the current implementation. Would it be hard to do so for the proposed block-statements? If nothing else, we might at least clearly document what well-behaved iterators should do... STeVe [1] http://docs.python.org/lib/typeiter.html -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

On Wed, Apr 27, 2005 at 03:58:14PM -0700, Guido van Rossum wrote:
Time to update the PEP; I'm pretty much settled on these semantics now...
[I'm trying to do a bit of Guido channeling here. I fear I may not be entirely successful.] The the __error__ method seems to simplify things a lot. The purpose of the __error__ method is to notify the iterator that the loop has been exited in some unusual way (i.e. not via a StopIteration raised by the iterator itself). The translation of a block-statement could become: itr = EXPR1 arg = None while True: try: VAR1 = next(itr, arg) except StopIteration: break try: arg = None BLOCK1 except Exception, exc: err = getattr(itr, '__error__', None) if err is None: raise exc err(exc) The translation of "continue EXPR2" would become: arg = EXPR2 continue The translation of "break" inside a block-statement would become: err = getattr(itr, '__error__', None) if err is not None: err(StopIteration()) break The translation of "return EXPR3" inside a block-statement would become: err = getattr(itr, '__error__', None) if err is not None: err(StopIteration()) return EXPR3 For generators, calling __error__ with a StopIteration instance would execute any 'finally' block. Any other argument to __error__ would get re-raised by the generator instance. You could then write: def opened(filename): fp = open(filename) try: yield fp finally: fp.close() and use it like this: block opened(filename) as fp: .... The main difference between 'for' and 'block' is that more iteration may happen after breaking or returning out of a 'for' loop. An iterator used in a block statement is always used up before the block is exited. Maybe __error__ should be called __break__ instead. StopIteration is not really an error. If it is called something like __break__, does it really need to accept an argument? Of hand I can't think of what an iterator might do with an exception. Neil

Neil Schemenauer wrote:
Seems great to me. Clean separation of when the block wants things to keep going if it can and when it wants to let the generator it's all done.
This constant use of the phrase "used up" for these blocks is bugging me slightly. It isn't like the passed-in generator is having next() called on it until it stops, it is just finishing up (or cleaning up, choose your favorite term). It may have had more iterations to go, but the block signaled it was done and thus the generator got its chance to finish up and wipe pick up after itself.
Maybe __error__ should be called __break__ instead.
I like that.
Could just make the default value be StopIteration. Is there really a perk to __break__ only raising StopIteration and not accepting an argument? The real question of whether people would use the ability of raising other exceptions passed in from the block. If you view yield expressions as method calls, then being able to call __break__ with other exceptions makes sense since you might code up try/except statements within the generator and that will care about what kind of exception gets raised. -Brett

Neil Schemenauer wrote:
This is only one case right? Any exception (including StopIteration) passed to a generator's __error__ method will just be re-raised at the point of the last yield, right? Or is there a need to special-case StopIteration? STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Neil Schemenauer wrote:
That can't be right. When __error__ is called, if the iterator catches the exception and goes on to do another yield, the yielded value needs to be assigned to VAR1 and the block executed again. It looks like your version will ignore the value from the second yield and only execute the block again on the third yield. So something like Guido's safe_loop() would miss every other yield. I think Guido was right in the first place, and __error__ really is just a minor variation on __next__ that shouldn't have a separate entry point. Greg

On 4/28/05, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Could you do something like: itr = EXPR1 arg = None next_func = next while True: try: VAR1 = next_func(itr, arg) except StopIteration: break try: arg = None next_func = next BLOCK1 except Exception, arg: try: next_func = type(itr).__error__ except AttributeError: raise arg ? STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Guido van Rossum wrote:
Fredrik, what does your intuition tell you?
having been busy with other stuff for nearly a week, and seeing that the PEP is now at version 1.22, my intuition tells me that it's time to read the PEP again before I have any opinion on anything ;-) </F>

Guido van Rossum wrote:
An optional third argument (raise=False) seems a lot friendlier (and more flexible) than a typecheck. Yet another alternative would be for the default behaviour to be to raise Exceptions, and continue with anything else, and have the third argument be "raise_exc=True" and set it to False to pass an exception in without raising it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

[Guido]
[Nick]
An optional third argument (raise=False) seems a lot friendlier (and more flexible) than a typecheck.
I think I agree, especially since Phillip's alternative (a different method) is even worse IMO.
You've lost me there. If you care about this, can you write it up in more detail (with code samples or whatever)? Or we can agree on a 2nd arg to __next__() (and a 3rd one to next()). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Brett C. wrote:
Pretty close, although I'd say 'could' rather than 'should', as it was an idle thought, rather than something I actually consider a good idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Guido van Rossum <gvanrossum@gmail.com> wrote:
Your code for the translation of a standard for loop is flawed. From the PEP: for VAR1 in EXPR1: BLOCK1 else: BLOCK2 will be translated as follows: itr = iter(EXPR1) arg = None while True: try: VAR1 = next(itr, arg) finally: break arg = None BLOCK1 else: BLOCK2 Note that in the translated version, BLOCK2 can only ever execute if next raises a StopIteration in the call, and BLOCK1 will never be executed because of the 'break' in the finally clause. Unless it is too early for me, I believe what you wanted is... itr = iter(EXPR1) arg = None while True: VAR1 = next(itr, arg) arg = None BLOCK1 else: BLOCK2 - Josiah

Ouch. Another bug in the PEP. It was late. ;-) The "finally:" should have been "except StopIteration:" I've updated the PEP online.
No, this would just propagate the StopIteration when next() raises it. StopIteration is not caught implicitly except around the next() call made by the for-loop control code. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 4/27/05, Guido van Rossum <gvanrossum@gmail.com> wrote:
I've written a PEP about this topic. It's PEP 340: Anonymous Block Statements (http://python.org/peps/pep-0340.html).
So block-statements would be very much like for-loops, except: (1) iter() is not called on the expression (2) the fact that break, continue, return or a raised Exception occurred can all be intercepted by the block-iterator/generator, though break, return and a raised Exception all look the same to the block-iterator/generator (they are signaled with a StopIteration) (3) the while loop can only be broken out of by next() raising a StopIteration, so all well-behaved iterators will be exhausted when the block-statement is exited Hope I got that mostly right. I know this is looking a little far ahead, but is the intention that even in Python 3.0 for-loops and block-statements will still be separate statements? It seems like there's a pretty large section of overlap. Playing with for-loop semantics right now isn't possible due to backwards compatibility, but when that limitation is removed in Python 3.0, are we hoping that these two similar structures will be expressed in a single statement? STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Guido van Rossum wrote:
I am at least +0 on all of this now, with a slow warming up to +1 (but then it might just be the cold talking =). I still prefer the idea of arguments to __next__() be raised if they are exceptions and otherwise just be returned through the yield expression. But I do realize this is easily solved with a helper function now:: def raise_or_yield(val): """Return the argument if not an exception, otherwise raise it. Meant to have a yield expression as an argument. Worries about Iteration subclasses are invalid since they will have been handled by the __next__() method on the generator already. """ if isinstance(val, Exception): raise val else: return val My objections that I had earlier to 'continue' and 'break' being somewhat magical in block statements has subsided. It all seems reasonable now within the context of a block statement. And while the thought is in my head, I think block statements should be viewed less as a tweaked version of a 'for' loop and more as an extension to generators that happens to be very handy for resource management (while allowing iterators to come over and play on the new swing set as well). I think if you take that view then the argument that they are too similar to 'for' loops loses some luster (although I doubt Nick is going to be buy this =) . Basically block statements are providing a simplified, syntactically supported way to control a generator externally from itself (or at least this is the impression I am getting). I just had a flash of worry about how this would work in terms of abstractions of things to functions with block statements in them, but then I realized you just push more code into the generator and handle it there with the block statement just driving the generator. Seems like this might provide that last key piece for generators to finally provide cool flow control that we all know they are capable of but just required extra work beforehand. -Brett

Brett C. wrote:
I'm surprisingly close to agreeing with you, actually. I've worked out that it isn't the looping that I object to, it's the inability to get out of the loop without exhausting the entire iterator. I need to think about some ideas involving iterator factories, then my objections may disappear. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
'break' isn't' enough for you as laid out by the proposal? The raising of StopIteration, which is what 'break' does according to the standard, should be enough to stop the loop without exhausting things. Same way you stop a 'for' loop from executing entirely. -Brett

Brett C. wrote:
The StopIteration exception effectively exhausted the generator, though. However, I've figured out how to deal with that, and my reservations about PEP 340 are basically gone. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Guido van Rossum wrote:
+1 A minor sticking point - I don't like that the generator has to re-raise any ``StopIteration`` passed in. Would it be possible to have the semantics be: If a generator is resumed with ``StopIteration``, the exception is raised at the resumption point (and stored for later use). When the generator exits normally (i.e. ``return`` or falls off the end) it re-raises the stored exception (if any) or raises a new ``StopIteration`` exception. So a generator would become effectively:: try: stopexc = None exc = None BLOCK1 finally: if exc is not None: raise exc if stopexc is not None: raise stopexc raise StopIteration where within BLOCK1: ``raise <exception>`` is equivalent to:: exc = <exception> return The start of an ``except`` clause sets ``exc`` to None (if the clause is executed of course). Calling ``__next__(exception)`` with ``StopIteration`` is equivalent to:: stopexc = exception (raise exception at resumption point) Calling ``__next__(exception)`` with ``ContinueIteration`` is equivalent to:: (resume exception with exception.value) Calling ``__next__(exception)__`` with any other value just raises that value at the resumption point - this allows for calling with arbitrary exceptions. Also, within a for-loop or block-statement, we could have ``raise <exception>`` be equivalent to:: arg = <exception> continue This also takes care of Brett's concern about distinguishing between exceptions and values passed to the generator. Anything except StopIteration or ContinueIteration will be presumed to be an exception and will be raised. Anything passed via ContinueIteration is a value. Tim Delaney

I don't like the idea of storing exceptions. Let's just say that we don't care whether it re-raises the very same StopIteration exception that was passed in or a different one -- it's all moot anyway because the StopIteration instance is thrown away by the caller of next(). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

It forbids returning a value, since that would mean the generator could "refuse" a break or return statement, which is a little bit too weird (returning a value instead would turn these into continue statements). I'll change this to clarify that I don't care about the identity of the StopException instance. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, Apr 27, 2005 at 12:30:22AM -0700, Guido van Rossum wrote:
I've written a PEP about this topic. It's PEP 340: Anonymous Block Statements (http://python.org/peps/pep-0340.html).
[Note: most of these comments are based on version 1.2 of the PEP] It seems like what you are proposing is a limited form of coroutines. Just as Python's generators are limited (yield can only jump up one stack frame), these coroutines have a similar limitation. Someone mentioned that we are edging closer to continuations. I think that may be a good thing. One big difference between what you propose and general continuations is in finalization semantics. I don't think anyone has figured out a way for try/finally to work with continuations. The fact that try/finally can be used inside generators is a significant feature of this PEP, IMO. Regarding the syntax, I actually quite like the 'block' keyword. It doesn't seem so surprising that the block may be a loop. Allowing 'continue' to have an optional value is elegant syntax. I'm a little bit concerned about what happens if the iterator does not expect a value. If I understand the PEP, it is silently ignored. That seems like it could hide bugs. OTOH, it doesn't seem any worse then a caller not expecting a return value. It's interesting that there is such similarity between 'for' and 'block'. Why is it that block does not call iter() on EXPR1? I guess that fact that 'break' and 'return' work differently is a more significant difference. After thinking about this more, I wonder if iterators meant for 'for' loops and iterators meant for 'block' statements are really very different things. It seems like a block-iterator really needs to handle yield-expressions. I wonder if generators that contain a yield-expression should properly be called coroutines. Practically, I suspect it would just cause confusion. Perhaps passing an Iteration instance to next() should not be treated the same as passing None. It seems like that would implementing the iterator easier. Why not treat Iterator like any normal value? Then only None, StopIteration, and ContinueIteration would be special. Argh, it took me so long to write this that you are already up to version 1.6 of the PEP. Time to start a new message. :-) Neil

It seems like what you are proposing is a limited form of coroutines.
Well, I though that's already what generators were -- IMO there isn't much news there. We're providing a more convenient way to pass a value back, but that's always been possible (see Fredrik's examples).
Exactly.
Well, perhaps block *should* call iter()? I'd like to hear votes about this. In most cases that would make a block-statement entirely equivalent to a for-loop, the exception being only when there's an exception or when breaking out of an iterator with resource management. I initially decided it should not call iter() so as to emphasize that this isn't supposed to be used for looping over sequences -- EXPR1 is really expected to be a resource management generator (or iterator).
But who knows, they might be useful for for-loops as well. After all, passing values back to the generator has been on some people's wish list for a long time.
I have to admit that I haven't looked carefully for use cases for this! I just looked at a few Ruby examples and realized that it would be a fairly simple extension of generators. You can call such generators coroutines, but they are still generators. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote: [SNIP]
I am -0 on changing it to call iter(). I do like the distinction from a 'for' loop and leaving an emphasis for template blocks (or blocks, or whatever hip term you crazy kids are using for these things at the moment) to use generators. As I said before, I am viewing these blocks as a construct for external control of generators, not as a snazzy 'for' loop. -Brett

At 05:43 PM 4/27/05 -0700, Guido van Rossum wrote:
Which is why I vote for not calling iter(), and further, that blocks not use the iteration protocol, but rather use a new "block template" protocol. And finally, that a decorator be used to convert a generator function to a "template function" (i.e., a function that returns a block template). I think it's less confusing to have two completely distinct concepts, than to have two things that are very similar, yet different in a blurry kind of way. If you want to use a block on an iterator, you can always explicitly do something like this: @blocktemplate def iterate(iterable): for value in iterable: yield value block iterate([1,2,3]) as x: print x
Anything that wants to do co-operative multitasking, basically.

Guido van Rossum wrote:
I think perhaps I'm not expressing myself very well. What I'm after is a high-level explanation that actually tells people something useful, and *doesn't* cop out by just saying "you're not experienced enough to understand this yet". If such an explanation can't be found, I strongly suspect that this doesn't correspond to a cohesive enough concept to be made into a built-in language feature. If you can't give a short, understandable explanation of it, then it's probably a bad idea. Greg

[Greg Ewing]
[Ping]
I don't know. What exactly is the audience supposed to be of this high-level statement? It would be pretty darn impossible to explain even the for-statement to people who are new to programming, let alone generators. And yet explaining the block-statement *must* involve a reference to generators. I'm guessing most introductions to Python, even for experienced programmers, put generators off until the "advanced" section, because this is pretty wild if you're not used to a language that has something similar. (I wonder how you'd explain Python generators to an experienced Ruby programmer -- their mind has been manipulated to the point where they'd be unable to understand Python's yield no matter how hard they tried. :-) If I weren't limited to newbies (either to Python or to programming in general) but simply had to explain it to Python programmers pre-Python-2.5, I would probably start with a typical example of the try/finally idiom for acquiring and releasing a lock, then explain how for software engineering reasons you'd want to templatize that, and show the solution with a generator and block-statement. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
I think this concept can be explained clearly. I'd like to try explaining PEP 340 to someone new to Python but not new to programming. I'll use the term "block iterator" to refer to the new type of iterator. This is according to my limited understanding. "Good programmers move commonly used code into reusable functions. Sometimes, however, patterns arise in the structure of the functions rather than the actual sequence of statements. For example, many functions acquire a lock, execute some code specific to that function, and unconditionally release the lock. Repeating the locking code in every function that uses it is error prone and makes refactoring difficult. "Block statements provide a mechanism for encapsulating patterns of structure. Code inside the block statement runs under the control of an object called a block iterator. Simple block iterators execute code before and after the code inside the block statement. Block iterators also have the opportunity to execute the controlled code more than once (or not at all), catch exceptions, or receive data from the body of the block statement. "A convenient way to write block iterators is to write a generator. A generator looks a lot like a Python function, but instead of returning a value immediately, generators pause their execution at "yield" statements. When a generator is used as a block iterator, the yield statement tells the Python interpreter to suspend the block iterator, execute the block statement body, and resume the block iterator when the body has executed. "The Python interpreter behaves as follows when it encounters a block statement based on a generator. First, the interpreter instantiates the generator and begins executing it. The generator does setup work appropriate to the pattern it encapsulates, such as acquiring a lock, opening a file, starting a database transaction, or starting a loop. Then the generator yields execution to the body of the block statement using a yield statement. When the block statement body completes, raises an uncaught exception, or sends data back to the generator using a continue statement, the generator resumes. At this point, the generator can either clean up and stop or yield again, causing the block statement body to execute again. When the generator finishes, the interpreter leaves the block statement." Is it understandable so far? Shane

Shane Hathaway wrote:
That actually looks pretty reasonable. Hmmm. "Patterns of structure." Maybe we could call it a "struct" statement. struct opening(foo) as f: ... Then we could confuse both C *and* Ruby programmers at the same time! :-) [No, I don't really mean this. I actually prefer "block" to this.] -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

On Thu, 28 Apr 2005, Shane Hathaway wrote: [...]
Yes, excellent. Speaking as somebody who scanned the PEP and this thread and only half-understood either, that was quite painless to read. Still not sure whether thunks or PEP 340 are better, but I'm at least confused on a higher level now. John

Guido van Rossum wrote:
If the use of block-statements becomes common for certain tasks such as opening files, it seems to me that people are going to encounter their use around about the same time they encounter for-statements. We need *something* to tell these people to enable them to understand the code they're reading. Maybe it would be sufficient just to explain the meanings of those particular uses, and leave the full general explanation as an advanced topic. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Right. The block statement is a bit like a chameleon: it adapts its meaning to the generator you supply. (Or maybe it's like a sewer: what you get out of it depends on what you put into it. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Greg Ewing wrote:
this makes sense to me, also because a new control statement will not be usually as hidden as metaclasses and some other possibly obscure corners can be. OTOH I have the impression that the new toy is too shiny to have a lucid discussion whether it could have sharp edges or produce dizziness for the unexperienced.

On 4/28/05, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
How about: """ A block-statement is much like a for-loop, and is also used to iterate over the elements of an iterable object. In a block-statement however, the iterable object is notified whenever a 'continue', 'break', or 'return' statement is executed inside the block-statement. Most iterable objects do not need to be notified of such statement executions, so for most iteration over iterable objects, you should use a for-loop. Functions that return iterable objects that should be used in a block-statement will be documented as such. """ If you need more information, you could also include something like: """ When generator objects are used in a block-statement, they are guaranteed to be "exhausted" at the end of the block-statement. That is, any additional call to next() with the generator object will produce a StopIteration. """ STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

On 4/28/05, Steven Bethard <steven.bethard@gmail.com> wrote:
however, the iterable object is notified whenever a 'continue', 'break', or 'return' statement is executed inside the block-statement.
This should read: however, the iterable object is notified whenever a 'continue', 'break' or 'return' statement is executed *or an exception is raised* inside the block-statement. Sorry! STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Steven Bethard wrote:
No, no, no. Similarity to a for-loop is the *last* thing we want to emphasise, because the intended use is very different from the intended use of a for-loop. This is going to give people the wrong idea altogether. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Elegant as the idea behind PEP 340 is, I can't shake the feeling that it's an abuse of generators. It seems to go to a lot of trouble and complication so you can write a generator and pretend it's a function taking a block argument. I'd like to reconsider a thunk implementation. It would be a lot simpler, doing just what is required without any jiggery pokery with exceptions and break/continue/return statements. It would be easy to explain what it does and why it's useful. Are there any objective reasons to prefer a generator implementation over a thunk implementation? If for-loops had been implemented with thunks, we might never have created generators. But generators have turned out to be more powerful, because you can have more than one of them on the go at once. Is there a use for that capability here? I can think of one possible use. Suppose you want to acquire multiple resources; one way would be to nest block-statements, like block opening(file1) as f: block opening(file2) as g: ... If you have a lot of resources to acquire, the nesting could get very deep. But with the generator implementation, you could do something like block iterzip(opening(file1), opening(file2)) as f, g: ... provided iterzip were modified to broadcast __next__ arguments to its elements appropriately. You couldn't do this sort of thing with a thunk implementation. On the other hand, a thunk implementation has the potential to easily handle multiple block arguments, if a suitable syntax could ever be devised. It's hard to see how that could be done in a general way with the generator implementation. [BTW, I've just discovered we're not the only people with numbered things called PEPs. I typed "PEP 340" into Google and got "PEP 340: Prevention and Care of Athletic Injuries"!] Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
Are there any objective reasons to prefer a generator implementation over a thunk implementation?
I, too, would like to see an answer to this question. I'd like to see an answer in the PEP, too. Cheers, mwh -- All obscurity will buy you is time enough to contract venereal diseases. -- Tim Peters, python-dev

Greg Ewing wrote:
"Simple is better than Complex." Is there a thunk PEP? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

[Greg Ewing]
Maybe. You're not the first one saying this and I'm not saying "no" outright, but I'd like to defend the PEP. There are a number of separate ideas that all contribute to PEP 340. One is turning generators into more general coroutines: continue EXPR passes the expression to the iterator's next() method (renamed to __next__() to work around a compatibility issue and because it should have been called that in the first place), and in a generator this value can be received as the return value of yield. Incidentally this makes the generator *syntax* more similar to Ruby (even though Ruby uses thunks, and consequently uses return instead of continue to pass a value back). I'd like to have this even if I don't get the block statement. The second is a solution for generator resource cleanup. There are already two PEPs proposing a solution (288 and 325) so I have to assume this addresses real pain! The only new twist offered by PEP 340 is a unification of the next() API and the resource cleanup API: neither PEP 288 nor PEP 325 seems to specify rigorously what should happen if the generator executes another yield in response to a throw() or close() call (or whether that should even be allowed); PEP 340 takes the stance that it *is* allowed and should return a value from whatever call sent the exception. This feels "right", especially together with the previous feature: if yield can return a value as if it were a function call, it should also be allowed to raise an exception, and catch or propagate it with impunity. Even without a block-statement, these two changes make yield look a lot like invoking a thunk -- but it's more efficient, since calling yield doesn't create a frame. The main advantage of thunks that I can see is that you can save the thunk for later, like a callback for a button widget (the thunk then becomes a closure). You can't use a yield-based block for that (except in Ruby, which uses yield syntax with a thunk-based implementation). But I have to say that I almost see this as an advantage: I think I'd be slightly uncomfortable seeing a block and not knowing whether it will be executed in the normal control flow or later. Defining an explicit nested function for that purpose doesn't have this problem for me, because I already know that the 'def' keyword means its body is executed later. The other problem with thunks is that once we think of them as the anonymous functions they are, we're pretty much forced to say that a return statement in a thunk returns from the thunk rather than from the containing function. Doing it any other way would cause major weirdness when the thunk were to survive its containing function as a closure (perhaps continuations would help, but I'm not about to go there :-). But then an IMO important use case for the resource cleanup template pattern is lost. I routinely write code like this: def findSomething(self, key, default=None): self.lock.acquire() try: for item in self.elements: if item.matches(key): return item return default finally: self.lock.release() and I'd be bummed if I couldn't write this as def findSomething(self, key, default=None): block synchronized(self.lock): for item in self.elements: if item.matches(key): return item return default This particular example can be rewritten using a break: def findSomething(self, key, default=None): block synchronized(self.lock): for item in self.elements: if item.matches(key): break else: item = default return item but it looks forced and the transformation isn't always that easy; you'd be forced to rewrite your code in a single-return style which feels too restrictive.
I don't know. In order to obtain the required local variable sharing between the thunk and the containing function I believe that every local variable used or set in the thunk would have to become a 'cell' (our mechanism for sharing variables between nested scopes). Cells slow down access somewhat compared to regular local variables. Perhaps not entirely coincidentally, the last example above (findSomething() rewritten to avoid a return inside the block) shows that, unlike for regular nested functions, we'll want variables *assigned to* by the thunk also to be shared with the containing function, even if they are not assigned to outside the thunk. I swear I didn't create the example for this purpose -- it just happened.
I think the async event folks like to use this (see the Mertz references in PEP 288).
Right, but the use cases for multiple blocks seem elusive. If you really want to have multiple blocks with yield, I suppose we could use "yield/n" to yield to the n'th block argument, or perhaps yield>>n. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Or pass it on to another function. This is something we haven't considered -- what if one resource-acquision- generator (RAG?) wants to delegate to another RAG? With normal generators, one can always use the pattern for x in sub_generator(some_args): yield x But that clearly isn't going to work if the generators involved are RAGs, because the exceptions passed in are going to be raised at the point of the yield in the outer RAG, and the inner RAG isn't going to get finalized (assuming the for-loop doesn't participate in the finalization protocol). To get the finalization right, the inner generator needs to be invoked as a RAG, too: block sub_generator(some_args): yield But PEP 340 doesn't say what happens when the block contains a yield. A thunk implementation wouldn't have any problem with this, since the thunk can be passed down any number of levels before being called, and any exceptions raised in it will be propagated back up through all of them.
Urg, you're right. Unless return is turned into an exception in that case. And then I suppose break and return (and yield?) will have to follow suit. I'm just trying to think how Smalltalk handles this, since it must have a similar problem, but I can't remember the details.
True, but is the difference all that great? It's just one more C-level indirection, isn't it?
Agreed. We'd need to add a STORE_CELL bytecode or something for this. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

(BTW, I'm trying to update the PEP with a discussion of thunks.) [Guido]
[Greg]
The same as when a for-loop contains a yield. The sub_generator is entirely unaware of this yield, since the local control flow doesn't actually leave the block (i.e., it's not like a break, continue or return statement). When the loop that was resumed by the yield calls next(), the block is resumed back after the yield. The generator finalization semantics guarantee (within the limitations of all finalization semantics) that the block will be resumed eventually. I'll add this to the PEP, too. I'd say that a yield in a thunk would be more troublesome: does it turn the thunk into a generator or the containing function? It would have to be the thunk, but then things get weird quickly (the caller of the thunk has to treat the result of the call as an iterator).
But wasn't that exactly what you were trying to avoid? :-)
Alas not. It becomes a call to PyCell_Set() or PyCell_Get().
This actually exists -- it is used for when an outer function stores into a local that it shares with an inner function. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
I like PEP 340 a lot, probably as much or more than any thunk ideas I've seen. But I want to defend thunks here a little. It is possible to implement thunks without them creating their own frame. They can reuse the frame of the surrounding function. So a new frame does not need to be created when the thunk is called, and, much like with a yield statement, the frame is not taken down when the thunk completes running. The implementation just needs to take care to save and restore members of the frame that get clobbered when the thunk is running. Cells would of course not be required if the thunk does not create its own frame.
I would also be uncomfortable if the thunk could be called at a later time. This can be disallowed by raising an exception if such an attempt is made. Such a restriction would not be completely arbitrary. One consequence of having the thunk borrow its surrounding function's frame is that it does not make much sense, implementationally speaking, to allow the thunk to be called at a later time (although I do realize that "it's difficult to implement" is not a good argument for anything).
If it is accepted that the thunk won't be callable at a later time, then I think it would seem normal that a return statement would return from the surrounding function. -Brian

Brian Sabbey <sabbey@u.washington.edu> writes:
Woo. That's cute. Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods

On Thu, Apr 28, 2005, Brian Sabbey wrote:
Maybe. It's not clear whether your thunks are lexical (I haven't been following the discussion closely). If it's not lexical, how do locals get handled without cells? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It's 106 miles to Chicago. We have a full tank of gas, a half-pack of cigarettes, it's dark, and we're wearing sunglasses." "Hit it."

Guido van Rossum <gvanrossum@gmail.com> writes:
This is kind of my point too; I'm not saying that I really prefer the thunk solution, just that I want to see it mentioned. I think the making-generators-more-sexy thing is nice, but I'm think that's almost orthogonal. [...]
I also find them somewhat easier to understand.
I'm not so sure about this. Did you read this mail: http://mail.python.org/pipermail/python-dev/2005-April/052970.html ? In this proposal, you have to go to some effort to make the thunk survive the block, and I think if weirdness results, that's the programmer's problem.
If you can't write it this way, the thunk proposal is dead.
Yes.
Cells slow down access somewhat compared to regular local variables.
So make them faster. I'm not sure I think this is a good argument. You could also do some analysis and treat variables that are only accessed or written in the block as normal locals. This all makes a block-created thunk somewhat different from an anonymous function, to be sure. But the creating syntax is different, so I don't know if I care (hell, the invoking syntax could be made different too, but I really don't think that's a good idea).
Oh, absolutely.
Hmm, it's nearly *May* 1... :) Cheers, mwh -- I'm a keen cyclist and I stop at red lights. Those who don't need hitting with a great big slapping machine. -- Colin Davidson, cam.misc

[Michael Hudson]
I think the making-generators-more-sexy thing is nice, but I'm think that's almost orthogonal.
Not entirely. I agree that "continue EXPR" calling next(EXPR) which enables yield-expressions is entirely orthogonal. But there are already two PEPs asking for passing exceptions and/or cleanup into generators and from there it's only a small step to using them as resource allocation/release templates. The "small step" part is important -- given that we're going to do that work on generators anyway, I expect the changes to the compiler and VM to support the block statement are actually *less* than the changes needed to support thunks. No language feature is designed in isolation.
It's not a complete proposal though. You say "And grudgingly, I guess you'd need to make returns behave like that anyway" (meaning they should return from the containing function). But you don't give a hint on how that could be made to happen, and I expect that by the time you've figured out a mechanism, thunks aren't all that simple any more. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote: [snip]
- I think there's a better word than Flow, but I'll keep using it until we find something better.
How about simply reusing Iteration (ala StopIteration)? Pass in 'ContinueIteration' for 'continue' Pass in 'BreakIteration' for 'break' Pass in 'AbortIteration' for 'return' and finalisation. And advise strongly *against* intercepting AbortIteration with anything other than a finally block.
As suggested above, perhaps the exception used here should be the exception that is raised when a 'return' statement is encountered inside the block, rather than the more-likely-to-be-messed-with 'break' statement. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
Hmmm... another idea: If break and continue return keep exactly the current semantics (break or continue the innermost for/while-loop), do we need different exceptions at all? AFAICS AbortIteration (+1 on the name) would be sufficient for all three interrupting statements, and this would prevent misuse too, I think. yours, Reinhold -- Mail address is perfectly valid!

Reinhold Birkenfeld wrote:
No, the iterator should be able to keep state around in the case of BreakIteration and ContinueIteration, whereas AbortIteration should shut the whole thing down. In particular "VAR = yield None" is likely to become syntactic sugar for: try: yield None except ContinueIteration, exc: VAR = ContinueIteration.value We definitely don't want that construct swallowing AbortIteration. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

On Tue, Apr 26, 2005, Guido van Rossum wrote:
That's precisely why I think we should keep the ``with``: the point of Python is to have a restricted syntax and requiring a prefix for these constructs makes it easier to read the code. You'll soon start to gloss over the ``with`` but it will be there as a marker for your subconscious. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It's 106 miles to Chicago. We have a full tank of gas, a half-pack of cigarettes, it's dark, and we're wearing sunglasses." "Hit it."

At 04:37 AM 4/26/05 -0700, Guido van Rossum wrote:
Actually, this isn't my interest at all. It's the part where you can pass values or exceptions *in* to a generator with *less* magic than is currently required. This interest is unrelated to anonymous blocks in any case; it's about being able to simulate lightweight pseudo-threads ala Stackless, for use with Twisted. I can do this now of course, but "yield expressions" as described in PEP 340 would eliminate the need for the awkward syntax and frame hackery I currently use.

Whew! This is a bit long... On 25 Apr 2005, at 00:57, Guido van Rossum wrote:
After reading a lot of contributions (though perhaps not all -- this thread seems to bifurcate every time someone has a new idea :-)
I haven't read all the posts around the subject, I'll have to admit. I've read the one I'm replying and its followups to pretty carefully, though.
The history of iterators and generators could be summarized by saying that an API was invented, then it turned out that in practice one way of implementing them -- generators -- was almost universally useful. This proposal seems a bit like an effort to make generators good at doing something that they aren't really intended -- or dare I say suited? -- for. The tail wagging the dog so to speak.
This is a syntax error today, of course. When does the finally: clause execute with your proposal? [I work this one out below :)]
This is a non-starter, I hope. I really meant what I said in PEP 310 about loops being loops.
More than that: if I'm implementing an iterator for, uh, iterating, why would one dream of needing to handle an 'err' argument in the next() method?
Ah, this answers my 'when does finally' execute question above.
Well, this is quite a big thing.
My main objection to all this is that it conflates iteration and a more general kind of execution control (I guess iteration is a kind of execution control, but I contend that it's a sufficiently common case to get special treatment and also that names like 'for' and 'next' are only applicable to iteration). So, here's a counterproposal! with expr as var: ... code ... is roughly: def _(var): ... code ... __private = expr __private(_) (var optional as in other proposals). so one might write: def open_file(f): def inner(block): try: block(f) finally: f.close() return inner and have with auto_closing(open("/tmp/foo")) as f: f.write('bob') The need for approximation in the above translation is necessary because you'd want to make assignments in '...code...' affect the scope their written in, and also one might want to allow breaks and continues to be handled as in the end of your proposal. And grudgingly, I guess you'd need to make returns behave like that anyway. Has something like this been argued out somewhere in this thread? As another example, here's how you'd implement something very like a for loop: def as_for_loop(thing): it = iter(thing) def inner(thunk): while 1: try: v = it.next() except StopIteration: break try: thunk(v) except Continue: continue except Break: break so for x in s: and with as_for_loop(s) as x: are now equivalent (I hope :). Cheers, mwh

Michael Hudson wrote:
This is a non-starter, I hope. I really meant what I said in PEP 310 about loops being loops.
The more I play with this, the more I want the 'with' construct to NOT be a loop construct. The main reason is that it would be really nice to be able to write and use a multipart code template as: def template(): # pre_part_1 yield None # post_part_1 yield None # pre_part_2 yield None # post_part_2 yield None # pre_part_3 yield None # post_part_3 def user(): block = template() with block: # do_part_1 with block: # do_part_2 with block: # do_part_3 If 'with' is a looping construct, the above won't work, since the first usage will drain the template. Accordingly, I would like to suggest that 'with' revert to something resembling the PEP 310 definition: resource = EXPR if hasattr(resource, "__enter__"): VAR = resource.__enter__() else: VAR = None try: try: BODY except: raise # Force realisation of sys.exc_info() for use in __exit__() finally: if hasattr(resource, "__exit__"): VAR = resource.__exit__() else: VAR = None Generator objects could implement this protocol, with the following behaviour: def __enter__(): try: return self.next() except StopIteration: raise RuntimeError("Generator exhausted, unable to enter with block") def __exit__(): try: return self.next() except StopIteration: return None def __except__(*exc_info): pass def __no_except__(): pass Note that the code template can deal with exceptions quite happily by utilising sys.exc_info(), and that the result of the call to __enter__ is available *inside* the with block, while the result of the call to __exit__ is available *after* the block (useful for multi-part blocks). If I want to drain the template, then I can use a 'for' loop (albeit without the cleanup guarantees). Taking this route would mean that: * PEP 310 and the question of passing values or exceptions into iterators would again become orthogonal * Resources written using generator syntax aren't cluttered with the repetitive try/finally code PEP 310 is trying to eliminate * 'for' remains TOOW to write an iterative loop * it is possible to execute _different_ suites between each yield in the template block, rather than being constrained to a single suite as in the looping case. * no implications for the semantics of 'return', 'break', 'continue' * 'yield' would not be usable inside a with block, unless the AbortIteration concept was adopting for forcible generator termination. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
One peculiarity of this is that every other 'yield' would not be allowed in the 'try' block of a try/finally statement (TBOATFS). Specifically, a 'yield' reached through the call to __exit__ would not be allowed in the TBOATFS. It gets even more complicated when one considers that 'next' may be called inside BODY. In such a case, it would not be sufficient to just disallow every other 'yield' in the TBOATFS. It seems like 'next' would need some hidden parameter that indicates whether 'yield' should be allowed in the TBOATFS. (I assume that if a TBOATFS contains an invalid 'yield', then an exception will be raised immediately before its 'try' block is executed. Or would the exception be raised upon reaching the 'yield'?)
I think your example above is a good reason to *allow* 'with' to loop. Writing 'auto_retry' with a looping 'with' would be pretty straightforward and intuitive. But the above, non-looping 'with' example requires two fairly advanced techniques (inner functions, variables-as-arrays trick) that would probably be lost on some python users (and make life more difficult for the rest). But I do see the appeal to having a non-looping 'with'. In many (most?) uses of generators, 'for' and looping 'with' could be used interchangeably. This seems ugly-- more than one way to do it and all that. -Brian

Nick Coghlan wrote:
That's an interesting idea, but do you have any use cases in mind? I worry that it will be too restrictive to be really useful. Without the ability for the iterator to control which blocks get executed and when, you wouldn't be able to implement something like a case statement, for example. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Greg Ewing wrote:
I was trying to address a use case which looked something like: do_begin() # code if some_condition: do_pre() # more code do_post() do_end() It's actually doable with a non-looping block statement, but I have yet to come up with a version which isn't as ugly as hell.
We can't write a case statement with a looping block statement either, since we're restricted to executing the same suite whenever we encounter a yield expression. At least the non-looping version offers some hope, since each yield can result in the execution of different code. For me, the main sticking point is that we *already* have a looping construct to drain an iterator - a 'for' loop. The more different the block statement's semantics are from a regular loop, the more powerful I think the combination will be. Whereas if the block statement is just a for loop with slightly tweaked exception handling semantics, then the potential combinations will be far less interesting. My current thinking is that we would be better served by a block construct that guaranteed it would call __next__() on entry and on exit, but did not drain the generator (e.g. by supplying appropriate __enter__() and __exit__() methods on generators for a PEP 310 style block statement, or __enter__(), __except__() and __no_except__() for the enhanced version posted elsewhere in this rambling discussion). However, I'm currently scattering my thoughts across half-a-dozen different conversation threads. So I'm going to stop doing that, and try to put it all into one coherent post :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Michael Hudson wrote:
it is fun because the two of us sort of already had this discussion in compressed form a lot of time ago: http://groups-beta.google.com/groups?q=with+generators+pedronis&hl=en not that I was really conviced about my idea at the time which was very embrional, and in fact I'm bit skeptical right now about how much bending or not of generators makes sense, especially for a learnability point of view.

Samuele Pedroni <pedronis@strakt.com> writes:
Oh yes. That was the discussion that led to PEP 310 being written.
http://groups-beta.google.com/groups?q=with+generators+pedronis&hl=en
At least I'm consistent :)
As am I, obviously. Cheers, mwh -- Arrrrgh, the braindamage! It's not unlike the massively non-brilliant decision to use the period in abbreviations as well as a sentence terminator. Had these people no imagination at _all_? -- Erik Naggum, comp.lang.lisp

Guido van Rossum wrote:
[SNIP - using 'for' syntax to delineate the block and resource]
So I think that this is probably not the right thing to pursue,
I totally agree with your reasoning on this.
Sure, but is the redundancy *that* bad? You should be able to pick up visually that something is an anonymous block from the indentation but I don't know how obvious it would be. Probably, in the end, this minimal syntax would be fine, but it just seems almost too plain in terms of screaming at me that something special is going on there (the '=' in an odd place just quite cut if for me for my meaning of "special").
I think I agree with Samuele that it would be more pertinent to put all of this effort into trying to come up with some way to handle cleanup in a generator. -Brett

[Brett]
I think I agree with Samuele that it would be more pertinent to put all of this effort into trying to come up with some way to handle cleanup in a generator.
I.e. PEP 325. But (as I explained, and you agree) that still doesn't render PEP 310 unnecessary, because abusing the for-loop for implied cleanup semantics is ugly and expensive, and would change generator semantics; and it bugs me that the finally clause's reachability depends on the destructor executing. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
yes, PEP325 would work in combination with PEP310, whether a combined thing (which cannot be the current for as dicussed) is desirable is a different issue: these anyway f = file(...): for line in f: ... vs. it = gen(): for val in it: ... would be analogous in a PEP310+325 world.

Guido van Rossum <gvanrossum@gmail.com> wrote:
Yes and no. PEP 325 offers a method to generators that handles cleanup if necessary and calls it close(). Obviously calling it close is a mistake. Actually, calling it anything is a mistake, and trying to combine try/finally handling in generators with __exit__/close (inside or outside of generators) is also a mistake. Start by saying, "If a non-finalized generator is garbage collected, it will be finalized." Whether this be by an exception or forcing a return, so be it. If this were to happen, we have generator finalization handled by the garbage collector, and don't need to translate /any/ for loop. As long as the garbage collection requirement is documented, we are covered (yay!). What about ... i.__enter__() try: ... finally: i.__exit__() ... types of things? Well, you seem to have offered a syntax ... [VAR '=']* EXPR: BODY ... which seems to translate into ... [VAR = ] __var = EXPR try: BODY finally: __var.__exit__() ... or something like that. Great! We've got a syntax for resource allocation/freeing outside of generators, and a non-syntax for resource allocation/freeing inside of generators. - Josiah

On Apr 21, 2005, at 8:59 PM, Josiah Carlson wrote:
Well, for the CPython implementation, couldn't you get away with using garbage collection to do everything? Maybe I'm missing something.. import weakref class ResourceHandle(object): def __init__(self, acquire, release): acquire() # if I understand correctly, this is safer than __del__ self.ref = weakref.ref(self, lambda o:release()) class FakeLock(object): def acquire(self): print "acquired" def release(self): print "released" def with_lock(lock): r = ResourceHandle(lock.acquire, lock.release) yield None del r
I could imagine someone complaining about generators that are never used missing out on the acquire/release. That could be solved with a trivial rewrite: def with_lock(lock): def _with_lock(r): yield None del r return _with_lock(ResourceHandle(lock.acquire, lock.release))
Of course, this just exaggerates Guido's "it bugs me that the finally clause's reachability depends on the destructor executing".. but it does work, in CPython. It seems to me that this pattern would be painless enough to use without a syntax change... -bob

Bob Ippolito wrote:
[SNIP] Well, if you are missing something then so am I since your suggestion is basically correct. The only issue is that people will want more immediate execution of the cleanup code which gc cannot guarantee. That's why the ability to call a method with the PEP 325 approach gets rid of that worry. -Brett

On Apr 22, 2005, at 12:28 AM, Brett C. wrote:
Well in CPython, if you are never assigning the generator to any local or global, then you should be guaranteed that it gets cleaned up at the right time unless it's alive in a traceback somewhere (maybe you WANT it to be!) or some insane trace hook keeps too many references to frames around.. It seems *reasonably* certain that for reasonable uses this solution WILL clean it up optimistically. -bob

Guido van Rossum wrote:
Right, I'm not saying PEP 310 shouldn't also be considered. It just seems like we are beginning to pile a lot on this discussion by bringing in PEP 310 and PEP 325 in at the same time since, as pointed out, there is no guarantee that anything will be called in a generator and thus making PEP 310 work in generators does not seem guaranteed to solve that problem (although I might have missed something; just started really following the thread today). At this point anonymous blocks just don't seem to be happening, at least not like in Ruby. Fine, I didn't want them anyway. Now we are trying to simplify resource cleanup and handling. What I am trying to say is that generators differ just enough as to possibly warrant a separate discussion from all of this other resource handling "stuff". So I am advocating a more focused generator discussion since resource handling in generators is much more difficult than the general case in non-generator situations. I mean obviously in the general case all of this is handled already in Python today with try/finally. But with generators you have to jump through some extra hoops to get similar support (passing in anything that needs to be cleaned up, hoping that garbage collection will eventually handle things, etc.).
and it bugs me that the finally clause's reachability depends on the destructor executing.
Yeah, I don't like it either. I would rather see something like: def gen(): FILE = open("stuff.txt", 'rU') for line in FILE: yield line cleanup: FILE.close() and have whatever is in the 'cleanup' block be either accessible from a method in the generator or have it become the equivalent of a __del__ for the generator, or maybe even both (which would remove contention that whatever needs to be cleaned up is done too late thanks to gc not guaranteeing immediate cleanup). This way you get the guaranteed cleanup regardless and you don't have to worry about creating everything outside of the generator, passing it in, and then handling cleanup in a try/finally that contains the next() calls to the generator (or any other contortion you might have to go through). Anyway, my random Python suggestion for the day. -Brett

On Thu, 21 Apr 2005, Guido van Rossum wrote:
It seems to me that, in general, Python likes to use keywords for statements and operators for expressions. Maybe the reason lambda looks like such a wart is that it uses a keyword in the middle of an expression. It also uses the colon *not* to introduce an indented suite, which is a strange thing to the Pythonic eye. This suggests that an operator might fit better. A possible operator for lambda might be ->. sort(items, key=x -> x.lower()) Anyway, just a thought. -- ?!ng

Ka-Ping Yee wrote:
It seems to me that, in general, Python likes to use keywords for statements and operators for expressions.
Probably worth noting that 'for', 'in' and 'if' in generator expressions and list comprehensions blur this distinction somewhat... Steve -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

On Thu, Apr 21, 2005, Guido van Rossum wrote:
Yes, it could. The question then becomes whether it should. Because it's easy to indent Python code when you're not using a block (consider function calls with lots of args), my opinion is that like the "optional" colon after ``for`` and ``if``, the resource block *should* have a keyword. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code -- not in reams of trivial code that bores the reader to death." --GvR

I do not know that I have ever needed 'anonymous blocks', and I have therefore not followed this discussion in detail, but I appreciate Python's beauty and want to see it maintained. So I have three comments and yet-another syntax proposal that I do not remember seeing (but could have missed). 1. Python's integration of for loops, iterators, and generators are, to me, a gem of program language design that distinguishes Python from other languages I have used. Using them to not iterate but to do something else may be cute, but in a perverted sort of way. I would rather have 'something else' done some other way. 2. General-purpose passable block objects with parameters look a lot like general-purpose anonymous functions ('full lambdas'). I bet they would be used a such if at all possible. This seems to me like the wrong direction. 3. The specific use-cases for Python not handled better by current syntax seem to be rather specialized: resource management around a block. So I cautiously propose: with <resource type> <specific resource>: <suite> with the exact semantics dependent on <resource type>. In particular: with lock somelock: codeblock could abbreviate and mean somelock.acquire() try: codeblock finally: somelock.release() (Guido's example). with file somefile: codeblock might translate to (the bytecode equivalent of) if isinstance(somefile, basestring?): somefile = open(somefile,defaults) codeblock somefile.close The compound keywords could be 'underscored' but I presume they could be parsed as is, much like 'not in'. Terry J. Reedy

Guido> or perhaps even (making "for VAR" optional in the for-loop syntax) Guido> with Guido> in synchronized(the_lock): Guido> BODY This could be a new statement, so the problematic issue of implicit try/finally in every for statement wouldn't be necessary. That complication would only be needed for the above form. (Of course, if you've dispensed with this I am very likely missing something fundamental.) Skip

Skip Montanaro wrote:
s/in/with/ to get PEP 310. A parallel which has been bugging me is the existence of the iterator protocol (__iter__, next()) which you can implement manually if you want, and the existence of generators, which provide a nice clean way of writing iterators as functions. I'm wondering if something similar can't be found for the __enter__/__exit__ resource protocol. Guido's recent screed crystallised the idea of writing resources as two-part generators: def my_resource(): print "Hi!" # Do entrance code yield None # Go on with the contents of the 'with' block print "Bye!" # Do exit code Giving the internal generator object an enter method that calls self.next() (expecting None to be returned), and an exit method that does the same (but expects StopIteration to be raised) should suffice to make this possible with a PEP 310 style syntax. Interestingly, with this approach, "for dummy in my_resource()" would still wrap the block of code in the entrance/exit code (because my_resource *is* a generator), but it wouldn't get the try/finally semantics. An alternative would be to replace the 'yield None' with a 'break' or 'continue', and create an object which supports the resource protocol and NOT the iterator protocol. Something like: def my_resource(): print "Hi!" # Do entrance code continue # Go on with the contents of the 'with' block print "Bye!" # Do exit code (This is currently a SyntaxError, so it isn't ambiguous in any way) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
That's a very interesting suggestion. I've been lurking, thinking about a way to use something like PEP 310 to help manage database transactions. Here is some typical code that changes something under transaction control: begin_transaction() try: changestuff() changemorestuff() except: abort_transaction() raise else: commit_transaction() There's a lot of boilerplate code there. Using your suggestion, I could write that something like this: def transaction(): begin_transaction() try: continue except: abort_transaction() raise else: commit_transaction() with transaction(): changestuff() changemorestuff() Shane

Shane Hathaway wrote:
For that to work, the behaviour would need to differ slightly from what I envisioned (which was that the 'continue' would be behaviourally equivalent to a 'yield None'). Alternatively, something equivalent to the above could be written as: def transaction(): begin_transaction() continue ex = sys.exc_info() if ex[0] is not None: abort_transaction(): else: commit_transaction(): Note that you could do this with a normal resource, too: class transaction(object): def __enter__(): begin_transaction() def __exit__(): ex = sys.exc_info() if ex[0] is not None: abort_transaction(): else: commit_transaction(): Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Nick Coghlan wrote:
Oh, it is ambiguous, as soon as you insert a for/while statement in your resource function and want to call continue in there. Other than that, it's very neat. Maybe "yield" alone (which is always a SyntaxError) could be used. Reinhold -- Mail address is perfectly valid!

On 4/21/05, Guido van Rossum <gvanrossum@gmail.com> wrote:
How is this different from: def synchronized(lock): def synch_fn(block): lock.acquire() try: block() finally: lock.release() return synch_fn @synchronized def foo(): BLOCK True, it's non-obvious that foo is being immediately executed, but regardless I like the way synchronized is defined, and doesn't use yield (which in my opinion is a non-obvious solution)

After reading a lot of contributions (though perhaps not all -- this thread seems to bifurcate every time someone has a new idea :-) I'm back to liking yield for the PEP 310 use case. I think maybe it was Doug Landauer's post mentioning Beta, plus scanning some more examples of using yield in Ruby. Jim Jewett's post on defmacro also helped, as did Nick Coghlan's post explaining why he prefers 'with' for PEP 310 and a bare expression for the 'with' feature from Pascal (and other languages :-). It seems that the same argument that explains why generators are so good for defining iterators, also applies to the PEP 310 use case: it's just much more natural to write def with_file(filename): f = open(filename) try: yield f finally: f.close() than having to write a class with __entry__ and __exit__ and __except__ methods (I've lost track of the exact proposal at this point). At the same time, having to use it as follows: for f in with_file(filename): for line in f: print process(line) is really ugly, so we need new syntax, which also helps with keeping 'for' semantically backwards compatible. So let's use 'with', and then the using code becomes again this: with f = with_file(filename): for line in f: print process(line) Now let me propose a strawman for the translation of the latter into existing semantics. Let's take the generic case: with VAR = EXPR: BODY This would translate to the following code: it = EXPR err = None while True: try: if err is None: VAR = it.next() else: VAR = it.next_ex(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" if not hasattr(it, "next_ex"): raise (The variables 'it' and 'err' are not user-visible variables, they are internal to the translation.) This looks slightly awkward because of backward compatibility; what I really want is just this: it = EXPR err = None while True: try: VAR = it.next(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" pass but for backwards compatibility with the existing argument-less next() API I'm introducing a new iterator API next_ex() which takes an exception argument. If that argument is None, it should behave just like next(). Otherwise, if the iterator is a generator, this will raised that exception in the generator's frame (at the point of the suspended yield). If the iterator is something else, the something else is free to do whatever it likes; if it doesn't want to do anything, it can just re-raise the exception. Also note that, unlike the for-loop translation, this does *not* invoke iter() on the result of EXPR; that's debatable but given that the most common use case should not be an alternate looping syntax (even though it *is* technically a loop) but a more general "macro statement expansion", I think we can expect EXPR to produce a value that is already an iterator (rather than merely an interable). Finally, I think it would be cool if the generator could trap occurrences of break, continue and return occurring in BODY. We could introduce a new class of exceptions for these, named ControlFlow, and (only in the body of a with statement), break would raise BreakFlow, continue would raise ContinueFlow, and return EXPR would raise ReturnFlow(EXPR) (EXPR defaulting to None of course). So a block could return a value to the generator using a return statement; the generator can catch this by catching ReturnFlow. (Syntactic sugar could be "VAR = yield ..." like in Ruby.) With a little extra magic we could also get the behavior that if the generator doesn't handle ControlFlow exceptions but re-raises them, they would affect the code containing the with statement; this means that the generator can decide whether return, break and continue are handled locally or passed through to the containing block. Note that EXPR doesn't have to return a generator; it could be any object that implements next() and next_ex(). (We could also require next_ex() or even next() with an argument; perhaps this is better.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Might this be a good time to introduce __next__ (having the same signature and semantics as your proposed next_ex) and builtin next(obj, exception=None)? def next(obj, exception=None): if hasattr(obj, '__next__'): return obj.__next__(exception) if exception is not None: return obj.next(exception) # Will raise an appropriate exception return obj.next() Tim Delaney

On 4/25/05, Tim Delaney <tcdelaney@optusnet.com.au> wrote:
Hmm, it took me a while to get this, but what you're ssaying is that if you modify Guido's "what I really want" solution to use VAR = next(it, exc) then this builtin next makes "API v2" stuff using __next__ work while remaining backward compatible with old-style "API v1" stuff using 0-arg next() (as long as old-style stuff isn't used in a context where an exception gets passed back in). I'd suggest that the new builtin have a "magic" name (__next__ being the obvious one :-)) to make it clear that it's an internal implementation detail. Paul. PS The first person to replace builtin __next__ in order to implement a "next hook" of some sort, gets shot :-)

Paul Moore wrote:
Yes, but it could also be used (almost) anywhere an explicit obj.next() is used. it = iter(seq) while True: print next(it) for loops would also change to use builtin next() rather than calling it.next() directly.
There aren't many builtins that have magic names, and I don't think this should be one of them - it has obvious uses other than as an implementation detail.
PS The first person to replace builtin __next__ in order to implement a "next hook" of some sort, gets shot :-)
Damn! There goes the use case ;) Tim Delaney

Tim Delaney wrote:
I think there's some confusion here. As I understood the suggestion, __next__ would be the Python name of the method corresponding to the tp_next typeslot, analogously with __len__, __iter__, etc. There would be a builtin function next(obj) which would invoke obj.__next__(), for use by Python code. For loops wouldn't use it, though; they would continue to call the tp_next typeslot directly.
I think he meant next(), not __next__. And it wouldn't work anyway, since as I mentioned above, C code would bypass next() and call the typeslot directly. I'm +1 on moving towards __next__, BTW. IMO, that's the WISHBDITFP. :-) -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Guido van Rossum wrote: [SNIP]
Can I suggest the name next_exc() instead? Everything in the sys module uses "exc" as the abbreviation for "exception". I realize you might be suggesting using the "ex" as the suffix because of the use of that as the suffix in the C API for an extended API, but that usage is not prominent in the stdlib. Also, would this change in Python 3000 so that both next_ex() and next() are merged into a single method? As for an opinion of the need of 'with', I am on the fence, leaning towards liking it. To make sure I am understanding the use case, it is to help encapsulate typical resource management with proper cleanup in another function instead of having to constantly pasting in boilerplate into your code, right? So the hope is to be able to create factory functions, typically implemented as a generator, that encapsulate the obtaining, temporary lending out, and cleanup of a resource? Is there some other use that I am totally missing that is obvious?
Honestly, I am not very comfortable with this magical meaning of 'break', 'continue', and 'return' in a 'with' block. I realize 'return' already has special meaning in an generator, but I don't think that is really needed either. It leads to this odd dichotomy where a non-exception-related statement directly triggers an exception in other code. It seems like code doing something behind my back; "remember, it looks like a 'continue', but it really is a method call with a specific exception instance. Surprise!" Personally, what I would rather see, is to have next_ex(), for a generator, check if the argument is a subclass of Exception. If it is, raise it as such. If not, have the 'yield' statement return the passed-in argument. This use of it would make sense for using the next_ex() name. Then again I guess having exceptions triggering a method call instead of hitting an 'except' statement is already kind of "surprise" semantics anyway. =) Still, I would like to minimize the surprises that we could spring. And before anyone decries the fact that this might confuse a newbie (which seems to happen with every advanced feature ever dreamed up), remember this will not be meant for a newbie but for someone who has experience in Python and iterators at the minimum, and hopefully with generators. Not exactly meant for someone for which raw_input() still holds a "wow" factor for. =)
Yes, that requirement would be good. Will make sure people don't try to use an iterator with the 'with' statement that has not been designed properly for use within the 'with'. And the precedence of requiring an API is set by 'for' since it needs to be an iterable or define __getitem__() as it is. -Brett

"Brett C." <bac@OCF.Berkeley.EDU> wrote in message news:426C54BF.2010906@ocf.berkeley.edu...
I have accepted the fact that Python has become a two-level language: basic Python for expressing algorithms + advanced features (metaclasses, decorators, CPython-specific introspection and hacks, and now possibly 'with' or whatever) for solving software engineering issues. Perhaps there should correspondingly be two tutorials. Terry J. Reedy

Brett C. wrote:
This is dangerously close to the "you don't need to know about it if you're not going to use it" argument, which is widely recognised as false. Newbies might not need to know all the details of the implementation, but they will need to know enough about the semantics of with-statements to understand what they're doing when they come across them in other people's code. Which leads me to another concern. How are we going to explain the externally visible semantics of a with-statement in a way that's easy to grok, without mentioning any details of the implementation? You can explain a for-loop pretty well by saying something like "It executes the body once for each item from the sequence", without having to mention anything about iterators, generators, next() methods, etc. etc. How the items are produced is completely irrelevant to the concept of the for-loop. But what is the equivalent level of description of the with-statement going to say? "It executes the body with... ???" And a related question: What are we going to call the functions designed for with-statements, and the objects they return? Calling them generators and iterators (even though they are) doesn't seem right, because they're being used for a purpose very different from generating and iterating. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

Greg Ewing wrote:
I am not saying it is totally to be ignored by people staring at Python code, but we don't need to necessarily spell out the intricacies.
It executes the body, calling next() on the argument name on each time through until the iteration stops.
I like "managers" since they are basically managing resources most of the time for the user. -Brett

On Mon, 25 Apr 2005, Brett C. wrote:
It executes the body, calling next() on the argument name on each time through until the iteration stops.
There's a little more to it than that. But on the whole I do support the goal of finding a simple, short description of what this construct is intended to do. If it can be described accurately in a sentence or two, that's a good sign that the semantics are sufficiently clear and simple.
I like "managers" since they are basically managing resources most of the time for the user.
No, please let's not call them that. "Manager" is a very common word to describe all kinds of classes in object-oriented designs, and it is so generic as to hardly mean anything. (Sorry, i don't have a better alternative at the moment.) -- ?!ng

Brett C. wrote:
It executes the body, calling next() on the argument name on each time through until the iteration stops.
But that's no good, because (1) it mentions next(), which should be an implementation detail, and (2) it talks about iteration, when most of the time the high-level intent has nothing to do with iteration. In other words, this is too low a level of explanation. Greg

Guido van Rossum wrote: [snip illustration of how generators (and other iterators) can be modified to be used in with-blocks]
I'm sure I could get used to it, but my intuition for with f = with_file(filename): for line in f: print process(line) is that the f = with_file(filename) executes only once. That is, as you said, I don't expect this to be a looping syntax. Of course, as long as the generators (or other objects) here yield only one value (like with_file does), then the with-block will execute only once. But because the implementation lets you make the with-block loop if you want, it makes be nervous... I guess it would be helpful to see example where the looping with-block is useful. So far, I think all the examples I've seen have been like with_file, which only executes the block once. Of course, the loop allows you to do anything that you would normally do in a for-loop, but my feeling is that this is probably better done by composing a with-block that executes the block only once with a normal Python for-loop. I'd almost like to see the with-block translated into something like it = EXPR try: VAR = it.next() except StopIteration: raise WithNotStartedException err = None try: BODY except Exception, err: # Pretend "except Exception:" == "except:" pass try: it.next_ex(err) except StopIteration: pass else: raise WithNotEndedException where there is no looping at all, and the iterator is expected to yield exactly one item and then terminate. Of course this looks a lot like: it = EXPR VAR = it.__enter__() err = None try: BODY except Exception, err: # Pretend "except Exception:" == "except:" pass it.__exit__(err) So maybe I'm just still stuck on the enter/exit semantics. ;-) STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

At 09:12 PM 4/24/05 -0600, Steven Bethard wrote:
I guess it would be helpful to see example where the looping with-block is useful.
Automatically retry an operation a set number of times before hard failure: with auto_retry(times=3): do_something_that_might_fail() Process each row of a database query, skipping and logging those that cause a processing error: with x,y,z = log_errors(db_query()): do_something(x,y,z) You'll notice, by the way, that some of these "runtime macros" may be stackable in the expression. I'm somewhat curious what happens to yields in the body of the macro block, but I assume they'll just do what would normally occur. Somehow it seems strange, though, to be yielding to something other than the enclosing 'with' object. In any case, I'm personally more excited about the part where this means we get to build co-routines with less magic. The 'with' statement itself is of interest mainly for acquisition/release and atomic/rollback scenarios, but being able to do retries or skip items that cause errors is often handy. Sometimes you have a list of things (such as event callbacks) where you need to call all of them, even if one handler fails, but you can't afford to silence the errors either. Code that deals with that scenario well is a bitch to write, and a looping 'with' would make it a bit easier to write once and reuse many.

On 4/24/05, Phillip J. Eby <pje@telecommunity.com> wrote:
Thanks for the examples! If I understand your point here right, the examples that can't be easily rewritten by composing a single-execution with-block with a for-loop are examples where the number of iterations of the for-loop depends on the error handling of the with-block. Could you rewrite these with PEP 288 as something like: gen = auto_retry(times=3) for _ in gen: try: do_something_that_might_fail() except Exception, err: # Pretend "except Exception:" == "except:" gen.throw(err) gen = log_errors(db_query()) for x,y,z in gen: try: do_something(x,y,z) except Exception, err: # Pretend "except Exception:" == "except:" gen.throw(err) Obviously, the code is cleaner using the looping with-block. I'm just trying to make sure I understand your examples right. So assuming we had looping with-blocks, what would be the benefit of using a for-loop instead? Just efficiency? Or is there something that a for-loop could do that a with-block couldn't? STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy

Phillip J. Eby wrote:
These are also possible by combining a normal for loop with a non-looping with (but otherwise using Guido's exception injection semantics): def auto_retry(attempts): success = [False] failures = [0] except = [None] def block(): try: yield None except: failures[0] += 1 else: success[0] = True while not success[0] and failures[0] < attempts: yield block() if not success[0]: raise Exception # You'd actually propagate the last inner failure for attempt in auto_retry(3): with attempt: do_something_that_might_fail() The non-looping version of with seems to give the best of both worlds - multipart operation can be handled by multiple with statements, and repeated use of the same suite can be handled by nesting the with block inside iteration over an appropriate generator. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

At 04:57 PM 4/24/05 -0700, Guido van Rossum wrote:
[uncontrolled drooling, followed by much rejoicing] If this were available to generators in general, you could untwist Twisted. I'm basically simulating this sort of exception/value passing in peak.events to do exactly that, except I have to do: yield somethingBlocking(); result=events.resume() where events.resume() magically receives a value or exception from outside the generator and either returns or raises it. If next()-with-argument and next_ex() are available normally on generators, this would allow you to simulate co-routines without the events.resume() magic; the above would simply read: result = yield somethingBlocking() The rest of the peak.events coroutine simulation would remain around to manage the generator stack and scheduling, but the syntax would be cleaner and the operation of it entirely unmagical.

Guido van Rossum wrote:
or with with_file(filename) as f: ... ? (assignment inside block-opening constructs aren't used in Python today, as far as I can tell...)
slightly weird, but useful enough to be cool. (maybe "return value" is enough, though. the others may be slightly too weird... or should that return perhaps be a "continue value"? you're going back to the top of loop, after all). </F>

"Fredrik Lundh" <fredrik@pythonware.com> wrote in message news:d4i6hg$q88$1@sea.gmane.org...
with <target> as <value>: would parallel the for-statement header and read smoother to me. for <target> as <value>: would not need new keyword, but would require close reading to distinguish 'as' from 'in'. Terry J. Reedy

[ Simon Percivall ]:
I do not have strong feelings about this issue, but for completeness sake... Mixing both suggestions: from <target> as <value>: <BODY> That resembles an import statement which some may consider good (syntax/keyword reuse) or very bad (confusion?, value focus). cheers, Senra -- Rodrigo Senra -- MSc Computer Engineer rodsenra(at)gpr.com.br GPr Sistemas Ltda http://www.gpr.com.br/ Personal Blog http://rodsenra.blogspot.com/

I have just noticed that this whole notion is fairly similar to the "local" statement in ML, the syntax for which looks like this: local <declarations> in <declarations> end The idea is that the first declarations, whatever they are, are processed without putting their names into the surrounding scope, then the second declarations are processed *with* putting their names into the surrounding scope. For example: local fun add(x:int, y:int) = x+y in fun succ(x) = add(x, 1) end This defines succ in the surrounding scope, but not add. So in Python terms, I think this would be local: <suite> in: <suite> or, for example: local: <target> = value in: blah blah blah

Guido van Rossum wrote:
Indeed - the transaction example is very easy to write this way: def transaction(): begin_transaction() try: yield None except: abort_transaction() raise else: commit_transaction()
Not supporting iterables makes it harder to write a class which is inherently usable in a with block, though. The natural way to make iterable classes is to use 'yield' in the definition of __iter__ - if iter() is not called, then that trick can't be used.
Perhaps 'continue' could be used to pass a value into the iterator, rather than 'return'? (I believe this has been suggested previously in the context of for loops) This would permit 'return' to continue to mean breaking out of the containing function (as for other loops).
So, "VAR = yield x" would expand to something like: try: yield x except ReturnFlow, ex: VAR = ReturnFlow.value ?
That seems a little bit _too_ magical - it would be nice if break and continue were defined to be local, and return to be non-local, as for the existing loop constructs. For other non-local control flow, application specific exceptions will still be available. Regardless, the ControlFlow exceptions do seem like a very practical way of handling the underlying implementation.
With this restriction (i.e. requiring next_ex, next_exc, or Terry's suggested __next__), then the backward's compatible version would be simply your desired semantics, plus an attribute check to exclude old-style iterators: it = EXPR if not hasattr(it, "__next__"): raise TypeError("'with' block requires 2nd gen iterator API support") err = None while True: try: VAR = it.next(err) except StopIteration: break try: err = None BODY except Exception, err: # Pretend "except Exception:" == "except:" pass The generator objects created by using yield would supply the new API, so would be usable immediately inside such 'with' blocks. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

Terry Reedy wrote:
If you're defining it by means of a generator, you don't need a class at all -- just make the whole thing a generator function. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

"Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote in message news:426DC75A.1010005@canterbury.ac.nz...
Terry Reedy wrote:
The part you quoted was by Nick Coghlan, not me, as indicated by the >> (now >>>) instead of > (which would now be >>) in front of the lines.
Not supporting iterables makes it harder to write a class which is ...

Guido> At the same time, having to use it as follows: Guido> for f in with_file(filename): Guido> for line in f: Guido> print process(line) Guido> is really ugly, so we need new syntax, which also helps with Guido> keeping 'for' semantically backwards compatible. So let's use Guido> 'with', and then the using code becomes again this: Guido> with f = with_file(filename): Guido> for line in f: Guido> print process(line) How about deferring major new syntax changes until Py3K when the grammar and semantic options might be more numerous? Given the constraints of backwards compatibility, adding more syntax or shoehorning new semantics into what's an increasingly crowded space seems to always result in an unsatisfying compromise. Guido> Now let me propose a strawman for the translation of the latter Guido> into existing semantics. Let's take the generic case: Guido> with VAR = EXPR: Guido> BODY What about a multi-variable case? Will you have to introduce a new level of indentation for each 'with' var? Skip
participants (30)
-
Aahz
-
Andrew Koenig
-
Bob Ippolito
-
Brett C.
-
Brian Sabbey
-
David Ascher
-
Duncan Booth
-
Fredrik Lundh
-
Greg Ewing
-
Guido van Rossum
-
Jim Fulton
-
John J Lee
-
Josiah Carlson
-
Ka-Ping Yee
-
Luis Bruno
-
Michael Hudson
-
Neil Schemenauer
-
Nick Coghlan
-
Paul Moore
-
Phillip J. Eby
-
Reinhold Birkenfeld
-
Rodrigo Dias Arruda Senra
-
Samuele Pedroni
-
Shane Hathaway
-
Simon Percivall
-
Skip Montanaro
-
Steven Bethard
-
Terry Reedy
-
Tim Delaney
-
Timothy Fitz