Update of /cvsroot/python/python/nondist/peps In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv22078 Modified Files: pep-0343.txt Log Message: Specify generator enhancements. Change keyword to 'with'. Index: pep-0343.txt =================================================================== RCS file: /cvsroot/python/python/nondist/peps/pep-0343.txt,v retrieving revision 1.16 retrieving revision 1.17 diff -u -d -r1.16 -r1.17 --- pep-0343.txt 31 May 2005 20:27:15 -0000 1.16 +++ pep-0343.txt 1 Jun 2005 15:13:37 -0000 1.17 @@ -1,5 +1,5 @@ PEP: 343 -Title: Anonymous Block Redux +Title: Anonymous Block Redux and Generator Enhancements Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum @@ -11,17 +11,13 @@ Introduction - After a lot of discussion about PEP 340 and alternatives, I've - decided to withdraw PEP 340 and propose a slight variant on - PEP 310. - -Evolutionary Note - - After ample discussion on python-dev, I'll add back a mechanism + After a lot of discussion about PEP 340 and alternatives, I + decided to withdraw PEP 340 and proposed a slight variant on PEP + 310. After more discussion, I have added back a mechanism for raising an exception in a suspended generator using a throw() method, and a close() method which throws a new GeneratorExit - exception. Until I get a chance to update the PEP, see reference - [2]. I'm also leaning towards 'with' as the keyword. + exception; these additions were first proposed in [2] and + universally approved of. I'm also changing the keyword to 'with'. Motivation and Summary @@ -53,7 +49,7 @@ with VAR = EXPR: BLOCK - which roughly translates into + which roughly translates into this: VAR = EXPR VAR.__enter__() @@ -74,7 +70,7 @@ goto (a break, continue or return), BLOCK2 is *not* reached. The magic added by the with-statement at the end doesn't affect this. - (You may ask, what if a bug in the __exit__ method causes an + (You may ask, what if a bug in the __exit__() method causes an exception? Then all is lost -- but this is no worse than with other exceptions; the nature of exceptions is that they can happen *anywhere*, and you just have to live with that. Even if you @@ -89,16 +85,18 @@ Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried to create a decorator that would turn a suitable generator into an - object with the necessary __entry__ and __exit__ methods. Here I - ran into a snag: while it wasn't too hard for the locking example, - it was impossible to do this for the opening example. The idea - was to define the template like this: + object with the necessary __enter__() and __exit__() methods. + Here I ran into a snag: while it wasn't too hard for the locking + example, it was impossible to do this for the opening example. + The idea was to define the template like this: @with_template def opening(filename): f = open(filename) - yield f - f.close() + try: + yield f + finally: + f.close() and used it like this: @@ -106,21 +104,21 @@ ...read data from f... The problem is that in PEP 310, the result of calling EXPR is - assigned directly to VAR, and then VAR's __exit__ method is called - upon exit from BLOCK1. But here, VAR clearly needs to receive the - opened file, and that would mean that __exit__ would have to be a - method on the file. + assigned directly to VAR, and then VAR's __exit__() method is + called upon exit from BLOCK1. But here, VAR clearly needs to + receive the opened file, and that would mean that __exit__() would + have to be a method on the file. While this can be solved using a proxy class, this is awkward and made me realize that a slightly different translation would make writing the desired decorator a piece of cake: let VAR receive the - result from calling the __enter__ method, and save the value of - EXPR to call its __exit__ method later. Then the decorator can - return an instance of a wrapper class whose __enter__ method calls - the generator's next() method and returns whatever next() returns; - the wrapper instance's __exit__ method calls next() again but - expects it to raise StopIteration. (Details below in the section - Optional Generator Decorator.) + result from calling the __enter__() method, and save the value of + EXPR to call its __exit__() method later. Then the decorator can + return an instance of a wrapper class whose __enter__() method + calls the generator's next() method and returns whatever next() + returns; the wrapper instance's __exit__() method calls next() + again but expects it to raise StopIteration. (Details below in + the section Optional Generator Decorator.) So now the final hurdle was that the PEP 310 syntax: @@ -128,41 +126,61 @@ BLOCK1 would be deceptive, since VAR does *not* receive the value of - EXPR. Given PEP 340, it was an easy step to: + EXPR. Borrowing from PEP 340, it was an easy step to: with EXPR as VAR: BLOCK1 - or, using an alternate keyword that has been proposed a number of - times: + Additional discussion showed that people really liked being able + to "see" the exception in the generator, even if it was only to + log it; the generator is not allowed to yield another value, since + the with-statement should not be usable as a loop (raising a + different exception is marginally acceptable). To enable this, a + new throw() method for generators is proposed, which takes three + arguments representing an exception in the usual fashion (type, + value, traceback) and raises it at the point where the generator + is suspended. - do EXPR as VAR: - BLOCK1 + Once we have this, it is a small step to proposing another + generator method, close(), which calls throw() with a special + exception, GeneratorExit. This tells the generator to exit, and + from there it's another small step to proposing that close() be + called automatically when the generator is garbage-collected. + + Then, finally, we can allow a yield-statement inside a try-finally + statement, since we can now guarantee that the finally-clause will + (eventually) be executed. The usual cautions about finalization + apply -- the process may be terminated abruptly without finalizing + any objects, and objects may be kept alive forever by cycles or + memory leaks in the application (as opposed to cycles or leaks in + the Python implementation, which are taken care of by GC). + + Note that we're not guaranteeing that the finally-clause is + executed immediately after the generator object becomes unused, + even though this is how it will work in CPython. This is similar + to auto-closing files: while a reference-counting implementation + like CPython deallocates an object as soon as the last reference + to it goes away, implementations that use other GC algorithms do + not make the same guarantee. This applies to Jython, IronPython, + and probably to Python running on Parrot. Use Cases See the Examples section near the end. -Specification +Specification: The 'with' Statement A new statement is proposed with the syntax: - do EXPR as VAR: + with EXPR as VAR: BLOCK - Here, 'do' and 'as' are new keywords; EXPR is an arbitrary + Here, 'with' and 'as' are new keywords; EXPR is an arbitrary expression (but not an expression-list) and VAR is an arbitrary assignment target (which may be a comma-separated list). The "as VAR" part is optional. - The choice of the 'do' keyword is provisional; an alternative - under consideration is 'with'. - - A yield-statement is illegal inside BLOCK. This is because the - do-statement is translated into a try/finally statement, and yield - is illegal in a try/finally statement. - The translation of the above statement is: abc = EXPR @@ -209,94 +227,184 @@ non-local goto should be considered unexceptional for the purposes of a database transaction roll-back decision. -Optional Generator Decorator +Specification: Generator Enhancements + + Let a generator object be the iterator produced by calling a + generator function. Below, 'g' always refers to a generator + object. + + New syntax: yield allowed inside try-finally + + The syntax for generator functions is extended to allow a + yield-statement inside a try-finally statement. + + New generator method: throw(type, value, traceback) + + g.throw(type, value, traceback) causes the specified exception to + be thrown at the point where the generator g is currently + suspended (i.e. at a yield-statement, or at the start of its + function body if next() has not been called yet). If the + generator catches the exception and yields another value, that is + the return value of g.throw(). If it doesn't catch the exception, + the throw() appears to raise the same exception passed it (it + "falls through"). If the generator raises another exception (this + includes the StopIteration produced when it returns) that + exception is raised by the throw() call. In summary, throw() + behaves like next() except it raises an exception at the + suspension point. If the generator is already in the closed + state, throw() just raises the exception it was passed without + executing any of the generator's code. + + The effect of raising the exception is exactly as if the + statement: + + raise type, value, traceback + + was executed at the suspension point. The type argument should + not be None. + + New standard exception: GeneratorExit + + A new standard exception is defined, GeneratorExit, inheriting + from Exception. A generator should handle this by re-raising it + or by raising StopIteration. + + New generator method: close() + + g.close() is defined by the following pseudo-code: + + def close(self): + try: + self.throw(GeneratorExit, GeneratorExit(), None) + except (GeneratorExit, StopIteration): + pass + else: + raise TypeError("generator ignored GeneratorExit") + # Other exceptions are not caught + + New generator method: __del__() + + g.__del__() is an alias for g.close(). This will be called when + the generator object is garbage-collected (in CPython, this is + when its reference count goes to zero). If close() raises an + exception, a traceback for the exception is printed to sys.stderr + and further ignored; it is not propagated back to the place that + triggered the garbage collection. This is consistent with the + handling of exceptions in __del__() methods on class instances. + + If the generator object participates in a cycle, g.__del__() may + not be called. This is the behavior of CPython's current garbage + collector. The reason for the restriction is that the GC code + needs to "break" a cycle at an arbitrary point in order to collect + it, and from then on no Python code should be allowed to see the + objects that formed the cycle, as they may be in an invalid state. + Objects "hanging off" a cycle are not subject to this restriction. + Note that it is unlikely to see a generator object participate in + a cycle in practice. However, storing a generator object in a + global variable creates a cycle via the generator frame's + f_globals pointer. Another way to create a cycle would be to + store a reference to the generator object in a data structure that + is passed to the generator as an argument. Neither of these cases + are very likely given the typical pattern of generator use. + +Generator Decorator It is possible to write a decorator that makes it possible to use - a generator that yields exactly once to control a do-statement. + a generator that yields exactly once to control a with-statement. Here's a sketch of such a decorator: class Wrapper(object): + def __init__(self, gen): self.gen = gen - self.state = "initial" + def __enter__(self): - assert self.state == "initial" - self.state = "entered" try: return self.gen.next() except StopIteration: - self.state = "error" - raise RuntimeError("template generator didn't yield") - def __exit__(self, *args): - assert self.state == "entered" - self.state = "exited" - try: - self.gen.next() - except StopIteration: - return + raise RuntimeError("generator didn't yield") + + def __exit__(self, type, value, traceback): + if type is None: + try: + self.gen.next() + except StopIteration: + return + else: + raise RuntimeError("generator didn't stop") else: - self.state = "error" - raise RuntimeError("template generator didn't stop") + try: + self.gen.throw(type, value, traceback) + except (type, StopIteration): + return + else: + raise RuntimeError("generator caught exception") - def do_template(func): + def with_template(func): def helper(*args, **kwds): return Wrapper(func(*args, **kwds)) return helper This decorator could be used as follows: - @do_template + @with_template def opening(filename): f = open(filename) # IOError here is untouched by Wrapper yield f f.close() # Ditto for errors here (however unlikely) - A robust implementation of such a decorator should be made part of - the standard library. + A robust implementation of this decorator should be made part of + the standard library, but not necessarily as a built-in function. + (I'm not sure which exception it should raise for errors; + RuntimeError is used above as an example only.) -Other Optional Extensions +Optional Extensions It would be possible to endow certain objects, like files, - sockets, and locks, with __enter__ and __exit__ methods so that - instead of writing: + sockets, and locks, with __enter__() and __exit__() methods so + that instead of writing: - do locking(myLock): + with locking(myLock): BLOCK one could write simply: - do myLock: + with myLock: BLOCK I think we should be careful with this; it could lead to mistakes like: f = open(filename) - do f: + with f: BLOCK1 - do f: + with f: BLOCK2 - which does not do what one might think (f is closed when BLOCK2 is - entered). + which does not do what one might think (f is closed before BLOCK2 + is entered). + + OTOH such mistakes are easily diagnosed. Examples - Several of these examples contain "yield None". If PEP 342 is - accepted, these can be changed to just "yield". + (Note: several of these examples contain "yield None". If PEP 342 + is accepted, these can be changed to just "yield".) 1. A template for ensuring that a lock, acquired at the start of a block, is released when the block is left: - @do_template + @with_template def locking(lock): lock.acquire() - yield None - lock.release() + try: + yield None + finally: + lock.release() Used as follows: - do locking(myLock): + with locking(myLock): # Code here executes with myLock held. The lock is # guaranteed to be released when the block is left (even # if via return or by an uncaught exception). @@ -304,29 +412,32 @@ 2. A template for opening a file that ensures the file is closed when the block is left: - @do_template + @with_template def opening(filename, mode="r"): f = open(filename, mode) - yield f - f.close() + try: + yield f + finally: + f.close() Used as follows: - do opening("/etc/passwd") as f: + with opening("/etc/passwd") as f: for line in f: print line.rstrip() 3. A template for committing or rolling back a database transaction; this is written as a class rather than as a - decorator since it requires access to the exception information: + decorator since it requires access to the exception + information: class transactional: def __init__(self, db): self.db = db def __enter__(self): self.db.begin() - def __exit__(self, *args): - if args and args[0] is not None: + def __exit__(self, type, value, tb): + if type is not None: self.db.rollback() else: self.db.commit() @@ -338,47 +449,51 @@ self.lock = lock def __enter__(self): self.lock.acquire() - def __exit__(self, *args): + def __exit__(self, type, value, tb): self.lock.release() (This example is easily modified to implement the other - examples; it shows how much simpler generators are for the same - purpose.) + examples; it shows the relative advantage of using a generator + template.) 5. Redirect stdout temporarily: - @do_template + @with_template def redirecting_stdout(new_stdout): save_stdout = sys.stdout sys.stdout = new_stdout - yield None - sys.stdout = save_stdout + try: + yield None + finally: + sys.stdout = save_stdout Used as follows: - do opening(filename, "w") as f: - do redirecting_stdout(f): + with opening(filename, "w") as f: + with redirecting_stdout(f): print "Hello world" This isn't thread-safe, of course, but neither is doing this - same dance manually. In a single-threaded program (e.g., a - script) it is a totally fine way of doing things. + same dance manually. In single-threaded programs (for example, + in scripts) it is a popular way of doing things. 6. A variant on opening() that also returns an error condition: - @do_template + @with_template def opening_w_error(filename, mode="r"): try: f = open(filename, mode) except IOError, err: yield None, err else: - yield f, None - f.close() + try: + yield f, None + finally: + f.close() Used as follows: - do opening_w_error("/etc/passwd", "a") as f, err: + with opening_w_error("/etc/passwd", "a") as f, err: if err: print "IOError:", err else: @@ -389,7 +504,7 @@ import signal - do signal.blocking(): + with signal.blocking(): # code executed without worrying about signals An optional argument might be a list of signals to be blocked; @@ -401,19 +516,21 @@ import decimal - @do_template - def with_extra_precision(places=2): + @with_template + def extra_precision(places=2): c = decimal.getcontext() saved_prec = c.prec c.prec += places - yield None - c.prec = saved_prec + try: + yield None + finally: + c.prec = saved_prec Sample usage (adapted from the Python Library Reference): def sin(x): "Return the sine of x as measured in radians." - do with_extra_precision(): + with extra_precision(): i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1 while s != lasts: lasts = s @@ -423,31 +540,33 @@ sign *= -1 s += num / fact * sign # The "+s" rounds back to the original precision, - # so this must be outside the do-statement: + # so this must be outside the with-statement: return +s 9. Here's a more general Decimal-context-switching template: - @do_template - def with_decimal_context(newctx=None): + @with_template + def decimal_context(newctx=None): oldctx = decimal.getcontext() if newctx is None: newctx = oldctx.copy() decimal.setcontext(newctx) - yield newctx - decimal.setcontext(oldctx) + try: + yield newctx + finally: + decimal.setcontext(oldctx) - Sample usage (adapted from the previous one): + Sample usage: def sin(x): - do with_decimal_context() as ctx: + with decimal_context() as ctx: ctx.prec += 2 - # Rest of algorithm the same + # Rest of algorithm the same as above return +s - (Nick Coghlan has proposed to add __enter__ and __exit__ + (Nick Coghlan has proposed to add __enter__() and __exit__() methods to the decimal.Context class so that this example can - be simplified to "do decimal.getcontext() as ctx: ...".) + be simplified to "with decimal.getcontext() as ctx: ...".) References