Author: nick.coghlan Date: Sat Oct 29 08:08:12 2005 New Revision: 41350 Modified: peps/trunk/pep-0343.txt Log: Update with outcome of recent python-dev discussions Modified: peps/trunk/pep-0343.txt ============================================================================== --- peps/trunk/pep-0343.txt (original) +++ peps/trunk/pep-0343.txt Sat Oct 29 08:08:12 2005 @@ -7,17 +7,17 @@ Type: Standards Track Content-Type: text/plain Created: 13-May-2005 -Post-History: 2-Jun-2005 +Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005 Abstract This PEP adds a new statement "with" to the Python language to make it possible to factor out standard uses of try/finally statements. - The PEP has been approved in principle by the BDFL, but there are + The PEP was approved in principle by the BDFL, but there were still a couple of implementation details to be worked out (see the - section on Open Issues). It's been reverted to Draft status until - those issues have been settled to Guido's satisfaction. + section on Resolved Issues). It's still at Draft status until + Guido gives a final blessing to the updated PEP. Author's Note @@ -225,7 +225,7 @@ The translation of the above statement is: - abc = (EXPR).__with__() + abc = (EXPR).__context__() exc = (None, None, None) VAR = abc.__enter__() try: @@ -241,14 +241,18 @@ accessible to the user; they will most likely be implemented as special registers or stack positions. - The call to the __with__() method serves a similar purpose to that - of the __iter__() method of iterator and iterables. An object with - with simple state requirements (such as threading.RLock) may provide - its own __enter__() and __exit__() methods, and simply return - 'self' from its __with__ method. On the other hand, an object with - more complex state requirements (such as decimal.Context) may - return a distinct context manager object each time its __with__ - method is invoked. + The above translation is fairly literal - if any of the relevant + methods are not found as expected, the interpreter will raise + AttributeError. + + The call to the __context__() method serves a similar purpose to + that of the __iter__() method of iterator and iterables. An + object with with simple state requirements (such as + threading.RLock) may provide its own __enter__() and __exit__() + methods, and simply return 'self' from its __context__ method. On + the other hand, an object with more complex state requirements + (such as decimal.Context) may return a distinct context manager + object each time its __context__ method is invoked. If the "as VAR" part of the syntax is omitted, the "VAR =" part of the translation is omitted (but abc.__enter__() is still called). @@ -284,12 +288,12 @@ that makes it possible to use a generator that yields exactly once to control a with-statement. Here's a sketch of such a decorator: - class GeneratorContext(object): + class GeneratorContextManager(object): def __init__(self, gen): self.gen = gen - def __with__(self): + def __context__(self): return self def __enter__(self): @@ -314,14 +318,14 @@ else: raise RuntimeError("generator caught exception") - def context(func): + def contextmanager(func): def helper(*args, **kwds): - return GeneratorContext(func(*args, **kwds)) + return GeneratorContextManager(func(*args, **kwds)) return helper This decorator could be used as follows: - @context + @contextmanager def opening(filename): f = open(filename) # IOError is untouched by GeneratorContext try: @@ -329,16 +333,18 @@ finally: f.close() # Ditto for errors here (however unlikely) - A robust implementation of this decorator should be made part of - the standard library. Refer to Open Issues regarding its name and - location. + A robust builtin implementation of this decorator will be made + part of the standard library. Just as generator-iterator functions are very useful for writing __iter__() methods for iterables, generator-context functions will - be very useful for writing __with__() methods for contexts. It is - proposed that the invocation of the "context" decorator be - considered implicit for generator functions used as __with__() - methods (again, refer to the Open Issues section). + be very useful for writing __context__() methods for contexts. + These methods will still need to be decorated using the + contextmanager decorator. To ensure an obvious error message if the + decorator is left out, generator-iterator objects will NOT be given + a native context - if you want to ensure a generator is closed + promptly, use something similar to the duck-typed "closing" context + manager in the examples. Optional Extensions @@ -371,6 +377,15 @@ second with-statement calls f.__enter__() again. A similar error can be raised if __enter__ is invoked on a closed file object. + For Python 2.5, the following candidates have been identified for + native context managers: + - file + - decimal.Context + - thread.LockType + - threading.Lock + - threading.RLock + - threading.Condition + Standard Terminology Discussions about iterators and iterables are aided by the standard @@ -384,7 +399,7 @@ This PEP proposes that the protocol used by the with statement be known as the "context management protocol", and that objects that implement that protocol be known as "context managers". The term - "context" then encompasses all objects with a __with__() method + "context" then encompasses all objects with a __context__() method that returns a context manager (this means that all context managers are contexts, but not all contexts are context managers). @@ -395,50 +410,13 @@ In cases where the general term "context" would be ambiguous, it can be made explicit by expanding it to "manageable context". -Open Issues - - Discussion on python-dev revealed some open issues. These are listed - here and will be resolved either by consensus on python-dev or by - BDFL fiat. - - 1. The name of the decorator used to convert a generator-iterator - function into a generator-context function is still to be - finalised. - The proposal in this PEP is that it be called simply "context" - with the following reasoning: - - A "generator function" is an undecorated function containing - the 'yield' keyword, and the objects produced by - such functions are "generator-iterators". The term - "generator" may refer to either a generator function or a - generator-iterator depending on the situation. - - A "generator context function" is a generator function to - which the "context" decorator is applied and the objects - produced by such functions are "generator-context-managers". - The term "generator context" may refer to either a generator - context function or a generator-context-manager depending on - the situation. - - 2. Should the decorator to convert a generator function into a - generator context function be a builtin, or located elsewhere in - the standard library? This PEP suggests that it should be a - builtin, as generator context functions are the recommended way - of writing new context managers. - - 3. Should a generator function used to implement a __with__ method - always be considered to be a generator context function, without - requiring the context decorator? This PEP suggests that it - should, as applying a decorator to a slot just looks strange, - and omitting the decorator would be a source of obscure bugs. - The __new__ slot provides some precedent for special casing of - certain slots when processing slot methods. - Resolved Issues - The following issues were resolved either by BDFL fiat, consensus on - python-dev, or a simple lack of objection to proposals in the - original version of this PEP. + The following issues were resolved either by BDFL approval, + consensus on python-dev, or a simple lack of objection to + proposals in the original version of this PEP. - 1. The __exit__() method of the GeneratorContext class + 1. The __exit__() method of the GeneratorContextManager class catches StopIteration and considers it equivalent to re-raising the exception passed to throw(). Is allowing StopIteration right here? @@ -458,10 +436,10 @@ finally-clause (the one implicit in the with-statement) which re-raises the original exception anyway. - 2. What exception should GeneratorContext raise when the underlying - generator-iterator misbehaves? The following quote is the reason - behind Guido's choice of RuntimeError for both this and for the - generator close() method in PEP 342 (from [8]): + 2. What exception should GeneratorContextManager raise when the + underlying generator-iterator misbehaves? The following quote is + the reason behind Guido's choice of RuntimeError for both this + and for the generator close() method in PEP 342 (from [8]): "I'd rather not introduce a new exception class just for this purpose, since it's not an exception that I want people to catch: @@ -477,24 +455,27 @@ on python-dev [4] settled on the term "context manager" for objects which provide __enter__ and __exit__ methods, and "context management protocol" for the protocol itself. With the - addition of the __with__ method to the protocol, a natural - extension is to call all objects which provide a __with__ method - "contexts" (or "manageable contexts" in situations where the - general term "context" would be ambiguous). + addition of the __context__ method to the protocol, a natural + extension is to call all objects which provide a __context__ + method "contexts" (or "manageable contexts" in situations where + the general term "context" would be ambiguous). This is now documented in the "Standard Terminology" section. 4. The originally approved version of this PEP did not include a - __with__ method - the method was only added to the PEP after + __context__ method - the method was only added to the PEP after Jason Orendorff pointed out the difficulty of writing appropriate __enter__ and __exit__ methods for decimal.Context [5]. This approach allows a class to define a native context manager using generator syntax. It also allows a class to use an existing independent context manager as its native context manager by applying the independent context manager to 'self' in - its __with__ method. It even allows a class written in C to use - a generator context manager written in Python. - The __with__ method parallels the __iter__ method which forms + its __context__ method. It even allows a class written in C to + use a generator context manager written in Python. + The __context__ method parallels the __iter__ method which forms part of the iterator protocol. + An earlier version of this PEP called this the __with__ method. + This was later changed to match the name of the protocol rather + than the keyword for the statement [9]. 5. The suggestion was made by Jason Orendorff that the __enter__ and __exit__ methods could be removed from the context @@ -514,18 +495,56 @@ works without having to first understand the mechanics of how generator context managers are implemented. + 6. The decorator to make a context manager from a generator will be + a builtin called "contextmanager". The shorter term "context" was + considered too ambiguous and potentially confusing [9]. + The different flavours of generators can then be described as: + - A "generator function" is an undecorated function containing + the 'yield' keyword, and the objects produced by + such functions are "generator-iterators". The term + "generator" may refer to either a generator function or a + generator-iterator depending on the situation. + - A "generator context function" is a generator function to + which the "contextmanager" decorator is applied and the + objects produced by such functions are "generator-context- + managers". The term "generator context" may refer to either a + generator context function or a generator-context-manager + depending on the situation. + + 7. A generator function used to implement a __context__ method will + need to be decorated with the contextmanager decorator in order + to have the correct behaviour. Otherwise, you will get an + AttributeError when using the class in a with statement, as + normal generator-iterators will NOT have __enter__ or __exit__ + methods. + Getting deterministic closure of generators will require a + separate context manager such as the closing example below. + As Guido put it, "too much magic is bad for your health" [10]. + + 8. It is fine to raise AttributeError instead of TypeError if the + relevant methods aren't present on a class involved in a with + statement. The fact that the abstract object C API raises + TypeError rather than AttributeError is an accident of history, + rather than a deliberate design decision [11]. + Examples - (The generator based examples assume PEP 342 is implemented. Also, - some of the examples are likely to be unnecessary in practice, as - the appropriate objects, such as threading.RLock, will be able to - be used directly in with statements) + The generator based examples rely on PEP 342. Also, some of the + examples are likely to be unnecessary in practice, as the + appropriate objects, such as threading.RLock, will be able to be + used directly in with statements. + + The tense used in the names of the example context managers is not + arbitrary. Past tense ("-ed") is used when the name refers to an + action which is done in the __enter__ method and undone in the + __exit__ method. Progressive tense ("-ing") is used when the name + refers to an action which is to be done in the __exit__ method. 1. A template for ensuring that a lock, acquired at the start of a block, is released when the block is left: - @context - def locking(lock): + @contextmanager + def locked(lock): lock.acquire() try: yield @@ -534,20 +553,20 @@ Used as follows: - with locking(myLock): + with locked(myLock): # Code here executes with myLock held. The lock is # guaranteed to be released when the block is left (even # if via return or by an uncaught exception). - PEP 319 gives a use case for also having an unlocking() + PEP 319 gives a use case for also having an unlocked() template; this can be written very similarly (just swap the acquire() and release() calls). 2. A template for opening a file that ensures the file is closed when the block is left: - @context - def opening(filename, mode="r"): + @contextmanager + def opened(filename, mode="r"): f = open(filename, mode) try: yield f @@ -556,15 +575,15 @@ Used as follows: - with opening("/etc/passwd") as f: + with opened("/etc/passwd") as f: for line in f: print line.rstrip() 3. A template for committing or rolling back a database transaction: - @context - def transactional(db): + @contextmanager + def transaction(db): db.begin() try: yield None @@ -575,10 +594,10 @@ 4. Example 1 rewritten without a generator: - class locking: + class locked: def __init__(self, lock): self.lock = lock - def __with__(self, lock): + def __context__(self): return self def __enter__(self): self.lock.acquire() @@ -586,13 +605,14 @@ self.lock.release() (This example is easily modified to implement the other - examples; it shows that is is easy to avoid the need for a - generator if no special state needs to be preserved.) + relatively stateless examples; it shows that it is easy to avoid + the need for a generator if no special state needs to be + preserved.) 5. Redirect stdout temporarily: - @context - def redirecting_stdout(new_stdout): + @contextmanager + def stdout_redirected(new_stdout): save_stdout = sys.stdout sys.stdout = new_stdout try: @@ -602,18 +622,18 @@ Used as follows: - with opening(filename, "w") as f: - with redirecting_stdout(f): + with opened(filename, "w") as f: + with stdout_redirected(f): print "Hello world" This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example, in scripts) it is a popular way of doing things. - 6. A variant on opening() that also returns an error condition: + 6. A variant on opened() that also returns an error condition: - @context - def opening_w_error(filename, mode="r"): + @contextmanager + def opened_w_error(filename, mode="r"): try: f = open(filename, mode) except IOError, err: @@ -626,7 +646,7 @@ Used as follows: - with opening_w_error("/etc/passwd", "a") as (f, err): + with opened_w_error("/etc/passwd", "a") as (f, err): if err: print "IOError:", err else: @@ -637,7 +657,7 @@ import signal - with signal.blocking(): + with signal.blocked(): # code executed without worrying about signals An optional argument might be a list of signals to be blocked; @@ -679,7 +699,8 @@ 9. Here's a proposed native context manager for decimal.Context: # This would be a new decimal.Context method - def __with__(self): + @contextmanager + def __context__(self): # We set the thread context to a copy of this context # to ensure that changes within the block are kept # local to the block. This also gives us thread safety @@ -710,7 +731,7 @@ 10. A generic "object-closing" template: - @context + @contextmanager def closing(obj): try: yield obj @@ -737,6 +758,78 @@ for datum in data: process(datum) + 11. Native contexts for objects with acquire/release methods: + + # This would be a new method of e.g., threading.RLock + def __context__(self): + return locked(self) + + def released(self): + return unlocked(self) + + Sample usage: + + with my_lock: + # Operations with the lock held + with my_lock.released(): + # Operations without the lock + # e.g. blocking I/O + # Lock is held again here + + 12. A "nested" context manager that automatically nests the + supplied contexts from left-to-right to avoid excessive + indentation: + + class nested(object): + def __init__(*contexts): + self.contexts = contexts + self.entered = None + + def __context__(self): + return self + + def __enter__(self): + if self.entered is not None: + raise RuntimeError("Context is not reentrant") + self.entered = deque() + vars = [] + try: + for context in self.contexts: + mgr = context.__context__() + vars.append(mgr.__enter__()) + self.entered.appendleft(mgr) + except: + self.__exit__(*sys.exc_info()) + raise + return vars + + def __exit__(self, *exc_info): + # Behave like nested with statements + # first in, last out + # New exceptions override old ones + ex = exc_info + for mgr in self.entered: + try: + mgr.__exit__(*ex) + except: + ex = sys.exc_info() + self.entered = None + if ex is not exc_info: + raise ex[0], ex[1], ex[2] + + Sample usage: + + with nested(a, b, c) as (x, y, z): + # Perform operation + + Is equivalent to: + + with a as x: + with b as y: + with c as z: + # Perform operation + + References [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx @@ -760,6 +853,15 @@ [8] http://mail.python.org/pipermail/python-dev/2005-June/054064.html + [9] + http://mail.python.org/pipermail/python-dev/2005-October/057520.html + + [10] + http://mail.python.org/pipermail/python-dev/2005-October/057535.html + + [11] + http://mail.python.org/pipermail/python-dev/2005-October/057625.html + Copyright This document has been placed in the public domain.