[Python-checkins] commit of r41350 - peps/trunk
nick.coghlan@python.org
nick.coghlan at python.org
Sat Oct 29 08:08:13 CEST 2005
Author: nick.coghlan
Date: Sat Oct 29 08:08:12 2005
New Revision: 41350
Modified:
peps/trunk/pep-0343.txt
Log:
Update with outcome of recent python-dev discussions
Modified: peps/trunk/pep-0343.txt
==============================================================================
--- peps/trunk/pep-0343.txt (original)
+++ peps/trunk/pep-0343.txt Sat Oct 29 08:08:12 2005
@@ -7,17 +7,17 @@
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
-Post-History: 2-Jun-2005
+Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005
Abstract
This PEP adds a new statement "with" to the Python language to make
it possible to factor out standard uses of try/finally statements.
- The PEP has been approved in principle by the BDFL, but there are
+ The PEP was approved in principle by the BDFL, but there were
still a couple of implementation details to be worked out (see the
- section on Open Issues). It's been reverted to Draft status until
- those issues have been settled to Guido's satisfaction.
+ section on Resolved Issues). It's still at Draft status until
+ Guido gives a final blessing to the updated PEP.
Author's Note
@@ -225,7 +225,7 @@
The translation of the above statement is:
- abc = (EXPR).__with__()
+ abc = (EXPR).__context__()
exc = (None, None, None)
VAR = abc.__enter__()
try:
@@ -241,14 +241,18 @@
accessible to the user; they will most likely be implemented as
special registers or stack positions.
- The call to the __with__() method serves a similar purpose to that
- of the __iter__() method of iterator and iterables. An object with
- with simple state requirements (such as threading.RLock) may provide
- its own __enter__() and __exit__() methods, and simply return
- 'self' from its __with__ method. On the other hand, an object with
- more complex state requirements (such as decimal.Context) may
- return a distinct context manager object each time its __with__
- method is invoked.
+ The above translation is fairly literal - if any of the relevant
+ methods are not found as expected, the interpreter will raise
+ AttributeError.
+
+ The call to the __context__() method serves a similar purpose to
+ that of the __iter__() method of iterator and iterables. An
+ object with with simple state requirements (such as
+ threading.RLock) may provide its own __enter__() and __exit__()
+ methods, and simply return 'self' from its __context__ method. On
+ the other hand, an object with more complex state requirements
+ (such as decimal.Context) may return a distinct context manager
+ object each time its __context__ method is invoked.
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
the translation is omitted (but abc.__enter__() is still called).
@@ -284,12 +288,12 @@
that makes it possible to use a generator that yields exactly once
to control a with-statement. Here's a sketch of such a decorator:
- class GeneratorContext(object):
+ class GeneratorContextManager(object):
def __init__(self, gen):
self.gen = gen
- def __with__(self):
+ def __context__(self):
return self
def __enter__(self):
@@ -314,14 +318,14 @@
else:
raise RuntimeError("generator caught exception")
- def context(func):
+ def contextmanager(func):
def helper(*args, **kwds):
- return GeneratorContext(func(*args, **kwds))
+ return GeneratorContextManager(func(*args, **kwds))
return helper
This decorator could be used as follows:
- @context
+ @contextmanager
def opening(filename):
f = open(filename) # IOError is untouched by GeneratorContext
try:
@@ -329,16 +333,18 @@
finally:
f.close() # Ditto for errors here (however unlikely)
- A robust implementation of this decorator should be made part of
- the standard library. Refer to Open Issues regarding its name and
- location.
+ A robust builtin implementation of this decorator will be made
+ part of the standard library.
Just as generator-iterator functions are very useful for writing
__iter__() methods for iterables, generator-context functions will
- be very useful for writing __with__() methods for contexts. It is
- proposed that the invocation of the "context" decorator be
- considered implicit for generator functions used as __with__()
- methods (again, refer to the Open Issues section).
+ be very useful for writing __context__() methods for contexts.
+ These methods will still need to be decorated using the
+ contextmanager decorator. To ensure an obvious error message if the
+ decorator is left out, generator-iterator objects will NOT be given
+ a native context - if you want to ensure a generator is closed
+ promptly, use something similar to the duck-typed "closing" context
+ manager in the examples.
Optional Extensions
@@ -371,6 +377,15 @@
second with-statement calls f.__enter__() again. A similar error
can be raised if __enter__ is invoked on a closed file object.
+ For Python 2.5, the following candidates have been identified for
+ native context managers:
+ - file
+ - decimal.Context
+ - thread.LockType
+ - threading.Lock
+ - threading.RLock
+ - threading.Condition
+
Standard Terminology
Discussions about iterators and iterables are aided by the standard
@@ -384,7 +399,7 @@
This PEP proposes that the protocol used by the with statement be
known as the "context management protocol", and that objects that
implement that protocol be known as "context managers". The term
- "context" then encompasses all objects with a __with__() method
+ "context" then encompasses all objects with a __context__() method
that returns a context manager (this means that all context managers
are contexts, but not all contexts are context managers).
@@ -395,50 +410,13 @@
In cases where the general term "context" would be ambiguous, it
can be made explicit by expanding it to "manageable context".
-Open Issues
-
- Discussion on python-dev revealed some open issues. These are listed
- here and will be resolved either by consensus on python-dev or by
- BDFL fiat.
-
- 1. The name of the decorator used to convert a generator-iterator
- function into a generator-context function is still to be
- finalised.
- The proposal in this PEP is that it be called simply "context"
- with the following reasoning:
- - A "generator function" is an undecorated function containing
- the 'yield' keyword, and the objects produced by
- such functions are "generator-iterators". The term
- "generator" may refer to either a generator function or a
- generator-iterator depending on the situation.
- - A "generator context function" is a generator function to
- which the "context" decorator is applied and the objects
- produced by such functions are "generator-context-managers".
- The term "generator context" may refer to either a generator
- context function or a generator-context-manager depending on
- the situation.
-
- 2. Should the decorator to convert a generator function into a
- generator context function be a builtin, or located elsewhere in
- the standard library? This PEP suggests that it should be a
- builtin, as generator context functions are the recommended way
- of writing new context managers.
-
- 3. Should a generator function used to implement a __with__ method
- always be considered to be a generator context function, without
- requiring the context decorator? This PEP suggests that it
- should, as applying a decorator to a slot just looks strange,
- and omitting the decorator would be a source of obscure bugs.
- The __new__ slot provides some precedent for special casing of
- certain slots when processing slot methods.
-
Resolved Issues
- The following issues were resolved either by BDFL fiat, consensus on
- python-dev, or a simple lack of objection to proposals in the
- original version of this PEP.
+ The following issues were resolved either by BDFL approval,
+ consensus on python-dev, or a simple lack of objection to
+ proposals in the original version of this PEP.
- 1. The __exit__() method of the GeneratorContext class
+ 1. The __exit__() method of the GeneratorContextManager class
catches StopIteration and considers it equivalent to re-raising
the exception passed to throw(). Is allowing StopIteration
right here?
@@ -458,10 +436,10 @@
finally-clause (the one implicit in the with-statement) which
re-raises the original exception anyway.
- 2. What exception should GeneratorContext raise when the underlying
- generator-iterator misbehaves? The following quote is the reason
- behind Guido's choice of RuntimeError for both this and for the
- generator close() method in PEP 342 (from [8]):
+ 2. What exception should GeneratorContextManager raise when the
+ underlying generator-iterator misbehaves? The following quote is
+ the reason behind Guido's choice of RuntimeError for both this
+ and for the generator close() method in PEP 342 (from [8]):
"I'd rather not introduce a new exception class just for this
purpose, since it's not an exception that I want people to catch:
@@ -477,24 +455,27 @@
on python-dev [4] settled on the term "context manager" for
objects which provide __enter__ and __exit__ methods, and
"context management protocol" for the protocol itself. With the
- addition of the __with__ method to the protocol, a natural
- extension is to call all objects which provide a __with__ method
- "contexts" (or "manageable contexts" in situations where the
- general term "context" would be ambiguous).
+ addition of the __context__ method to the protocol, a natural
+ extension is to call all objects which provide a __context__
+ method "contexts" (or "manageable contexts" in situations where
+ the general term "context" would be ambiguous).
This is now documented in the "Standard Terminology" section.
4. The originally approved version of this PEP did not include a
- __with__ method - the method was only added to the PEP after
+ __context__ method - the method was only added to the PEP after
Jason Orendorff pointed out the difficulty of writing
appropriate __enter__ and __exit__ methods for decimal.Context
[5]. This approach allows a class to define a native context
manager using generator syntax. It also allows a class to use an
existing independent context manager as its native context
manager by applying the independent context manager to 'self' in
- its __with__ method. It even allows a class written in C to use
- a generator context manager written in Python.
- The __with__ method parallels the __iter__ method which forms
+ its __context__ method. It even allows a class written in C to
+ use a generator context manager written in Python.
+ The __context__ method parallels the __iter__ method which forms
part of the iterator protocol.
+ An earlier version of this PEP called this the __with__ method.
+ This was later changed to match the name of the protocol rather
+ than the keyword for the statement [9].
5. The suggestion was made by Jason Orendorff that the __enter__
and __exit__ methods could be removed from the context
@@ -514,18 +495,56 @@
works without having to first understand the mechanics of
how generator context managers are implemented.
+ 6. The decorator to make a context manager from a generator will be
+ a builtin called "contextmanager". The shorter term "context" was
+ considered too ambiguous and potentially confusing [9].
+ The different flavours of generators can then be described as:
+ - A "generator function" is an undecorated function containing
+ the 'yield' keyword, and the objects produced by
+ such functions are "generator-iterators". The term
+ "generator" may refer to either a generator function or a
+ generator-iterator depending on the situation.
+ - A "generator context function" is a generator function to
+ which the "contextmanager" decorator is applied and the
+ objects produced by such functions are "generator-context-
+ managers". The term "generator context" may refer to either a
+ generator context function or a generator-context-manager
+ depending on the situation.
+
+ 7. A generator function used to implement a __context__ method will
+ need to be decorated with the contextmanager decorator in order
+ to have the correct behaviour. Otherwise, you will get an
+ AttributeError when using the class in a with statement, as
+ normal generator-iterators will NOT have __enter__ or __exit__
+ methods.
+ Getting deterministic closure of generators will require a
+ separate context manager such as the closing example below.
+ As Guido put it, "too much magic is bad for your health" [10].
+
+ 8. It is fine to raise AttributeError instead of TypeError if the
+ relevant methods aren't present on a class involved in a with
+ statement. The fact that the abstract object C API raises
+ TypeError rather than AttributeError is an accident of history,
+ rather than a deliberate design decision [11].
+
Examples
- (The generator based examples assume PEP 342 is implemented. Also,
- some of the examples are likely to be unnecessary in practice, as
- the appropriate objects, such as threading.RLock, will be able to
- be used directly in with statements)
+ The generator based examples rely on PEP 342. Also, some of the
+ examples are likely to be unnecessary in practice, as the
+ appropriate objects, such as threading.RLock, will be able to be
+ used directly in with statements.
+
+ The tense used in the names of the example context managers is not
+ arbitrary. Past tense ("-ed") is used when the name refers to an
+ action which is done in the __enter__ method and undone in the
+ __exit__ method. Progressive tense ("-ing") is used when the name
+ refers to an action which is to be done in the __exit__ method.
1. A template for ensuring that a lock, acquired at the start of a
block, is released when the block is left:
- @context
- def locking(lock):
+ @contextmanager
+ def locked(lock):
lock.acquire()
try:
yield
@@ -534,20 +553,20 @@
Used as follows:
- with locking(myLock):
+ with locked(myLock):
# Code here executes with myLock held. The lock is
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
- PEP 319 gives a use case for also having an unlocking()
+ PEP 319 gives a use case for also having an unlocked()
template; this can be written very similarly (just swap the
acquire() and release() calls).
2. A template for opening a file that ensures the file is closed
when the block is left:
- @context
- def opening(filename, mode="r"):
+ @contextmanager
+ def opened(filename, mode="r"):
f = open(filename, mode)
try:
yield f
@@ -556,15 +575,15 @@
Used as follows:
- with opening("/etc/passwd") as f:
+ with opened("/etc/passwd") as f:
for line in f:
print line.rstrip()
3. A template for committing or rolling back a database
transaction:
- @context
- def transactional(db):
+ @contextmanager
+ def transaction(db):
db.begin()
try:
yield None
@@ -575,10 +594,10 @@
4. Example 1 rewritten without a generator:
- class locking:
+ class locked:
def __init__(self, lock):
self.lock = lock
- def __with__(self, lock):
+ def __context__(self):
return self
def __enter__(self):
self.lock.acquire()
@@ -586,13 +605,14 @@
self.lock.release()
(This example is easily modified to implement the other
- examples; it shows that is is easy to avoid the need for a
- generator if no special state needs to be preserved.)
+ relatively stateless examples; it shows that it is easy to avoid
+ the need for a generator if no special state needs to be
+ preserved.)
5. Redirect stdout temporarily:
- @context
- def redirecting_stdout(new_stdout):
+ @contextmanager
+ def stdout_redirected(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
@@ -602,18 +622,18 @@
Used as follows:
- with opening(filename, "w") as f:
- with redirecting_stdout(f):
+ with opened(filename, "w") as f:
+ with stdout_redirected(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this
same dance manually. In single-threaded programs (for example,
in scripts) it is a popular way of doing things.
- 6. A variant on opening() that also returns an error condition:
+ 6. A variant on opened() that also returns an error condition:
- @context
- def opening_w_error(filename, mode="r"):
+ @contextmanager
+ def opened_w_error(filename, mode="r"):
try:
f = open(filename, mode)
except IOError, err:
@@ -626,7 +646,7 @@
Used as follows:
- with opening_w_error("/etc/passwd", "a") as (f, err):
+ with opened_w_error("/etc/passwd", "a") as (f, err):
if err:
print "IOError:", err
else:
@@ -637,7 +657,7 @@
import signal
- with signal.blocking():
+ with signal.blocked():
# code executed without worrying about signals
An optional argument might be a list of signals to be blocked;
@@ -679,7 +699,8 @@
9. Here's a proposed native context manager for decimal.Context:
# This would be a new decimal.Context method
- def __with__(self):
+ @contextmanager
+ def __context__(self):
# We set the thread context to a copy of this context
# to ensure that changes within the block are kept
# local to the block. This also gives us thread safety
@@ -710,7 +731,7 @@
10. A generic "object-closing" template:
- @context
+ @contextmanager
def closing(obj):
try:
yield obj
@@ -737,6 +758,78 @@
for datum in data:
process(datum)
+ 11. Native contexts for objects with acquire/release methods:
+
+ # This would be a new method of e.g., threading.RLock
+ def __context__(self):
+ return locked(self)
+
+ def released(self):
+ return unlocked(self)
+
+ Sample usage:
+
+ with my_lock:
+ # Operations with the lock held
+ with my_lock.released():
+ # Operations without the lock
+ # e.g. blocking I/O
+ # Lock is held again here
+
+ 12. A "nested" context manager that automatically nests the
+ supplied contexts from left-to-right to avoid excessive
+ indentation:
+
+ class nested(object):
+ def __init__(*contexts):
+ self.contexts = contexts
+ self.entered = None
+
+ def __context__(self):
+ return self
+
+ def __enter__(self):
+ if self.entered is not None:
+ raise RuntimeError("Context is not reentrant")
+ self.entered = deque()
+ vars = []
+ try:
+ for context in self.contexts:
+ mgr = context.__context__()
+ vars.append(mgr.__enter__())
+ self.entered.appendleft(mgr)
+ except:
+ self.__exit__(*sys.exc_info())
+ raise
+ return vars
+
+ def __exit__(self, *exc_info):
+ # Behave like nested with statements
+ # first in, last out
+ # New exceptions override old ones
+ ex = exc_info
+ for mgr in self.entered:
+ try:
+ mgr.__exit__(*ex)
+ except:
+ ex = sys.exc_info()
+ self.entered = None
+ if ex is not exc_info:
+ raise ex[0], ex[1], ex[2]
+
+ Sample usage:
+
+ with nested(a, b, c) as (x, y, z):
+ # Perform operation
+
+ Is equivalent to:
+
+ with a as x:
+ with b as y:
+ with c as z:
+ # Perform operation
+
+
References
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
@@ -760,6 +853,15 @@
[8]
http://mail.python.org/pipermail/python-dev/2005-June/054064.html
+ [9]
+ http://mail.python.org/pipermail/python-dev/2005-October/057520.html
+
+ [10]
+ http://mail.python.org/pipermail/python-dev/2005-October/057535.html
+
+ [11]
+ http://mail.python.org/pipermail/python-dev/2005-October/057625.html
+
Copyright
This document has been placed in the public domain.
More information about the Python-checkins
mailing list