[Python-Dev] PEP 343: Resource Composition and Idempotent __exit__
Ka-Ping Yee
python-dev at zesty.ca
Sun May 15 06:58:12 CEST 2005
Yikes, this turned out to be rather long. Here's a summary:
- The expansion suggested by Shane (moving __enter__ outside the
try-finally) fits the expectation that __exit__ will always be
paired with a successful __enter__. This "paired-exit" expansion
is fairly intuitive and makes simple resources easy to write,
but they are not very composable.
- The expansion that Guido currently has in PEP 343 encourages an
implementation style where __exit__ is idempotent. If we use it,
we should document this fact, since it may seem a little unusual
at first; we should also rename "enter"/"exit" so they do not
mislead programmers into believing that they are paired. Simple
resources are a little more work to write than with a paired-exit
expansion, but they are easier to compose and reuse.
- The generator style in PEP 340 is the easiest to compose and
reuse, but its implementation is the most complex to understand.
I lean (but only slightly) toward the second option, because it seems
to be a reasonable compromise. The increased complexity of writing
resources for PEP 343 over PEP 340 becomes less of an issue if we have
a good do_template function in the standard library; on the other hand,
the use of do_template may complicate debugging.
For idempotent-exit, possible renamings of enter/exit might be
enter/finish, enter/cleanup, enter/finally, start/finally, begin/finally.
* * *
Okay. Here's how i arrived at the above conclusions. (In the following,
i'll just use "with" for "do/with".)
PEP 343 (rev 1.8) currently expands
with EXPR as VAR:
BLOCK
to this, which i'll call the "idempotent-exit" expansion:
resource = EXPR
exc = (None, None, None)
try:
try:
VAR = resource.__enter__()
BLOCK
except:
exc = sys.exc_info()
raise
finally:
resource.__exit__(*exc)
If there are problems during __enter__, then __enter__ is expected to
record this fact so that __exit__ can clean up. Since __exit__ is
called regardless of whether __enter__ succeeded, this encourages a
style of writing resources where __exit__ is idempotent.
An alternative, advocated by Shane (and by my first instincts), is this
expansion, which i'll call the "paired-exit" expansion:
resource = EXPR
exc = (None, None, None)
VAR = resource.__enter__()
try:
try:
BLOCK
except:
exc = sys.exc_info()
raise
finally:
resource.__exit__(*exc)
If there are problems during __enter__, __enter__ must clean them up
before propagating an exception, because __exit__ will not be called.
To evaluate these options, we could look at a few scenarios where we're
trying to write a resource wrapper for some lock objects. Each lock
object has two methods, .acquire() and .release().
Scenario 1. You have two resource objects and you want to acquire both.
Scenario 2. You want a single resource object that acquires two locks.
Scenario 3. Your resource object acquires one of two locks depending
on some runtime condition.
Scenario 1 (Composition by client)
==================================
The client writes this:
with resource1:
with resource2:
BLOCK
The idempotent-exit expansion would yield this:
exc1 = (None, None, None)
try:
try:
resource1.__enter__()
exc2 = (None, None, None)
try:
try:
resource2.__enter__()
BLOCK
except:
exc2 = sys.exc_info()
raise
finally:
resource2.__exit__(*exc2)
except:
exc1 = sys.exc_info()
raise
finally:
resource1.__exit__(*exc1)
Because __exit__ is always called even if __enter__ fails, the resource
wrapper must record whether __enter__ succeeded:
class ResourceI:
def __init__(self, lock):
self.lock = lock
self.acquired = False
def __enter__(self):
self.lock.acquire()
self.acquired = True
def __exit__(self, *exc):
if self.acquired:
self.lock.release()
self.acquired = False
The paired-exit expansion would yield this:
exc1 = (None, None, None)
resource1.__enter__()
try:
try:
exc2 = (None, None, None)
resource2.__enter__()
try:
try:
BLOCK
except:
exc2 = sys.exc_info()
raise
finally:
resource2.__exit__(*exc2)
except:
exc1 = sys.exc_info()
raise
finally:
resource1.__exit__(*exc1)
In this case the lock can simply be implemented as:
class ResourceP:
def __init__(self, lock):
self.lock = lock
def __enter__(self):
self.lock.acquire()
def __exit__(self, *exc):
self.lock.release()
With PEP 340, assuming no return values and the presence of an __exit__
method, we would get this expansion:
exc1 = None
while True:
try:
if exc1:
resource1.__exit__(*exc1) # may re-raise *exc1
else:
resource1.next() # may raise StopIteration
except StopIteration:
break
try:
exc1 = None
exc2 = None
while True:
try:
if exc2:
resource2.__exit__(*exc2) # may re-raise *exc2
else:
resource2.next() # may raise StopIteration
except StopIteration:
break
try:
exc2 = None
BLOCK
except:
exc2 = sys.exc_info()
except:
exc1 = sys.exc_info()
Assuming that the implementations of resource1 and resource2 invoke
'yield' exactly once, this reduces to:
exc1 = None
resource1.next() # first time, will not raise StopIteration
try:
exc2 = None
resource2.next() # first time, will not raise StopIteration
try:
BLOCK
except:
exc2 = sys.exc_info()
try:
if exc2:
resource2.__exit__(*exc2)
else:
resource2.next() # second time, will raise StopIteration
except StopIteration:
pass
except:
exc1 = sys.exc_info()
try:
if exc1:
resource1.__exit__(*exc1)
else:
resource1.next() # second time, will raise StopIteration
except StopIteration:
pass
For this expansion, it is sufficient to implement the resource as:
def ResourceG(lock):
lock.acquire()
try:
yield
finally:
lock.release()
Scenario 2 (Composition by implementor)
=======================================
The client writes this:
with DoubleResource(lock1, lock2):
BLOCK
With the idempotent-exit expansion, we could implement DoubleResource
directly like this:
class DoubleResourceI:
def __init__(self, lock1, lock2):
self.lock1, self.lock2 = lock1, lock2
self.got1 = self.got2 = False
def __enter__(self):
self.lock1.acquire()
self.got1 = True
try:
self.lock2.acquire()
self.got2 = True
except:
self.lock1.release()
self.got1 = False
def __exit__(self, *exc):
try:
if self.got2:
self.lock2.release()
self.got2 = False
finally:
if self.got1:
self.lock1.release()
self.got1 = False
or it could be implemented in terms of ResourceA like this:
class DoubleResourceI:
def __init__(self, lock1, lock2):
self.resource1 = ResourceI(lock1)
self.resource2 = ResourceI(lock2)
def __enter__(self):
self.resource1.__enter__()
self.resource2.__enter__()
def __exit__(self, *exc):
try:
self.resource2.__exit__()
finally:
self.resource1.__exit__()
On the other hand, if we use the paired-exit expansion, the
DoubleResource would be implemented like this:
class DoubleResourceP:
def __init__(self, lock1, lock2):
self.lock1, self.lock2 = lock1, lock2
def __enter__(self):
self.lock1.acquire()
try:
self.lock2.acquire()
except:
self.lock1.release()
raise
def __exit__(self):
try:
self.lock2.release()
finally:
self.lock1.release()
As far as i can tell, the implementation of DoubleResourceP is
made no simpler by the definition of ResourceP.
With PEP 340, the DoubleResource could be written directly like this:
def DoubleResourceG(lock1, lock2):
lock1.acquire()
try:
lock2.acquire()
except:
lock1.release()
raise
try:
yield
finally:
try:
lock2.release()
finally:
lock1.release()
Or, if ResourceG were already defined, it could simply be written:
def DoubleResourceG(lock1, lock2):
with ResourceG(lock1):
with ResourceG(lock2):
yield
This should also work with PEP 343 if decorated with "@do_template",
though i don't have the patience to verify that carefully. When
written this way, the Boolean flags disappear, as their purpose is
replaced by the internal generator state.
Scenario 3 (Conditional acquisition)
====================================
The client writes this:
with ConditionalResource(condition, lock1, lock2):
BLOCK
For the idempotent-exit expansion, we could implement ConditionalResource
directly like this:
class ConditionalResourceI:
def __init__(self, condition, lock1, lock2):
self.condition = condition
self.lock1, self.lock2 = lock1, lock2
self.got1 = self.got2 = False
def __enter__(self):
if self.condition():
self.lock1.acquire()
self.got1 = True
else:
self.lock2.acquire()
self.got2 = True
def __exit__(self, *exc):
try:
if self.got1:
self.lock1.release()
self.got1 = False
finally:
if self.got2:
self.lock2.release()
self.got2 = False
Or we could implement it more simply in terms of ResourceI like this:
class ConditionalResourceI:
def __init__(self, condition, lock1, lock2):
self.condition = condition
self.resource1 = ResourceI(lock1)
self.resource2 = ResourceI(lock2)
def __enter__(self):
if self.condition():
self.resource1.__enter__()
else:
self.resource2.__enter__()
def __exit__(self, *exc):
try:
self.resource2.__exit__()
finally:
self.resource1.__exit__()
For the paired-exit expansion, we would implement ConditionalResource
directly like this:
class ConditionalResourceP:
def __init__(self, condition, lock1, lock2):
self.condition = condition
self.lock1, self.lock2 = lock1, lock2
self.flag = None
def __enter__(self):
self.flag = self.condition()
if self.flag:
self.lock1.acquire()
else:
self.lock2.acquire()
def __exit__(self, *exc):
if self.flag:
self.lock1.release()
else:
self.lock2.release()
And using PEP 340, we would write it as a generator like this:
def ConditionalResourceG(condition, lock1, lock2):
if condition:
with ResourceG(lock1):
yield
else:
with ResourceG(lock2):
yield
Again, i would expect this to also work with PEP 343 if "@do_template"
were inserted in front.
-- ?!ng
More information about the Python-Dev
mailing list