Re: [Python-Dev] PEP 343 - Abstract Block Redux
Guido van Rossum wrote:
I've written up the specs for my "PEP 340 redux" proposal as a separate PEP, PEP 343.
http://python.org/peps/pep-0343.html
Those who have been following the thread "Merging PEP 310 and PEP 340-redux?" will recognize my proposal in that thread, which received mostly positive responses there.
Please review and ask for clarifications of anything that's unclear.
There's a typo in the code snippets at the moment. The translation of the above statement is: abc = EXPR exc = () # Or (None, None, None) ? try: try: VAR = abc.__enter__() BLOCK except: exc = sys.exc_info() raise finally: abc.__exit__(exc) I think you meant "abc.__exit__(*exc)". Assuming that, then "exc = (None, None, None)" makes the most sense. If exc_info() is going to be passed as a single arg, then I'd rather have the default "exc = ()", so I can simply check "if exc:" in the __exit__ method. Robert Brewer System Architect Amor Ministries fumanchu@amor.org
Robert Brewer wrote:
There's a typo in the code snippets at the moment.
The translation of the above statement is:
abc = EXPR exc = () # Or (None, None, None) ? try: try: VAR = abc.__enter__() BLOCK except: exc = sys.exc_info() raise finally: abc.__exit__(exc)
I think you meant "abc.__exit__(*exc)". Assuming that, then "exc = (None, None, None)" makes the most sense. If exc_info() is going to be passed as a single arg, then I'd rather have the default "exc = ()", so I can simply check "if exc:" in the __exit__ method.
Also, the call to __enter__() needs to be before the try/finally block (as it is in PEP 310). Otherwise we get the "releasing a lock you failed to acquire" problem. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
[Nick Coghlan]
Also, the call to __enter__() needs to be before the try/finally block (as it is in PEP 310). Otherwise we get the "releasing a lock you failed to acquire" problem.
I did that on purpose. There's a separate object ('abc' in the pseudo-code of the translation) whose __enter__ and __exit__ methods are called, and in __enter__ it can keep track of the reversible actions it has taken. Consider an application where you have to acquire *two* locks regularly: def lockBoth(): got1 = got2 = False lock1.acquire(); got1 = True lock2.acquire(); got2 = True yield None if got2: lock2.release() if got1: lock1.release() If this gets interrupted after locking lock1 but before locking lock2, it still has some cleanup to do. I know that this complicates simpler use cases, and I'm not 100% sure this is the right solution; but I don't know how else to handle this use case. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
[Nick Coghlan]
Also, the call to __enter__() needs to be before the try/finally block (as it is in PEP 310). Otherwise we get the "releasing a lock you failed to acquire" problem.
I did that on purpose. There's a separate object ('abc' in the pseudo-code of the translation) whose __enter__ and __exit__ methods are called, and in __enter__ it can keep track of the reversible actions it has taken.
Consider an application where you have to acquire *two* locks regularly:
def lockBoth(): got1 = got2 = False lock1.acquire(); got1 = True lock2.acquire(); got2 = True yield None if got2: lock2.release() if got1: lock1.release()
If this gets interrupted after locking lock1 but before locking lock2, it still has some cleanup to do.
That code is incorrect, though. Say lockBoth() acquires lock1 but then lock2.acquire() throws an exception. (Maybe the lock requires some I/O operation, and the operation fails.) The interpreter will never reach the yield statement and lock1 will never be released. You really have to write it like this: def lockBoth(): lock1.acquire() try: lock2.acquire() except: lock1.release() raise yield None try: lock2.release() finally: lock1.release()
I know that this complicates simpler use cases, and I'm not 100% sure this is the right solution; but I don't know how else to handle this use case.
If __enter__ raises an exception, it has to clean up after itself before propagating the exception. __exit__ shouldn't be called if __enter__ fails. Shane
Yikes, this turned out to be rather long. Here's a summary: - The expansion suggested by Shane (moving __enter__ outside the try-finally) fits the expectation that __exit__ will always be paired with a successful __enter__. This "paired-exit" expansion is fairly intuitive and makes simple resources easy to write, but they are not very composable. - The expansion that Guido currently has in PEP 343 encourages an implementation style where __exit__ is idempotent. If we use it, we should document this fact, since it may seem a little unusual at first; we should also rename "enter"/"exit" so they do not mislead programmers into believing that they are paired. Simple resources are a little more work to write than with a paired-exit expansion, but they are easier to compose and reuse. - The generator style in PEP 340 is the easiest to compose and reuse, but its implementation is the most complex to understand. I lean (but only slightly) toward the second option, because it seems to be a reasonable compromise. The increased complexity of writing resources for PEP 343 over PEP 340 becomes less of an issue if we have a good do_template function in the standard library; on the other hand, the use of do_template may complicate debugging. For idempotent-exit, possible renamings of enter/exit might be enter/finish, enter/cleanup, enter/finally, start/finally, begin/finally. * * * Okay. Here's how i arrived at the above conclusions. (In the following, i'll just use "with" for "do/with".) PEP 343 (rev 1.8) currently expands with EXPR as VAR: BLOCK to this, which i'll call the "idempotent-exit" expansion: resource = EXPR exc = (None, None, None) try: try: VAR = resource.__enter__() BLOCK except: exc = sys.exc_info() raise finally: resource.__exit__(*exc) If there are problems during __enter__, then __enter__ is expected to record this fact so that __exit__ can clean up. Since __exit__ is called regardless of whether __enter__ succeeded, this encourages a style of writing resources where __exit__ is idempotent. An alternative, advocated by Shane (and by my first instincts), is this expansion, which i'll call the "paired-exit" expansion: resource = EXPR exc = (None, None, None) VAR = resource.__enter__() try: try: BLOCK except: exc = sys.exc_info() raise finally: resource.__exit__(*exc) If there are problems during __enter__, __enter__ must clean them up before propagating an exception, because __exit__ will not be called. To evaluate these options, we could look at a few scenarios where we're trying to write a resource wrapper for some lock objects. Each lock object has two methods, .acquire() and .release(). Scenario 1. You have two resource objects and you want to acquire both. Scenario 2. You want a single resource object that acquires two locks. Scenario 3. Your resource object acquires one of two locks depending on some runtime condition. Scenario 1 (Composition by client) ================================== The client writes this: with resource1: with resource2: BLOCK The idempotent-exit expansion would yield this: exc1 = (None, None, None) try: try: resource1.__enter__() exc2 = (None, None, None) try: try: resource2.__enter__() BLOCK except: exc2 = sys.exc_info() raise finally: resource2.__exit__(*exc2) except: exc1 = sys.exc_info() raise finally: resource1.__exit__(*exc1) Because __exit__ is always called even if __enter__ fails, the resource wrapper must record whether __enter__ succeeded: class ResourceI: def __init__(self, lock): self.lock = lock self.acquired = False def __enter__(self): self.lock.acquire() self.acquired = True def __exit__(self, *exc): if self.acquired: self.lock.release() self.acquired = False The paired-exit expansion would yield this: exc1 = (None, None, None) resource1.__enter__() try: try: exc2 = (None, None, None) resource2.__enter__() try: try: BLOCK except: exc2 = sys.exc_info() raise finally: resource2.__exit__(*exc2) except: exc1 = sys.exc_info() raise finally: resource1.__exit__(*exc1) In this case the lock can simply be implemented as: class ResourceP: def __init__(self, lock): self.lock = lock def __enter__(self): self.lock.acquire() def __exit__(self, *exc): self.lock.release() With PEP 340, assuming no return values and the presence of an __exit__ method, we would get this expansion: exc1 = None while True: try: if exc1: resource1.__exit__(*exc1) # may re-raise *exc1 else: resource1.next() # may raise StopIteration except StopIteration: break try: exc1 = None exc2 = None while True: try: if exc2: resource2.__exit__(*exc2) # may re-raise *exc2 else: resource2.next() # may raise StopIteration except StopIteration: break try: exc2 = None BLOCK except: exc2 = sys.exc_info() except: exc1 = sys.exc_info() Assuming that the implementations of resource1 and resource2 invoke 'yield' exactly once, this reduces to: exc1 = None resource1.next() # first time, will not raise StopIteration try: exc2 = None resource2.next() # first time, will not raise StopIteration try: BLOCK except: exc2 = sys.exc_info() try: if exc2: resource2.__exit__(*exc2) else: resource2.next() # second time, will raise StopIteration except StopIteration: pass except: exc1 = sys.exc_info() try: if exc1: resource1.__exit__(*exc1) else: resource1.next() # second time, will raise StopIteration except StopIteration: pass For this expansion, it is sufficient to implement the resource as: def ResourceG(lock): lock.acquire() try: yield finally: lock.release() Scenario 2 (Composition by implementor) ======================================= The client writes this: with DoubleResource(lock1, lock2): BLOCK With the idempotent-exit expansion, we could implement DoubleResource directly like this: class DoubleResourceI: def __init__(self, lock1, lock2): self.lock1, self.lock2 = lock1, lock2 self.got1 = self.got2 = False def __enter__(self): self.lock1.acquire() self.got1 = True try: self.lock2.acquire() self.got2 = True except: self.lock1.release() self.got1 = False def __exit__(self, *exc): try: if self.got2: self.lock2.release() self.got2 = False finally: if self.got1: self.lock1.release() self.got1 = False or it could be implemented in terms of ResourceA like this: class DoubleResourceI: def __init__(self, lock1, lock2): self.resource1 = ResourceI(lock1) self.resource2 = ResourceI(lock2) def __enter__(self): self.resource1.__enter__() self.resource2.__enter__() def __exit__(self, *exc): try: self.resource2.__exit__() finally: self.resource1.__exit__() On the other hand, if we use the paired-exit expansion, the DoubleResource would be implemented like this: class DoubleResourceP: def __init__(self, lock1, lock2): self.lock1, self.lock2 = lock1, lock2 def __enter__(self): self.lock1.acquire() try: self.lock2.acquire() except: self.lock1.release() raise def __exit__(self): try: self.lock2.release() finally: self.lock1.release() As far as i can tell, the implementation of DoubleResourceP is made no simpler by the definition of ResourceP. With PEP 340, the DoubleResource could be written directly like this: def DoubleResourceG(lock1, lock2): lock1.acquire() try: lock2.acquire() except: lock1.release() raise try: yield finally: try: lock2.release() finally: lock1.release() Or, if ResourceG were already defined, it could simply be written: def DoubleResourceG(lock1, lock2): with ResourceG(lock1): with ResourceG(lock2): yield This should also work with PEP 343 if decorated with "@do_template", though i don't have the patience to verify that carefully. When written this way, the Boolean flags disappear, as their purpose is replaced by the internal generator state. Scenario 3 (Conditional acquisition) ==================================== The client writes this: with ConditionalResource(condition, lock1, lock2): BLOCK For the idempotent-exit expansion, we could implement ConditionalResource directly like this: class ConditionalResourceI: def __init__(self, condition, lock1, lock2): self.condition = condition self.lock1, self.lock2 = lock1, lock2 self.got1 = self.got2 = False def __enter__(self): if self.condition(): self.lock1.acquire() self.got1 = True else: self.lock2.acquire() self.got2 = True def __exit__(self, *exc): try: if self.got1: self.lock1.release() self.got1 = False finally: if self.got2: self.lock2.release() self.got2 = False Or we could implement it more simply in terms of ResourceI like this: class ConditionalResourceI: def __init__(self, condition, lock1, lock2): self.condition = condition self.resource1 = ResourceI(lock1) self.resource2 = ResourceI(lock2) def __enter__(self): if self.condition(): self.resource1.__enter__() else: self.resource2.__enter__() def __exit__(self, *exc): try: self.resource2.__exit__() finally: self.resource1.__exit__() For the paired-exit expansion, we would implement ConditionalResource directly like this: class ConditionalResourceP: def __init__(self, condition, lock1, lock2): self.condition = condition self.lock1, self.lock2 = lock1, lock2 self.flag = None def __enter__(self): self.flag = self.condition() if self.flag: self.lock1.acquire() else: self.lock2.acquire() def __exit__(self, *exc): if self.flag: self.lock1.release() else: self.lock2.release() And using PEP 340, we would write it as a generator like this: def ConditionalResourceG(condition, lock1, lock2): if condition: with ResourceG(lock1): yield else: with ResourceG(lock2): yield Again, i would expect this to also work with PEP 343 if "@do_template" were inserted in front. -- ?!ng
Ka-Ping Yee wrote:
- The generator style in PEP 340 is the easiest to compose and reuse, but its implementation is the most complex to understand.
The latest version of my PEP 3XX aims to get (most of) the power of PEP 340, with the easy comprehensibility of PEP 310. What magic it requires is almost entirely contained in the statement_template decorator. It can be looked at as PEP 340 without the looping or ability to suppress exceptions, or as PEP 343, with PEP 340's style of using generators to write templates. It falls into the category where __enter__ and __exit__ are paired, as it uses the same expansion as Shane describes (an exception in __enter__ means that __exit__ is never executed).
To evaluate these options, we could look at a few scenarios where we're trying to write a resource wrapper for some lock objects. Each lock object has two methods, .acquire() and .release().
Scenario 1. You have two resource objects and you want to acquire both.
Scenario 2. You want a single resource object that acquires two locks.
Scenario 3. Your resource object acquires one of two locks depending on some runtime condition.
For your three scenarios, PEP 3XX usage and implementation are as you describe for PEP 343. PEP 343 itself doesn't work as you describe, as it still prohibits yielding inside a try/finally block (and, by extension, inside a with statement, as that is just syntactic sugar for a particular type of try/finally). Scenario 1 (two locks, handled manually): PEP 3XX actually recommends supplying __enter__ and __exit__ directly on lock objects, so no additional 'resource' wrapper is required: with lock1: with lock2: BLOCK And the relevant lock methods are: def __enter__(self): self.acquire() def __exit__(self, *exc): self.release() However, if that didn't happen, and an external wrapper was needed, it could be optimally implemented as: class Resource(object): def __init__(self, lock): self.lock = lock def __enter__(self): self.lock.acquire() def __exit__(self, *exc): self.lock.release() Or less efficiently as: @statement_template def Resource(lock): lock.acquire() try: yield finally: lock.release() Scenario 2 (two locks, handled by resource): Used as: with DoubleResource(lock1, lock2): BLOCK Implemented as: @statement_template def DoubleResource(resource1, resource2): with resource1: with resource2: yield Scenario 3 (runtime choice of lock): Used as: with ConditionalResource(condition, lock1, lock2): BLOCK Implemented as: @statement_template def ConditionalResource(condition, resource1, resource2): if condition: with resource1: yield else: with resource2: yield Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Shane Hathaway wrote:
Guido van Rossum wrote:
Consider an application where you have to acquire *two* locks regularly:
You really have to write it like this:
Shane, you've already solved this one more elegantly: def lockBoth(): return combining(lock1.locking(), lock2.locking()) using the combining function you wrote earlier, which I assume will make it into the library. - Anders
Guido van Rossum wrote:
[Nick Coghlan]
Also, the call to __enter__() needs to be before the try/finally block (as it is in PEP 310). Otherwise we get the "releasing a lock you failed to acquire" problem.
I did that on purpose. There's a separate object ('abc' in the pseudo-code of the translation) whose __enter__ and __exit__ methods are called, and in __enter__ it can keep track of the reversible actions it has taken.
Consider an application where you have to acquire *two* locks regularly:
def lockBoth(): got1 = got2 = False lock1.acquire(); got1 = True lock2.acquire(); got2 = True yield None if got2: lock2.release() if got1: lock1.release()
If this gets interrupted after locking lock1 but before locking lock2, it still has some cleanup to do.
I know that this complicates simpler use cases, and I'm not 100% sure this is the right solution; but I don't know how else to handle this use case.
If we retained the ability to inject exceptions into generators, this would be written with the extremely natural: @with template: def lockboth(): lock1.acquire() try: lock2.acquire() try: yield finally: lock2.release() finally: lock1.release() Or, even more simply: @with_template: def lockboth(): with lock1: with lock2: yield I think Fredrik's intuition is on to something - PEP 343 has scaled the idea back *too* far by throwing away the injection of exceptions into generator templates, when the only major objection was to the looping nature of the proposal. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
participants (6)
-
Anders J. Munch -
Guido van Rossum -
Ka-Ping Yee -
Nick Coghlan -
Robert Brewer -
Shane Hathaway