[Python-Dev] More on contextlib - adding back a contextmanager decorator

[Nick Coghlan]

Guido van Rossum wrote:
> On 4/30/06, Nick Coghlan <ncoghlan at iinet.net.au> wrote:
>> A few things from the pre-alpha2 context management terminology review 
>> have
>> had a chance to run around in the back of my head for a while now, and 
>> I'd
>> like to return to a topic Paul Moore brought up during that discussion.
> 
> I believe the context API design has gotten totally out of hand.
> Regardless of the merits of the "with" approach to HTML generation
> (which I personally believe to be an abomination),

The example is tempting because it's easy to follow. I agree actually doing it 
in real code would almost certainly be nuts :)

> I don't see why the
> standard library should support every possible use case with a
> custom-made decorator. Let the author of that tag library provide the
> decorator.

The HTML tag was just an example. The underlying idea is being able to easily 
create a re-usable object that can be passed to multiple with statements 
(potentially nested within each other or within distinct threads). Without the 
__context__ method, the naive version of such an object looks like:

class reusable(object):
     def __init__(self, factory):
         self.factory = factory
         factory() # Check the factory works at definition time
     def __enter__(self):
         current = self.current = factory()
         return current.__enter__()
     def __exit__(self, *exc_info):
         return self.current.__exit__(*exc_info)

The downside of this over the __context__ method is that it is neither nesting 
nor thread-safe.  Because the storage is on the object rather than in the 
execution frame, sharing such objects between threads or using one for nested 
with statements will break (as self.current gets overwritten).

> I have a counter-proposal: let's drop __context__. Nearly all use
> cases have __context__ return self. In the remaining cases, would it
> really be such a big deal to let the user make an explicit call to
> some appropriately named method? The only example that I know of where
> __context__ doesn't return self is the decimal module.

It would also prevent threading.Condition from using its underlying lock 
object as the managed context.

The real problem I have with removing __context__() is that it pushes the 
burden of handling thread-safety and nesting-safety issues onto the developers 
of context managers without giving them any additional tools beyond 
threading.locals(). This was the problem Jason brought up for decimal.Context 
that lead to the introduction of __context__ in the first place.

Without the __context__() method, *users* of the with statement will be forced 
to create a new object with __enter__()/__exit__() methods every time, either 
by invoking a method (whose name will vary from object to object, depending on 
the whim of the designer) or by calling a factory function (which is likely to 
be created either as a zero-argument lambda returning an object with 
enter/exit methods, or else by using PEP 309's partial function).

So if you see a with statement with a bare variable name as the context 
expression, it will probably be wrong, unless:
  a) the implementor of that type provided thread-safety and nesting-safety; or
  b) the object is known to be neither thread-safe nor nesting-safe

The synchronisation objects in threading being examples of category a, file 
objects being examples of category b. In this scenario, generator contexts 
defined using @contextfactory should always be invoked directly in the context 
expression, as attempting to cache them in order to be reused won't work (you 
would need to put them in a zero-argument lambda and call it in the context 
expression, so that you get a new generator object each time).

Documenting all of the thread-safety and nesting-safety issues and how to deal 
with them would be a serious pain. I consider it much easier to provide the 
__context__() method and explain how to use that as the one obvious way to 
deal with such problems. Then only implementors need to care about it - from a 
user's point of view, you just provide a context expression that resolves to a 
context manager, and everything works as intended, including being able to 
cache that expression in a local variable and use it multiple times. (That 
last point obviously not applying to context managers like files that leave 
themselves in an unusable state after __exit__, and don't restore themselves 
to a usable state in __enter__).

Essentially, I don't think dropping __context__ would gain us anything - the 
complexity associated with it is real, and including that method in the API 
let's us deal with that complexity in one place, once and for all.

Removing the method from the statement definition just pushes the 
documentation burden out to all of the context managers where it matters (like 
decimal.Context, the documentation for which would get stuck with trying to 
explain why you have to call a method in order to get a usable context manager).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org