[Python-Dev] More on contextlib - adding back a contextmanager decorator

[Guido van Rossum]

[I'm cutting straight to the chase here]
On 4/30/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The downside of this over the __context__ method is that it is neither nesting
> nor thread-safe.

This argument is bogus.

We currently have two types of objects involved in with-statements:
those whose __context__ returns self and those whose __context__
returns some other object. In all of the latter cases, the original
object doesn't have __enter__ or __exit__ methods, so using it in the
"reduced-with-statement" will be an immediate run-time error. Writing
a method that returns the correct type of object with the correct
behavior (thread-safe, or nesting, or whatever is required by that
specific object) is no harder whether the method name is __context__
or not.

> The real problem I have with removing __context__() is that it pushes the
> burden of handling thread-safety and nesting-safety issues onto the developers
> of context managers without giving them any additional tools beyond
> threading.locals(). This was the problem Jason brought up for decimal.Context
> that lead to the introduction of __context__ in the first place.

Again, I don't see how writing the thread-safe version is easier when
the method is called __context__.

> Without the __context__() method, *users* of the with statement will be forced
> to create a new object with __enter__()/__exit__() methods every time,

You seem to be missing the evidence that 9 out of 10 objects currently
have a __context__ that returns self. In all those cases the user of
the with-statement won't have to make any changes at all compared to
code that works with 2.5a2.

> either
> by invoking a method (whose name will vary from object to object, depending on
> the whim of the designer) or by calling a factory function (which is likely to
> be created either as a zero-argument lambda returning an object with
> enter/exit methods, or else by using PEP 309's partial function).

Having the name being different in each situation may actually be an
advantage -- it will give an additional clue as to what is happening,
and it will let us design different APIs for use in a with-statement
(just like dicts have iterkeys() and iteritems()).

> So if you see a with statement with a bare variable name as the context
> expression, it will probably be wrong, unless:
>   a) the implementor of that type provided thread-safety and nesting-safety; or
>   b) the object is known to be neither thread-safe nor nesting-safe
>
> The synchronisation objects in threading being examples of category a, file
> objects being examples of category b. In this scenario, generator contexts
> defined using @contextfactory should always be invoked directly in the context
> expression, as attempting to cache them in order to be reused won't work (you
> would need to put them in a zero-argument lambda and call it in the context
> expression, so that you get a new generator object each time).

Now we get to the crux of the matter. When I recommend that people write

  with foo.some_method():
      <BLOCK>

you are worried that if they need two separate blocks like that, they
are tempted to write

  x = foo.some_method()
  with x:
      <BLOCK1>
  with x:
      <BLOCK2>

But there's an obvious solution for that: make sure that the object
returned by foo.some_method() can only be used once. The second "with
x" will raise an exception explaining what went wrong. (We could even
rig it so that nesting the second "with x" inside the first one will
produce a different error message.)

So the "with foo.some_method()" idiom is the only one that works, and
that is just as thread-safe and nesting-resilient as is writing
__context__() today.

If you object against the extra typing, we'll first laugh at you
(proposals that *only* shave a few characters of a common idiom aren't
all that popular in these parts), and then suggest that you can spell
foo.some_method() as foo().

> Essentially, I don't think dropping __context__ would gain us anything - the
> complexity associated with it is real, and including that method in the API
> let's us deal with that complexity in one place, once and for all.

It's not real in 9 out of 10 current cases.

(And the Condition variable could easily be restored to its previous
case where it had its own __enter__ and __exit__ methods that called
the corresponding methods of the underlying lock. The code currently
in svn is merely an optimization.)

--
--Guido van Rossum (home page: http://www.python.org/~guido/)