[Python-ideas] Letting context managers react to yields inside their scope

[Eric Snow]

On Wed, Apr 29, 2015 at 3:01 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> Hi Nathanial,
>
> Sorry for not replying to your last email promptly.
> FWIW I was going to do it later today ;)
>
> My opinion on this subject (and I've implemented lots
> of local-context kind of objects in different frameworks) is
> that *just* inserting some kind of suspend/resume points
> before/after yields does not work.

Good point.

>
> Why:
>
> 1. Greenlets, gevent, eventlet, stackless python: they do
> not have 'yield'. Context switches are invisible to the
> interpreter.  And those frameworks are popular.  You want
> numpy.errstate/decimal.localcontext to work there too.
>
> 2. I'm curious how this will be implemented and what's
> the performance impact.
>
> 3. I think that mechanism should be more generic than
> just 'with' staments. What if you want to have a decorator
> that applies some context? What if you can't write it as
> a generator with @contextlib.contextmanager?
>
> What I would propose:
>
> I think that the right approach here would be to have a
> standard protocol; something similar to how we defined
> WSGI -- it doesn't require any changes in the interpreter,
> it is a protocol.
>
> For instance, we could have a module in the stdlib, with a
> class Context:
>
>    class Context:
>       @classmethod
>       def __suspend_contexts(cls):
>          for child in cls.__subclasses__():
>             child.__suspend_context()
>
>       @classmethod
> def __resume_contexts(cls): ..
>
>
> To inform context objects that the context is about to change,
> asyncio event loop would call Context.__suspend_contexts()
>
> I think that it's up to the event loop/library/framework
> to manage context switches properly.  In asyncio, for instance,
> you can attach callbacks to Futures. I think it would be
> great if such callbacks are executed in the right context.
>
> I would also suggest to think about a universal context --
> the one that works both for asyncio coroutines and
> threadpools they might use.
>
> This way any framework can implement the context in the
> most efficient way.
>
> All in all, I'm in favor of API and a couple functions
> added to the stdlib for this.

Great idea.  I've been thinking of something along those lines.  I
think there also needs to be a recognition of active vs. inactive
context.  Only active contexts would be used by the event loop/etc.  A
generic context would also make it easy to clone and to use in context
managers.

One approach:

class Context:
    def activate(self): ...  # or open
    def deactivate(self): ...  # or close
    def clone(self): ...
    def __suspend__(self, cs, id): ...
    def __resume__(self, cs, id): ...
    def __enter__(self):
        self.activate()
        return self
    def __exit__(self, ...):
        self.deactivate()

def active(ctx=Context):
    return ...

The various "context switchers" (event loop, thread pool, etc.) would
then call __suspend__ and __resume__ on all the active contexts.  The
two args (cs for "context switcher") to those methods would allow each
context object to adjust to the targeted "linear execution context"
uniquely identified by the (cs, id) pair.  For example:
thread/async-local namespaces.

I've often found it to be cleaner to split the "spec" from the "instance", so:

class InactiveContext:  # or GlobalContext or DefaultContext
    def activate(self): ...  # return a new cloned Context
    def clone(self): ...  # merge with activate?

class Context:
    def deactivate(self): ...
    def clone(self): ...
    def __suspend__(self, cm, id): ...
    def __resume__(self, cm, id): ...
    def __enter__(self):
        return self
    def __exit__(self, ...):
        self.deactivate()

This is all rough and the result of divided attention over the last
few hours, but the idea of leaving the switching up to the "context
switcher" is the key thing.  Also, I smell some overlapping concepts
with existing types and patterns that could probably distill the idea
of a generic Context type into something cleaner.

-eric