[Python-ideas] concurrency-local namespaces

[Yury Selivanov]

Guido, Nathaniel,

On 2015-04-22 2:45 PM, Guido van Rossum wrote:
> I'll have to leave this up to you and Nathaniel and others to ponder, I've
> run out of time to think about more issues. Sorry! (I do think that the
> conclusion of that patch was to go with current_task() intead.)

Yes, that was the conclusion.  Because I was interested to add
this functionality for web frameworks, where, as Nathaniel
mentioned, you control the top-level Task object.  So I gave up
on pushing the 'context' idea.

If we want to add a generic thread-local-like object to
coroutines, I think that it has to be done on an asyncio
event loop level.  Because event loop is not just about
coroutines, it's also about callbacks, and it is useful
to *avoid* loosing context in them.

Nathaniel, I'd be glad if you could glance over my old proposal.
It's based on ideas I expressed in this thread:
https://groups.google.com/forum/#!topic/python-tulip/zix5HQxtElg

Thanks,
Yury

>
> On Wed, Apr 22, 2015 at 11:37 AM, Yury Selivanov <yselivanov.ml at gmail.com>
> wrote:
>
>> Guido,
>>
>> On 2015-04-22 1:46 PM, Guido van Rossum wrote:
>>
>>> Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like
>>> it would be super great if the async with statement would (optionally?)
>>> notify the context manager of switching to a different task and back.
>>>
>> I'm not sure if it's possible to implement.  PEP492 coroutines
>> know nothing about the loop that schedules them.
>>
>> One way to solve this problem of TLS-like storage for coroutines
>> in asyncio is to add a notion of 'context' to the loop.
>>
>> In this code review: https://codereview.appspot.com/87940044/#ps1
>> please take a look at patch set 1, where I add
>> 'loop.get_context_id()' method.
>>
>> I also implemented a PoC of local storage based on that patch
>> (I can't remember where is it, but it was the reason of the code
>> review :) I'll implement it again if we resurrect this idea).
>>
>> With context being implemented on the loop level (with context_id
>> passed along with callbacks) it allowed to implement local storage
>> concepts on a lower level than Task object, solving some of
>> the problems Nathaniel raised.
>>
>> All in all, I don't think that to implement local storage we
>> need to do something to `async with`, but rather we should
>> implement this feature in asyncio and have the local storage
>> object shipped with it.
>>
>> Thanks,
>> Yury
>>
>>
>>
>>> On Tue, Apr 21, 2015 at 10:36 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>>   On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow <ericsnowcurrently at gmail.com>
>>>> wrote:
>>>>
>>>>> threading.local provides thread-local namespaces.  As far as I can
>>>>> tell, there isn't an equivalent for coroutines (asyncio), even though
>>>>> I would expect they would provide the same benefits.  I'd like to see
>>>>> coroutine-local namespaces added and would be happy to work on the
>>>>> problem.  I plan on working on a feature that will rely on applying a
>>>>> thread-local context and realized that coroutines would need a similar
>>>>> treatment.  Also, there are probably a few spots in the stdlib that
>>>>> would benefit (e.g. decimal contexts).
>>>>>
>>>> Possibly this is off-topic for this thread but -- it's a good point
>>>> that, currently, when implementing a global state variable (like the
>>>> decimal context), one can bind it to a single thread by storing the
>>>> state in TLS, but AFAIK there's no way to do this with coroutines.
>>>> It's somewhat unpleasant that
>>>>
>>>>     with decimal.localcontext() as ctx:
>>>>         ctx.prec = 10
>>>>         values = (yield from db.fetchstuff())
>>>>         morecodehere()
>>>>
>>>> can cause arbitrary other code to run with the given decimal context
>>>> in effect, while also failing to guarantee that morecodehere() will
>>>> run with any particular decimal context.
>>>>
>>>> I hoped for a second that "async with" might help with this, but
>>>> checking the PEP it doesn't -- to do that it would need support
>>>> context manager callbacks that were called whenever the coroutine got
>>>> suspended/resumed while the 'with' block was in effect, but that's not
>>>> what it does, it just allows the __enter__ and __exit__ methods
>>>> themselves to be coroutines.
>>>>
>>>> And AFAICT the thread Guido linked
>>>> (https://github.com/python/asyncio/issues/165) only considers cases
>>>> where an async framework (e.g. a web server framework) is the one who
>>>> wants to set global variables that can be accessed from within a set
>>>> of coroutines. This works out, b/c when the same code controls both
>>>> the spawning of coroutines (by creating a new Task) and setting the
>>>> variables, so it can bind them together. But that doesn't seem to help
>>>> for something like the decimal module context, or numpy's error
>>>> handling context
>>>> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html
>>>> ).
>>>>
>>>> Obviously one can just document that users should always bracket any
>>>> yield/await calls with explicit state setting/restore functions, but
>>>> is there any better solution?
>>>>
>>>> -n
>>>>
>>>> --
>>>> Nathaniel J. Smith -- http://vorpus.org
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>