[Python-Dev] PEP 567 v3

[Nathaniel Smith]

On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum <guido at python.org> wrote:
>> On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> On Tue, 16 Jan 2018 17:44:14 -0500
>>> Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>>>
>>> > Offloading execution to other threads
>>> > -------------------------------------
>>> >
>>> > It is possible to run code in a separate OS thread using a copy
>>> > of the current thread context::
>>> >
>>> >     executor = ThreadPoolExecutor()
>>> >     current_context = contextvars.copy_context()
>>> >
>>> >     executor.submit(
>>> >         lambda: current_context.run(some_function))
>>>
>>> Does it also support offloading to a separate process (using
>>> ProcessPoolExecutor in the example above)?  This would require the
>>> Context to support pickling.
>>
>>
>> I don't think that's a requirement. The transparency between the two
>> different types of executor is mostly misleading anyway -- it's like the old
>> RPC transparency problem, which was never solved IIRC. There are just too
>> many things you need to be aware of before you can successfully offload
>> something to a different process.
>
> I agree.
>
> I think it would be a very fragile thing In practice: if you have even
> one variable in the context that isn't pickleable, your code that uses
> a ProcessPool would stop working.  I would defer Context pickleability
> to 3.8+.

There's also a more fundamental problem: you need some way to match up
the ContextVar objects across the two processes, and right now they
don't have an attached __module__ or __qualname__.

I guess we could do like namedtuple and (a) capture the module where
the ContextVar was instantiated, on the assumption that that's where
it will be stored, (b) require that users pass in the name of variable
where it will be stored as the 'name' argument to ContextVar.__init__.
I tend to agree that this is something to worry about for 3.8 though.
(If we need to retrofit pickle support, we could add a
pickleable=False argument to ContextVar, and require people to pass
pickleable=True to signal that they've done the appropriate setup to
make the ContextVar identifiable across processes, and that its
contents are safe to pickle.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org