On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:44:14 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Offloading execution to other threads -------------------------------------
It is possible to run code in a separate OS thread using a copy of the current thread context::
executor = ThreadPoolExecutor() current_context = contextvars.copy_context()
executor.submit( lambda: current_context.run(some_function))
Does it also support offloading to a separate process (using ProcessPoolExecutor in the example above)? This would require the Context to support pickling.
I don't think that's a requirement. The transparency between the two different types of executor is mostly misleading anyway -- it's like the old RPC transparency problem, which was never solved IIRC. There are just too many things you need to be aware of before you can successfully offload something to a different process.
I agree.
I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__. I guess we could do like namedtuple and (a) capture the module where the ContextVar was instantiated, on the assumption that that's where it will be stored, (b) require that users pass in the name of variable where it will be stored as the 'name' argument to ContextVar.__init__. I tend to agree that this is something to worry about for 3.8 though. (If we need to retrofit pickle support, we could add a pickleable=False argument to ContextVar, and require people to pass pickleable=True to signal that they've done the appropriate setup to make the ContextVar identifiable across processes, and that its contents are safe to pickle.) -n -- Nathaniel J. Smith -- https://vorpus.org