[Python-ideas] solving multi-core Python

Eric Snow ericsnowcurrently at gmail.com
Sun Jun 21 01:16:37 CEST 2015

On Jun 20, 2015 4:55 PM, "Devin Jeanpierre" <jeanpierreda at gmail.com> wrote:
> It's worthwhile to consider fork as an alternative.  IMO we'd get a
> lot out of making forking safer, easier, and more efficient. (e.g.
> respectively: adding an atfork registration mechanism; separating out
> the bits of multiprocessing that use pickle from those that don't;
> moving the refcount to a separate page, or allowing it to be frozen
> prior to a fork.)

So leverage a common base of code with the multiprocessing module?

> It sounds to me like this approach would use more memory than either
> regular threaded code or forking, so its main advantages are being
> cross-platform and less bug-prone. Is that right?

I would expect subinterpreters to use less memory.  Furthermore creating
them would be significantly faster.  Passing objects between them would be
much more efficient.  And, yes, cross-platform.

> Note: I don't count the IPC cost of forking, because at least on
> linux, any way to efficiently share objects between independent
> interpreters in separate threads can also be ported to independent
> interpreters in forked subprocesses,

How so?  Subinterpreters are in the same process.  For this proposal each
would be on its own thread.  Sharing objects between them through channels
would be more efficient than IPC.  Perhaps I've missed something?

> and *should* be.
> See also: multiprocessing.Value/Array. This is probably a good
> opportunity for that unification you mentioned. :)

I'll look.

> On Sat, Jun 20, 2015 at 3:04 PM, Yury Selivanov <yselivanov.ml at gmail.com>
> > On 2015-06-20 5:42 PM, Eric Snow wrote:
> >> * only allow immutable objects to be shared between subinterpreters
> >
> > Even if this is the only thing we have -- an efficient way
> > for sharing immutable objects (such as bytes, strings, ints,
> > and, stretching the definition of immutable, FDs) that will
> > allow us to do a lot.
> +1, this has a lot of utility, and can be extended naturally to other
> types and circumstances.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/a306568b/attachment-0001.html>

More information about the Python-ideas mailing list