[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 22 Jun 2015 01:45, "Sturla Molden" <sturla.molden at gmail.com> wrote:
>
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> > Sturla, expressing out and out contempt for entire communities of
> > capable, competent developers (both the creators of Windows and Java,
> > and the users of those platforms) has no place on the core Python
> > mailing lists. Please refrain from casually insulting entire groups of
> > people merely because you don't approve of their technical choices.
>
> I am not sure what you mean. Using threads on Windows and Java comes from
a
> necessity, not because developers are incompetent.

The folks *designing* Windows and Java are also people, and as creators of
development platforms go, it's hard to dispute their success in helping
folks solve real problems. We should be mindful of that when drawing
lessons from their experience.

> Windows does not provide
> a fork and processes are heavy-weight, hence multi-threading is the
obvious
> choice.

Windows actually has superior native parallel execution APIs to Linux in
some respects, but open source programming languages tend not to support
them, presumably due to a combination of Microsoft's longstanding hostile
perspective on open source licencing (which seems to finally be moderating
with their new CEO), and the even longer standing POSIX mindset that "fork
and file descriptors ought to be enough for anyone" (even if the workload
in the child processes is wildly different from that in the main process).

asyncio addresses that problem for Python in regards to IOCP vs select (et
al), and the configurable subprocess creation options addressed it for
multiprocessing, but I'm not aware of any efforts to get greenlets to use
fibres when they're available.

> > With
> > subinterpreters, we have a lot more options for that than we do with
> > any form of IPC, including shared references to immutable objects, and
> > the PEP 3118 buffer API.
>
> Perhaps. One could do this with shared memory as well, but a complicating
> factor is that the base address must be the same (or corrected for). But
> one could probably do low-level magic with memory mapping  to work around
> this. Particularly on 64-bit it is not really difficult to make sure a
page
> is mapped to the same address in two processes.
>
> It is certainly easier to achieve within a process. But if the plan for
> Erlang-style "share nothing" threads is to pickle and memcpy objects,
there
> is little or nothing to gain over using multiprocessing.

The Python level *semantics* should be as if the objects were being copied
(for ease of use), but the *implementation* should try to avoid actually
doing that (for speed of execution).

Assuming that can be done effectively *within* a process between
subinterpreters, then the possibility arises of figuring out how to use
shared memory to federate that approach across multiple processes. That
could then provide a significant performance improvement for
multiprocessing.

But since we have the option of tackling the simpler problem of
subinterpreters *first*, it makes sense to do that before diving into the
cross-platform arcana involved in similarly improving the efficiency of
multiprocessing's IPC.

Regards,
Nick.

>
>
> Sturla
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150622/79315fc4/attachment-0001.html>