[Python-ideas] [Python-Dev] PyParallel: alternate async I/O and GIL removal

[Nick Coghlan]

On 17 November 2013 12:56, Guido van Rossum <guido at python.org> wrote:
> On Sat, Nov 16, 2013 at 6:24 PM, Trent Nelson <trent at snakebite.org> wrote:
>>     And in hindsight, perhaps I need to put more emphasis on the fact
>>     that it *is* very experimental work with a long-term view, versus
>>     Tulip/asyncio, which was intended for *now*.  So although Tulip and
>>     PyParallel spawned from the same discussions and are attempting to
>>     attack the same problem -- it's really not fair for me to discredit
>>     Tulip/Twisted in favor of PyParallel because they're on completely
>>     different playing fields with vastly different implementation time
>>     frames (I'm thinking 5+ years before this work lands in a mainstream
>>     Python release -- if it ever does.  And if not, hey, it can live on
>>     as another interpreter, just like Stackless et al).
>
>
> I would love it if you could write a list of things a callback *cannot* do
> when it is in parallel mode. I believe that list includes mutating any kind
> of global/shared state (any object created in the main thread is read-only
> in parallel mode -- it seems you had to work hard to make string interning
> work, which is semantically transparent but mutates hidden global state). In
> addition (or, more likely, as a consequence!) a callback cannot create
> anything that lasts beyond the callback's lifetime, except for the brief
> time between the callback's return and the completion of the I/O operation
> involving the return value. (Actually, I missed how you do this -- doesn't
> this mean you cannot release the callback's heap until much later?)
>
> So it seems that the price for extreme concurrency is the same as always --
> you can only run purely functional code. Haskell fans won't mind, but for
> Python this seems to be putting the cart before the horse -- who wants to
> write Python with those constraints?

MapReduce fans already do :)

I think there's some interesting potential in Trent's PyParallel work,
but it needs something analogous to Rust's ability to transfer object
ownership between threads (thus enabling message passing) to expand
beyond the simple worker thread model which is really only interesting
on Windows (where processes are expensive - on *nix, processes are
generally cheap enough that PyParallel is unlikely to be worth the
hassle).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia