[Python-ideas] [Python-Dev] PyParallel: alternate async I/O and GIL removal

[Andrew Barnert]

On Nov 16, 2013, at 22:35, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I think there's some interesting potential in Trent's PyParallel work,
> but it needs something analogous to Rust's ability to transfer object
> ownership between threads (thus enabling message passing)

I wonder whether an explicit copy_to_main_thread (maybe both shallow and deep variants) function, maybe with a Queue subclass that called it automatically in the put method, would be sufficient for a decent class of applications to be built?

> to expand
> beyond the simple worker thread model which is really only interesting
> on Windows (where processes are expensive - on *nix, processes are
> generally cheap enough that PyParallel is unlikely to be worth the
> hassle).

Windows is fine at scheduling ncores separate processes; it's just slow at _starting_ each one. And most servers aren't constantly creating and reaping processes; they create ncores processes at startup or when they're first needed. So if it takes 0.9 seconds instead of 0.2 to restart your server, is that a big enough problem to rewrite the whole server (and the interpreter)?

As I said in my previous message, the benefit of being able to skip refcounting might make it worth doing. But avoiding process creation overhead isn't much if a win.