[Python-ideas] [Python-Dev] PyParallel: alternate async I/O and GIL removal

[Antoine Pitrou]

On Sun, 17 Nov 2013 16:35:23 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> I think there's some interesting potential in Trent's PyParallel work,
> but it needs something analogous to Rust's ability to transfer object
> ownership between threads (thus enabling message passing) to expand
> beyond the simple worker thread model which is really only interesting
> on Windows (where processes are expensive - on *nix, processes are
> generally cheap enough that PyParallel is unlikely to be worth the
> hassle).

This is a bit of an oversimplification. The cost of processes is not
only the cost of spawning them. There is also the CPU cost of
marshalling data between processes, and the memory cost of having
duplicate structures and data in your various processes.

(also, note that using a process pool generally amortizes the spawning
cost quite well)

Regards

Antoine.