[Python-ideas] [Python-Dev] PyParallel: alternate async I/O and GIL removal

[Charles-François Natali]

There's something which really bothers me (well, that has been
bothering me since the beginning):

"""
Memory Deallocation within
Parallel Contexts

•These parallel contexts aren’t intended to be long-running bits of
code/algorithm
•Let’s not free() anything…
•….and just blow away the entire heap via HeapFree() with one call,
once the context has finished

•Cons:
oYou technically couldn’t do this:
def work():
for x in xrange(0, 1000000000):
…
o(Why would you!)

•So… there’s no point referencing counting objects allocated within
parallel contexts!
"""

So basically, pyparallel solves the issue of garbage collection in a
multi-threaded process by not doing garbage collection: yeah, sure,
things get a lot simpler, but in real life, you do want to have loops
such as above, I don't see how one could pretend otherwise. That's
simply a show-stopper to me.

In fact, I find the whole programming model completely puzzling.
Depending on whether you're in the main thread or not:
- you're only able to write to thread-local data (thread-local is the
sense allocated by the current thread, not thread-specific): what
happens if some parallel context calls "import foo"?
- you won't be able to allocate/free many objects

So, in fact, in your parallel contexts, you can do so little that it's
IMO almost useless in practice. It's not Python - more like a cross
between Haskell and Python - and moreover, it means that some code
cannot be executed in parallel context.
Which means that you basically cannot use any library, since you don't
know what's doing under the hood (it might die with a MemoryError or
an invalid write to main thread memory).

Compare this to e.g. Go's goroutines and channels, and you'll see how
one might solve those issues in a sensible way (including user-level
thread multiplexing over kernel-level threads, and using epoll ;-).

In short, I'm really skeptical, to say the least...

cf