[Python-ideas] Python 3000 TIOBE -3%

[Nick Coghlan]

On Sat, Feb 11, 2012 at 1:40 PM, Matt Joiner <anacrolix at gmail.com> wrote:
> I'm asking if it'd actually be accepted in 3.

Why is that relevant? If free threading is the all-singing all dancing
wonderment you believe:

1. Fork CPython
2. Make it free-threaded (while retaining backwards compatibility with
all the C extensions out there!)
3. Watch the developer hordes flock to your door (after all, it's the
lack of free-threading that has held Python's growth back for the last
two decades, so everyone will switch in a heartbeat the second you, or
anyone else, publishes a free-threaded alternative where all their C
extensions work. Right?).

> I know well, and have seen how
> quickly things are blocked and rejected in core (dabeaz and shannon's
> attempts come to mind). I'm well familiar with previous attempts.

If that's what you think happened, then no, you're not familiar with
them at all.

python-dev has just a few simple rules for accepting a free-threading patch:

1. All current third party C extension modules must continue to work
(ask the folks working on Ironclad for IronPython and cpyext for PyPy
how much fun *that* requirement is)
2. Calls to builtin functions and methods should remain atomic (the
Jython experience can help a lot here)
3. The performance impact on single threaded scripts must be minimal
(which basically means eliding all the locks in single-threaded mode
the way CPython currently does with the GIL, but then creating those
elided locks in the correct state when Python's threading support gets
initialised)

That's it, that's basically all the criteria we have for accepting a
free-threading patch. However, while most people are quite happy to
say "Hey, someone should make CPython free-threaded!", they're
suddenly far less interested when faced with the task of implementing
it *themselves* while preserving backwards compatibility (and if you
think the Python 2 -> Python 3 transition is rough going, you are
definitely *not* prepared for the enormity of the task of trying to
move the entire C extension ecosystem away from the refcounting APIs.
The refcounting C API compatibility requirement is *not* optional if
you want a free-threaded CPython to be the least bit interesting in
real world terms).

When we can't even get enough volunteers willing to contribute back
their fixes and workarounds for the known flaws in multiprocessing, do
people think there is some magical concurrency fairy that will
sprinkle free threading pixie dust over CPython and the GIL will be
gone?

Removing the GIL *won't* be fun. Just like fixing multiprocessing, or
making improvements to the GIL itself, it will be a long, tedious
grind dealing with subtleties of the locking and threading
implementations on Windows, Linux, Mac OS X, *BSD, Solaris and various
other platforms where CPython is supported (or at least runs). For
extra fun, try to avoid breaking the world for CPython variants on
platforms that don't even *have* threading (e.g. PyMite). And your
reward for all that effort? A CPython with better support for what is
arguably one of the *worst* approaches to concurrency that computer
science has ever invented.

If a fraction of the energy that people put into asking for free
threading was instead put into asking "how can we make inter-process
communication better?", we'd no doubt have a good shared object
implementation in the mmap module by now (and someone might have
actually responded to Jesse's request for a new multiprocessing
maintainer when circumstances forced him to step down).

But no, this is the internet: it's much easier to incessantly berate
python-dev for pointing out that free threading would be
extraordinarily hard to implement correctly and isn't the panacea that
many folks seem to think it is than it is to go *do* something that's
more likely to offer a good return on the time investment required.

My own personal wishlist for Python's concurrency support?

* I want to see mmap2 up on PyPI, with someone working on fast shared
object IPC that can then be incorporated into the stdlib's mmap module
* I want to see multiprocessing2 on PyPI, with someone working on the
long list of multiprocessing bugs on the python.org bug tracker
(including adding support for Windows-style, non-fork based child
processes on POSIX platforms)
* I want to see progress on PEP 3153, so that some day we can have a
"Python event loop" instead of a variety of framework specific event
loops, as well as solid cross-platform async IO support in the stdlib.

As Jesse said earlier, asking for free threading in CPython is like
asking for free pie. Sure, free pie would be nice, but who's going to
bake it? And what else could those people be doing with their time if
they weren't busy baking pie?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia