[Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL

[Adam Olsen]

On Sat, Jun 26, 2010 at 10:39, John Nagle <nagle at animats.com> wrote:
>    The rationale behind "freezing" some of the language semantics
> when the program goes multi-thread comes from two sources -
> Adam Olsen's Safethread work, and the acceptance of the
> multiprocessing module.  Olsen tried to retain all the dynamism of
> the language in a multithreaded environment, but locking all the
> underlying dictionaries was a boat-anchor on the whole system,
> and slowed things down so much that he abandoned the project.
> The Unladen Swallow documentation indicates that early thinking
> on the project was that Olsen's approach would allow getting
> rid of the GIL, but later notes indicate that no path to a
> GIL-free JIT system is currently in development.

That's not true.  Refcounting was the boat-anchor, not dicts.  I was
unable to come up with a relatively simple replacement that scaled
fully.

The dicts shared as module globals and class dicts were a design
issue, but more of an ideological one: concurrency mentality says you
should only share immutable objects.  Python prefers ad-hoc design,
where you can do what you want so long as it's not particularly nasty.
 I was unable to find a way to have both, so I declared the python
mentality the winner.

The shareddict I came up with uses a read write lock, so that it's
safe when you do mutate and doesn't bottleneck when you don't mutate.
The only thing fancy was my method of checkpointing when doing a
readlock->writelock transition, but there's a hundred other ways to
accomplish that.