[ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL

[John Nagle]

On 6/25/2010 4:14 PM, Paul Rubin wrote:
> I only had a couple minutes to look at it (maybe more during the
> weekend).  It looks interesting.  I wonder whether Python is really the
> right host language for it.  How do you handle nested objects whose
> outermost layer is immutable but whose contents are potentially mutable?
> An obvious example is a list within a tuple
>
>     ( [1,2,3], )
>
> but conceivably the mutable stuff could exist at an arbitrary depth
> (making a thorough scan impractical) and give rise to a race condition.

    "freeze" freezes recursively.  So

	freeze(( [1,2,3], ))

returns

	( (1,2,3), )

So passing a big, mutable data object into a synchronized object is
inefficient, but it works.  (The same issue appears with
the multiprocessing module; if you pass in something big, there's
considerable copying overhead.)

    If you have some complex structure, like
a tree, that needs to be visible to multiple threads, it's appropriate
to put it inside a SynchronizedObject or AtomicObject and
manipulate it through methods.

    Python is a good language for this because so much Python data is
immutable.  Sharing immutable data between threads is thread-safe,
which can eliminate much copying.

> Also, I haven't followed PyPy development lately, but appraently stuff
> has been happening.  Maybe the issues of CPython and the GIL will be
> behind us not that long from now.  Is newthreading likely to be
> applicable to PyPy?

PyPy has a global lock.  See 
"http://morepypy.blogspot.com/2008/05/threads-and-gcs.html".  They've 
looked at replacing their global lock
with many local locks, but the overhead on that is high. 
"safethreading" tried, and it didn't work out well.  The "newthreading"
concept could be applied to PyPy; it's not tied to a particular
implementation.  I haven't looked much at PyPy though.

There's more on memory management issues in the "newthreading" theory
paper, "http://www.animats.com/papers/languages/pythonconcurrency.html".
Right now, I'm focusing on usability of the approach.  It can be made
to go fast; the question is whether it can be made popular.

				John Nagle