[pypy-dev] Would the following shared memory model be possible?

Alex Gaynor alex.gaynor at gmail.com
Thu Jul 29 03:44:23 CEST 2010


On Wed, Jul 28, 2010 at 8:33 PM, Kevin Ar18 <kevinar18 at hotmail.com> wrote:
> As a followup to my earlier post:
> "pre-emptive micro-threads utilizing shared memory message passing?"
>
> I am actually finding that the biggest hurdle to accomplishing what I want
> is the lack of ANY type of shared memory -- even if it is limited.  I wonder
> if I might ask a question:
>
> Would the following be a possible way to offer a limited type of shared
> memory:
>
> Summary: create a system very, very similar to POSH, but with differences:
>
> In detail, here's what I mean:
> * unlike POSH, utilize OS threads and shared memory (not processes)
> * Create a special shared memory location where you can place Python objects
> * Each Python object you place into this location can only be accessed
> (modified) by 1 thread.
> * You must manually assign ownership of an object to a particular thread.
> * The thread that "owns" the object is the only one that can modify it.
> * You can transfer ownership to another thread (but, as always only the
> owner can modify it).
>
> * There is no GIL when a thread interacts with these special objects.  You
> can have true thread parallelism if your code uses a lot of these special
> objects.
> * The GIL remains in place for all other data access.
> * If your code has a mixture of access to the special objects and regular
> data, then once you hit a point where a thread starts to interact with data
> not in the special storage, then that thread must follow GIL rules.
>
> Granted, there might be some difficulty with the GIL part... but I thought I
> might ask anyways. :)
>
>> Date: Wed, 28 Jul 2010 22:54:39 +1000
>> Subject: Re: [pypy-dev] pre-emptive micro-threads utilizing shared memory
>> message passing?
>> From: william.leslie.ttg at gmail.com
>> To: kevinar18 at hotmail.com
>> CC: pypy-dev at codespeak.net
>>
>> On 28 July 2010 04:20, Kevin Ar18 <kevinar18 at hotmail.com> wrote:
>> > I am attempting to experiment with FBP - Flow Based Programming
>> > (http://www.jpaulmorrison.com/fbp/ and book:
>> > http://www.jpaulmorrison.com/fbp/book.pdf)  There is something very similar
>> > in Python: http://www.kamaelia.org/MiniAxon.html  Also, there are some
>> > similarities to Erlang - the share nothing memory model... and on some very
>> > broad levels, there are similarities that can be found in functional
>> > languages.
>>
>> Does anyone know if there is a central resource for incompatible
>> python memory model proposals? I know of Jython, Python-Safethread,
>> and Mont-E.
>>
>> I do like the idea of MiniAxon, but let me mention a topic that has
>> slowly been bubbling to the front of my mind for the last few months.
>>
>> Concurrency in the face of shared mutable state is hard. It makes it
>> trivial to introduce bugs all over the place. Nondeterminacy related
>> bugs are far harder to test, diagnose, and fix than anything else that
>> I would almost mandate static verification (via optional typing,
>> probably) of task noninterference if I was moving to a concurrent
>> environment with shared mutable state. There might be a reasonable
>> middle ground where, if a task attempts to violate the required static
>> semantics, it fails dynamically. At least then, latent bugs make
>> plenty of noise. An example for MiniAxon (as I understand it, which is
>> not very well) would be verification that a "task" (including
>> functions that the task calls) never closes over and yields the same
>> mutable objects, and never mutates globally reachable objects.
>>
>> I wonder if you could close such tasks off with a clever subclass of
>> the proxy object space that detects and rejects such memory model
>> violations? With only semantics that make the program deterministic?
>>
>> The moral equivalent would be cooperating processes with a large
>> global (or not) shared memory store for immutable objects, queues for
>> communication, and the additional semantic that objects in a queue are
>> either immutable or the queue holds their only reference. The trouble
>> is that it is so hard to work out what immutable really means.
>> Non-optional annotations would be not very pythonian.
>>
>> --
>> William Leslie
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Honestly, that sounds really difficult, out and out removing the GIL
would probably be easier.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me



More information about the Pypy-dev mailing list