On Wed, Jul 28, 2010 at 8:33 PM, Kevin Ar18 <kevinar18@hotmail.com> wrote:
As a followup to my earlier post: "pre-emptive micro-threads utilizing shared memory message passing?"
I am actually finding that the biggest hurdle to accomplishing what I want is the lack of ANY type of shared memory -- even if it is limited. I wonder if I might ask a question:
Would the following be a possible way to offer a limited type of shared memory:
Summary: create a system very, very similar to POSH, but with differences:
In detail, here's what I mean: * unlike POSH, utilize OS threads and shared memory (not processes) * Create a special shared memory location where you can place Python objects * Each Python object you place into this location can only be accessed (modified) by 1 thread. * You must manually assign ownership of an object to a particular thread. * The thread that "owns" the object is the only one that can modify it. * You can transfer ownership to another thread (but, as always only the owner can modify it).
* There is no GIL when a thread interacts with these special objects. You can have true thread parallelism if your code uses a lot of these special objects. * The GIL remains in place for all other data access. * If your code has a mixture of access to the special objects and regular data, then once you hit a point where a thread starts to interact with data not in the special storage, then that thread must follow GIL rules.
Granted, there might be some difficulty with the GIL part... but I thought I might ask anyways. :)
Date: Wed, 28 Jul 2010 22:54:39 +1000 Subject: Re: [pypy-dev] pre-emptive micro-threads utilizing shared memory message passing? From: william.leslie.ttg@gmail.com To: kevinar18@hotmail.com CC: pypy-dev@codespeak.net
On 28 July 2010 04:20, Kevin Ar18 <kevinar18@hotmail.com> wrote:
I am attempting to experiment with FBP - Flow Based Programming (http://www.jpaulmorrison.com/fbp/ and book: http://www.jpaulmorrison.com/fbp/book.pdf) There is something very similar in Python: http://www.kamaelia.org/MiniAxon.html Also, there are some similarities to Erlang - the share nothing memory model... and on some very broad levels, there are similarities that can be found in functional languages.
Does anyone know if there is a central resource for incompatible python memory model proposals? I know of Jython, Python-Safethread, and Mont-E.
I do like the idea of MiniAxon, but let me mention a topic that has slowly been bubbling to the front of my mind for the last few months.
Concurrency in the face of shared mutable state is hard. It makes it trivial to introduce bugs all over the place. Nondeterminacy related bugs are far harder to test, diagnose, and fix than anything else that I would almost mandate static verification (via optional typing, probably) of task noninterference if I was moving to a concurrent environment with shared mutable state. There might be a reasonable middle ground where, if a task attempts to violate the required static semantics, it fails dynamically. At least then, latent bugs make plenty of noise. An example for MiniAxon (as I understand it, which is not very well) would be verification that a "task" (including functions that the task calls) never closes over and yields the same mutable objects, and never mutates globally reachable objects.
I wonder if you could close such tasks off with a clever subclass of the proxy object space that detects and rejects such memory model violations? With only semantics that make the program deterministic?
The moral equivalent would be cooperating processes with a large global (or not) shared memory store for immutable objects, queues for communication, and the additional semantic that objects in a queue are either immutable or the queue holds their only reference. The trouble is that it is so hard to work out what immutable really means. Non-optional annotations would be not very pythonian.
-- William Leslie
_______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Honestly, that sounds really difficult, out and out removing the GIL would probably be easier. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me