[pypy-dev] pre-emptive micro-threads utilizing shared memory message passing?

Paolo Giarrusso p.giarrusso at gmail.com
Wed Jul 28 15:37:07 CEST 2010


On Wed, Jul 28, 2010 at 14:54, William Leslie
<william.leslie.ttg at gmail.com> wrote:
> On 28 July 2010 04:20, Kevin Ar18 <kevinar18 at hotmail.com> wrote:
>> I am attempting to experiment with FBP - Flow Based Programming (http://www.jpaulmorrison.com/fbp/ and book: http://www.jpaulmorrison.com/fbp/book.pdf)  There is something very similar in Python: http://www.kamaelia.org/MiniAxon.html  Also, there are some similarities to Erlang - the share nothing memory model... and on some very broad levels, there are similarities that can be found in functional languages.

> Does anyone know if there is a central resource for incompatible
> python memory model proposals? I know of Jython, Python-Safethread,
> and Mont-E.

Add Unladen Swallow to your list - the "Jython memory model" is undocumented.
I don't know of Mont-E, can't find its website through Google (!), and
there seems to be no such central resource.

> I do like the idea of MiniAxon, but let me mention a topic that has
> slowly been bubbling to the front of my mind for the last few months.

> Concurrency in the face of shared mutable state is hard. It makes it
> trivial to introduce bugs all over the place. Nondeterminacy related
> bugs are far harder to test, diagnose, and fix than anything else that
> I would almost mandate static verification (via optional typing,
> probably) of task noninterference if I was moving to a concurrent
> environment with shared mutable state.

This is a general issue with concurrency, and usually I try to solve
this using more pencil-and-paper design than usual.

> There might be a reasonable
> middle ground where, if a task attempts to violate the required static
> semantics, it fails dynamically. At least then, latent bugs make
> plenty of noise.

In general, I've seen lots of research on this, and something
implemented in Valgrind - see here for links:
http://blaisorbladeprog.blogspot.com/2010/07/automatic-race-detection.html.
Given the interest on this, the lack of complete tools might mean that
it is just too hard currently.

> An example for MiniAxon (as I understand it, which is
> not very well) would be verification that a "task" (including
> functions that the task calls) never closes over and yields the same
> mutable objects, and never mutates globally reachable objects.

I guess that 'close over' here means 'getting as input'.

> I wonder if you could close such tasks off with a clever subclass of
> the proxy object space that detects and rejects such memory model
> violations? With only semantics that make the program deterministic?

> The moral equivalent would be cooperating processes with a large
> global (or not) shared memory store for immutable objects, queues for
> communication, and the additional semantic that objects in a queue are
> either immutable or the queue holds their only reference.

In C++ auto_ptr do it, but that's hard in Python.

> The trouble
> is that it is so hard to work out what immutable really means.
> Non-optional annotations would be not very pythonian.

If you want static guarantees, you need a statically typed language.
The usual argument for dynamic languages is that instead of static
typing, you need to write unit tests, and since you must do that
anyway, dynamic languages are a win. We have two incomplete attempts
to make programs correct:
- Types give strong guarantees against a subclass of errors (you
_never_ get certain errors from a program which compiles)
- Testing gives weak guarantees (which go just as far as you test),
but covers all classes of errors
- The middle ground would be to require annotations to prove
properties. One would need (once and for all) to annotate even strings
as immutable!

Cheers,
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/



More information about the Pypy-dev mailing list