python concurrency proposal
Michael
ms at cerenity.org
Tue Jan 3 18:05:47 EST 2006
corey.coughlin at comcast.net wrote:
> Alright, so I've been following some of the arguments about enhancing
> parallelism in python, and I've kind of been struck by how hard things
> still are. It seems like what we really need is a more pythonic
> approach.
[... major snippage ...]
> OK? So what do you all think?
On the surface of it, what you've described resembles Kamaelia[1] -
specifically in the way used in the Axon Shell [2]. In other ways it
differs wildy. I think it's interesting because this (or some of this)
could be used as syntactic sugar for Kamaelia.
[1] http://kamaelia.sourceforge.net/Home
[2] http://kamaelia.sourceforge.net/AxonShell.html
The similarity is this: a pardef appears to be a function that can run,
and be made to run in parallel with other functions. In Kamaelia we use
generators to achieve precisely this.
However, given pythonic is a loaded term (beauty is in the eye of the
beholder), I'd personally say that there's some elements of your syntax
that suprise me, especially given the (IMO valid) approach of not being
able to access inside the pardef.
Since there are some strong similarities between what you've written and
Kamaelia it seems sensible to discuss them.
If you have your pardef:
pardef <Name>(self, <par type>, arguments...):
self.send(<dest pardef>, <tag>, arguments)
self.receive(<tag>, arguments)
return arguments
yield arguments
This maps to this: (in your terms :) )
class <Name>(<par type>):
Inboxes = [ <tag>, <tag>, <tag> ] # Explicit is better than implicit
Outboxes = [ <tag>, <tag>, <tag> ] # There are defaults available...
def __init__(self, arguments ...):
// standard initialisation //
super(<Name>, self).__init__()
def main(self):
// local
// Initialisation
while 1:
do stuff
yield //some value// (relinquish control to scheduler)
return
// We don't have the concept of a result, though this
would be useful, though we do have something
similar so this would be doable //
Inboxes and Outboxes are used as declarations to allow the baseclass
(component) to initialise some datastructures for communications. These
declarations are actually any iterable, and these days we tend to use
dictionaries because it simplifies documentation.
An example here might control the behaviour of a sprite onscreen. So,
suppose we have a sprite:
(This is a simplified example of an existing component)
class SimpleSprite(component, pygame.sprite.Sprite):
Inboxes = { "position" : "Expect to receive (x:int,y:int) values",
"rotation" : "Expect to an int in range 0..359",
}
def __init__(self, image, pos):
pygame.sprite.Sprite.__init__(self)
component.__init__(self) # Can't use super here because we inherit
# from pygame.sprite.Sprite as well
self.original = image
self.image = image # self.image is displayed by pygame
self.pos = pos
def main(self):
pos = self.pos
rotation = 0
image = self.image
while 1:
self.image = self.original
if not self.anyReady():
self.pause() # Ask scheduler not to run us until we
# receive data on an inbox
yield 1
if self.dataReady("position"):
pos = self.recv("position")
if self.dataReady("rotation"):
angle = self.recv("rotation")
self.image = pygame.transform.rotate(self.image, angle)
self.rect.center = pos
yield 1
We could then have some game logic that sends out information that controls
this sprite over time:
class WalkInASquare(component):
Outboxes = {
"position" : "Sends out an (x:int, y:int) pair",
"orientation" : "Sends out an angular orientation",
}
def __init__(self, left, top, right, bottom, steps):
# We assume here that left < right and top < bottom
# In something real you'd want to check or enforce this
# (eg asserts or swapping)
self.left = left
self.top = top
self.right = right
self.bottom = bottom
def main(self):
# You'd ideally want to check for shutdown messages as well
x = self.left
y = self.top
while 1: # do this forever, normally we'd shutdown
# Walk right
self.send(90, "orientation")
for i in xrange(self.left, self.right, steps):
self.send((i,y), "position")
yield 1
x = right
# Walk down
self.send(180, "orientation")
for i in xrange(self.top, self.bottom, steps):
self.send((x,i), "position")
yield 1
y = self.bottom
# Walk left
self.send(270, "orientation")
for i in xrange(self.right, self.left, -steps):
self.send((i,y), "position")
yield 1
x = self.left
# Walk up
self.send(0, "orientation")
for i in xrange(self.bottom, self.top, -steps):
self.send((x,i), "position")
yield 1
y = self.top
The logic of walking around should be clear:
* We take a step, and send out our position
* If we reach the end, we turn to face the new direction and
send that value out.
Clearly for this to be interesting this can be joined with the sprite.
Unlike your system, our components (your pardefs), don't know
about each other and only talk to inboxes AND outboxes - these
are connected at a higher level by something enclosing them.
Graphline(
Walker = SimpleSprite(walkerImage, 10,10),
WalkerLogic = WalkInASquare(10,10,400,400,50),
linkages = {
("WalkerLogic", "position") : ("Walker", "position"),
("WalkerLogic", "orientation") : ("Walker", "rotation"),
}
)
This graphline can then be activated or run like any other component.
[[ For more complete examples, grab Kamaelia from the website,
download and run :) ]]
You'll note from the above that clearly there are aspects to it where
your ideas could be used as syntactic sugar. Some of the benefits (like
collapsing __init__ and main together) can be gained by using decorators
in python 2.4.
You'll also note there are some areas where your ideas increase parallel
safety (such as when sending mutable data structures), at the expense on
increasing the cost of communication.
I'm not saying the above is the paragon of virtue (it isn't) - we're looking
to see what's possible - but not rushing into creating syntax until we have
an idea that the approaches work. That said, we are interested in ideas for
syntaxes. At the moment I'd prefer these to be based on API enhancements
rather than changes to python since I personally think that's an extreme
approach until you know the change to the language will actually be
beneficial.
((In some respects we're held back from some improvements because we want
to stay supporting python on series 60 (which limits us to 2.2 late
features rather than things like decorators and PEP 342 enhancements).
We may drop that if necessary later or simply have a specialised version
there. This might explain the emphasis on ))
Regarding other points, of your suggestions though...
You allow the following to be equivalent initialisations:
h1 = vecadd(a[:500], b[:500], 'd')
# vs
h1 = vecadd()
h1.veca = a[:500]
h1.vecb = b[:500]
h1.arrtype = 'd'
To me this looks dangerous. (as in likely to confuse and cause bugs)
What if vecadd is implemented as a thread, and can run concurrently with
the main piece of code? (especially since you suggest reusing the same
active object) This allows the possibility:
h1 = vecadd()
h1.run() # does this background? Do we block? (appears to do the former)
h1.veca = a[:500] # What happens, is h1 blocked until we get here?
h1.vecb = b[:500]
h1.arrtype = 'd'
h1.veca = b[:500] # Uh oh, what's happening here?
c = h1.result + h2.result # Um, what does this now mean ?
were the values to veca queued? Were they overwritten? Would this be valid?
To me it's completely non-obvious what this code means on reading it. It
also *looks* like you're updating internal state of the object rather than
an external interface. (It also has the implicit suggestion that you can
read these values as well, which may or may not be valid)
I think the core to of your idea is that your new suite really introduces
a scheduler for generators and a means for communicating (you call them
tags). What you've done here is also make the arguments for creation mutable
which becomes confusing.
I'm not trying to discourage you, I like the ideas, and would like to see
you expand them more since they interest me, but you have some other
inconsistencies. For example, you say here:
Remember your statement that pardefs don't inherit any data from the global
scope. Data is only passed in through receive statements and the arguments.
In practice however, you also have in your examples this:
updef = not (isinstance(up, int) or isintance(up, float))
downdef = not (isinstance(down, int) or isintance(down, float))
rightdef = not (isinstance(right, int) or isintance(right, float))
leftdef = not (isinstance(left, int) or isintance(left, float))
In these lines you have passed over global scope data - specifically
int and float.
Unlike your approach (syntax first), we've created a large (growing) number
of components (like your pardefs) which have been put together for various
systems which have varying levels of concurrency which are probably a useful
testbed for testing your ideas:
* http://kamaelia.sourceforge.net/Components.html
Example systems created using this vary from a "everything broadcast" PVR
for radio [*], a networked audio mixer matrix, simple games, a presentation
tool (probably including soon a live video mixer to go with the video
playback), topology viewing tools, a simple "paint" program, etc)
* Actually a box for creating a record of transmission, which
is slightly different, but the former description is more
accessible if slightly inaccurate.)
If you're interested in prototyping your ideas in python, you can simulate
some of your ideas using decorators. Something that might help you with
prototyping your ideas is our tutorial for Kamaelia, which is a "build your
own core" system type tutorial. It might also help show that your pardefs
are very similar to python's generators. It can be found here:
* http://kamaelia.sourceforge.net/MiniAxon/
In many respects, I feel that the API we have still isn't "100% pythonic" as
many would call it, but it does try to be unsurprising and consistent. You
could say we're aiming for pythonic - though personally I favour easy and
unsurprising as more concrete, less abstract goals - even if it's not there
yet. (portability of ideas to other languages is important to me, which
again is another reason for an API based view rather than syntax).
If you're willing to take a look at it, and make suggestions as to how your
ideas might fit, (or what you think is dumb :-) I'd welcome it.
*Especially* if it simplifies the system (*or* the way it's used).
Finally though, as I say above,I'm not trying to discourage you, I like
the ideas, and would like to see you expand them more since they interest
me, and like you I think this is an area that needs work. I would suggest
playing with the ideas though and testing them against systems before
writing a PEP though. (real/useful systems rather than contrived ones! -)
Best Regards & Happy New Year,
Michael.
--
Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP
This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.
More information about the Python-list
mailing list