According to this list's welcome message, I should introduce myself. I'm Bruce Eckel, I write, consult and give seminars about computer programming, I use Python whenever I can get away with it, and I've spoken at Pycon a number of times. There are further URLs in my signature at the bottom. I'm joining this list because I was having a conversation with Guido about concurrency support in Python (in particular, using an actor/active-object approach) and he suggested I bring it over here. Here are the highlights of my most recent reply to him (Guido's remarks are marked by the '>' signs):
Agreed. IMO the pthreads-style solution doesn't work well no matter *what* the programming model.
Exactly. I think the problem may be that low-level threads people are one step behind (much like embedded systems programmers, which is where I began). They may be just now catching up to objects, but as far as concurrency goes their brains still think in terms of threads, so it seems natural to apply thread concepts to objects. But from an OO standpoint, pthread-style thinking is two steps backwards. You effectively throw open the innards of the object that you just spent time decoupling from the rest of your system, and the coupling is now unpredictable. It's the worst of all possible worlds.
I worked on Mobile Agents at CNRI from 1995 till 2000, but they were very heavy-weight. Yet, I think that any solution that moves objects between CPUs on a regular basis is going to suffer the same problem that process switching in general gets you -- the move is so expensive that it's hard to decide when it's justified.
The examples I've seen haven't relied on mobility, so that's a question: how important is mobility really to an agent system? And could it be done in a trivial fashion in Python, by sending source code. And if it was really necessary sometimes, could it be something where the cost of mobility only occurred when moving an object? It's possible that either an argument can be made against mobility and/or some kind of trivial mobility system could be developed that would work when necessary. Possibly a Linda-like put-take system with pickles (off the top of my head). I don't know that much about mobility, but it does seem like mobile agents could be a powerful solution to the problem solved by enterprise messaging systems.
I believe you pointed me to Active Objects before, right? (Years ago, maybe when you visited us at Zope.)
I may have mentioned them in the past.
If multiple active objects can co-exist in the same *process*, but nevertheless the language implementation prevents them from sharing data except via channels, and in addition allows dynamic reallocation of active objects across multiple CPUs, they just might be the ticket. But it would require a complete implementation to prove it.
Yes, defining an class as "active" would: 1) Install a worker thread and concurrent queue in each object of that class. 2) Automatically turn method calls into tasks and enqueue them 3) Prevent any other interaction other than enqueued messages At that point, the only time the GIL might be needed is to lock the queue at the point of putting ant taking objects. But there has been work done on lock-free data structures, which has been incorporated into Java 5 libraries, so it might even be possible to use a lock-free queue to do this and thus eliminate the need for the GIL at all. Since the enqueuing process serializes all requests to active objects, and since each message-task runs to completion before the next one starts, the problem of threads interfering with each other is eliminated. Also, the enqueuing process will always happen, so even if the queue blocks it will be for a very short, deterministic time. So the theory is that Active Object systems cannot deadlock (although I believe they can livelock). So yes, the best way for this to work might be some kind of enforced implementation -- but forcing you to do something is not particularly Pythonic, so I would think that it would be possible to build an active object framework along with some checking tools to warn you if you are breaking the rules. But what's not clear is whether this would require knowledge of the innards of the Python interpreter (which I don't have) or if it could be built using libraries. BTW: I think that Hoare's Communicating Sequential Processes (CSP) basically describes the idea of active objects but without objects. That is, I think active objects is CSP applied to OO. Bruce Eckel http://www.BruceEckel.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
At 11:18 AM 9/27/2005 -0600, Bruce Eckel wrote:
Yes, defining an class as "active" would: 1) Install a worker thread and concurrent queue in each object of that class. 2) Automatically turn method calls into tasks and enqueue them 3) Prevent any other interaction other than enqueued messages
#3 is the sticky bit. Enforcing such restrictions in Python is *hard*. Each active object would have to have its own sys.modules, for example, because modules and classes in Python are mutable and use dictionaries, and modifying them from two thread simultaneously would be fatal. For built-in types it's not so bad, so they could be shared... except for the fact that they have reference counts, which need lock protection as well to avoid fouling up GC. So really, this idea doesn't really help with GIL removal at all. What it *does* help with, is effective use of multiprocessor machines on platforms where fork() is available, if the API works across processes as well as threads.
So yes, the best way for this to work might be some kind of enforced implementation -- but forcing you to do something is not particularly Pythonic,
But keeping the interpreter from dumping core due to program bugs *is* Pythonic, which is why the GIL would need to stay. :)
so I would think that it would be possible to build an active object framework along with some checking tools to warn you if you are breaking the rules.
Well, you could pickle and unpickle the objects you send from one function to another, and for cross-process communication, you'll need to do something like that anyway, or else use that shared-memory objects thing. PySHM? I don't remember its name, but it's an extension that lets you store Python objects in shared memory and use them from multiple processes, modulo certain strict limitations.
But what's not clear is whether this would require knowledge of the innards of the Python interpreter (which I don't have) or if it could be built using libraries.
If you're not trying to get rid of the GIL or truly "enforce" things, you can do everything you need for CSP in plain old Python. peak.events has a generator-based microthread facility that's CSP-ish, for example, although it doesn't have decorators for async methods. They're not hard to write, though; Chandler has some for methods that get called across thread barriers now. That is, methods that get called from Chandler's UI thread but are run in the Twisted reactor thread. I've occasionally also thought about implementing async C#'s "chord" features in Python for peak.events, but I haven't actually had a use case that needed that much generality yet.
Phillip J. Eby wrote:
Well, you could pickle and unpickle the objects you send from one function to another, and for cross-process communication, you'll need to do something like that anyway, or else use that shared-memory objects thing. PySHM? I don't remember its name, but it's an extension that lets you store Python objects in shared memory and use them from multiple processes, modulo certain strict limitations.
"POSH": http://poshmodule.sourceforge.net/posh/html/ -- Benji York
Bruce Eckel wrote:
Since the enqueuing process serializes all requests to active objects, and since each message-task runs to completion before the next one starts, the problem of threads interfering with each other is eliminated. Also, the enqueuing process will always happen, so even if the queue blocks it will be for a very short, deterministic time. So the theory is that Active Object systems cannot deadlock (although I believe they can livelock).
I've done this at work (in C++, not Python), and it works very well. I had my doubts when my boss first put the architecture for it in place, but a couple of years of working with this architecture (and that boss!) persuaded me otherwise. Livelock is a definite issue, particular if you design the individual active objects as finite state machines - it is fairly easy to get into a situation where Object A is waiting for a particular message from Object B, while Object B is waiting for a particular message from Object A. This chain is generally indirect, which makes it hard to spot. It is, however, still easier to figure this out than it is to figure out normal threading bugs, because you can find it just by analysing a sequence diagram showing message exchange and state transitions, rather than caring about the actual underlying code. Threading bugs, on the other hand, can turn up any time you access a shared data structure.
BTW: I think that Hoare's Communicating Sequential Processes (CSP) basically describes the idea of active objects but without objects. That is, I think active objects is CSP applied to OO.
Or to put it another way, applying OO to CSP is a matter of hiding the addressing and sending of messages behind method calls. It becomes almost like doing in-process remote procedure calls. I think there's a definite synergy with PEP 342 here as well - one of the problems with handling Active Objects is how to allow other messages to be processed while making a blocking call, without losing your current state. One way is to hold any state that persists between messages in the object itself, but that's a seriously unnatural style of programming (it can be done, it's just a pain). PEP 342's yield expressions can probably be used to help address that problem, though: class SomeAO(ActiveObject): def processSomeMessage(self): msg = yield # Do something with the message next_msg = yield makeSomeBlockingCall(self) # Do something with the next message Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Nick Coghlan wrote:
PEP 342's yield expressions can probably be used to help address that problem, though:
class SomeAO(ActiveObject): def processSomeMessage(self): msg = yield # Do something with the message next_msg = yield makeSomeBlockingCall(self) # Do something with the next message
I don't see how that helps, since makeSomeBlockingCall() is evaluated (and therefore blocks) *before* the yield happens. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
On 9/28/05, Greg Ewing
Nick Coghlan wrote:
PEP 342's yield expressions can probably be used to help address that problem, though:
class SomeAO(ActiveObject): def processSomeMessage(self): msg = yield # Do something with the message next_msg = yield makeSomeBlockingCall(self) # Do something with the next message
I don't see how that helps, since makeSomeBlockingCall() is evaluated (and therefore blocks) *before* the yield happens.
Sounds like makeSomeBlockingCall is just misnamed (probably depending who you ask). I wrote a small library recently that wraps Twisted's Deferreds and asynchronous Failure objects such that you can do stuff like try: x = yield remoteObject.getSomething() except Foo: print "Oh no!" This is just a 2.5-ification of defgen, which is at twisted.internet.defer.{deferredGenerator,waitForDeferred}. So anyway, if your actor messages always return Deferreds, then this works quite nicely. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
Oops. I forgot to add that to the list. Yes, in the working example of Active Objects that I've written in Java J2SE5, when you send a message to an active object, you get back a Future<ReturnType>, which I suspect would be the same as your Deferred. Tuesday, September 27, 2005, 7:41:27 PM, Christopher Armstrong wrote:
On 9/28/05, Greg Ewing
wrote: Nick Coghlan wrote:
PEP 342's yield expressions can probably be used to help address that problem, though:
class SomeAO(ActiveObject): def processSomeMessage(self): msg = yield # Do something with the message next_msg = yield makeSomeBlockingCall(self) # Do something with the next message
I don't see how that helps, since makeSomeBlockingCall() is evaluated (and therefore blocks) *before* the yield happens.
Sounds like makeSomeBlockingCall is just misnamed (probably depending who you ask).
I wrote a small library recently that wraps Twisted's Deferreds and asynchronous Failure objects such that you can do stuff like
try: x = yield remoteObject.getSomething() except Foo: print "Oh no!"
This is just a 2.5-ification of defgen, which is at twisted.internet.defer.{deferredGenerator,waitForDeferred}. So anyway, if your actor messages always return Deferreds, then this works quite nicely.
-- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/bruceeckel-python3234%40ma...
Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234@mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar
Greg Ewing wrote:
Nick Coghlan wrote:
PEP 342's yield expressions can probably be used to help address that problem, though:
class SomeAO(ActiveObject): def processSomeMessage(self): msg = yield # Do something with the message next_msg = yield makeSomeBlockingCall(self) # Do something with the next message
I don't see how that helps, since makeSomeBlockingCall() is evaluated (and therefore blocks) *before* the yield happens.
Chris got it right - I just named the function badly. The idea is that there would be an interaction with the Active Object's event loop to get the actual result back once it was available from the thread that actually made the blocking call. In the meantime, *this* object could continue to handle other messages. What the approach allows is to have a single physical thread with a number of coroutiney-type event handlers running inside it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
Bruce Eckel wrote:
Since the enqueuing process serializes all requests to active objects, and since each message-task runs to completion before the next one starts, the problem of threads interfering with each other is eliminated. Also, the enqueuing process will always happen, so even if the queue blocks it will be for a very short, deterministic time. So the theory is that Active Object systems cannot deadlock (although I believe they can livelock).
My experience with active objects is with SDL (the Specification and Description Language). There, the primitive communication between objects is through one-way messages (called signals). Such a system can deadlock if an object doesn't produce "enough" new outgoing messages in response to incoming messages. If all communication is through RPCs, i.e. you owe your caller a response, then the only way to deadlock such a system is that all active objects terminate. The interesting question then is whether an object can terminate while processing a message, and whether this will cause a response to be sent back.
But what's not clear is whether this would require knowledge of the innards of the Python interpreter (which I don't have) or if it could be built using libraries.
As others have pointed out, you have to find a way to deal with the standard library. In principle, each active object should have its own copy of the standard library. For some modules/objects, this is not reasonable (e.g. stdout). In OCCAM, host IO itself is (conceptually) an occam process, so sys.stdout would have to become an active object. In that approach, you would essentially have to rewrite parts of the standard library, wrapping the underlying modules appropriately.
BTW: I think that Hoare's Communicating Sequential Processes (CSP) basically describes the idea of active objects but without objects. That is, I think active objects is CSP applied to OO.
I don't believe this is true. In CSP, communication between processes is through rendezvous, which is quite different from active objects, as there is no queue for incoming calls - instead, the receiving process must expect a specific communication partner just like the sending process; there is also no inherent request/response mechanism in CSP. In CSP, there is plenty of chance for deadlocks. If a process becomes STOP (i.e. the process that will never again interact), then any communication attempt with that process also cause the partner to become STOP. Regards, Martin
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far. What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads. Here are my own criteria for what such a solution would look like: 1) It works by default, so that novices can use it without falling into the deep well of threading. That is, a program that you write using threading is broken by default, and the tool you have to fix it is "inspection." I want something that allows me to say "this is a task. Go." and have it work without the python programmer having to study and understand several tomes on the subject. 2) Tasks can be automatically distributed among processors, so it solves the problems of (a) making python run faster (b) how to utilize multiprocessor systems. 3) Tasks are cheap enough that I can make thousands of them, to solve modeling problems (in which I also lump games). This is really a solution to a cerain type of program complexity -- if I can just assign a task to each logical modeling unit, it makes such a system much easier to program. 4) Tasks are "self-guarding," so they prevent other tasks from interfering with them. The only way tasks can communicate with each other is through some kind of formal mechanism (something queue-ish, I'd imagine). 5) Deadlock is prevented by default. I suspect livelock could still happen; I don't know if it's possible to eliminate that. 6) It's natural to make an object that is actor-ish. That is, this concurrency approach works intuitively with objects. 7) Complexity should be eliminated as much as possible. If it requires greater limitations on what you can do in exchange for a clear, simple, and safe programming model, that sounds pythonic to me. The way I see it, if we can't easily use tasks without getting into trouble, people won't use them. But if we have a model that allows people to (for example) make easy use of multiple processors, they will use that approach and the (possible) extra overhead that you pay for the simplicity will be absorbed by the extra CPUs. 8) It should not exclude the possibility of mobile tasks/active objects, ideally with something relatively straightforward such as Linda-style tuple spaces. One thing that occurs to me is that a number of items on this wish list may conflict with each other, which may require a different way of thinking about the problem. For example, it may require two approaches: for "ordinary" non-OO tasks, a functional programming approach ala Erlang, in combination with an actor approach for objects. Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234@mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
Bruce Eckel
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
Here are my own criteria for what such a solution would look like:
Just because I've been mentioning it everywhere else since I read it, have you seen this paper: http://research.microsoft.com/Users/simonpj/papers/stm/ ? I don't know how applicable it would be to Python but it's well worth the time it takes to read. Cheers, mwh -- This makes it possible to pass complex object hierarchies to a C coder who thinks computer science has made no worthwhile advancements since the invention of the pointer. -- Gordon McMillan, 30 Jul 1998
This paper looks very interesting and promises some good ideas. It also looks like it will require time and effort to digest. I've only read the first few pages, but one thing that does leap out is at the beginning of section 3, they say: "... a purely-declarative language is a perfect setting for transactional memory." What's not clear to me from this is whether STM will work in a non-declarative language like Python. Thursday, September 29, 2005, 8:12:23 AM, Michael Hudson wrote:
Bruce Eckel
writes:
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
Here are my own criteria for what such a solution would look like:
Just because I've been mentioning it everywhere else since I read it, have you seen this paper:
? I don't know how applicable it would be to Python but it's well worth the time it takes to read.
Cheers, mwh
Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234@mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar
FWIW, the Perl 6 community is also investigating STM, so it appears to
be a worthwhile idea for an impure, multi-paradigm language as well.
Regards,
Michael
On 9/29/05, Bruce Eckel
This paper looks very interesting and promises some good ideas. It also looks like it will require time and effort to digest.
I've only read the first few pages, but one thing that does leap out is at the beginning of section 3, they say:
"... a purely-declarative language is a perfect setting for transactional memory."
What's not clear to me from this is whether STM will work in a non-declarative language like Python.
Thursday, September 29, 2005, 8:12:23 AM, Michael Hudson wrote:
Bruce Eckel
writes: I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
Here are my own criteria for what such a solution would look like:
Just because I've been mentioning it everywhere else since I read it, have you seen this paper:
? I don't know how applicable it would be to Python but it's well worth the time it takes to read.
Cheers, mwh
Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234@mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/michael.walter%40gmail.com
At 09:30 AM 9/29/2005 -0600, Bruce Eckel wrote:
This paper looks very interesting and promises some good ideas. It also looks like it will require time and effort to digest.
I've only read the first few pages, but one thing that does leap out is at the beginning of section 3, they say:
"... a purely-declarative language is a perfect setting for transactional memory."
What's not clear to me from this is whether STM will work in a non-declarative language like Python.
I spent a few weekends studying that paper earlier this year in order to see if anything could be stolen for Python; my general impression was "not easily" at the least. One notable feature of the presented concept was that when code would otherwise block, they *rolled it back* to the last nonblocking execution point. In a sense, they ran the code backwards, albeit by undoing its effects. They then suspend execution until there's a change to at least one of the variables read during the forward execution, to avoid repeated retries. It was a really fascinating idea, because it was basically a way to write nonblocking code without explicit yielding constructs. But, it depends utterly on having all changes to data being transactional, and on being able to guarantee it in order to prevent bugs that would be just as bad as the ones you get from threading. Oddly enough, this paper actually demonstrates a situation where having static type checking is in fact a solution to a non-trivial problem! It uses static type checking of monads to ensure that you can't touch untransacted things inside a transaction.
I spent a few weekends studying that paper earlier this year in order to see if anything could be stolen for Python; my general impression was "not easily" at the least. One notable feature of the presented concept was that when code would otherwise block, they *rolled it back* to the last nonblocking execution point. In a sense, they ran the code backwards, albeit by undoing its effects. They then suspend execution until there's a change to at least one of the variables read during the forward execution, to avoid repeated retries.
I haven't spent the weekends on the paper yet (but it looks like that is what it would take), but I had the impression that they were talking about the lock-free techniques such as the ones used in Java 5. Basically, you start a write operation "in the background" without locking the data structure, so reads can continue while the calculation is taking place but before the result is committed. When the result is ready, an atomic "test and write" operation is used to determine whether any other task has modified the value in the meantime, and if not to commit the new value. If another task did modify the value, then the calculation begins anew. That was my take, but I haven't studied everything about STM yet, so I'm probably missing something. The one thing about this paper is that it seems to be an orthogonal perspective to anything about concurrency that *I* have seen before.
Oddly enough, this paper actually demonstrates a situation where having static type checking is in fact a solution to a non-trivial problem! It uses static type checking of monads to ensure that you can't touch untransacted things inside a transaction.
Yes, because of some of my diatribes against static checking people get the impression that I think it's just a bad thing. However, I'm really trying to get across the idea that "static type checking as the solution to all problems is a bad idea," and that the cost is often much greater than the benefit. But if there really is a clear payoff then I'm certainly not averse to it. In general, I really *do* like to be told when something has gone wrong -- I think there's a huge benefit in that. But if I can learn about it at runtime rather than compile time, then that is often a reasonable solution. So with concurrency, I would like to know when I do something wrong, but if I am told at runtime that's OK with me as long as I'm told. Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234@mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar
1) It works by default, so that novices can use it without falling into the deep well of threading. That is, a program that you write using threading is broken by default, and the tool you have to fix it is "inspection." I want something that allows me to say "this is a task. Go." and have it work without the python programmer having to study and understand several tomes on the subject.
Bruce, this seems to me like wishful thinking. Either the separate threads don't interact, in which case you are running separate programs, in which case os.system() already works well enough, or they do, in which case you have the various deadlock and livelock problems of threading. And this nonsense about threaded programs being "broken by default" -- might as well say the same about programs that use variables. If you don't know how to use threads, great, just don't use them. Spend the time reading Fred Brooks' NO SILVER BULLET, instead. It's available in essay form at http://www-inst.eecs.berkeley.edu/~maratb/readings/NoSilverBullet.html.
One thing that occurs to me is that a number of items on this wish list may conflict with each other, which may require a different way of thinking about the problem.
That seems correct to me. Bill
At 10:48 AM 9/29/2005 -0600, Bruce Eckel wrote:
I haven't spent the weekends on the paper yet (but it looks like that is what it would take), but I had the impression that they were talking about the lock-free techniques such as the ones used in Java 5. Basically, you start a write operation "in the background" without locking the data structure, so reads can continue while the calculation is taking place but before the result is committed. When the result is ready, an atomic "test and write" operation is used to determine whether any other task has modified the value in the meantime, and if not to commit the new value. If another task did modify the value, then the calculation begins anew.
That was my take, but I haven't studied everything about STM yet, so I'm probably missing something.
No, that's certainly the general idea. The issue for an imperative language like Python is that side-effects are the norm, rather than an exception. The Haskell implementation of the idea effectively relies on the fact that in Haskell you have to jump through monadic hoops to get side effects, so if you have a low-level transactional memory facility, you can create monadic combinators that restrict those side effects to occuring in the way you want them to. This is so utterly alien to Python's execution model - or indeed most imperative languages' execution models - that I don't see how it can be modelled in Python in a way that retains the significant benefits of the approach. Now, in PyPy, I suppose it might be possible to create a custom object-space that works that way, and I hadn't thought of that before. Maybe you should run that paper past one of the PyPy wizards and see if you can get them interested in the idea, then stand back and see what happens. :)
The one thing about this paper is that it seems to be an orthogonal perspective to anything about concurrency that *I* have seen before.
Yes - it's dramatically different, and seems like the One True Way to do imperative concurrency with side-effects. Certainly transactions work nicely for databases, anyway. :)
So with concurrency, I would like to know when I do something wrong, but if I am told at runtime that's OK with me as long as I'm told.
Yeah, but in current Python implementations - with the possible exception of PyPY - detecting the problem at all is virtually impossible. PyPy is an extensible VM, though, so in principle you could create some kind of transactional object space that could keep track of things and tell you if you mess up.
"Phillip J. Eby"
At 09:30 AM 9/29/2005 -0600, Bruce Eckel wrote:
This paper looks very interesting and promises some good ideas. It also looks like it will require time and effort to digest.
I've only read the first few pages, but one thing that does leap out is at the beginning of section 3, they say:
"... a purely-declarative language is a perfect setting for transactional memory."
What's not clear to me from this is whether STM will work in a non-declarative language like Python.
I spent a few weekends studying that paper earlier this year in order to see if anything could be stolen for Python; my general impression was "not easily" at the least. One notable feature of the presented concept was that when code would otherwise block, they *rolled it back* to the last nonblocking execution point. In a sense, they ran the code backwards, albeit by undoing its effects. They then suspend execution until there's a change to at least one of the variables read during the forward execution, to avoid repeated retries.
It was a really fascinating idea, because it was basically a way to write nonblocking code without explicit yielding constructs. But, it depends utterly on having all changes to data being transactional, and on being able to guarantee it in order to prevent bugs that would be just as bad as the ones you get from threading.
Oh yes, if implemented this really has be baked thoroughly into the implementation. It's not something that can be implemented as a library (as far as I can see, anyway).
Oddly enough, this paper actually demonstrates a situation where having static type checking is in fact a solution to a non-trivial problem! It uses static type checking of monads to ensure that you can't touch untransacted things inside a transaction.
Well, this is an extension of the way Haskell deals with side-effects in general... but yes, it's interesting to be sure. Cheers, mwh -- People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system. -- Linus Torvalds
Michael Hudson wrote:
Bruce Eckel
writes: I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
Here are my own criteria for what such a solution would look like:
Just because I've been mentioning it everywhere else since I read it, have you seen this paper:
http://research.microsoft.com/Users/simonpj/papers/stm/
? I don't know how applicable it would be to Python but it's well worth the time it takes to read.
I haven't read more than the abstract and the responses to the paper here, but I think I have the gist. (I look forward to reading it later.) I'll note that we have experience with something less rigorous than but otherwise similar to this with ZODB. We often have tens of processes working on the same data, synchronizing through a common object store. Each process executes transactions which are written without any threading code at all. Processes synchronize through transaction commits. This model has worked very well for Zope, at least as long as conflicts can be kept to a minimum. :) The ZODB approach is less rigorous as it only works when your processes operate soley on database objects, but it provides a useful example, I think, of applying this model in Python and it probably covers a reasonably large set of interesting applications. There are hundreds (possibly thousands) of programmers who have used this technique in Zope, probably without even realizing that they were doing concurrent programming. J2EE systems use a similar approach. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Bruce Eckel wrote:
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
I've pondered this problem. Python deals programmers a double whammy when it comes to threads: not only is threading unsafe like it is in other languages, but the GIL also prevents you from using multiple processors. Thus there's more pressure to improve concurrency in Python than there is elsewhere. I like to use fork(), but fork has its own set of surprises. In particular, in the programmer's view, forking creates a disassociated copy of every object except files. Also, there's no Pythonic way for the two processes to communicate once the child has started. It's tempting to create a library around fork() that solves the communication problem, but the copied objects are still a major source of bugs. Imagine what would happen if you forked a Zope process with an open ZODB. If both the parent and child change ZODB objects, ZODB is likely to corrupt itself, since the processes share file descriptors. Thus forking can just as dangerous as threading. Therefore, I think a better Python concurrency model would be a lot like the subprocess module, but designed for calling Python code. I can already think of several ways I would use such a module. Something like the following would solve problems I've encountered with threads, forking, and the subprocess module: import pyprocess proc = pyprocess.start('mypackage.mymodule', 'myfunc', arg1, arg2=5) while proc.running(): # do something else res = proc.result() This code doesn't specify whether the subprocess should continue to exist after the function completes (or throws an exception). I can think of two ways to deal with that: 1) Provide two APIs. The first API stops the subprocess upon function completion. The second API allows the parent to call other functions in the subprocess, but never more than one function at a time. 2) Always leave subprocesses running, but use a 'with' statement to guarantee the subprocess will be closed quickly. I prefer this option. I think my suggestion fits most of your objectives.
1) It works by default, so that novices can use it without falling into the deep well of threading. That is, a program that you write using threading is broken by default, and the tool you have to fix it is "inspection." I want something that allows me to say "this is a task. Go." and have it work without the python programmer having to study and understand several tomes on the subject.
Done, IMHO.
2) Tasks can be automatically distributed among processors, so it solves the problems of (a) making python run faster (b) how to utilize multiprocessor systems.
Done. The OS automatically maps subprocesses to other processors.
3) Tasks are cheap enough that I can make thousands of them, to solve modeling problems (in which I also lump games). This is really a solution to a cerain type of program complexity -- if I can just assign a task to each logical modeling unit, it makes such a system much easier to program.
Perhaps the suggested module should have a queue-oriented API. Usage would look like this: import pyprocess queue = pyprocess.ProcessQueue(max_processes=4) task = queue.put('mypackage.mymodule', 'myfunc', arg1, arg2=5) Then, you can create as many tasks as you like; parallelism will be limited to 4 concurrent tasks. A variation of ProcessQueue might manage the concurrency limit automatically.
4) Tasks are "self-guarding," so they prevent other tasks from interfering with them. The only way tasks can communicate with each other is through some kind of formal mechanism (something queue-ish, I'd imagine).
Done. Subprocesses have their own Python namespace. Subprocesses receive messages through function calls and send messages by returning from functions.
5) Deadlock is prevented by default. I suspect livelock could still happen; I don't know if it's possible to eliminate that.
No locking is done at all. (That makes me uneasy, though; have I just moved locking problems to the application developer?)
6) It's natural to make an object that is actor-ish. That is, this concurrency approach works intuitively with objects.
Anything pickleable is legal.
7) Complexity should be eliminated as much as possible. If it requires greater limitations on what you can do in exchange for a clear, simple, and safe programming model, that sounds pythonic to me. The way I see it, if we can't easily use tasks without getting into trouble, people won't use them. But if we have a model that allows people to (for example) make easy use of multiple processors, they will use that approach and the (possible) extra overhead that you pay for the simplicity will be absorbed by the extra CPUs.
I think the solution is very simple.
8) It should not exclude the possibility of mobile tasks/active objects, ideally with something relatively straightforward such as Linda-style tuple spaces.
The proposed module could serve as a guide for a very similar module that sends tasks to other machines. Shane
I'd like to restart this discussion; I didn't mean to put forth active objects as "the" solution, only that it seems to be one of the better, more OO solutions that I've seen so far.
Thanks for doing this. I think this is an issue that is going to be more and more important as Python continues to gain momentum, and I would love to see a PEP come out of this discussion.
What I'd really like to figure out is the "pythonic" solution for concurrency. Guido and I got as far as agreeing that it wasn't threads.
Evaluating the situation is often confusing for Python beginners because: 1. You won't often find threads being the recommended solution for concurrency in Python. There is some disagreement on this point, as the recent thread on the GIL reveals, but I think the point stands. 2. Multi-processing is often brought up as a good alternative to threading. 3. There are a decent number of built-in high level abstractions for threaded programming in Python (Queues, Thread objects, Lock objects, etc.) and plenty of documentation too. These abstractions also make it relatively straightforward to make your code work on multiple platforms. 4. The only support for multi-processing is quite low-level, and can be single platform. Forking isn't really an option on windows, and neither are named pipes. Shared memory? Forget it! Sockets could be used, but thats a bit low-level. Its really a shame. There seems to be some consensus about multi-processing, but not a whole lot of interest in making it easier out of the box. When it comes to multi-processing, batteries really _aren't_ included. Sure, you have lead dioxide and some sulphuric acid, but you have to put them together to make your battery. This isn't the end of the world, but I find it tedious, and I am sure it confuses and frustrates people new to Python.
Here are my own criteria for what such a solution would look like:
1) It works by default, so that novices can use it without falling into the deep well of threading. That is, a program that you write using threading is broken by default, and the tool you have to fix it is "inspection." I want something that allows me to say "this is a task. Go." and have it work without the python programmer having to study and understand several tomes on the subject.
2) Tasks can be automatically distributed among processors, so it solves the problems of (a) making python run faster (b) how to utilize multiprocessor systems.
3) Tasks are cheap enough that I can make thousands of them, to solve modeling problems (in which I also lump games). This is really a solution to a cerain type of program complexity -- if I can just assign a task to each logical modeling unit, it makes such a system much easier to program.
4) Tasks are "self-guarding," so they prevent other tasks from interfering with them. The only way tasks can communicate with each other is through some kind of formal mechanism (something queue-ish, I'd imagine).
5) Deadlock is prevented by default. I suspect livelock could still happen; I don't know if it's possible to eliminate that.
6) It's natural to make an object that is actor-ish. That is, this concurrency approach works intuitively with objects.
7) Complexity should be eliminated as much as possible. If it requires greater limitations on what you can do in exchange for a clear, simple, and safe programming model, that sounds pythonic to me. The way I see it, if we can't easily use tasks without getting into trouble, people won't use them. But if we have a model that allows people to (for example) make easy use of multiple processors, they will use that approach and the (possible) extra overhead that you pay for the simplicity will be absorbed by the extra CPUs.
8) It should not exclude the possibility of mobile tasks/active objects, ideally with something relatively straightforward such as Linda-style tuple spaces.
This all sounds pretty brilliant to me, although even a small subset of what you define above would be totally adequate I think. To me it breaks down simply into: 1. The ability to spawn/create "tasks" (which are really processes) easily, where tasks are isolated from each other. 2. The ability to send a message into a task. A formal queueing mechanism would be nice, but the simple ability to send messages is completely enough to roll your own queueing. Preventing deadlock is hard. Mobile tasks / active objects? Getting way ahead of ourselves, I think. Really, I just want those two things above, and I want it as a standard part of Python.
One thing that occurs to me is that a number of items on this wish list may conflict with each other, which may require a different way of thinking about the problem. For example, it may require two approaches: for "ordinary" non-OO tasks, a functional programming approach ala Erlang, in combination with an actor approach for objects.
I am not sure this is the case. I don't really think of concurrency in terms of objects or good "object-oriented" design all that much. I think of concurrency in terms of processes. Is this bad? I don't think so, really. The OS knows nothing of objects, only of threads and processes (or only of processes). If we had just the two items I mentioned above, it would be enough for people to build their own higher level abstractions on top of the solid built-in support. I haven't really done the research here, and the TSM paper that was sent earlier makes my brain hurt. It seems like its way more than is necessary to solve the problem. If we just accomplished the two things I stated above, I for one would be happy. Am I alone here? -- Jonathan -- http://cleverdevil.org
[ I don't post often, but hopefully the following is of interest in this discussion ] Bruce Eckel wrote:
Yes, defining an class as "active" would: 1) Install a worker thread and concurrent queue in each object of that class. 2) Automatically turn method calls into tasks and enqueue them 3) Prevent any other interaction other than enqueued messages
Here I think the point is an object that has some form of thread with enforced communications preventing (or limiting) shared data. This essentially describes the project we've been working on with Kamaelia. Kamaelia is divided into two halves: * A simple framework (Axon) for building components with the following characteristics: * Have a main() method which is a generator. * Have inbound and outbound named queues implemented using lists. (akin to stdin/out) * Access to an environmental system which is key/value lookup (similar purpose to the unix environment). (This could easily grow into a much more linda like system) * A collection of components which form a library. We also have a threaded component which uses a thread and queues rather than a generator & lists. With regard to the criteria you listed in a later post: 1) Has been tested against (so far) 2 novice programmers, who just picked up the framework and ran with in. (learnt python one week, axon the next, then implemented interesting things rapidly) 2) We don't deal with distribution (yet), but this is on the cards. Probably before EuroOSCON. (I want to have a program with two top level pygame windows) 3) We use python generators and have experimented initially with several tens of thousands of them, which works suprisingly well, even on an old 500Mhz machine. 4) Self guarding is difficult. However generators are pretty opaque, and getting at values inside their stack frame is beyond my python skills. 5) Deadlock can be forced in all scenarios, but you would actually have to work at it. In practice if we yield at all points or use the threaded components for everything then we'd simply livelock. 6) Regarding an actor. Our pygame sprite component actually is very, very actor like. This is to the extent we've considered implementing a script reading and director component for orchestrating them. (Time has prevented that though) 7) Since our target has been novices, this has come by default. 8) We don't do mobility of components/tasks as yet, however a colleague at work interested in mobility & agent software has proposed looking at that using Kamaelia for the coming year. We also ran the framework by people at the WoTUG (*) conference recently, and the general viewpoint came back that it's tractable from a formal analysis and mobility perspective(if we change x,y,z). (*) http://www.wotug.org/ I wrote a white paper based on my Python UK talk, which is here: * http://www.bbc.co.uk/rd/pubs/whp/whp11.shtml (BBC R&D white papers are aimed at imparting info in a such a way that it's useful and readable, unlike some white papers :) So far we have components for building things from network servers through to audio/video decoders and players, and presentation tools. (Using pygame and tkinter) Also we have a graphical editting tool for putting together systems and introspecting systems. An aim in terms of API was to enable novices to implement a microcosm version of the system. Specifically we tested this idea on a pre-university trainee who built a fairly highly parallel system which took video from an mpg file (simulating a PVR), and sent periodic snapshots to a mobile phone based client. He had minimal programming experience at the start, and implemented this over a 3 months period. We've since repeated that experience with a student who has had 2 years at university, but no prior concurrency, python or networks experience. In his 6 weeks with us he was able to do some interesting things. (Such as implement a system for joining multicast islands and sending content over a simple reliable multicast protocol). The tutorial we use to get people up to speed rapidly is here: * http://kamaelia.sourceforge.net/MiniAxon/ However that doesn't really cover the ideas of how to write components, use pipelines, graphlines and services. At the moment we're adding in the concept of a chassis into which you plug in existing components. The latest concept we're bouncing round is this: # (This was fleshed out in respect to a question by a novice on c.l.p - # periodically send the the same message to all connected clients) # start from Kamaelia.Chassis.ConnectedServer import SimpleServer from Kamaelia.Chassis.Splitter import splitter, publish, Subscriber class message_source(component): [ ... snippety ... ] def main(self): last = self.scheduler.time while 1: yield 1 if self.scheduler.time - last > 1: self.send(self.generateMessage, "outbox") last = self.scheduler.time splitter("client_fanout").activate() source = message_source().activate() publish("client_fanout", source) # perhaps something else? def client_protocol(): return Subscriber("client_fanout") SimpleServer(protocol=client_protocol, port=1400).run() # end
From the perspective of using multiple CPUs, currently we're thinking along the lines of this. I'lll use existing examples to make things more concrete.
Currently to build a player to decode and playback dirac [1] encoded video in python, we're doing this: [1] http://dirac.sf.net/ pipeline( ReadFileAdaptor(file, readmode="bitrate", bitrate = 300000*8/5), DiracDecoder(), RateLimit(framerate), VideoOverlay(), ).run() Sometimes, however, it would be nice to take a YUV file, encode, decode and view (since it simplifies the process of seeing changes): pipeline( ReadFileAdaptor(FILENAME, readmode="bitrate", bitrate= 1000000), RawYUVFramer( size=SIZE ), DiracEncoder(preset=DIRACPRESET, encParams=ENCPARAMS ), DiracDecoder(), VideoOverlay() ).run() Clearly on a multiple CPU machine, it would be useful to have the processing cross processes, so we're intending on using a process chassis (a chassis is a thing with holes or can have things attached) to run the components in a different python process. The likely syntax we'll be playing with would be: pipeline( ReadFileAdaptor(FILENAME, readmode="bitrate", bitrate= 1000000), RawYUVFramer( size=SIZE ), SubProcess( DiracEncoder(preset=DIRACPRESET, encParams=ENCPARAMS ), ), SubProcess( DiracDecoder()), VideoOverlay() ).run() This is intended to run over 3 processes. (One parent, two children) For the degenerate case: pipeline( SubProcess( ReadFileAdaptor(FILENAME, readmode="bitrate", bitrate= 1000000)), SubProcess( RawYUVFramer( size=SIZE ),) SubProcess( DiracEncoder(preset=DIRACPRESET, encParams=ENCPARAMS ), ), SubProcess( DiracDecoder()), SubProcess( VideoOverlay()) ).run() We're actively considering the possible sugar: DistributedProcessPipeline( ... original list of components ... ) But we're looking to walk before running there. Much like we implemented the underlying functionality before pipelines and that before implementing graphines. One issue you potentially raise in this scenario is how you find existing components that may exist without having tight coupling. The way we're working on this is to have named services which you look up in a linda-type way. One nasty thing we found there is that the code can start looking snarled up. That was until we realised because we have a scheduler that we can implementing almost-but-not-quite-greenlet style switching using generators. Specifically a pygame based component can do a lookup for an actve display thus: def main(self): yield WaitComplete( self.requestDisplay(DISPLAYREQUEST=True, callback = (self,"control"), size = (self.render_area.width, self.render_area.height), position = self.position ) ) display = self.display Where requestDisplay does this: def requestDisplay(self, **argd): displayservice = PygameDisplay.getDisplayService() self.link((self,"signal"), displayservice) self.send(argd, "signal") for _ in self.waitBox("control"): yield 1 display = self.recv("control") self.display = display That said, having seen your post further down the thread about a set of requirements you post, I think on many fronts we hit much of the aims you list. This is largely for one simple reason - our primary aim is that of making concurrency __easy__ to deal with, whilst not sacrificing the portable scalability. This is also why we've worked on graphical tools for putting together these pipelines, as well as looking inside. We have a deliberately naive C++ implementation based on the mini-axon tutorial which uses Duffs device to implement generators for example. I'd love to see that implementation expand since I think it'd generate a whole set of questions that need answering and would probably benefit the python implementation. Mind you hopefully pypy and shedskin will mitigate /our/ need to do that translation manually :-) If anyone's interested further, I'm happy to talk more (I'm going to Euro OSCON is anyone wants to discuss this there), but this feels like spamming the list so I'll leave things at that. However, so far we're finding the approach is a good one for our needs. The reason I'm posting about it here is because I think Kamaelia directly hits every point you made, but mainly in the hope it's of use to others (either directly or for cherry picking ideas). Best Regards, Michael. -- Michael Sparks, Senior R&D Engineer, Digital Media Group Michael.Sparks@rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This e-mail may contain personal views which are not the views of the BBC.
On Friday 30 September 2005 22:13, Michael Sparks (home address) wrote:
I wrote a white paper based on my Python UK talk, which is here: * http://www.bbc.co.uk/rd/pubs/whp/whp11.shtml
Oops that URL isn't right. It should be: * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml Sorry! (Thanks to LD 'Gus' Landis for pointing that out!) Regards, Michael. -- "Though we are not now that which in days of old moved heaven and earth, that which we are, we are: one equal temper of heroic hearts made weak by time and fate but strong in will to strive, to seek, to find and not to yield" -- "Ulysses", Tennyson
that stm paper isn't the end. there's a java implementation which seems to be exactly what we want: http://research.microsoft.com/~tharris/papers/2003-oopsla.pdf
At 02:35 AM 10/12/2005 +0000, Joshua Spoerri wrote:
that stm paper isn't the end.
there's a java implementation which seems to be exactly what we want: http://research.microsoft.com/~tharris/papers/2003-oopsla.pdf
There's already a Python implementation of what's described in the paper. It's called ZODB. :) Just use the memory backend if you don't want the objects to persist. Granted, if you want automatic retry you'll need to create a decorator that catches conflict errors. But basically, ZODB implements a similar optimistic conflict management transaction algorithm to that described in the paper. Certainly, it's the closest thing you can get in CPython without a complete redesign of the VM.
participants (16)
-
"Martin v. Löwis"
-
Benji York
-
Bill Janssen
-
Bruce Eckel
-
Christopher Armstrong
-
Greg Ewing
-
Jim Fulton
-
Jonathan LaCour
-
Joshua Spoerri
-
Michael Hudson
-
Michael Sparks
-
Michael Sparks (home address)
-
Michael Walter
-
Nick Coghlan
-
Phillip J. Eby
-
Shane Hathaway