Monitor implementation for the stdlib?

I was surprised to find that there is no "monitor construct" implementation for the stdlib. I wrote one that I feel is pretty Pythonic, and I'd like it to be torn apart :) It's fairly well documented -- for most standard monitor use cases you can simply inherit the Monitor class and use the Monitor-specific condition variables. Hopefully the accompanying examples will also help to clarify the usage. They're somewhat silly, but get the idea across. As an aside, because this seems to come up in every conversation I've had about monitors in python: if I'm not mistaken, monitors are useful with or without the GIL :) Monitors are nifty tools that make complex synchronization problems somewhat simpler (though more serialized). So far as I understand it, the GIL provides single-bytecode atomicity, and monitor methods are rarely single instructions. Plus, Jython and IronPython don't have a GIL, so I would argue that monitors can still be valuable to "Python the language" even if you won't allow that they can be valuable to "Python the standard implementation". Looking forward to hearing everybody's opinions. Cheers, Chris

Isn't this just syntactic sugar on top of recursive locks and condition variables? What's the big deal apart from conforming to a historically important API? Note that as of python 2.5, the with-statement lets you create critical sections simply by writing with <some_lock>: <critical_section> i.e. the acquire() and release() calls are implicit in the with-statement. --Guido 2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Though I do think that the with statement is an excellent addition to the language (it avoids much of the ugliness and chance for error in symmetric P and V), I don't believe that it adequately covers the use case that monitors are intended for; namely, encapsulation and abstraction of the mutual exclusion involved in locking around /several data entities/ with /several defined functionalities/. Due to how frequently this pattern is encountered, I think that monitors not only reduce clutter with syntactic sugar (simplicity?), but help encourage a more straightforward way of thinking -- several non re-entrant operations on shared data implies monitor usage, not lock acquisition all over the place. Thoughts? Chris Guido van Rossum wrote:

2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
Locks can do that too of course. It looks like a monitor is similar to 'synchronized' in Java except it appears the default (once inheriting from Monitor) is synchronized, and you need to use a decorator to declare an unsynchronized method. Have I got that right?
I detect a slight prejudice against lock acquisition in your wording. :-)
Thoughts?
It seems to be a matter of granularity. With monitors, one is required to create a new class for each group of variables that require synchronization. With locks, that is not necessary. Which is better depends on the complexity of the group of variables and (especially) the operations. A more "Pythonic" approach would be to implement an @synchronized decorator; Python users are more likely to be familiar with Java terms than with classic Hoare or P and V. (I'm sure someone has already implemented @synchronized, but it hasn't made it into the standard library -- perhaps an indication that its importance is more theoretical than practical, perhaps simply due to a time lag.) --Guido
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

I didn't mean to imply that locks /couldn't/ model any of the mentioned functionality, just that they weren't the right tool for the job that I was describing (which I believe is frequently occurring in multithreaded applications -- is that in dispute?). I debated putting the unsychronized decorator into the implementation. On one hand, a monitor is a monitor -- it's invariant is mutual exclusion between methods; however, on the other hand you can have related functionality to a "bridge" monitor that doesn't require mutual exclusion, though I figured it should probably be used very sparingly, or the point of a monitor approaches moot -- it just appealed to me more than forcing the equivalent functionality to move to global functions. Perhaps the alternative is better if it makes the monitor concept significantly weaker. That aside, I think the synchronized decorator silly (though I may be wrong, it happens all the time ;). I can envision very few situations where you would only want to make a couple of your methods mutually exclusive and none of the others. In my mind this would imply you're either using poor encapsulation or leaving yourself prone to errors... from a maintainability and understandability perspective I believe that you're losing value by selectively choosing which methods should be synchronized, whereas inheriting Monitor is a clear statement that the instance variables are protected in a thread safe manner. I wholly agree that locks are better suited for some (very complex xor very simple) multithreaded situations. I would think that in the common case where grouping data for thread safety is involved, you'll have very few, disjoint groups, in which case monitors are what you're looking for. I'd be interested in hearing more about your reasoning for synchronized. The monitor "everything is mutually exclusive" seems even /more/ simple to me than "selected functions are mutually exclusive" -- it's certainly less to worry about going forward. Chris Guido van Rossum wrote:
-- Christopher D. Leary ECE/CS Undergraduate Cornell University C: 914.844.1622 christopher.leary@cornell.edu

2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
What I'm trying to say (between the lines :-) is that the available tools are IMO good enough and that I don't see the great improvement in convenience you are perceiving with Monitors.
Perhaps you should focus on purity, and tell people who want unsynchronized access that they can't use a monitor. :-)
That aside, I think the synchronized decorator silly (though I may be wrong, it happens all the time ;).
Assuming a @synchronized decorator that works like Java's synchronized method, would it still be silly?
A clear lack of imagination on your part... :-) (Or lack of practical experience?)
I take it you don't like Java much. I wonder though if you realize how easy it is in Python to break through any abstraction?
Sounds like a matter of taste to me.
Maybe you should talk to Java's designers. --Guido
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Em 21/10/2007, às 23:17, Guido van Rossum escreveu:
Of course, as you know his code is writen using locks
Yep, any java class with synchronized is a monitor.
There is some implementations of @synchronized on the python cookbook at ASPN, but them don't implement monitors, just a way to make a non- reentrant method. I think Christopher implemented monitors that by default have all its methods as if they where java synchronized is because that is most common in python where you don't have the abundance of geters and static methods that java has. I would like to see monitors in the python std library if for no other reason to let concurrency teachers and java fanatics happy. I'm (althought still sitting on my hands about it) wanting to see STM on python. -- Leonardo Santagada

[BTW, Chris, Guido as BDFL gets a pass on complaints about top-posting, but you're not immune to them. ;-) Please use the standard formatting of mixed quoting as I do below.] On Sun, Oct 21, 2007, Christopher D. Leary wrote:
Seems to me that the "standard" Python technique for managing this is message passing to a centralized thread using Queue. What advantages do monitors offer? In any event, you will almost certainly not get monitors into the standard library until they have proven themselves in the wild as a module posted to PyPI. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The best way to get information on Usenet is not to ask a question, but to post the wrong information.

Em 22/10/2007, às 16:00, Aahz escreveu:
This is a great advice for everyone that has code and want it on stdlib, post it somewhere (googlecode or sourceforge comes to mind) package it with distutils (if you make a tarball and an egg everyone is happy), put it on PyPI and I think you can post about it in some key places like python.announce, this thread now (I for one would be willing to test my classes work with it) and some other places like freshmeat or something. -- Leonardo Santagada A: Because it breaks the logical sequence of the discussion. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?

On 10/21/07, Christopher D. Leary <christopher.leary@cornell.edu> wrote:
My threading extensions to Python (which I still haven't posted) will provide a Monitor class and use it fairly often. There's two major distinctions from what you suggest though: * Deadlocks will be detected and broken automatically * Entering the monitor will be enforced by the language. You'll never need to consult a memory model to determine correctness. This makes it more like Concurrent Pascal's monitors or Erlang's actors than Java's monitors. However, it won't drop a lock on every method automatically. You need a @monitormethod decorator for that. A method is often *NOT* the correct granularity to achieve a thread-safe API! Consider: l = ... some "thread-safe" list ... if l: x = l.pop() There's two method calls, so you cannot guarantee the object hasn't changed between them. I get the impression this is regarded as a major mistake in Java's extensive use of monitors - plastering locks everywhere just isn't good enough. -- Adam Olsen, aka Rhamphoryncus

Em 22/10/2007, às 03:21, Adam Olsen escreveu:
When are you planing on doing it? Why not put it on a googlecode svn repository? -- Leonardo Santagada A: Because it breaks the logical sequence of the discussion. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?

On 10/22/07, Leonardo Santagada <santagada@gmail.com> wrote:
I'm working on it now, although I take forever. Major functionality is still missing.
Why not put it on a googlecode svn repository?
They're picky about licensing. I don't think I can put a patch to python on there. I will eventually post on python's bug tracker. -- Adam Olsen, aka Rhamphoryncus

2007/10/22, Adam Olsen <rhamph@gmail.com>:
Beware; I think you're close to getting guilty of pre-announcing vaporware. :-)
Are you sure? Where do you read this? The way I see it, you can't make something available using the Python license (and for good reason, it's a crappy license even though it's OSI-approved and GPL-compatible), but there's no reason you can't release patches to Python under the Apache 2 license -- after all the PSF suggests that license for all contributions *to* Python as well.
I will eventually post on python's bug tracker.
Sounds hardly the place for a variant of the language. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 10/22/07, Guido van Rossum <guido@python.org> wrote:
I've been there so long I have a doormat that reads "welcome vaporware-land". ;)
Since the patch is based on code using the Python license I'm not sure how the diffs I post will be *completely* under my license. I planned to ask the PSF to help me sort out what *exactly* needed to do wrt licensing before posting my patch.
I will eventually post on python's bug tracker.
Sounds hardly the place for a variant of the language.
I haven't decided if I should view it as a language variant or as an enhancement. -- Adam Olsen, aka Rhamphoryncus

2007/10/22, Adam Olsen <rhamph@gmail.com>:
Since the patch is based on code using the Python license I'm not sure how the diffs I post will be *completely* under my license.
Doesn't matter. The PSF license allows you to relicense it.
I planned to ask the PSF to help me sort out what *exactly* needed to do wrt licensing before posting my patch.
Please do; they will be happy to explain this to you.
From the description you mailed me long ago it's clearly a language variant.
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Isn't this just syntactic sugar on top of recursive locks and condition variables? What's the big deal apart from conforming to a historically important API? Note that as of python 2.5, the with-statement lets you create critical sections simply by writing with <some_lock>: <critical_section> i.e. the acquire() and release() calls are implicit in the with-statement. --Guido 2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Though I do think that the with statement is an excellent addition to the language (it avoids much of the ugliness and chance for error in symmetric P and V), I don't believe that it adequately covers the use case that monitors are intended for; namely, encapsulation and abstraction of the mutual exclusion involved in locking around /several data entities/ with /several defined functionalities/. Due to how frequently this pattern is encountered, I think that monitors not only reduce clutter with syntactic sugar (simplicity?), but help encourage a more straightforward way of thinking -- several non re-entrant operations on shared data implies monitor usage, not lock acquisition all over the place. Thoughts? Chris Guido van Rossum wrote:

2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
Locks can do that too of course. It looks like a monitor is similar to 'synchronized' in Java except it appears the default (once inheriting from Monitor) is synchronized, and you need to use a decorator to declare an unsynchronized method. Have I got that right?
I detect a slight prejudice against lock acquisition in your wording. :-)
Thoughts?
It seems to be a matter of granularity. With monitors, one is required to create a new class for each group of variables that require synchronization. With locks, that is not necessary. Which is better depends on the complexity of the group of variables and (especially) the operations. A more "Pythonic" approach would be to implement an @synchronized decorator; Python users are more likely to be familiar with Java terms than with classic Hoare or P and V. (I'm sure someone has already implemented @synchronized, but it hasn't made it into the standard library -- perhaps an indication that its importance is more theoretical than practical, perhaps simply due to a time lag.) --Guido
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

I didn't mean to imply that locks /couldn't/ model any of the mentioned functionality, just that they weren't the right tool for the job that I was describing (which I believe is frequently occurring in multithreaded applications -- is that in dispute?). I debated putting the unsychronized decorator into the implementation. On one hand, a monitor is a monitor -- it's invariant is mutual exclusion between methods; however, on the other hand you can have related functionality to a "bridge" monitor that doesn't require mutual exclusion, though I figured it should probably be used very sparingly, or the point of a monitor approaches moot -- it just appealed to me more than forcing the equivalent functionality to move to global functions. Perhaps the alternative is better if it makes the monitor concept significantly weaker. That aside, I think the synchronized decorator silly (though I may be wrong, it happens all the time ;). I can envision very few situations where you would only want to make a couple of your methods mutually exclusive and none of the others. In my mind this would imply you're either using poor encapsulation or leaving yourself prone to errors... from a maintainability and understandability perspective I believe that you're losing value by selectively choosing which methods should be synchronized, whereas inheriting Monitor is a clear statement that the instance variables are protected in a thread safe manner. I wholly agree that locks are better suited for some (very complex xor very simple) multithreaded situations. I would think that in the common case where grouping data for thread safety is involved, you'll have very few, disjoint groups, in which case monitors are what you're looking for. I'd be interested in hearing more about your reasoning for synchronized. The monitor "everything is mutually exclusive" seems even /more/ simple to me than "selected functions are mutually exclusive" -- it's certainly less to worry about going forward. Chris Guido van Rossum wrote:
-- Christopher D. Leary ECE/CS Undergraduate Cornell University C: 914.844.1622 christopher.leary@cornell.edu

2007/10/21, Christopher D. Leary <christopher.leary@cornell.edu>:
What I'm trying to say (between the lines :-) is that the available tools are IMO good enough and that I don't see the great improvement in convenience you are perceiving with Monitors.
Perhaps you should focus on purity, and tell people who want unsynchronized access that they can't use a monitor. :-)
That aside, I think the synchronized decorator silly (though I may be wrong, it happens all the time ;).
Assuming a @synchronized decorator that works like Java's synchronized method, would it still be silly?
A clear lack of imagination on your part... :-) (Or lack of practical experience?)
I take it you don't like Java much. I wonder though if you realize how easy it is in Python to break through any abstraction?
Sounds like a matter of taste to me.
Maybe you should talk to Java's designers. --Guido
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Em 21/10/2007, às 23:17, Guido van Rossum escreveu:
Of course, as you know his code is writen using locks
Yep, any java class with synchronized is a monitor.
There is some implementations of @synchronized on the python cookbook at ASPN, but them don't implement monitors, just a way to make a non- reentrant method. I think Christopher implemented monitors that by default have all its methods as if they where java synchronized is because that is most common in python where you don't have the abundance of geters and static methods that java has. I would like to see monitors in the python std library if for no other reason to let concurrency teachers and java fanatics happy. I'm (althought still sitting on my hands about it) wanting to see STM on python. -- Leonardo Santagada

[BTW, Chris, Guido as BDFL gets a pass on complaints about top-posting, but you're not immune to them. ;-) Please use the standard formatting of mixed quoting as I do below.] On Sun, Oct 21, 2007, Christopher D. Leary wrote:
Seems to me that the "standard" Python technique for managing this is message passing to a centralized thread using Queue. What advantages do monitors offer? In any event, you will almost certainly not get monitors into the standard library until they have proven themselves in the wild as a module posted to PyPI. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ The best way to get information on Usenet is not to ask a question, but to post the wrong information.

Em 22/10/2007, às 16:00, Aahz escreveu:
This is a great advice for everyone that has code and want it on stdlib, post it somewhere (googlecode or sourceforge comes to mind) package it with distutils (if you make a tarball and an egg everyone is happy), put it on PyPI and I think you can post about it in some key places like python.announce, this thread now (I for one would be willing to test my classes work with it) and some other places like freshmeat or something. -- Leonardo Santagada A: Because it breaks the logical sequence of the discussion. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?

On 10/21/07, Christopher D. Leary <christopher.leary@cornell.edu> wrote:
My threading extensions to Python (which I still haven't posted) will provide a Monitor class and use it fairly often. There's two major distinctions from what you suggest though: * Deadlocks will be detected and broken automatically * Entering the monitor will be enforced by the language. You'll never need to consult a memory model to determine correctness. This makes it more like Concurrent Pascal's monitors or Erlang's actors than Java's monitors. However, it won't drop a lock on every method automatically. You need a @monitormethod decorator for that. A method is often *NOT* the correct granularity to achieve a thread-safe API! Consider: l = ... some "thread-safe" list ... if l: x = l.pop() There's two method calls, so you cannot guarantee the object hasn't changed between them. I get the impression this is regarded as a major mistake in Java's extensive use of monitors - plastering locks everywhere just isn't good enough. -- Adam Olsen, aka Rhamphoryncus

On 10/22/07, George Sakkis <george.sakkis@gmail.com> wrote:
aka: l = ... some "thread-safe" list ... with l.lock: if l: x = l.pop() But if this "list" doesn't provide a thread-safe API in the first place it has no business providing a lock. -- Adam Olsen, aka Rhamphoryncus

Em 22/10/2007, às 03:21, Adam Olsen escreveu:
When are you planing on doing it? Why not put it on a googlecode svn repository? -- Leonardo Santagada A: Because it breaks the logical sequence of the discussion. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?

On 10/22/07, Leonardo Santagada <santagada@gmail.com> wrote:
I'm working on it now, although I take forever. Major functionality is still missing.
Why not put it on a googlecode svn repository?
They're picky about licensing. I don't think I can put a patch to python on there. I will eventually post on python's bug tracker. -- Adam Olsen, aka Rhamphoryncus

2007/10/22, Adam Olsen <rhamph@gmail.com>:
Beware; I think you're close to getting guilty of pre-announcing vaporware. :-)
Are you sure? Where do you read this? The way I see it, you can't make something available using the Python license (and for good reason, it's a crappy license even though it's OSI-approved and GPL-compatible), but there's no reason you can't release patches to Python under the Apache 2 license -- after all the PSF suggests that license for all contributions *to* Python as well.
I will eventually post on python's bug tracker.
Sounds hardly the place for a variant of the language. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 10/22/07, Guido van Rossum <guido@python.org> wrote:
I've been there so long I have a doormat that reads "welcome vaporware-land". ;)
Since the patch is based on code using the Python license I'm not sure how the diffs I post will be *completely* under my license. I planned to ask the PSF to help me sort out what *exactly* needed to do wrt licensing before posting my patch.
I will eventually post on python's bug tracker.
Sounds hardly the place for a variant of the language.
I haven't decided if I should view it as a language variant or as an enhancement. -- Adam Olsen, aka Rhamphoryncus

2007/10/22, Adam Olsen <rhamph@gmail.com>:
Since the patch is based on code using the Python license I'm not sure how the diffs I post will be *completely* under my license.
Doesn't matter. The PSF license allows you to relicense it.
I planned to ask the PSF to help me sort out what *exactly* needed to do wrt licensing before posting my patch.
Please do; they will be happy to explain this to you.
From the description you mailed me long ago it's clearly a language variant.
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (6)
-
Aahz
-
Adam Olsen
-
Christopher D. Leary
-
George Sakkis
-
Guido van Rossum
-
Leonardo Santagada