On Thu, Oct 27, 2011 at 6:39 PM, Greg Ewing email@example.com wrote:
Another cofunctions revision:
I've added a formal expansion for cocalls
I think using _iter as the name for the result of the __cocall__ invocation helps make it clear what is going on - it's probably worth doing the same for the costart() definition as well.
and sharpened up the argument about critical sections to hopefully make it less handwavey and more persuasive.
Definitely less handwavey, but I'm not entirely sold on the "more persuasive" just yet. Consider that Java has a synchronized statement, and then synchronized methods are just a shorthand for wrapping the entire body in a synchronized statement. Consider that explicit locking (either manually or via a with statement) is more common with Python code than hiding the locks inside a decorator. The PEP currently proposes that, for coroutines, it's OK to *only* have the function level locking and no tool for statement level locking.
To see why this is potentially an issue, consider the case where you have a function that you've decided should really be a cofunction and you want to update your entire code base accordingly. The interpreter can help in this task, since you'll get errors wherever it is called from an ordinary function, but silent changes of behaviour in cofunctions. The latter is mostly a good thing in terms of making coroutine programming easier (and is pretty much the entire point of the PEP), but it is *not* acceptable if manual auditing is the proposed solution to the 'critical section for implicit cooperative concurrency' problem.
After all, one of the big things that makes cooperative threading easier than preemptive threading is that you have well defined points where you may be interrupted. The cofunction PEP in its current form gives up (some of) that advantage by making every explicit function call a potential yield point, but without providing any new tools to help manage that loss of control. That means "self.x += y" is safe, but "self.x = y(self.x)" is not (since y may yield control and self.x may be overwritten based on a call that started with a now out of date value for self.x), and there's no way to make explicit (either to the interpreter or to the reader) that 'y' is guaranteed to be an ordinary function.
I honestly believe the PEP would be improved if it offered a way to guarantee ordinary call semantics for a suite inside a cofunction so that if someone unexpectedly turns one of the functions called by that code into a cofunction, it will throw an exception rather than silently violating the assumption that that section of code won't yield control to the coroutine scheduler. Obviously, libraries should never make such a change without substantial warning (since it breaks backwards compatibility), but it would still allow a programmer to make an interpreter-enforced declaration that a particular section of code will never implicitly yield control. I think having such a construct will help qualm many of the fears people had about the original version of the implicit invocation proposal - just as try/except blocks help manage exceptions and explicit locks help manage thread preemption, being able to force ordinary call semantics for a suite would allow people to effectively manage implicit coroutine suspension in cases where they felt it mattered.
"with not codef:" was the first spelling I thought of to say "switch off coroutine call semantics for this suite". Another alternative would be "with def:" to say "switch ordinary call semantics back on for this suite", while a third would be to add explicit expressions for *both* "cocall f(*args, **kwds)" *and* "defcall f(*args, **kwds)", such that the choice of "def vs codef" merely changed the default behaviour of call expressions and you could explicitly invoke the other semantics whenever you wanted (although an explicit cocall would automatically turn something into a coroutine, just as yield turns one into a generator). I'm sure there are other colours that bike shed could be painted - my main point is that I think this particular bike shed needs to exist.