Cofunctions: It's alive! Its alive!
I've been doing some more hacking, and I now have a working implementation of cofunctions, which I'll upload somewhere soon. I have also translated my yield-from examples to use cofunctions. In the course of doing this, the additional restrictions that cofunctions impose have already proved their worth -- I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go! -- Greg
On Fri, Aug 6, 2010 at 9:12 PM, Greg Ewing wrote:
I've been doing some more hacking, and I now have a working implementation of cofunctions, which I'll upload somewhere soon.
I have also translated my yield-from examples to use cofunctions. In the course of doing this, the additional restrictions that cofunctions impose have already proved their worth -- I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go!
This is good to hear. Without being too critical, I feel like saying that so far I've been following the cofunctions threads and waiting for compelling use cases. So, I'll be happy to see what you have there. It seems like the main use case that comes to mind for cofunctions is, essentially, quick and dirty cooperative multitasking. Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out (I don't know about the embeded space. Probably it's hanging on there for real time purposes.) in favor of preemptive multitasking. But Python already has preemptive multitasking: it's called threads. Or, if one prefers, there is multiprocessing. Of course, those are relatively heavy-weight, but then again, so is adding new keywords and syntax. So, other use cases would be appreciated.
I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go!
Hmm. I think this can be pushed even farther. For example, we could use mandatory function annotations to mark what classes a function is capable of receiving and we could mark the classes of variable names using some new syntax. Then when, for example, you accidentally try to send a string to sum, you could be told at compile time "TypeError." I propose we call this "Static TypeErroring". ;-) OTOH, "Explicit is better than implicit." So, maybe the explicit syntax for cocalling is worth the pain. Again, I'd like to see more motivating examples. Cautiously-optimistically-yrs, -- Carl Johnson
On Sat, Aug 7, 2010 at 3:05 AM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
On Fri, Aug 6, 2010 at 9:12 PM, Greg Ewing wrote:
I've been doing some more hacking, and I now have a working implementation of cofunctions, which I'll upload somewhere soon.
I have also translated my yield-from examples to use cofunctions. In the course of doing this, the additional restrictions that cofunctions impose have already proved their worth -- I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go!
This is good to hear. Without being too critical, I feel like saying that so far I've been following the cofunctions threads and waiting for compelling use cases. So, I'll be happy to see what you have there. It seems like the main use case that comes to mind for cofunctions is, essentially, quick and dirty cooperative multitasking. Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out (I don't know about the embeded space. Probably it's hanging on there for real time purposes.) in favor of preemptive multitasking. But Python already has preemptive multitasking: it's called threads. Or, if one prefers, there is multiprocessing. Of course, those are relatively heavy-weight, but then again, so is adding new keywords and syntax. So, other use cases would be appreciated.
I am excited about the cofunctions PEP and features, since it will greatly improve the ability for monocle to have cooperative async tasks. An extension in the yield-from PEP allows us to return normal values without confusing semantics, cocall and codef allow us to avoid several types of errors, and in general the idea of a cofunction is promoted to a first-class tool, instead of an overloaded trick on top of generators. This need for these monocle chains is a much larger topic, but threads and processes introduce locking and state sharing complications which are much more involved than simply pausing a function until data is received. Not to mention the GIL. Cooperative multitasking in the OS was set aside, but async IO is alive and well. Allowing programmers to use async IO with a linear blocking look-alike syntax seems to be an important compromise which makes it easier to write efficient concurrent networking code. -Greg
On Sat, Aug 7, 2010 at 3:05 AM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out [...]
I don't think you can see this as reflecting badly on cooperative multitasking. Rather, it is now understood better and has moved entirely to user space, leaving the OS to deal with preemptive threads, which cannot work well without OS support. As an example of cooperative multitasking being alive and well, see Stackless, Greenlets, and, in fact, coroutines built out of generators (possible in Python 2.5 and later). -- --Guido van Rossum (python.org/~guido)
On Sun, Aug 8, 2010 at 12:51 AM, Guido van Rossum <guido@python.org> wrote:
On Sat, Aug 7, 2010 at 3:05 AM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out [...]
I don't think you can see this as reflecting badly on cooperative multitasking. Rather, it is now understood better and has moved entirely to user space, leaving the OS to deal with preemptive threads, which cannot work well without OS support. As an example of cooperative multitasking being alive and well, see Stackless, Greenlets, and, in fact, coroutines built out of generators (possible in Python 2.5 and later).
Even simpler: GUI event loops are fundamentally about cooperative multi-tasking. To me, the cofunction idea is really about making it easier to write event loop code. I think PEP 380 works well on its own, but will work better when paired with a separate cofunctions PEP (similar to how PEP 342 and PEP 343 were best considered as a pair of co-proposals that went together, even though they were technically independent of each other). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Aug 7, 2010 at 5:20 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 8, 2010 at 12:51 AM, Guido van Rossum <guido@python.org> wrote:
On Sat, Aug 7, 2010 at 3:05 AM, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out [...]
I don't think you can see this as reflecting badly on cooperative multitasking. Rather, it is now understood better and has moved entirely to user space, leaving the OS to deal with preemptive threads, which cannot work well without OS support. As an example of cooperative multitasking being alive and well, see Stackless, Greenlets, and, in fact, coroutines built out of generators (possible in Python 2.5 and later).
Even simpler: GUI event loops are fundamentally about cooperative multi-tasking. To me, the cofunction idea is really about making it easier to write event loop code.
I actually added a reference to Twisted to my email, and then took it out because (while I agree to a point) the programming model with callbacks is so different that the word "task" doesn't really cover it for me. But it is indeed all about total app control over where to suspend execution in favor of another activity.
I think PEP 380 works well on its own, but will work better when paired with a separate cofunctions PEP
We'll see. I still cannot get my head around why cofunctions are so great. (Also the name sucks for me.)
(similar to how PEP 342 and PEP 343 were best considered as a pair of co-proposals that went together, even though they were technically independent of each other).
That's a rather one-sided relationship though. While PEP 343 deeply relies on the improvements to yield (especially close() and the new exception semantics), the main subject of PEP 342 (coroutines) couldn't care less about with-statements (unless there's a pattern I'm missing). -- --Guido van Rossum (python.org/~guido)
Guido van Rossum wrote:
We'll see. I still cannot get my head around why cofunctions are so great.
I think I can offer some evidence. I've been playing around with two versions of a discrete-event simulation kernel, one using yield-from and the other using cofunctions. Here's the main function of one of my test cases. I've introduced a deliberate bug -- can you spot it? def customer(i): print("Customer", i, "arriving at", now()) yield from tables.acquire(1) print("Customer", i, "sits down at a table at", now()) yield from waiters.acquire(1) print("Customer", i, "orders spam at", now()) hold(random.normalvariate(20, 2)) waiters.release(1) print("Customer", i, "gets served spam at", now()) yield from hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) tables.release(1) The bug is that the first call to hold() is missing a 'yield from' in front of it. If I run this, I don't get any exception -- it produces plausible-looking but incorrect results. Here's another version, with a very similar bug in a different place -- this time it's the second call to hold() that's missing a 'yield from'. def customer(i): print("Customer", i, "arriving at", now()) yield from tables.acquire(1) print("Customer", i, "sits down at a table at", now()) yield from waiters.acquire(1) print("Customer", i, "orders spam at", now()) yield from hold(random.normalvariate(20, 2)) waiters.release(1) print("Customer", i, "gets served spam at", now()) hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) tables.release(1) If I run this one, I do get an exception, but it's a rather unhelpful one: Traceback (most recent call last): File "restaurant2.py", line 35, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 32, in customer tables.release(1) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/resource.py", line 25, in release wakeup(self.queue[0]) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 34, in wakeup schedule(process, now()) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 16, in schedule heappush(event_queue, (time, process)) TypeError: unorderable types: generator() < generator() If you examine the traceback, you'll find that *nowhere* does it mention the location where the error actually is! Instead, a mysterious error emanates from some place deep inside the scheduler. I would hate to have to track down a problem like this in a large program. Here's the equivalent thing using cofunctions, complete with a corresponding missing 'cocall': codef customer(i): print("Customer", i, "arriving at", now()) cocall tables.acquire(1) print("Customer", i, "sits down at a table at", now()) cocall waiters.acquire(1) print("Customer", i, "orders spam at", now()) cocall hold(random.normalvariate(20, 2)) cocall waiters.release(1) print("Customer", i, "gets served spam at", now()) hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) cocall tables.release(1) The exception and traceback resulting from this is crystal clear: Traceback (most recent call last): File "restaurant2.py", line 34, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/Cofunctions-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 29, in customer hold(random.normalvariate(10, 5)) TypeError: Cofunction must be called with cocall or costart If this doesn't convince you of the utility of cofunctions or something like them, I don't know what will. -- Greg
On Tue, Aug 10, 2010 at 3:22 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
We'll see. I still cannot get my head around why cofunctions are so great.
[snip]
Here's the equivalent thing using cofunctions, complete with a corresponding missing 'cocall':
[snip]
The exception and traceback resulting from this is crystal clear:
Traceback (most recent call last): File "restaurant2.py", line 34, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/Cofunctions-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 29, in customer hold(random.normalvariate(10, 5)) TypeError: Cofunction must be called with cocall or costart
If this doesn't convince you of the utility of cofunctions or something like them, I don't know what will.
So the benefit of cocalls is runtime type checking? Are your unit tests broken? I was -0 on ABCs and function annotations because I was promised by people that liked them that I could safely ignore them. I can't safely ignore this so I'm -1. -Jack
On Tue, Aug 10, 2010 at 5:10 AM, Jack Diederich <jackdied@gmail.com> wrote:
On Tue, Aug 10, 2010 at 3:22 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
We'll see. I still cannot get my head around why cofunctions are so great.
[snip]
Here's the equivalent thing using cofunctions, complete with a corresponding missing 'cocall':
[snip]
The exception and traceback resulting from this is crystal clear:
Traceback (most recent call last): File "restaurant2.py", line 34, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/Cofunctions-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 29, in customer hold(random.normalvariate(10, 5)) TypeError: Cofunction must be called with cocall or costart
If this doesn't convince you of the utility of cofunctions or something like them, I don't know what will.
So the benefit of cocalls is runtime type checking? Are your unit tests broken?
Not entirely. I think "cocall" is a better term than "yield from" for this task, and I see benefit in having an explicit way to declare a cofunction, instead of the "if 0: yield" trick. The runtime type checking benefits are certainly helpful, though, even if all they do is make it clear why your unit tests are failing.
I was -0 on ABCs and function annotations because I was promised by people that liked them that I could safely ignore them. I can't safely ignore this so I'm -1.
How were you hoping to ignore them? You can't safely ignore yield or generators, either. If a function you are calling which returned a list decided to change its implementation and be a generator, then your list subscript access would stop working, since generators are not subscriptable. -Greg
On Tue, 10 Aug 2010 08:10:44 -0400 Jack Diederich <jackdied@gmail.com> wrote:
On Tue, Aug 10, 2010 at 3:22 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
We'll see. I still cannot get my head around why cofunctions are so great.
[snip]
Here's the equivalent thing using cofunctions, complete with a corresponding missing 'cocall':
[snip]
The exception and traceback resulting from this is crystal clear:
Traceback (most recent call last): File "restaurant2.py", line 34, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/Cofunctions-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 29, in customer hold(random.normalvariate(10, 5)) TypeError: Cofunction must be called with cocall or costart
If this doesn't convince you of the utility of cofunctions or something like them, I don't know what will.
So the benefit of cocalls is runtime type checking? Are your unit tests broken?
Isn't "runtime type checking" just another way to say "duck typing"? Would you be happier if the error message was "'Cofunction' object is not callable", so it was matched the error you get when you call other non-callable objects? Given two features that are otherwise equivalent - which seems to be the case with yield-from vs cocalls - I'll take the one that makes debugging easier.
I was -0 on ABCs and function annotations because I was promised by people that liked them that I could safely ignore them. I can't safely ignore this so I'm -1.
I don't see how you can ignore any feature that some program you're trying to debug makes use of. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Jack Diederich wrote:
So the benefit of cocalls is runtime type checking? Are your unit tests broken?
The benefit is *diagnosis* of certain types of errors that otherwise produce very mysterious symptoms. Unit tests would tell me that *something* was wrong, but not what it is or where it is.
I was -0 on ABCs and function annotations because I was promised by people that liked them that I could safely ignore them. I can't safely ignore this so I'm -1.
I don't see any difference in ignorability. Just like ABCs, you can ignore it if you don't use it in your own code. You have to understand it in order to deal with someone else's code that uses it, but the same is true of ABCs or anything else. -- Greg
I'm convinced of the utility. I still find the mechanism somehow odd or clumsy; the need for two new keywords (codef and cocall), a new builtin (costart), and a new api (__cocall__) doesn't sit well. Please don't consider this a -1; I think there's something that can be done (either to my mind or to the proposal :-). I suppose I should look again at goroutines and see what syntax and other rules they use. In the mean time let me ask a few more questions (sorry if these have been answered before, my attention is focusing in and out of this thread): - Is it possible to mix and match yield, yield from, and cocall in the same function? Should / shouldn't it be? - Would it be sufficient if codef was a decorator instead of a keyword? (This new keyword in particular chafes me, since we've been so successful at overloading 'def' for so many meanings -- functions, methods, class methods, static methods, properties...) - If we had cocall, would yield from still be useful? (I suppose yield from is the thing of choice when using generators-as-iterators, e.g. when walking a tree. But what bout yield from for coroutines?) - The cocall keyword eerily reminds me of Fortran. I know that's not fair, but still... - The syntax worries me. Your PEP suggests that cocall binds tightly to an atom. That would mean that if the cofunction is really a comethod, you'd have to parenthesis it, like cocall (obj.method)(args) ? Huuuu, ugly. Also things lke 'cocall foo' (no call syntax) weird me out. - How much of the improved error flagging of codef/cocall can be obtained by judicious use of decorators and helper functions? (I need this in Python 2.5 *now*. :-) --Guido On Tue, Aug 10, 2010 at 12:22 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
We'll see. I still cannot get my head around why cofunctions are so great.
I think I can offer some evidence. I've been playing around with two versions of a discrete-event simulation kernel, one using yield-from and the other using cofunctions. Here's the main function of one of my test cases. I've introduced a deliberate bug -- can you spot it?
def customer(i): print("Customer", i, "arriving at", now()) yield from tables.acquire(1) print("Customer", i, "sits down at a table at", now()) yield from waiters.acquire(1) print("Customer", i, "orders spam at", now()) hold(random.normalvariate(20, 2)) waiters.release(1) print("Customer", i, "gets served spam at", now()) yield from hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) tables.release(1)
The bug is that the first call to hold() is missing a 'yield from' in front of it. If I run this, I don't get any exception -- it produces plausible-looking but incorrect results.
Here's another version, with a very similar bug in a different place -- this time it's the second call to hold() that's missing a 'yield from'.
def customer(i): print("Customer", i, "arriving at", now()) yield from tables.acquire(1) print("Customer", i, "sits down at a table at", now()) yield from waiters.acquire(1) print("Customer", i, "orders spam at", now()) yield from hold(random.normalvariate(20, 2)) waiters.release(1) print("Customer", i, "gets served spam at", now()) hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) tables.release(1)
If I run this one, I do get an exception, but it's a rather unhelpful one:
Traceback (most recent call last): File "restaurant2.py", line 35, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 32, in customer tables.release(1) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/resource.py", line 25, in release wakeup(self.queue[0]) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 34, in wakeup schedule(process, now()) File "/Local/Projects/D/Python/YieldFrom/3.1/YieldFrom-3.1.2/Examples/Simulation/simulation.py", line 16, in schedule heappush(event_queue, (time, process)) TypeError: unorderable types: generator() < generator()
If you examine the traceback, you'll find that *nowhere* does it mention the location where the error actually is! Instead, a mysterious error emanates from some place deep inside the scheduler. I would hate to have to track down a problem like this in a large program.
Here's the equivalent thing using cofunctions, complete with a corresponding missing 'cocall':
codef customer(i): print("Customer", i, "arriving at", now()) cocall tables.acquire(1) print("Customer", i, "sits down at a table at", now()) cocall waiters.acquire(1) print("Customer", i, "orders spam at", now()) cocall hold(random.normalvariate(20, 2)) cocall waiters.release(1) print("Customer", i, "gets served spam at", now()) hold(random.normalvariate(10, 5)) print("Customer", i, "finished eating at", now()) cocall tables.release(1)
The exception and traceback resulting from this is crystal clear:
Traceback (most recent call last): File "restaurant2.py", line 34, in <module> run() File "/Local/Projects/D/Python/YieldFrom/3.1/Cofunctions-3.1.2/Examples/Simulation/simulation.py", line 25, in run next(current_process) File "restaurant2.py", line 29, in customer hold(random.normalvariate(10, 5)) TypeError: Cofunction must be called with cocall or costart
If this doesn't convince you of the utility of cofunctions or something like them, I don't know what will.
-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
I think I like this. I'd like to see it without new keywords. It's easy to imagine @cofunction def ... For cocall if this were some other language we might see: <cocall>foo() cocall:foo() cocall$foo() =>foo() Is there a pythonesque alternative? --- Bruce (via android)
Bruce Leban wrote:
For cocall if this were some other language we might see:
<cocall>foo() cocall:foo() cocall$foo() =>foo()
Is there a pythonesque alternative?
It's generally considered unpythonic to assign arbitary meanings to randomly chosen punctuation (decorator syntax notwithstanding!) so it will be difficult to find one. -- Greg
Guido van Rossum wrote:
I'm convinced of the utility. I still find the mechanism somehow odd or clumsy; the need for two new keywords (codef and cocall), a new builtin (costart), and a new api (__cocall__) doesn't sit well.
The costart function isn't strictly necessary, because you can always invoke __cocall__ directly. It's just there for convenience and for symmetry with all the other function/typeslot pairs. It's possible that 'codef' could be eliminated by following a similar rule to generators, i.e. if it contains 'codef' anywhere then it's a cofunction. But then we're back to the situation of having to insert a dummy statement to force cofunctionness some of the time. I expect this be needed much more frequently with cofunctions than with generators, because it's very rare that you call a generator for a side effect, whereas with coroutines it's quite normal. I also feel strongly that it would be too implicit. We manage to get away with it in generators because from the outside a generator is just a normal function that happens to return an iterator. But a cofunction has a very different interface, and the caller must be aware of that, so I would much rather make it explicit at the point of definition. A decorator could be provided to take care of the no-cocall case, but since it wouldn't be required in most cases, and people wouldn't bother to use it when they didn't need to, so it wouldn't serve to make the interface explicit in general. Maybe something could be done to force the decorator to be used on all cofunctions, such as making them neither callable nor cocallable until the decorator has been applied, but things are getting terribly convoluted by then. There would be something perverse about a definition that looks to all the world like a plain function, except that you can't actually do anything with it until it's been wrapped in a specific decorator.
- Is it possible to mix and match yield, yield from, and cocall in the same function? Should / shouldn't it be?
Yes, it is. A cofunction is a kind of generator, and 'yield' and 'yield from' work just the same way in a cofunction as they do in an ordinary generator.
(This new keyword in particular chafes me, since we've been so successful at overloading 'def' for so many meanings -- functions, methods, class methods, static methods, properties...)
I understand how you feel, but this case seems fundamentally different to me. All of those decorators are pretty much agnostic about what they wrap -- they just take a callable object and externally modify its behaviour. A decorator with the same properties as codef would need to be much more intimately connected with the thing it's wrapping.
- If we had cocall, would yield from still be useful?
You need *some* way to suspend a cofunction, so if not yield, some other keyword would need to be invented. There doesn't seem to be any point to that. If you're asking whether it needs to be able to send and receive values when used in a cofunction, I suppose it's not strictly necessary, but again there doesn't seem to be any point in disallowing these things.
- The syntax worries me. Your PEP suggests that cocall binds tightly to an atom. That would mean that if the cofunction is really a comethod, you'd have to parenthesis it,
No, if you examine the grammar in the PEP you'll see that the atom can be followed by a subset of the trailers allowed after atoms in other contexts, so it's possible to write things like x = cocall foo.blarg[42].stuff(y) which parses as x = cocall (foo.blarg[42].stuff)(y)
Also things lke 'cocall foo' (no call syntax) weird me out.
That's a syntax error -- a cocall *must* ultimately terminate with an argument list.
- How much of the improved error flagging of codef/cocall can be obtained by judicious use of decorators and helper functions? (I need this in Python 2.5 *now*. :-)
I'll have to think about that and get back to you. -- Greg
On Wed, Aug 11, 2010 at 10:49 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
- The syntax worries me. Your PEP suggests that cocall binds tightly to an atom. That would mean that if the cofunction is really a comethod, you'd have to parenthesis it,
No, if you examine the grammar in the PEP you'll see that the atom can be followed by a subset of the trailers allowed after atoms in other contexts, so it's possible to write things like
x = cocall foo.blarg[42].stuff(y)
which parses as
x = cocall (foo.blarg[42].stuff)(y)
I would expect the grammatical rules for cocall expressions to be similar to those for yield expressions. And if they weren't, I'd want to hear a really good excuse for the inconsistency :) Also, a possible trick to make a @cofunction decorator work: class cofunction: # Obviously missing a bunch of stuff to tidy up the metadata def __init__(self, f): self._f = f def __cocall__(*args, **kwds): self, *args = *args return yield from self._f(*args, **kwds) Cofunctions then wouldn't even *have* a __call__ slot, so you couldn't call them normally by mistake, and ordinary functions wouldn't define __cocall__ so you couldn't invadvertently use them with the new keyword. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 10, 2010 at 7:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Aug 11, 2010 at 10:49 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
- The syntax worries me. Your PEP suggests that cocall binds tightly to an atom. That would mean that if the cofunction is really a comethod, you'd have to parenthesis it,
No, if you examine the grammar in the PEP you'll see that the atom can be followed by a subset of the trailers allowed after atoms in other contexts, so it's possible to write things like
x = cocall foo.blarg[42].stuff(y)
which parses as
x = cocall (foo.blarg[42].stuff)(y)
I would expect the grammatical rules for cocall expressions to be similar to those for yield expressions. And if they weren't, I'd want to hear a really good excuse for the inconsistency :)
This I can explain -- a cocall *must* be a call, syntactically, so that it can take the callable, check that it has a __cocall__ method, and call it with the given argument list. But I have to say, I don't really like it, it's very odd syntax (even worse than decorators).
Also, a possible trick to make a @cofunction decorator work:
class cofunction: # Obviously missing a bunch of stuff to tidy up the metadata def __init__(self, f): self._f = f
def __cocall__(*args, **kwds): self, *args = *args return yield from self._f(*args, **kwds)
Cofunctions then wouldn't even *have* a __call__ slot, so you couldn't call them normally by mistake, and ordinary functions wouldn't define __cocall__ so you couldn't invadvertently use them with the new keyword.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
On 08/10/2010 09:48 PM, Nick Coghlan wrote:
On Wed, Aug 11, 2010 at 10:49 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
- The syntax worries me. Your PEP suggests that cocall binds tightly to an atom. That would mean that if the cofunction is really a comethod, you'd have to parenthesis it,
No, if you examine the grammar in the PEP you'll see that the atom can be followed by a subset of the trailers allowed after atoms in other contexts, so it's possible to write things like
x = cocall foo.blarg[42].stuff(y)
which parses as
x = cocall (foo.blarg[42].stuff)(y)
I would expect the grammatical rules for cocall expressions to be similar to those for yield expressions. And if they weren't, I'd want to hear a really good excuse for the inconsistency :)
Also, a possible trick to make a @cofunction decorator work:
class cofunction: # Obviously missing a bunch of stuff to tidy up the metadata def __init__(self, f): self._f = f
def __cocall__(*args, **kwds): self, *args = *args return yield from self._f(*args, **kwds)
Cofunctions then wouldn't even *have* a __call__ slot, so you couldn't call them normally by mistake, and ordinary functions wouldn't define __cocall__ so you couldn't invadvertently use them with the new keyword.
I was wondering if a class would work. Could using conext() and cosend() methods instead of next() and send() give the better error messages and distinguish cofunctions from generators without the need for __cocall__ ? Can that be done with a decorator or class? Ron
Ron Adam wrote:
Could using conext() and cosend() methods instead of next() and send() give the better error messages and distinguish cofunctions from generators without the need for __cocall__ ?
I don't think it would help much. If you forget to use yield-from in a call that doesn't use the return value, the returned generator just gets thrown away without anything trying to call any of its methods. -- Greg
On 08/11/10 01:57, Guido van Rossum wrote:
- Would it be sufficient if codef was a decorator instead of a keyword? (This new keyword in particular chafes me, since we've been so successful at overloading 'def' for so many meanings -- functions, methods, class methods, static methods, properties...)
+1. I'd like to see this implemented as decorator (perhaps with special casing by the VM if necessary), and see how this cofunction will be used in wider practice before deciding whether the syntax sugar is necessary. The decorator could live as a built-in function or as stdlib module (from cofunction import cofunction), and be clearly marked as experimental.
Carl M. Johnson wrote:
It seems like the main use case that comes to mind for cofunctions is, essentially, quick and dirty cooperative multitasking. Of course, as we all know on the OS side of things, cooperative multitasking has been more or less phased out ... in favor of preemptive multitasking.
They're meant for different things. Preemptive multitasking is for concurrency -- you have N cores and they can all physically run at the same time. Cooperative multitasking is a program structuring technique, for when it helps to think in terms of several separate state machines interacting with each other. The existence of widely-used frameworks such as Twisted attests that there are very real use cases for non-preemptive multitasking. Recently I've come across a discrete-event simulation package called SimPy that does process-oriented simulation using the generator/trampoline technique. This seems like an ideal use case for cofunctions, and I'm going to look into creating a cofunction-based version of it.
Hmm. I think this can be pushed even farther. ... Then when, for example, you accidentally try to send a string to sum, you could be told at compile time "TypeError." I propose we call this "Static TypeErroring". ;-)
Well, I think you're indulging in extrapolation ad absurdum here... My initial proposal for cofunctions was considerably more dynamic, but it got criticised for not being explicit enough. It also wasn't capable of detecting and clearly diagnosing all of the errors that the current one can.
OTOH, "Explicit is better than implicit." So, maybe the explicit syntax for cocalling is worth the pain.
When you consider that the alternative to writing 'cocall' in certain places is to write 'yield from' or some equivalent thing in all the same places, I'd say the cofunction way is actually *less* painful. "Doctor, it hurts when I do this." "Don't do that, then." "But if I don't, it hurts twice as badly!" -- Greg
On 08/07/2010 02:12 AM, Greg Ewing wrote:
I've been doing some more hacking, and I now have a working implementation of cofunctions, which I'll upload somewhere soon.
I have also translated my yield-from examples to use cofunctions. In the course of doing this, the additional restrictions that cofunctions impose have already proved their worth -- I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go!
Would it be even remotely possible... ... to write a co-function program in a way where it could be switched from cooperative multitasking to preemptive multitasking by the use of a single flag? (I'd be +10,000 for this.) If so, it would enable a way to hide a lot of details of multi-tasking and multi-processing in a convenient to use api. No I haven't thought it though all that far. But it seems to me it might be able to work if generators could work in a suspend before yield, instead of a suspend after yield mode. Ideally pie in the sky, Ron Hey, this is the idea list after all. ;-)
Ron Adam, 08.08.2010 05:17:
Would it be even remotely possible...
... to write a co-function program in a way where it could be switched from cooperative multitasking to preemptive multitasking by the use of a single flag? (I'd be +10,000 for this.)
I wouldn't.
If so, it would enable a way to hide a lot of details of multi-tasking and multi-processing in a convenient to use api.
Totally not. Cooperative multitasking is about predictable interaction between parts of a program. Preemptive multitasking (in the sense of threading) is about non-deterministic concurrency. Except for some very special cases, there is no way you can take a piece of code that uses cooperative multitasking, switch it over to run concurrently, and still have it execute safely and correctly. I may end up liking the idea of using yield statements for thread synchronisation points, though. Stefan
participants (11)
-
Bruce Leban -
Carl M. Johnson -
ghazel＠gmail.com -
Greg Ewing -
Guido van Rossum -
Jack Diederich -
Lie Ryan -
Mike Meyer -
Nick Coghlan -
Ron Adam -
Stefan Behnel