Sam, BTW, kudos for shrapnel! On 2012-10-23, at 5:25 PM, Sam Rushing <sam-pydeas@rushing.nightmare.com> wrote: [snip]

On 10/23/12 12:33 PM, Yury Selivanov wrote:

...

What we really need now (and I think Guido has already mentioned that) is a callback-based (Deferreds, Futures, plain callbacks) design that is easy to plug-and-play in any coroutine-framework. It has to be low-level and simple. Sort of WSGI for async frameworks ;)

I've been trying to play catch-up since being told about this thread a couple of days ago. If I understand it correctly, 'yield-from' looks like it can help make generator-based-concurrency a little more sane by cutting back on endless chains of 'for x in ...: yield ...', right? That certainly sounds like an improvement, but does the generator nature of the API bubble all the way up to the top? Can you send an email with a function call?

Well, I guess so. Let's say, urllib is rewritten internally in async-style, exposing publicly its old API, like:: def urlopen(*args, **kwargs): return run_coro(urlopen_async, args, kwargs) where 'run_coro' takes care of setting up a Scheduler/event-loop and running yield-style or callback-style private code. So that 'urllib' is blocking, but there is an option of using 'urlopen_async' for those who need it. For basic library functions that will work. And that's already a huge win. But developing a complicated library will become twice as hard, as you'll need to maintain two versions of API - sync & async all the way through the code. There is only one way to 'magically' make existing code both sync- & async- friendly--greenlets, but I think there is no chance for them (or stackless) to land in cpython in the foreseeable future (although it would be awesome.) BTW, why didn't you use greenlets in shrapnel and ended up with your own implementation?

...

We also need to work on the stdlib, so that it is easy to inject a custom socket in any object. Ideally, by passing it in the constructor (as everybody hates monkey-patching.)

I second this one. Having a way to [optionally] pass in a factory for sockets would help with portability, and would cut down on the temptation to monkey-patch.

Great. Let's see - if nobody is opposed to this we can start with submitting patches :) Or is there a need for a separate small PEP? Thanks, Yury

On 2012-10-23, at 7:00 PM, Sam Rushing <sam-pydeas@rushing.nightmare.com> wrote:

On 10/23/12 3:05 PM, Yury Selivanov wrote:

...

Sam,

BTW, kudos for shrapnel! Thanks!

For basic library functions that will work. And that's already a huge win. But developing a complicated library will become twice as hard, as you'll need to maintain two versions of API - sync & async all the way through the code.

This is really difficult, if you want to see a great example of trying to make all parties happy, look at Pika (an AMQP implementation).

Thanks, will take a look!

Actually this reminds me, it would be really great if there was a standardized with_timeout()API. It's better than adding timeout args to all the functions. I'm sure that systems like Twisted & gevent could also implement it (if they don't already have it):

In shrapnel it is simply:

coro.with_timeout (<seconds>, <fun>, *args, **kwargs)

Timeouts are caught thus:

try: coro.with_timeout (...) except coro.TimeoutError: ...

You're right--if we want to ship some "standard" async API in python, API for timeouts is a must. We will at least need to handle timeouts in async code in the stdlib, won't we... A question: How do you protect finally statements in shrapnel? If we have a following coroutine (greenlet style): def foo(): connection = open_connection() try: spam() finally: [some code] connection.close() What happens if you run 'foo.with_timeout(1)' and timeout occurs at "[some code]" point? Will you just abort 'foo', possibly preventing 'connection' from being closed? - Yury

Hi Greg, On 2012-10-23, at 8:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

def foo(): connection = open_connection() try: spam() finally: [some code] connection.close() What happens if you run 'foo.with_timeout(1)' and timeout occurs at "[some code]" point?

I would say that vital cleanup code probably shouldn't do anything that could block. If you really need to do that, it should be protected by a finally clause of its own:

def foo(): connection = open_connection() try: spam() finally: try: [some code] finally: connection.close()

Please take a look at the problem definition in PEP 419. It's not about try..finally nesting, it's about Scheduler being aware that a coroutine is in its 'finally' block and thus shouldn't be interrupted at the moment (a problem that doesn't exist in a non-coroutine world). Speaking about your solution, imagine if you have three connections to close, what will you write? finally: try: c1.close() # coroutine call finally: try: c2.close() # coroutine call finally: c3.close() # coroutine call But if you somehow make scheduler aware of 'finally' block, through PEP 419 (which I don't like), or like in my framework where we inline special code in finally statement by modifying coroutine opcodes (which I don't like too), you can simply write:: finally: c1.close() c2.close() c3.close() And scheduler will gladly wait until finally is over. And the code snippet above is something, that is familiar to every user of python--nobody expects code in the finally section to be interrupted from the *outside* world. If we fail to guarantee 'finally' block safety, then coroutine-style programming is going to be much tougher. Or we have to abandon timeouts and coroutines interruptions. So eventually, we'll need to figure out the best mechanism/approach for this. Now, I don't think it's the right moment to shift discussion into this particular problem, but I would rather like to bring up the point, that implementing 'yield'-style coroutines is a very hard thing, and I'm not sure that we should implement them in 3.4. Setting guidelines and standard protocols, adding socket-factories support where necessary in the stdlib is a better approach (in my humble opinion.) - Yury

On Wed, Oct 24, 2012 at 2:30 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

It's not about try..finally nesting, it's about Scheduler being aware that a coroutine is in its 'finally' block and thus shouldn't be interrupted at the moment

It would be a bad idea to make a close() method, or anything else that might be needed for cleanup purposes, be a 'yield from' call. If it's an ordinary function, it can't be interrupted in the world we're talking about, so the PEP 419 problem doesn't apply.

If I were feeling in a radical mood, I might go as far as suggesting that 'yield' and 'yield from' be syntactically forbidden inside a finally clause. That would force you to design your cleanup code to be safe from interruptions.

What's the problem with just letting the cleanup take as long as it wants to and do whatever it wants? That's how try/finally works in regular Python code. -- --Guido van Rossum (python.org/~guido)

On Wed, Oct 24, 2012 at 4:03 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

Hi Guido,

On 2012-10-24, at 6:43 PM, Guido van Rossum <guido@python.org> wrote:

...

What's the problem with just letting the cleanup take as long as it wants to and do whatever it wants? That's how try/finally works in regular Python code.

The problem appears when you add timeouts support.

Let me show you an abstract example (I won't use yield_froms, but I'm sure that the problem is the same with them):

@coroutine def fetch_comments(app): session = yield app.new_session() try: return (yield session.query(...)) finally: yield session.close()

and now we execute that with:

#: Get a list of comments; throw a TimeoutError if it #: takes more than 1 second comments = yield fetch_comments(app).with_timeout(1.0)

Now, scheduler starts with 'fetch_comments', then executes 'new_session', then executes 'session.query' in a round-robin fashion.

Imagine, that database query took a bit less than a second to execute, scheduler pushes the result in coroutine, and then a timeout event occurs. So scheduler throws a 'TimeoutError' in the coroutine, thus preventing the 'session.close' to be executed. There is no way for a scheduler to understand, that there is no need in pushing the exception right now, as the coroutine is in its finally block.

And this situation is a pretty common when you have such timeouts mechanism in place and widely used.

Ok, I can understand. But still, this is a problem with timeouts in general, not just with timeouts in a yield-based environment. How does e.g. Twisted deal with this? As a work-around, I could imagine some kind of with-statement that tells the scheduler we're already in the finally clause (it could still send you a timeout if your cleanup takes way too long): try: yield <regular code> finally: with protect_finally(): yield <cleanup code> Of course this could be abused, but at your own risk -- the scheduler only gives you a fixed amount of extra time and then it's quits. -- --Guido van Rossum (python.org/~guido)

On 2012-10-24, at 7:25 PM, Steve Dower <Steve.Dower@microsoft.com> wrote: [snip]

Could another workaround be to spawn the cleanup code without yielding - in effect saying "go and do this, but don't come back"? Then there is nowhere for the scheduler to throw the exception.

I ask because this falls out naturally with my implementation (code is coming, but work is taking priority right now): "do_cleanup()" instead of "yield do_cleanup()". I haven't tried it in this context yet, so no idea whether it works, but I don't see why it wouldn't. In a system without the @async decorator you'd need a "scheduler.current.spawn(do_cleanup)" instead of yield [from]s, but it can still be done.

Well, yes, this will work. If we have the following: # "async()" is a way to launch coroutines in my framework without # "coming back"; with it they just return a promise/future that needs # to be yielded again finally: yield c.close().async() The solution is very limited though. Imagine if you have lots of cleanup code finally: yield c1.close().async() # go and do this, but don't come back yield c2.close().async() The above won't work, as scheduler would have an opportunity to break everything on the second 'yield'. You may solve it by grouping cleanup code in a separate inner coroutine, like: @coroutine def do_stuff(): try: ... finally: @coroutine def cleanup(): yield c1.close() yield c2.close() yield cleanup().async() # go and do this, but don't come back But that looks even worse than using 'with protect_finally()'. - Yury

On Wed, Oct 24, 2012 at 4:26 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

On 2012-10-24, at 7:12 PM, Guido van Rossum <guido@python.org> wrote:

...

Ok, I can understand. But still, this is a problem with timeouts in general, not just with timeouts in a yield-based environment. How does e.g. Twisted deal with this?

I don't know, I hope someone with an expertise in Twisted can tell us.

But I would imagine that they don't have this particular problem, as it should be related only to coroutines and schedulers that run them. I.e. it's a problem when you run some code and may interrupt it. And you can't interrupt a plain python code that uses callbacks without yields and greenlets.

Well, but in the Twisted world, if a cleanup callback requires more blocking calls, it has to spawn more deferred callbacks. So I think they *do* have the problem, unless they don't have a way at all to constrain the total running time of an action involving cascading callbacks. Also, they have inlineCallbacks which does use yield.

...

As a work-around, I could imagine some kind of with-statement that tells the scheduler we're already in the finally clause (it could still send you a timeout if your cleanup takes way too long):

try: yield <regular code> finally: with protect_finally(): yield <cleanup code>

Of course this could be abused, but at your own risk -- the scheduler only gives you a fixed amount of extra time and then it's quits.

Right, that's the basic approach. But it also gives you a feeling of a "broken" language feature. I.e. we have coroutines, but we can not implement timeouts on top of them without making 'finally' blocks look ugly. And if we assume that you can run any coroutine with a timeout - you'll need to use 'protect_finally' in virtually every 'finally' statement.

I think the problem may be with timeouts, or with doing blocking I/O in cleanup clauses. I suspect that any system implementing timeouts has subtle bugs.

I solved the problem by dynamically inlining 'with protect_finally()' code in @coroutine decorator (something that I would never suggest to put in the stdlib, btw). There is also PEP 419, but I don't like it as well, as it is tied to frames--two low level (and I'm not sure how it will work with future CPython optimizations and PyPy's JIT.)

BUT, the concept is nice. I've implemented a number of protocols with yield-coroutines, and managing timeouts with a simple ".with_timeout()" call is a very handy and readable feature. So, I hope, that we can all brainstorm this problem to make coroutines "complete", if we decide to start using them widely.

I think the with-clause is the solution. Note that in a world with only blocking calls this *can* be a problem (despite your repeated claims that it's not a problem there) -- a common approach to giving operations a timeout is sending it a SIGTERM (which you can easily call with a signal handler in Python) when the deadline is over, then sending it more SIGTERM signals every few seconds until it dies, and sending SIGKILL (which can't be caught) if it takes too long to die. -- --Guido van Rossum (python.org/~guido)

On 2012-10-24, at 8:00 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

On 2012-10-24, at 7:43 PM, Guido van Rossum <guido@python.org> wrote:

...

On Wed, Oct 24, 2012 at 4:26 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

...

On 2012-10-24, at 7:12 PM, Guido van Rossum <guido@python.org> wrote:

...

Ok, I can understand. But still, this is a problem with timeouts in general, not just with timeouts in a yield-based environment. How does e.g. Twisted deal with this?

...

I don't know, I hope someone with an expertise in Twisted can tell us.

But I would imagine that they don't have this particular problem, as it should be related only to coroutines and schedulers that run them. I.e. it's a problem when you run some code and may interrupt it. And you can't interrupt a plain python code that uses callbacks without yields and greenlets.

Well, but in the Twisted world, if a cleanup callback requires more blocking calls, it has to spawn more deferred callbacks. So I think they *do* have the problem, unless they don't have a way at all to constrain the total running time of an action involving cascading callbacks. Also, they have inlineCallbacks which does use yield.

Right.

I was under impression that you don't just use 'finally' stmt but rather setup a Deferred with a cleanup callback. Anyways, I'm now curious enough so I'll take a look...

Well, that wasn't too hard to find: Timeouts: http://stackoverflow.com/questions/221745/is-it-possible-to-set-a-timeout-on... - Yury

On 25/10/12 12:43, Guido van Rossum wrote:

Note that in a world with only blocking calls this *can* be a problem... a common approach to giving operations a timeout is sending it a SIGTERM

Well, yes, if you have preemptive interruptions of some kind, then things are a lot trickier. But I'm assuming we're using cooperative scheduling *instead* of things like that. (Note that in the face of preemption, I don't think it's possible to solve this problem completely without language support, because there will always be a small window of opportunity between entering the finally clause and getting into the with-statement or whatever that you're using to block asynchronous signals.) -- Greg

On Wed, Oct 24, 2012 at 7:28 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

On 2012-10-24, at 9:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [...]

...

(Note that in the face of preemption, I don't think it's possible to solve this problem completely without language support, because there will always be a small window of opportunity between entering the finally clause and getting into the with-statement or whatever that you're using to block asynchronous signals.)

Agree.

In my experience, though, broken finally blocks due to interruption by a signal is a very rare thing (again, that maybe different for someone else.)

We're far from our starting point: in a the yield-from (or yield) world, there are no truly async interrupts, but anything that yields may be interrupted, if we decide to implement timeouts by throwing an exception into the generator (which seems the logical thing to do). The with-statement can deal with this fine (there's no yield between entering the finally and entering the with-block) but making the cleanup into its own task (like Steve proposed) sounds fine too. In any case this sounds like something that each framework should decide for itself. -- --Guido van Rossum (python.org/~guido)

On 2012-10-24, at 10:51 PM, Guido van Rossum <guido@python.org> wrote:

On Wed, Oct 24, 2012 at 7:28 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

...

On 2012-10-24, at 9:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [...]

...

(Note that in the face of preemption, I don't think it's possible to solve this problem completely without language support, because there will always be a small window of opportunity between entering the finally clause and getting into the with-statement or whatever that you're using to block asynchronous signals.)

Agree.

In my experience, though, broken finally blocks due to interruption by a signal is a very rare thing (again, that maybe different for someone else.)

We're far from our starting point: in a the yield-from (or yield) world, there are no truly async interrupts, but anything that yields may be interrupted, if we decide to implement timeouts by throwing an exception into the generator (which seems the logical thing to do). The with-statement can deal with this fine (there's no yield between entering the finally and entering the with-block) but making the cleanup into its own task (like Steve proposed) sounds fine too.

In any case this sounds like something that each framework should decide for itself.

BTW, is there a way of adding a read-only property to generator objects - 'in_finally'? Will it actually slow down things? - Yury

Hi Yury, On Thu, Oct 25, 2012 at 9:18 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

Well, I couldn't resist and just implemented a *proof of concept* myself. The patch is here: https://dl.dropbox.com/u/21052/gen_in_finally.patch

The patch adds 'gi_in_finally' read-only property to generator objects.

Why haven't you used my implementation? http://bugs.python.org/issue14730 -- Paul

On Thu, Oct 25, 2012 at 6:37 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

]On 2012-10-25, at 3:49 AM, Paul Colomiets <paul@colomiets.name> wrote:

...

Hi Yury,

On Thu, Oct 25, 2012 at 9:18 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

...

Well, I couldn't resist and just implemented a *proof of concept* myself. The patch is here: https://dl.dropbox.com/u/21052/gen_in_finally.patch

The patch adds 'gi_in_finally' read-only property to generator objects.

Why haven't you used my implementation?

http://bugs.python.org/issue14730

Because it's a different thing. Yours is a PEP 419 implementation -- 'sys.setcleanuphook'. Mine is a quick hack to add 'gi_in_finally' property to generators and see how good/bad it is.

I feel it's a code smell if you need to use this feature a lot. If you need it rarely, well, use one of the existing work-arounds. -- --Guido van Rossum (python.org/~guido)

...

...

Mine is a quick hack to add 'gi_in_finally' property to generators and see how good/bad it is.

I feel it's a code smell if you need to use this feature a lot. If you need it rarely, well, use one of the existing work-arounds. But the feature isn't going to be used by users directly. It will be used only in scheduler implementations. Users will just write 'finally' blocks and they will work as expected. This just makes coroutines look and behave more like ordinary functions. Isn't it one of our goals--to make it convenient and reliable?

I'm agree with the intent, but I'm more worried about the broadness of this approach. What happens in this case? try: try: yield some_op() finally: yield cleanup_that_raises_network_error() except NetworkError: # will we ever see this? Basically, I don't think we can handle the "don't raise" cases entirely automatically, though I'd like to be able to. Cheers, Steve

...

...

...

...

Mine is a quick hack to add 'gi_in_finally' property to generators and see how good/bad it is.

I feel it's a code smell if you need to use this feature a lot. If you need it rarely, well, use one of the existing work-arounds. But the feature isn't going to be used by users directly. It will be used only in scheduler implementations. Users will just write 'finally' blocks and they will work as expected. This just makes coroutines look and behave more like ordinary functions. Isn't it one of our goals--to make it convenient and reliable?

I'm agree with the intent, but I'm more worried about the broadness of this approach. What happens in this case?

try: try: yield some_op() finally: yield cleanup_that_raises_network_error() except NetworkError: # will we ever see this?

Basically, I don't think we can handle the "don't raise" cases entirely automatically, though I'd like to be able to.

We can. You can experiment with the approach--I've implemented it a bit differently and it proved to work. Now we're just talking about making this feature supported on the interpreter level.

As for your example - I'm not sure what's the NetworkError is and how it relates to TimeoutError...

But if you have something like this:

try: try: yield some_op().with_timeout(0.1) finally: yield something_else() except TimeoutError: # Then everything would be just fine here.

Look, it all the same as if you just drop yields. Generators already support 'finally' clause perfectly.

The type of the error is irrelevant - if something_else() might raise an exception that is expected, it won't be passed in because the scheduler is suppressing exceptions inside finally blocks. Or perhaps I've misunderstood the point of gi_in_finally? Cheers, Steve

Yuri, please give up this particular issue (trying to patch CPython to record whether a generator is in a finally clause). I have failed to explain my reasons why I think it is a bad idea, but you haven't convinced me it's a good idea, and we have at least two decent work-arounds. So let me just use the release cycle as an argument: your patch is a new feature, 3.3 just came out, so it cannot be introduced until 3.4. I don't want to wait for that. -- --Guido van Rossum (python.org/~guido)

One question: what do we actually want to get? What're the goals?

- A specification (PEP?) of how to make stdlib more async-friendly?

- To develop a separate library that may be included in the stdlib one day?

- And what's your opinion on writing a PEP about making it possible to pass a custom socket-factory to stdlib objects?

I'm (and I think it's not just me) a bit lost here, after reading 100s of emails on python-ideas. And I just want to know where to channel my energy and expertise ;)

It's not just you, I'm not entirely clear on what we expect to end up with either. My current view is that we'll get a PEP that defines a convention for user code and an interface for schedulers. Adding *_async() methods to the entire standard library could take a long time and should probably be divided up so we can have really experienced devs on particular areas (e.g. someone on Windows sockets, someone else on Linux sockets, etc.) and may need individual PEPs. My hope is that the first PEP provides a protocol for users to defer the rest of a task until after some/any operation has completed - I don't really want sockets/networking/files/threads/etc. to leak through at all, though these are all important use cases that need to be tried. This is the way I'm approaching it, so please let me know if I'm off the mark :) Cheers, Steve

Guido, Thank you for such a detailed and deep response. Lots of good thoughts to digest. One idea: the scope of the problem is enormously big. It may take months/years to synchronize all ideas and thoughts by just communicating ideas over mail list without a concrete thing and subject to discuss. How about you/we create a repository with a draft implementation of scheduler/io loop/coroutines engine and we simply start tweaking an discussing that particular design? That way people will see where to start the discussion, what's done, and some will even participate? The goal is not to write a production-quality software, but rather to have a common place to discuss/try things/benchmark etc. I'm not sure, but maybe places like bitbucket, where you can have a wiki, issues, and the actual code is a better place, than a mail-list. I also think that there's need to move concurrency-related discussions to a separate mail-list, as everything else on python-ideas is lost now. On 2012-10-25, at 1:58 PM, Guido van Rossum <guido@python.org> wrote: [...]

...

- And what's your opinion on writing a PEP about making it possible to pass a custom socket-factory to stdlib objects?

That sounds like it might be jumping to a specific solution. I agree that the stdlib often, unfortunately, couples classes too tightly, where a class that needs an instance of another class just instantiates that other class rather than having an instance passed in (at least as an option). We're doing better with files these days -- most APIs (that I can think of) that work with streams let you pass one in. So maybe you're on to something. Perhaps, as a step towards the exploration of this PEP, you could come up with a concrete list of modules and classes (or other API elements) that you think would benefit from being able to pass in a socket? Please start another thread -- python-ideas is fine. I will read it.

OK, I will, in a week or two. Need some time for a research. [...]

- For the task scheduler I am piling all my hopes on PEP-380, i.e. yield from. I have not found a single thing that is harder to do using this style than using the PEP-342 yield <future> style, and I really don't like mixing the two up (despite what Steve Dower says :-). But I don't want the event loop interface to know about this at all -- howver the scheduler has to know about the event loop (at least its interface). I am currently refactoring my ideas in this area; I think I'll end up with a Task object that smells a bit like a Future, but represents a whole stack of generator invocations linked via yield-from, and which allows suspension of the entire stack at once; user code only needs to use Tasks when it wants to schedule multiple activities concurrently, not when it just wants to be able to yield. (This may be the core insight in favor of PEP 380.)

The only problem I have with PEP-380, is that to me it's not entirely clear when you should use 'yield' or 'yield from' (please correct me if I am wrong). I'll try to demonstrate it by example: class Socket: def sendall(self, payload): f = Future() IOLoop.sendall(payload, future=f) return f class SMTP: def send(self, s): ... # yield the returned future to the scheduler yield self.sock.sendall(s) ... # And later: s = SMTP() yield from s.send('spam') Is it (roughly) how you want it all to look like? I.e. using 'yield' to send a future/task to the scheduler, and 'yield from' to delegate? If I guessed correctly, and that's how you envision it, I have a question: What if you decide to refactor 'Socket.sendall' to be a coroutine? In that case you'd want users to call it 'yield from Socket.sendall', and not 'yield Socket.sendall'. Thank you, Yury

On 2012-10-25, at 3:25 PM, Guido van Rossum <guido@python.org> wrote:

On Thu, Oct 25, 2012 at 11:39 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

[...]

...

One idea: the scope of the problem is enormously big. It may take months/years to synchronize all ideas and thoughts by just communicating ideas over mail list without a concrete thing and subject to discuss. How about you/we create a repository with a draft implementation of scheduler/io loop/coroutines engine and we simply start tweaking an discussing that particular design? That way people will see where to start the discussion, what's done, and some will even participate? The goal is not to write a production-quality software, but rather to have a common place to discuss/try things/benchmark etc. I'm not sure, but maybe places like bitbucket, where you can have a wiki, issues, and the actual code is a better place, than a mail-list.

I am currently working on code. Steve Dower has also said he's going to write some code. I'm just not quite ready to show my code (I need to do a few more iterations on each component). As long as I can use Mercurial I'm happy; bitbucket or Google Code Hosting both work fine for me.

OK. Let's wait until we have a somewhat stable platform to work with. [...]

...

Is it (roughly) how you want it all to look like? I.e. using 'yield' to send a future/task to the scheduler, and 'yield from' to delegate?

I think that's the style that Steve Dower prefers. Greg Ewing would rather see all public APIs use yield from, and reserve plain yield exclusively as an implementation detail of the scheduler. In my own experimental code I am using Greg's style and it is working out great. My main reason for taking a hard stance on this is that it would otherwise be too confusing for users -- should they use yield, yield from, or a plain call? I'd like to tell them "if it blocks, use yield from".

BTW, if you haven't read Greg's introduction to this style, here it is -- worth reading! http://www.cosc.canterbury.ac.nz/greg.ewing/python/tasks/SimpleScheduler.htm...

...

If I guessed correctly, and that's how you envision it, I have a question: What if you decide to refactor 'Socket.sendall' to be a coroutine? In that case you'd want users to call it 'yield from Socket.sendall', and not 'yield Socket.sendall'.

That's why using yield from all the way is better!

Yes, that now makes sense! I'll definitely take a look at Greg's article. Thanks, Yury

On 2012-10-25, at 4:39 PM, Terry Reedy <tjreedy@udel.edu> wrote:

On 10/25/2012 12:10 PM, Yury Selivanov wrote:

...

- And what's your opinion on writing a PEP about making it possible to pass a custom socket-factory to stdlib objects?

I think this is probably a good idea quite aside from async issues. For one thing, it would make testing with a mock-socket class easier. Issues to decide: name of parameter (should be same for all socket using classes); keyword only? (ditto).

Right, good catch on mocking sockets! As for the issues: I think that the parameter name should be the same/very consistent, and surely keyword-only.

I am not sure this needs a PEP. Most parameter additions are just tracker issues. But I would be worthwhile to decide on the details here first.

We'll see. I'll start with a detailed post on python-ideas, and if the PEP looks like an overkill - I'd be glad to skip the PEP step. Thanks, Yury

Please start a new thread for this sub-topic. Note that for mocking, you won't need to pass in a socket object; you can just mock out socket.socket() directly using Michael Foord's all-singing all-dancing unittest.mock module (now in the Python 3 stdlib). On Thu, Oct 25, 2012 at 2:06 PM, Terry Reedy <tjreedy@udel.edu> wrote:

On 10/25/2012 4:51 PM, Yury Selivanov wrote:

...

On 2012-10-25, at 4:39 PM, Terry Reedy <tjreedy@udel.edu> wrote:

...

On 10/25/2012 12:10 PM, Yury Selivanov wrote:

...

- And what's your opinion on writing a PEP about making it possible to pass a custom socket-factory to stdlib objects?

I think this is probably a good idea quite aside from async issues. For one thing, it would make testing with a mock-socket class easier. Issues to decide: name of parameter (should be same for all socket using classes); keyword only? (ditto).

Right, good catch on mocking sockets!

As for the issues: I think that the parameter name should be the same/very consistent, and surely keyword-only.

I left out the following issue: should the argument be a socket-returning callable (a 'socket-factory' as you called it above) or an opened socket?

For files, we variously pass file names to be used with the default opener, opened files, and file descriptors, but never an alternate opener (such as StringIO). One reason is the the user typically needs a handle on the file object in order to later retrieve the contents.

I am not sure that the same applies to sockets. If I ask the ftp module to get or send a file, I should not ever need to see the socket used for the transport.

-- Terry Jan Reedy

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

Steve Dower wrote:

The type of the error is irrelevant - if something_else() might raise an exception that is expected, it won't be passed in because the scheduler is suppressing exceptions inside finally blocks. Or perhaps I've misunderstood the point of gi_in_finally?

IIUC, it's only *asynchronous* exceptions that would be blocked -- i.e. ones thrown in from a different task, or arising from an external event such as a timeout. An exception raised explicity by the task's own code would be unaffected. -- Greg

Hi Guido, On Thu, Oct 25, 2012 at 2:43 AM, Guido van Rossum <guido@python.org> wrote:

On Wed, Oct 24, 2012 at 4:26 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

...

On 2012-10-24, at 7:12 PM, Guido van Rossum <guido@python.org> wrote:

...

Ok, I can understand. But still, this is a problem with timeouts in general, not just with timeouts in a yield-based environment. How does e.g. Twisted deal with this?

...

I don't know, I hope someone with an expertise in Twisted can tell us.

But I would imagine that they don't have this particular problem, as it should be related only to coroutines and schedulers that run them. I.e. it's a problem when you run some code and may interrupt it. And you can't interrupt a plain python code that uses callbacks without yields and greenlets.

Well, but in the Twisted world, if a cleanup callback requires more blocking calls, it has to spawn more deferred callbacks. So I think they *do* have the problem, unless they don't have a way at all to constrain the total running time of an action involving cascading callbacks. Also, they have inlineCallbacks which does use yield.

AFAIR, in twisted there is no timeout on coroutine, there is a timeout on request, which is usually just a socket timeout. So there is no problem of interrupting the code in arbitrary places. Another twisted thing, is doing all writes asynchronously with respect to user code, so if you want to write something and close a connection for finalization you just call: transport.write('something') transport.loseConnection() And they do not return deferreds, so it returns immediately even if the socket is not writable at the moment. (IIRC, it never writes right now, but rather from reactor callback) -- Paul

On Thu, Oct 25, 2012 at 4:46 AM, Laurens Van Houtven <_@lvh.cc> wrote:

Sorry, working really long hours these days; just wanted to chime in that yes, you can call transport.write with large strings, and the reactor will do the right thing under the hood: loseConnection is the polite way of dropping a connection, which should wait for all pending writes to finish etc.

This seems a decent enough pattern. It also makes it possible to use one of these things as a substitute for a writable file object, so you can e.g. use it as sys.stdout or the stream for a logging.StreamHandler. Still, I wonder what happens if the socket/pipe/whatever that is written to is very slow and the program produces too much data. Does memory just balloon up, or is there some kind of throttling of the writer? Or a buffer overflow exception? For a totally general solution I would at least like to have the *option* of doing synchronous writes. (I'm asking these questions because I'd like to copy this useful pattern -- but I want to get the end cases right.) -- --Guido van Rossum (python.org/~guido)

On Wed, Oct 24, 2012 at 7:43 PM, Guido van Rossum <guido@python.org> wrote:

...

I don't know, I hope someone with an expertise in Twisted can tell us.

But I would imagine that they don't have this particular problem, as it should be related only to coroutines and schedulers that run them. I.e. it's a problem when you run some code and may interrupt it. And you can't interrupt a plain python code that uses callbacks without yields and greenlets.

Well, but in the Twisted world, if a cleanup callback requires more blocking calls, it has to spawn more deferred callbacks. So I think they *do* have the problem, unless they don't have a way at all to constrain the total running time of an action involving cascading callbacks. Also, they have inlineCallbacks which does use yield.

Deferreds don't do anything to prevent blocking. They're just a nice abstraction for callbacks. And yes, if you call 1000 functions that do lots of CPU in a row, that will keep other stuff from happening. However, consider how a timeout works: the event loop notices enough time has passed, and so calls some code that tells the Deferred to cancel its operation. So you're *not* adding the cancellation operations to the stack of the original operation, you're starting from the event loop. And so timeouts are just normal event loop world, where you need to be careful not to do to much CPU-intensive processing in any given call, and you can't call blocking system calls (except using a thread). Of course, you can't timeout a function that's just looping using CPU, or a blocking system call, and so code needs to be structured to deal with this, but that's a different issue.

There's an exception for "a deferred has been cancelled". Cancelling a deferred fires that down its errback chain just like any exception. Since @inlineCallbacks works on top of deferreds, it magically works:

...

...

from twisted.internet import defer d = defer.Deferred() @defer.inlineCallbacks ... def f(): ... yield d ... r = f() r <Deferred at 0x1019df950> d.cancel() r <Deferred at 0x1019df950 current result: <twisted.python.failure.Failure <class 'twisted.internet.defer.CancelledError'>>>

On Fri, Oct 26, 2012 at 5:25 PM, Guido van Rossum <guido@python.org> wrote:

On Fri, Oct 26, 2012 at 7:12 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:

...

On Wed, Oct 24, 2012 at 7:43 PM, Guido van Rossum <guido@python.org>

...

...

...

I don't know, I hope someone with an expertise in Twisted can tell us.

But I would imagine that they don't have this particular problem, as

it

...

...

should be related only to coroutines and schedulers that run them. I.e. it's a problem when you run some code and may interrupt it. And you can't interrupt a plain python code that uses callbacks without yields and greenlets.

Well, but in the Twisted world, if a cleanup callback requires more blocking calls, it has to spawn more deferred callbacks. So I think they *do* have the problem, unless they don't have a way at all to constrain the total running time of an action involving cascading callbacks. Also, they have inlineCallbacks which does use yield.

Deferreds don't do anything to prevent blocking. They're just a nice abstraction for callbacks. And yes, if you call 1000 functions that do lots of CPU in a row, that will keep other stuff from happening.

However, consider how a timeout works: the event loop notices enough time has passed, and so calls some code that tells the Deferred to cancel its operation. So you're *not* adding the cancellation operations to the stack of the original operation, you're starting from the event loop. And so timeouts are just normal event loop world, where you need to be careful not to do to much CPU-intensive processing in any given call, and you can't call blocking system calls (except using a thread).

Of course, you can't timeout a function that's just looping using CPU, or a blocking system call, and so code needs to be structured to deal with

wrote: this,

...

but that's a different issue.

So, basically, it's just "after T seconds you get this second callback and it's up to you to deal with it"? I guess the timeout callback can inspect the state of the operation, and cancel any pending operations?

Do you have a way to translate timeouts into exceptions in inlineCallbacks? If so, how is that working out?

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- cheers lvh

On Fri, Oct 26, 2012 at 8:52 AM, Laurens Van Houtven <_@lvh.cc> wrote:

err, I suppose the missing bit there is that you'll probably want to:

reactor.callLater(timeout, d.cancel)

As opposed to calling d.cancel() directly. (That snippet was in bpython-urwid with the reactor running in the background, but I doubt it'd work well anywhere else outside of manholes :))

So I think that Yuri's original problem statement, transformed to Twisted+Deferred, might still apply, depending on how you implement it. Yuri essentially did this: def foobar(): # a task try: yield <blocking action> finally: # must clean up regardless of whether action succeeded or failed: yield <blocking cleanup> He then calls this with a timeout, with the semantics that if the generator is blocked in a yield when the timeout arrives, that yield raises a Timeout exception (and at no other time is Timeout raised). The problem with this is that if the action succeeds within the timeout, but barely, there's a chance that the cleanup of a *successful* action receives the Timeout exception. Apparently this bit Yuri. I'm not sure how you'd model that using just Deferreds, but using inlineCallbacks it seems the same thing might happen. Using Deferreds, I assume there's a common pattern to implement this that doesn't have this problem. Of course, using coroutines, there is too -- spawn the cleanup as an independent task. -- --Guido van Rossum (python.org/~guido)

On 2012-10-26, at 12:57 PM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:

On Fri, Oct 26, 2012 at 12:36 PM, Guido van Rossum <guido@python.org> wrote: On Fri, Oct 26, 2012 at 8:52 AM, Laurens Van Houtven <_@lvh.cc> wrote:

...

err, I suppose the missing bit there is that you'll probably want to:

reactor.callLater(timeout, d.cancel)

As opposed to calling d.cancel() directly. (That snippet was in bpython-urwid with the reactor running in the background, but I doubt it'd work well anywhere else outside of manholes :))

So I think that Yuri's original problem statement, transformed to Twisted+Deferred, might still apply, depending on how you implement it. Yuri essentially did this:

def foobar(): # a task try: yield <blocking action> finally: # must clean up regardless of whether action succeeded or failed: yield <blocking cleanup>

He then calls this with a timeout, with the semantics that if the generator is blocked in a yield when the timeout arrives, that yield raises a Timeout exception (and at no other time is Timeout raised). The problem with this is that if the action succeeds within the timeout, but barely, there's a chance that the cleanup of a *successful* action receives the Timeout exception. Apparently this bit Yuri. I'm not sure how you'd model that using just Deferreds, but using inlineCallbacks it seems the same thing might happen. Using Deferreds, I assume there's a common pattern to implement this that doesn't have this problem. Of course, using coroutines, there is too -- spawn the cleanup as an independent task.

If you call cancel() on a Deferred that already has a result, nothing happens. So you don't get a TimeoutError if the operation has succeeded (or failed some other way). This would also be true when using inlineCallbacks, so there's no issue.

In general I'm not clear why this is a problem: in a single-threaded program only one thing happens at a time. Your code for triggering a timeout always has the option to check if the operation has succeeded, without worrying about race conditions.

Let me ask you a question that may help me and others to understand how inlineCallbacks works. If you write the following: def func(): try: yield one_thing() yield and_another() finally: yield and_finally() Then each of those yields will create a separate Deferred object, that 'inlineCallbacks' transparently dispatches via generator send/throw, right? And if you 'yield func()' the same will happen--'inlineCallbacks' will return a Deferred, that will have a result of 'func' execution? Thanks, Yury

On 2012-10-26, at 1:14 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:

On Fri, Oct 26, 2012 at 1:06 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

... snip ...

...

Let me ask you a question that may help me and others to understand how inlineCallbacks works.

If you write the following:

def func(): try: yield one_thing() yield and_another() finally: yield and_finally()

Then each of those yields will create a separate Deferred object, that 'inlineCallbacks' transparently dispatches via generator send/throw, right?

one_thing() and and_another() and and_finally() should return Deferreds. inlineCallbacks gets those Deferreds, adds callbacks for completion/error, and resumes the generator at the appropriate time. You don't use the results from either Deferreds, so the values will just be thrown out. The yield/trampoline doesn't create any Deferreds for those operations itself.

...

And if you 'yield func()' the same will happen--'inlineCallbacks' will return a Deferred, that will have a result of 'func' execution?

You didn't decorate func with inlineCallbacks, but if you do, func() will give you Deferred. Note that func itself doesn't return any value. In Twisted land, this is done by defer.returnValue(), which uses exceptions to return a value to the trampoline. This maps well to the new sugar in 3.3.

Right, I forgot to decorate the 'func' with 'inlineCallbacks'. If it is decorated, though, how can I invoke it with a timeout? - Yury

On Fri, Oct 26, 2012 at 9:57 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:

On Fri, Oct 26, 2012 at 12:36 PM, Guido van Rossum <guido@python.org> wrote:

...

On Fri, Oct 26, 2012 at 8:52 AM, Laurens Van Houtven <_@lvh.cc> wrote:

...

err, I suppose the missing bit there is that you'll probably want to:

reactor.callLater(timeout, d.cancel)

As opposed to calling d.cancel() directly. (That snippet was in bpython-urwid with the reactor running in the background, but I doubt it'd work well anywhere else outside of manholes :))

So I think that Yuri's original problem statement, transformed to Twisted+Deferred, might still apply, depending on how you implement it. Yuri essentially did this:

def foobar(): # a task try: yield <blocking action> finally: # must clean up regardless of whether action succeeded or failed: yield <blocking cleanup>

He then calls this with a timeout, with the semantics that if the generator is blocked in a yield when the timeout arrives, that yield raises a Timeout exception (and at no other time is Timeout raised). The problem with this is that if the action succeeds within the timeout, but barely, there's a chance that the cleanup of a *successful* action receives the Timeout exception. Apparently this bit Yuri. I'm not sure how you'd model that using just Deferreds, but using inlineCallbacks it seems the same thing might happen. Using Deferreds, I assume there's a common pattern to implement this that doesn't have this problem. Of course, using coroutines, there is too -- spawn the cleanup as an independent task.

If you call cancel() on a Deferred that already has a result, nothing happens. So you don't get a TimeoutError if the operation has succeeded (or failed some other way). This would also be true when using inlineCallbacks, so there's no issue.

In general I'm not clear why this is a problem: in a single-threaded program only one thing happens at a time. Your code for triggering a timeout always has the option to check if the operation has succeeded, without worrying about race conditions.

But the example is not single-threaded (in the informal sense that you use it here). Each yield is a suspension point where other things can happen, and one of those things could be a cancellation of *this* task (because of a timeout or otherwise). The example would have to set some flag indicating it has a result after the first yield (i.e. before entering the finally, or at least before yielding in the finally clause). And the timeout callback would have to check this flag. This makes it slightly awkward to design a general-purpose timeout mechanism for tasks written in this style -- if you expect a timeout or cancellation you must protect your cleanup code from it by using some API. Anyway, no need to respond: I think I understand how Twisted deals with this, and translating that into the world of PEP 380 is not your job. -- --Guido van Rossum (python.org/~guido)

On 2012-10-27, at 7:21 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Guido van Rossum wrote:

...

The example would have to set some flag indicating it has a result after the first yield (i.e. before entering the finally, or at least before yielding in the finally clause). And the timeout callback would have to check this flag. This makes it slightly awkward to design a general-purpose timeout mechanism for tasks written in this style -- if you expect a timeout or cancellation you must protect your cleanup code from it by using some API.

This is where having a way to find out whether a generator is in a finally clause would help. It would allow the scheduler to take care of this transparently.

Right. But now I'm not sure this approach will work with yield-froms. As when you yield-fromming scheduler knows nothing about the chain of generators, as it's all hidden in the yield-from implementation. - Yury

On 2012-10-27, at 7:52 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

But now I'm not sure this approach will work with yield-froms. As when you yield-fromming scheduler knows nothing about the chain of generators, as it's all hidden in the yield-from implementation.

I think this just means that the implementation would involve more than looking at a single bit. Something like an in_finally() method that looks along the yield-from chain and returns true if any of the generators are in a finally section.

That would not be a solution either. Imagine that we have two coroutines: @coroutine def c1(): try: yield c2().with_timeout(1.0) # p1 finally: try: yield c2().with_timeout(1.0) # p2 except TimeoutError: pass @coroutine def c2(): try: yield c3().with_timeout(2.0) # p3 finally: yield c4() # p4 In the above example scheduler *can* safely interrupt "c2" when it is invoked from "c1" at "p2". I.e. scheduler can't interrupt the coroutine when it is itself in its finally statement, but it's fine to interrupt it when it is not, even if it is invoked from other coroutine's finally block. If you translate this example in yield-from form, then checking 'in_finally()' result on "c1" when it is at "p2" will prevent you to raise TimeoutError, but you clearly should. In other words, we want coroutines behaviour to be closer to the regular python code. - Yury

On 2012-10-28, at 1:55 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

In the above example scheduler *can* safely interrupt "c2" when it is invoked from "c1" at "p2". I.e. scheduler can't interrupt the coroutine when it is itself in its finally statement, but it's fine to interrupt it when it is not, even if it is invoked from other coroutine's finally block.

I'm confused about the relationship between c1 and c2 here, and what you mean by one coroutine "invoking" another.

Can you post a version that uses yield-from instead of yielding objects with unknown (to me) semantics?

The reason I kept using my version is because I'm not sure how we will set timeouts for yield-from style coroutines. But let's assume that we can do that with a context manager. Let's also assume that generator object has 'in_finally()' method, as you defined: "Something like an in_finally() method that looks along the yield-from chain and returns true if any of the generators are in a finally section." def coro1(): try: with timeout(1.0): yield from coro2() # 1 finally: try: with timeout(1.0): yield from coro2() # 2 except TimeoutError: pass def coro2(): try: block() yield # 3 action() finally: block() yield # 4 another_action() Now, if "coro2" is suspended at #4 -- it shouldn't be interrupted with TimeoutError. If, however, "coro2" is at #3 -- it can be, and it doesn't matter was it called from #1 or #2. IIUC, yield-from supporting scheduler, won't know about "coro2". All it will have is a generator for "coro1". All dispatching will be handled by "yield from" statement automatically. In this case, you can't rely on "coro1.in_finally()", because it will return: - True, when "coro1" is at #1 & "coro2" is at #4 (it's unsafe to interrupt) - True, when "coro1" is at #2 & "coro2" is at #3 (safe to interrupt) The fundamental problem here, is that scheduler knows nothing about coroutines call chain. It doesn't even know at what generator 'with timeout' was called. - Yury

On 2012-10-29, at 1:05 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

def coro1(): try: with timeout(1.0): yield from coro2() # 1 finally: try: with timeout(1.0): yield from coro2() # 2 except TimeoutError: pass def coro2(): try: block() yield # 3 action() finally: block() yield # 4 another_action() Now, if "coro2" is suspended at #4 -- it shouldn't be interrupted with TimeoutError. If, however, "coro2" is at #3 -- it can be, and it doesn't matter was it called from #1 or #2.

What is your reasoning behind asserting this? Because it's inside a try block of its own? Because it's subject to a nested timeout? Something else?

Because scheduler, when it is deciding to interrupt a coroutine or not, should only question whether that particular coroutine is in its finally, and not the one which called it. - Yury

On 2012-10-29, at 8:06 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

Because scheduler, when it is deciding to interrupt a coroutine or not, should only question whether that particular coroutine is in its finally, and not the one which called it.

So given this:

def c1(): try: something() finally: yield from c2() very_important_cleanup()

def c2(): yield from block() # 1

it should be okay to interrupt at point 1, even though it will prevent very_important_cleanup() from being done?

That doesn't seem right to me.

-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Yury Selivanov wrote:

So in your example scheduler would never ever has a question of interrupting c2(), because it wasn't called with any restriction/timeout. There simply no reason to interrupt it ever.

But there's nothing to stop someone writing def c3(): try: yield from with_timeout(10.0, c1()) except TimeoutError: print("That's cool, I can cope with that") Also, it's not just TimeoutErrors that are a potential problem, it's any asynchronous exception. For example, the task calling c1() might get cancelled by another task while c2() is blocked. If cancelling is implemented by throwing in an exception, you have the same problem.

Then you need scheduler to know if it is in its finally or not. Because its c2() which was run with a timeout. It's c2() code that may be subject to aborting.

I'm really not following your reasoning here. You seem to be speaking as if with_timeout() calls only have an effect one level deep. But that's not the case -- the frame that a TimeoutError gets thrown into by with_timeout() can be nested any number of yield-from calls deep. -- Greg

Guido, Greg, On 2012-10-27, at 7:45 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:

Right. But now I'm not sure this approach will work with yield-froms. As when you yield-fromming scheduler knows nothing about the chain of generators, as it's all hidden in the yield-from implementation.

I think I've come up with a solution that should work for yield-froms too (if we accept my in_finally idea in 3.4). And there should be a way of writing a 'protect_finally' context manager too. I'll illustrate the approach on Guido's tulip micro-framework (consider it a pseudo code to illustrate the idea): class Interrupt(BaseException): """Should penetrate all try..excepts""" def call_with_timeout(timeout, gen): context.current_task._add_timeout(timeout, gen) try: return (yield from gen) except Interrupt: raise TimeoutError() from None class Task: def _add_timeout(timeout, gen): self.eventloop.call_later( timeout, partial(self._interrupt, gen)) def _interrupt(self, gen): if not gen.in_finally: gen.throw(Interrupt, Interrupt(), None) else: # So we set a flag to watch for gen's in_finally value # on each 'step' call. And when it's 0 - Task.step # will call '_interrupt' again. self._watch_finally(gen) I defined a new function 'call_with_timeout', because tulip's 'with_timeout' starts a new Task, whereas the former works in any generator inside the task. So, after that you'd be able to do the following: yield from call_with_timeout(1.0, something()) And something's 'finally' won't ever be aborted. - Yury

On 25/10/12 12:12, Guido van Rossum wrote:

Of course this could be abused, but at your own risk -- the scheduler only gives you a fixed amount of extra time and then it's quits.

Which is another good reason to design your cleanup code so that it can't take an arbitrarily long time. -- Greg

Greg, On 2012-10-24, at 5:30 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Yury Selivanov wrote:

...

It's not about try..finally nesting, it's about Scheduler being aware that a coroutine is in its 'finally' block and thus shouldn't be interrupted at the moment

It would be a bad idea to make a close() method, or anything else that might be needed for cleanup purposes, be a 'yield from' call. If it's an ordinary function, it can't be interrupted in the world we're talking about, so the PEP 419 problem doesn't apply.

If I were feeling in a radical mood, I might go as far as suggesting that 'yield' and 'yield from' be syntactically forbidden inside a finally clause. That would force you to design your cleanup code to be safe from interruptions.

I'm not sure it would be a good idea... Cleanup code for a DB connection *will* need to run queries to the database (at least in some circumstances). And we can't make them blocking. - Yury

On 2012-10-24, at 8:52 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 25/10/12 11:47, Yury Selivanov wrote:

...

Cleanup code for a DB connection *will* need to run queries to the database (at least in some circumstances).

That smells like a design problem to me. If something goes wrong, the most you should have to do is roll back any transactions you were in the middle of. Trying to perform further queries is just inviting more trouble.

Right. And that rolling back - a tiny db query "rollback" - is an async code, and where there is an async code, no matter how tiny and fast, - scheduler has an opportunity to screw it up. Guido's 'with protected_finally' should work, although it probably will look weird for for people unfamiliar with coroutines and this particular problem. - Yury

Greg, On 2012-10-24, at 9:29 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 25/10/12 14:07, Yury Selivanov wrote:

...

Right. And that rolling back - a tiny db query "rollback" - is an async code,

Only if we implement it as a blocking operation as far as our task scheduler is concerned. I wouldn't do it that way -- I'd perform it synchronously and assume it'll be fast enough for that not to be a problem.

In a non-blocking application there is no way of running a blocking code, even if it's anticipated to block for a mere millisecond. Because if something gets out of control and it blocks for a longer period of time - everything just stops, right? Or did you mean something else with "synchronously" (perhaps Steve Dower's approach)?

BTW, we seem to be using different definitions for the term "query". To my way of thinking, a rollback is *not* a query, even if it happens to be triggered by sending a "rollback" command to the SQL interpreter. At the Python API level, it should appear as a distinct operation with its own method.

Right. I meant that "sending a rollback command to the SQL interpreter" part--this should be done through a non-blocking socket. To invoke an operation on a non-blocking socket we have to do it through 'yield' or 'yield from', hence - give scheduler a chance to interrupt the coroutine. Given the fact that we know, that the clean-up code should be simple and fast, it still contains coroutine context switches in real world code, be it due to the need of sending some information via a socket, or just by calling some other coroutine. If you write a single 'yield' in your finally block, and that (or caller) coroutine is called with a timeout, there is a chance that its 'finally' block execution will be aborted by a scheduler. Writing this yield/non-blocking type of code in finally blocks is a necessity, unfortunately. And even if that cleanup code is incredibly fast, if you have a webserver that runs for days/weeks/months, bad things will happen. So if we decide to adopt Guido's approach with explicitly marking critical finally blocks (well, they are all critical) with 'with protected_finally()' - allright. If we somehow invent a mechanism that would allow us to hide this all from user and protect finally blocks implicitly in scheduler - that's even better. Or we should design a totally different approach of handling timeouts, and try to not to interrupt coroutines at all. - Yury

On 24.10.12 01:00, Sam Rushing wrote:

On 10/23/12 3:05 PM, Yury Selivanov wrote: ...

...

There is only one way to 'magically' make existing code both sync- & async- friendly--greenlets, but I think there is no chance for them (or stackless) to land in cpython in the foreseeable future (although it would be awesome.)

BTW, why didn't you use greenlets in shrapnel and ended up with your own implementation? I think shrapnel predates greenlets... some of the core asm code for greenlets may have come from one of shrapnel's precursors at ironport... Unfortunately it took many years to get shrapnel open-sourced - I remember talking with Guido about it over lunch in ~2006.

Hi Sam, greenlets were developed in 2004 by Armin Rigo, on the first (and maybe only) Stackless sprint here in Berlin. The greenlet asm code was ripped out of Stackless and slightly improved, but has the same old stack-slicing idea. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

-----Original Message----- From: Python-ideas [mailto:python-ideas- bounces+kristjan=ccpgames.com@python.org] On Behalf Of Sam Rushing Sent: 23. október 2012 23:01 To: Yury Selivanov Cc: python-ideas@python.org Subject: Re: [Python-ideas] Async API

In shrapnel it is simply:

coro.with_timeout (<seconds>, <fun>, *args, **kwargs)

Timeouts are caught thus:

try: coro.with_timeout (...) except coro.TimeoutError: ...

Hi Sam ( I rember our talk about Shrapnel here at CCP some years back) , others: Jumping in here with some random stuff, in case anyone cares: A few years ago, I started trying to create a standard library for stackless python. we use it internally at ccp and it is open source, at https://bitbucket.org/krisvale/stacklesslib What it provides is 1) some utility classes for stackless (context managers mostly) but also synchronization primitives. 2) a basic "main" functionality: A main loop and an event scheduler 3) a set of replacement modules for threading/socket, etc 4) Monkeypatching tools, to monkeypatch in the replacements, and even run monkeypatched scripts. On the basis of the event scheduler, I also implemented timeout for socket.receive() functions. These used to allow e.g. timeouts for locking operations Timeouts are indeed implemented as exceptions raised. There are some minor race issues to think about but that's it. Notice the need for a stacklesslib.main module. The issue I have found with this sort of event driven model, is that composability suffers when everyone has their own idea about what a "main" loop should be. In threaded programming, the OS provides the main loop and the event scheduler. For something like Python, a whole application has to agree on what the main loop is, and how to schedule future events. Hopefully this discussion is an attempt to settle that in a standard manner. Cheers, Kristján p.s. stacklesslib is in a state of protracted and procrastinated development. I promised that I would fix it up at last pycon. Mostly I'm working on restructuring and making the main loop work more "out of the box."

On 2012-10-26, at 6:27 AM, Sturla Molden <sturla@molden.no> wrote:

On 23.10.2012 21:33, Yury Selivanov wrote:

...

topics = FE.select([ FE.publication_date, FE.body, FE.category, (FE.creator, [ (FE.creator.subject, [ (gpi, [ gpi.avatar ]) ]) ]) ]).filter(FE.publication_date< FE.publication_date.now(), FE.category == self.category)

Why use Python when you clearly want Java?

And why do you think so? ;) - Yury