[Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
Hello. I've read the PEP and some things raise questions in my consciousness. Here they are. 1. Series of sock_ methods can be organized into a wrapper around sock object. This wrappers can then be saved and used later in async-aware code. This way code like: sock = socket(...) # later, e.g. in connect() yield from tulip.get_event_loop().sock_connect(sock, ...) # later, e.g. in read() data = yield from tulip.get_event_loop().sock_recv(sock, ...) will look like: sock = socket(...) async_sock = tulip.get_event_loop().wrap_socket(sock) # later, e.g. in connect() yield from async_sock.connect(...) # later, e.g. in read() data = yield from async_sock.recv(...) Interface looks cleaner while plain calls (if they ever needed) will be only 5 chars longer. 2. Not as great, but still possible to wrap fd in similar way to make interface simpler. Instead of: add_reader(fd, callback, *args) remove_reader(fd) We can do: wrap_fd(fd).reader = functools.partial(callback, *args) wrap_fd(fd).reader = None # or del wrap_fd(fd).reader 3. Why not use properties (or fields) instead of methods for cancelled, running and done in Future class? I think, it'll be easier to use since I expect such attributes to be accessed as properties. I see it as some javaism since in Java Future have getters for this fields but they are prefixed with 'is'. 4. Why separate exception() from result() for Future class? It does the same as result() but with different interface (return instead of raise). Doesn't this violate the rule "There should be one obvious way to do it"? 5. I think, protocol and transport methods' names are not easy or understanding enough: - write_eof() does not write anything but closes smth, should be close_writing or smth alike; - the same way eof_received() should become smth like receive_closed; - pause() and resume() work with reading only, so they should be suffixed (prefixed) with read(ing), like pause_reading(), resume_reading(). Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 11:14 AM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
The exception() method exists for the same reason that we support both "key in mapping" and raising KeyError from "mapping[key]": sometimes you want "Look Before You Leap", other times you want to let the exception fly. If you want the latter, just call .result() directly, if you want the former, check .exception() first. Regardless, the Future API isn't really being defined in PEP 3156, as it is mostly inheritied from the previously implemented PEP 3148 (http://www.python.org/dev/peps/pep-3148/#future-objects) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 9, 2013 at 8:28 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
You need to combine it with the other LBYL checks (f.done() and f.cancelled()) to be sure it won't throw an exception. if f.done() and not f.cancelled(): # Since we now know neither TimeoutError nor CancelledError can happen, # we can check for exceptions either by calling f.exception() or # by calling f.result() inside a try/except block # The latter will usually be the better option Just calling f.result() is by far the most common, but the other can be convenient in some cases (e.g. if you're writing a scheduler that needs to check if it should be calling send() or throw() on a generator). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 09/01/2013 10:54am, Nick Coghlan wrote:
Which goes to show that it cannot be used with LBYL. For exception() to be usable with LBYL one would need to be able to check that exception() returns a value without having to catch any exceptions -- either from exception() or from result(). But you can only check that exception() doesn't raise an error by calling result() to ensure that it does raise an error. But then you might as well catch the exception from result(). And the idea of calling exception() first and then result() if it fails is just crazy. As things stand, exception() is pointless. -- Richard
On Wed, Jan 9, 2013 at 4:13 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
exception() will raise only TimeoutError or CancelledError, exceptions from the Future computation are not raised, they are returned. So to verify that a Future is properly computed, you should write: f.done() and not f.cancelled() and f.exception() is None and you won't have to catch any exceptions. -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 4:13 AM, Richard Oudkerk <shibturn@gmail.com> wrote:
Not true -- if the future has a callback associated with it, the callback (or callbacks) is called when it becomes "done", and if the callback wants to check for an exception it can use exception(). The callback is guaranteed that the future is done so it doesn't have to worry about the exception that is raised if the future isn't done. (Of course a callback can also just call result() and catch the exception, or let it bubble out -- in that case it will be logged by the event loop and then dropped.) -- --Guido van Rossum (python.org/~guido)
On Tue, Jan 8, 2013 at 5:14 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
I've read the PEP and some things raise questions in my consciousness. Here they are.
Thanks!
This is a semi-internal API that is mostly useful to Transport implementers, and there won't be many of those. So I prefer the API that has the fewest classes.
Ditto.
Too late, this is how PEP 3148 defined it. It was indeed inspired by Java Futures. However I would defend using methods here, since these are not all that cheap -- they have to acquire and release a lock.
Because it is quite awkward to check for an exception if you have to catch it (4 lines instead of 1).
I am indeed struggling a bit with these names, but "writing an EOF" is actually how I think of this (maybe I am dating myself to the time of mag tapes though :-).
- pause() and resume() work with reading only, so they should be suffixed (prefixed) with read(ing), like pause_reading(), resume_reading().
Agreed. -- --Guido van Rossum (python.org/~guido)
On Tue, Jan 8, 2013 at 8:31 PM, Guido van Rossum <guido@python.org> wrote:
I think I want to take that back. I think it is more common for a protocol to want to pause the transport (i.e. hold back data_received() calls) than it is for a transport to want to pause the protocol (i.e. hold back write() calls). So the more common method can have a shorter name. Also, pause_reading() is almost confusing, since the protocol's method is named data_received(), not read_data(). Also, there's no reason for the protocol to want to pause the *write* (send) actions of the transport -- if wanted to write less it should not have called write(). The reason to distinguish between the two modes of pausing is because it is sometimes useful to "stack" multiple protocols, and then a protocol in the middle of the stack acts as a transport to the protocol next to it (and vice versa). See the discussion on this list previously, e.g. http://mail.python.org/pipermail/python-ideas/2013-January/018522.html (search for the keyword "stack" in this long message to find the relevant section). -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 8:50 AM, Guido van Rossum <guido@python.org> wrote:
I totally agree with protocol/transport stacking, anyone should be able to do some ugly thing like FTP over SSL over SOCKS over SSL over HTTP (j/k). Just take a look at what you can do with netgraph in *BSD (anything over anything with any number of layers). But still we shouldn't sacrifice ease of understanding (both docs and code) for couple extra chars (10 actually). Yes, 'reading' is misleading, pause_receiving and resume_receiving are better. -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 8:31 AM, Guido van Rossum <guido@python.org> wrote:
Ok, I see. Should transports be bound to event loop on creation? I wonder, what would happen if someone changes current event loop between these calls.
I understand why it should be a method, but still if it's a getter, it should have either get_ or is_ prefix. Are there any way to change this with 'Final' PEP? they do. I've just imagined the amount of words I'll have to say to students about EOFs instead of simple "it closes our end of one half of a socket".
-- Kind regards, Yuriy.
On Tue, Jan 8, 2013 at 9:02 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
Yes, this is what the transport implementation does.
Why? That's not a universal coding standard. The names seem clear enough to me.
Are there any way to change this with 'Final' PEP?
No, the concurrent.futures package has been released (I forget if it was Python 3.2 or 3.3) and we're bound to backwards compatibility. Also I really don't think it's a big deal at all.
But which half? A socket is two independent streams, one in each direction. Twisted uses half_close() for this concept but unless you already know what this is for you are left wondering which half. Which is why I like using 'write' in the name. -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 9:14 AM, Guido van Rossum <guido@python.org> wrote:
But in theory every sock_ call is independent and returns Future bound to current event loop. So if one change event loop with active transport, nothing bad should happen. Or I'm missing something.
When I see (in autocompletion, for example) or remember name like "running", it triggers thought that it's a field. When I remember smth like is_running, it definitely associates with method.
Yes, not a big deal.
Yes, 'write' part is good, I should mention it. I meant to say that I won't need to explain that there were days when we had to handle a special marker at the end of file. -- Kind regards, Yuriy.
Is this thread really ready to migrate to python-dev when we're still bikeshedding method names? Yuriy Taraday writes:
Mystery is good for students.<wink/> Getting serious, "close_writer" occured to me as a possibility.
On Tue, Jan 8, 2013 at 9:26 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
It is bound to the event loop whose sock_<call>() method you called.
So if one change event loop with active transport, nothing bad should happen. Or I'm missing something.
Changing event loops in the middle of event processing is not a common (or even useful) pattern. You start the event loop and then leave it alone.
That must pretty specific to your personal experience.
But even today you have to mark the end somehow, to distinguish it from "not done yet, more could be coming". The equivalent is typing ^D into a UNIX terminal (or ^Z on Windows). -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 10:02 AM, Guido van Rossum <guido@python.org> wrote:
Yes. It was not-so-great morning idea.
My interns told me that they remember EOF as special object only from high school when they had to study Pascal. I guess, in 5 years students won't understand how one can write an EOF. (and schools will finally replace Pascal with Python) -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 8:55 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
Python really doesn't try to avoid the concept of an End-of-file marker. ================ $ python3 Python 3.2.3 (default, Jun 8 2012, 05:36:09) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Only makes one system call, so less data may be returned than requested In non-blocking mode, returns None if no data is available. On end-of-file, returns ''. ================ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 9, 2013 at 11:14 AM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
The exception() method exists for the same reason that we support both "key in mapping" and raising KeyError from "mapping[key]": sometimes you want "Look Before You Leap", other times you want to let the exception fly. If you want the latter, just call .result() directly, if you want the former, check .exception() first. Regardless, the Future API isn't really being defined in PEP 3156, as it is mostly inheritied from the previously implemented PEP 3148 (http://www.python.org/dev/peps/pep-3148/#future-objects) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Jan 9, 2013 at 8:28 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
You need to combine it with the other LBYL checks (f.done() and f.cancelled()) to be sure it won't throw an exception. if f.done() and not f.cancelled(): # Since we now know neither TimeoutError nor CancelledError can happen, # we can check for exceptions either by calling f.exception() or # by calling f.result() inside a try/except block # The latter will usually be the better option Just calling f.result() is by far the most common, but the other can be convenient in some cases (e.g. if you're writing a scheduler that needs to check if it should be calling send() or throw() on a generator). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 09/01/2013 10:54am, Nick Coghlan wrote:
Which goes to show that it cannot be used with LBYL. For exception() to be usable with LBYL one would need to be able to check that exception() returns a value without having to catch any exceptions -- either from exception() or from result(). But you can only check that exception() doesn't raise an error by calling result() to ensure that it does raise an error. But then you might as well catch the exception from result(). And the idea of calling exception() first and then result() if it fails is just crazy. As things stand, exception() is pointless. -- Richard
On Wed, Jan 9, 2013 at 4:13 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
exception() will raise only TimeoutError or CancelledError, exceptions from the Future computation are not raised, they are returned. So to verify that a Future is properly computed, you should write: f.done() and not f.cancelled() and f.exception() is None and you won't have to catch any exceptions. -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 4:13 AM, Richard Oudkerk <shibturn@gmail.com> wrote:
Not true -- if the future has a callback associated with it, the callback (or callbacks) is called when it becomes "done", and if the callback wants to check for an exception it can use exception(). The callback is guaranteed that the future is done so it doesn't have to worry about the exception that is raised if the future isn't done. (Of course a callback can also just call result() and catch the exception, or let it bubble out -- in that case it will be logged by the event loop and then dropped.) -- --Guido van Rossum (python.org/~guido)
On Tue, Jan 8, 2013 at 5:14 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
I've read the PEP and some things raise questions in my consciousness. Here they are.
Thanks!
This is a semi-internal API that is mostly useful to Transport implementers, and there won't be many of those. So I prefer the API that has the fewest classes.
Ditto.
Too late, this is how PEP 3148 defined it. It was indeed inspired by Java Futures. However I would defend using methods here, since these are not all that cheap -- they have to acquire and release a lock.
Because it is quite awkward to check for an exception if you have to catch it (4 lines instead of 1).
I am indeed struggling a bit with these names, but "writing an EOF" is actually how I think of this (maybe I am dating myself to the time of mag tapes though :-).
- pause() and resume() work with reading only, so they should be suffixed (prefixed) with read(ing), like pause_reading(), resume_reading().
Agreed. -- --Guido van Rossum (python.org/~guido)
On Tue, Jan 8, 2013 at 8:31 PM, Guido van Rossum <guido@python.org> wrote:
I think I want to take that back. I think it is more common for a protocol to want to pause the transport (i.e. hold back data_received() calls) than it is for a transport to want to pause the protocol (i.e. hold back write() calls). So the more common method can have a shorter name. Also, pause_reading() is almost confusing, since the protocol's method is named data_received(), not read_data(). Also, there's no reason for the protocol to want to pause the *write* (send) actions of the transport -- if wanted to write less it should not have called write(). The reason to distinguish between the two modes of pausing is because it is sometimes useful to "stack" multiple protocols, and then a protocol in the middle of the stack acts as a transport to the protocol next to it (and vice versa). See the discussion on this list previously, e.g. http://mail.python.org/pipermail/python-ideas/2013-January/018522.html (search for the keyword "stack" in this long message to find the relevant section). -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 8:50 AM, Guido van Rossum <guido@python.org> wrote:
I totally agree with protocol/transport stacking, anyone should be able to do some ugly thing like FTP over SSL over SOCKS over SSL over HTTP (j/k). Just take a look at what you can do with netgraph in *BSD (anything over anything with any number of layers). But still we shouldn't sacrifice ease of understanding (both docs and code) for couple extra chars (10 actually). Yes, 'reading' is misleading, pause_receiving and resume_receiving are better. -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 8:31 AM, Guido van Rossum <guido@python.org> wrote:
Ok, I see. Should transports be bound to event loop on creation? I wonder, what would happen if someone changes current event loop between these calls.
I understand why it should be a method, but still if it's a getter, it should have either get_ or is_ prefix. Are there any way to change this with 'Final' PEP? they do. I've just imagined the amount of words I'll have to say to students about EOFs instead of simple "it closes our end of one half of a socket".
-- Kind regards, Yuriy.
On Tue, Jan 8, 2013 at 9:02 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
Yes, this is what the transport implementation does.
Why? That's not a universal coding standard. The names seem clear enough to me.
Are there any way to change this with 'Final' PEP?
No, the concurrent.futures package has been released (I forget if it was Python 3.2 or 3.3) and we're bound to backwards compatibility. Also I really don't think it's a big deal at all.
But which half? A socket is two independent streams, one in each direction. Twisted uses half_close() for this concept but unless you already know what this is for you are left wondering which half. Which is why I like using 'write' in the name. -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 9:14 AM, Guido van Rossum <guido@python.org> wrote:
But in theory every sock_ call is independent and returns Future bound to current event loop. So if one change event loop with active transport, nothing bad should happen. Or I'm missing something.
When I see (in autocompletion, for example) or remember name like "running", it triggers thought that it's a field. When I remember smth like is_running, it definitely associates with method.
Yes, not a big deal.
Yes, 'write' part is good, I should mention it. I meant to say that I won't need to explain that there were days when we had to handle a special marker at the end of file. -- Kind regards, Yuriy.
Is this thread really ready to migrate to python-dev when we're still bikeshedding method names? Yuriy Taraday writes:
Mystery is good for students.<wink/> Getting serious, "close_writer" occured to me as a possibility.
On Tue, Jan 8, 2013 at 9:26 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
It is bound to the event loop whose sock_<call>() method you called.
So if one change event loop with active transport, nothing bad should happen. Or I'm missing something.
Changing event loops in the middle of event processing is not a common (or even useful) pattern. You start the event loop and then leave it alone.
That must pretty specific to your personal experience.
But even today you have to mark the end somehow, to distinguish it from "not done yet, more could be coming". The equivalent is typing ^D into a UNIX terminal (or ^Z on Windows). -- --Guido van Rossum (python.org/~guido)
On Wed, Jan 9, 2013 at 10:02 AM, Guido van Rossum <guido@python.org> wrote:
Yes. It was not-so-great morning idea.
My interns told me that they remember EOF as special object only from high school when they had to study Pascal. I guess, in 5 years students won't understand how one can write an EOF. (and schools will finally replace Pascal with Python) -- Kind regards, Yuriy.
On Wed, Jan 9, 2013 at 8:55 PM, Yuriy Taraday <yorik.sar@gmail.com> wrote:
Python really doesn't try to avoid the concept of an End-of-file marker. ================ $ python3 Python 3.2.3 (default, Jun 8 2012, 05:36:09) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Only makes one system call, so less data may be returned than requested In non-blocking mode, returns None if no data is available. On end-of-file, returns ''. ================ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Dustin J. Mitchell -
Glyph -
Guido van Rossum -
Jasper St. Pierre -
Nick Coghlan -
Richard Oudkerk -
Stephen J. Turnbull -
Yuriy Taraday