[Python-ideas] Async API: some code to review

[Giampaolo Rodolà]

2012/10/29 Guido van Rossum <guido at python.org>
>
> I'm most interested in feedback on the design of polling.py and
> scheduling.py, and to a lesser extent on the design of sockets.py;
> main.py is just an example of how this style works out in practice.

Follows my comments.

=== About polling.py ===

1 - I think DelayedCall should have a reset() method, other than just cancel().

2 - EventLoopMixin should have a call_every() method other than just
call_later()

3 - call_later() and call_every() should also take **kwargs other than
just *args

4 - I think PollsterBase should provide a method to modify() the
events registered for a certain fd (both poll() and epoll() have such
a method and it's faster compared to un/registering a fd).

Feel free to take a look at my scheduler implementation which looks
quite similar to what you've done in polling.py:
http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop.py?spec=svn1115&r=1115#85


=== About sockets.py ===

1 - In SocketTransport it seems there's no error handling provisioned
for send() and recv().
You should expect these errors
http://hg.python.org/cpython/file/95931c48a76f/Lib/asyncore.py#l60
signaling disconnection plus EWOULDBLOCK and EAGAIN for "retry"

2 - SslTransport's send() and recv() methods should suffer the same problem.

3 - I don't fully understand how data transfer works exactly but keep
in mind that the transport should interact with the pollster.
What I mean is that generally speaking a connected socket should
*always* be readable ("r"), even when it's idle, then switch to "rw"
events when sending data, then get back to "r" when all the data has
been sent.
This is *crucial* if you want to achieve high performances/scalability
and that is why PollsterBase should probably provide a modify()
method.
Please take a look at what I've done here:
http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop.py?spec=svn1115&r=1115#809


===  Other considerations ===

This 'yield' / 'yield from' approach is new to me (I'm more of a
"callback guy") so I can't say I fully understand what's going on just
by reading the code.
What I would like to see instead of main.py is a bunch of code samples
/ demos showing how this library is supposed to be used in different
circumstances.
In details I'd like to see at least:

1 - a client example (connect(), send() a string, recv() a response, close())
2 - an echo server example (accept(), recv() string,  send() it back(), close()
3 - how to use a different transport (e.g. UDP)?
4 - how to run long running tasks in a thread?

Also:

5 - is it possible to use multiple "reactors" in different threads?
How?  (asyncore for example achieves this by providing a separate
'map' argument for both the 'reactor' and the dispatchers)


I understand you just started with this so I'm probably asking too
much at this point in time.
Feel free to consider this a kind of a "long term review".


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/