[Python-Dev] [Python-checkins] r88395 - python/branches/py3k/Lib/asyncore.py

[Giampaolo Rodolà]

2011/2/13 Antoine Pitrou <solipsis at pitrou.net>:
>
>> It would then be subject to python-dev development policy rather than
>> twisted dev policy (which is even stricter!). Would the twisted devs
>> *really* want that? We could use the same processes we have for
>> "externally maintained" libraries, but they have without fail caused us
>> problems.
>
> Oh, I agree with you. -1 on any new externally maintained library.
>
>> The other issue is that just because we provide an alternative doesn't
>> mean that everyone automatically stops using asyncore and migrates.
>
> Of course. asyncore's problem is not that its a maintenance burden, it's
> that it's really subpar compared to everything else out there.
> That said, Giampaolo has committed to taking it forward, so perhaps the
> 3.3 version of asyncore will be much (?) better.

I must say that asyncore can surely be improved but not so that it can
reach the flexibility offered by Twisted.
Or at least, not without changing some aspects of the current API and
break backward compatibility.
I'll try to summarize what I think is wrong with asyncore so that
maybe someone can chime in and propose ideas.

Guido was right when he stated that providing a future-proof and
maximally flexible design of such an API is not easy, and this is
exactly the problem with asyncore.
It provides a subclass-based API which doesn't work well in all those
cases where I want to mix different functionallities as in:

- I want a base TCP dispatcher
- ...with buffered output capabilities a-la asynchat
- ...which is able to limit the speed for incoming data
- ...and that can also switch to SSL

Although I don't use it, it seems that Twisted managed to do this by
splitting the concepts of "transport" and "protocol" / "application"
and by using zope.interface.
At the current state, asyncore is not able to do this flexibly. It
should at least separate transport and protocol, but again, I can't
imagine how exactly and it would surely have a cost in terms of
backward compatibility.

Another problem is that asyncore is pretty low-level, and this is why
the outcome is a code which looks monkey patched.

Where Twisted provides a dataReceived() method, asyncore provides
handle_read(): the user is supposed to override handle_read() and
manually call recv() which might either fail (e.g. "retry later" or
"disconnected") or succeed.
The same applies for all other aspects of a TCP connection:
handle_accept() -> accept(), handle_connect() -> connect() and
handle_write -> send().
They all might fail and all need to be handled with care individually
(see for example: http://bugs.python.org/issue6706 ).

This aspect might be mitigated by providing a serie of higher lever
classes (e.g. TCPServer, UDPServer, TCPConnection, UDPConnection,
SSLTCPConnection) providing an API similar to:
http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.interfaces.IProtocol.html
...but the need of a separation between transport and protocol layers
is still needed.

Last but not least, the asyncore "reactor" (asyncore.loop()) is not
tied with the dispatcher.
>From the dispatcher we have no reference to the reactor, hence we
cannot register/unregister file descriptors, forcing the main loop to
iterate over all dispatcher instances and making impossible to benefit
of epoll() and kqueue(), which are crucial for scalable servers
handling thousands simultaneous requests:
http://bugs.python.org/issue6692

As for what we can *currently* do without going into too much trouble
I can mention:
http://bugs.python.org/issue10084
http://bugs.python.org/issue1641

As for Twisted core, I think it would be a nice addition for the
stdlib, but for me it should also fit one crucial requirement: it
should be *simple* and reflect the simplicity and "taste" of all other
stdlib modules, and to fulfill such a requirement I think Twisted
probably needs to be "adapted" a bit.
The main reason why I decided to use asyncore is that, despite it's
huge gaps and lack of base functionnalities, I can read its source
code, understand what's going on and extend it via monkey patching.
It might seem a poor approach but it worked for me and couldn't do the
same when I tried to use Twisted.


Regards,

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/