[Python-ideas] asyncore: included batteries don't fit

[Josiah Carlson]

Temporarily un-lurking to reply to this thread (which I'll actually be reading).

Giampaolo and I talked about this for a bit over the weekend, and I
have to say that I agree with his perspective.

In particular, to get something better than asyncore, there must be
something minimally better to build upon. I don't really have an
opinion on what that minimally better thing should be named, but I do
agree that having a simple reactor API that has predictable behavior
over the variety of handlers (select, poll, epoll, kqueue, WSAEvent in
Windows, etc.) is necessary.

Now, let's get to brass tacks...

1. Whatever reactors are available, you need to be able to instantiate
multiple of different types of reactors and multiple instances of the
same type of reactor simultaneously (to support multiple threads
handling different groups of reactors, or different reactors for
different types of objects on certain platforms). While this allows
for insanity in the worst-case, we're all consenting adults here, so
shouldn't be limited by reactor singletons. There should be a default
reactor class, which is defined on module/package import (use the
"best" one for the platform).

2. The API must be simple. I am not sure that it can get easier than
Idea #3 from:
http://mail.python.org/pipermail/python-ideas/2012-May/015245.html
I personally like it because it offers a simple upgrade path for
asyncore users (create your asyncore-derived classes, pass it into the
new reactor), while simultaneously defining a relatively easy API for
any 3rd party to integrate with. By offering an easy-to-integrate
method for 3rd parties (that is also sane), there is the added bonus
that 3rd parties are more likely to integrate, rather than replace,
which means more use in the "real world", better bug reports, etc. To
simplify integration further, make the API register(fd, handler,
events=singleton). Passing no events from the caller means "register
me for all events", which will help 3rd parties that aren't great with
handling read/write registration.

3. I don't have a 3rd tack, you can hang things on the wall with 2 ;)

Regards,
 - Josiah

On Mon, Sep 24, 2012 at 3:31 PM, Giampaolo Rodolà <g.rodola at gmail.com> wrote:
>
> I still think this proposal is too vaguely defined and any effort towards
> adding async IO support to existing batteries is premature for different
> reasons, first of which the inadequacy of asyncore as the base async
> framework to fulfill the task you're proposing.
>
> asyncore is so old and difficult to fix/enhance without breaking backward
> compatibility (see for example http://bugs.python.org/issue11273#msg156439)
> that relying on it for any modern work is inevitably a bad idea.
>
> From a chronological standpoint I still think the best thing to do in order
> to fix the "python async problem" once and for all is to first define and
> possibly implement an "async WSGI interface" describing what a standard
> async IO loop/reactor should look like (in terms of API) and how to
> integrate with it, see:
> http://mail.python.org/pipermail/python-ideas/2012-May/015223.html
> http://mail.python.org/pipermail/python-ideas/2012-May/015235.html
>
> From there the python stdlib *might* grow a new module implementing the
> "async WSGI interface" (let's call it asyncore2) and some of the stdlib
> batteries such as socketserver can possibly use it.
>
> In my mind this is the ideal long-term scenario but even managing to define
> an "async WSGI interface" alone would be a big step forward.
>
> Again, at this point in time what you're proposing looks too vague,
> ambitious and premature to me.
>
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
> http://code.google.com/p/pysendfile/
>
>
>
> 2012/9/22 chrysn <chrysn at fsfe.org>
>>
>> hello python-ideas,
>>
>> i'd like to start discussion about the state of asyncore/asynchat's
>> adaption in the python standard library, with the intention of finding a
>> roadmap for how to improve things, and of kicking off and coordinating
>> implementations.
>>
>> here's the problem (as previously described in [issue15978] and
>> redirected here, with some additions):
>>
>> the asyncore module would be much more useful if it were well integrated
>> in the standard library. in particular, it should be supported by:
>>
>> * subprocess
>>
>> * BaseHTTPServer / http.server (and thus, socketserver)
>>
>> * urllib2 / urllib, http.client
>>
>> * probably many other network libraries except smtpd, which already uses
>>   asyncore
>>
>> * third party libraries (if stdlib leads the way, the ecosystem will
>>   follow; eg pyserial)
>>
>> without widespread asyncore support, it is not possible to easily
>> integrate different servers and services with each other; with asyncore
>> support, it's just a matter of creating the objects and entering the
>> main loop. (eg, a http server for controlling a serial device, with a
>> telnet-like debugging interface).
>
>