[ python-Bugs-1161031 ] Neverending warnings from asyncore

SourceForge.net noreply at sourceforge.net
Tue May 31 21:34:00 CEST 2005

Bugs item #1161031, was opened at 2005-03-10 16:34
Message generated for change (Comment added) made by josiahcarlson
You can respond by visiting: 

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Tony Meyer (anadelonbrin)
Assigned to: Nobody/Anonymous (nobody)
Summary: Neverending warnings from asyncore

Initial Comment:
Changes in asyncore from 2.3 to 2.4 mean that
asyncore.poll() now passes all the sockets in the map
to select.select() to be checked for errors, which is
probably a good thing.  If an error occurs, then
handle_expt() is called, which by default logs the error.

asyncore.dispatcher creates nonblocking sockets.  When
connect_ex() is called on a nonblocking socket, it will
probably return EWOULDBLOCK (connecting takes time),
which may mean the connection is successful, or may not
(asyncore dispatcher keeps going assuming all is well).

If the connection is not successful, and then
asyncore.loop() is called, then select.select() will
indicate that there is an error with the socket (can't
connect) and the error will be logged.

The trouble is that asyncore.loop then keeps going, and
will log this error again.  My not-that-fast system
here gets about 10,000 logged messages per second with
a single socket in the asyncore map.

There are ways to avoid this:

  (1) if the socket is blocking when connect()ing (and
then nonblocking afterwards) an error is raised if the
connect fails.

  (2) Before calling asyncore.loop(), the caller can
run through all the sockets, checking that they are ok.

  (3) handle_expt() can be overridden with a function
that repairs or removes the socket from the map (etc)

However, I'm not convinced that this is good behavior
for asyncore to have, by default.  On Windows,
select.select() will only indicate an error when trying
to connect (nonblocking) or if SO_OOBINLINE is
disabled, but this may not be the case (i.e. errors can
occur at other times) with other platforms, right? 
Unless the error is temporary, asyncore will by default
start streaming (extremely fast) a lot of "warning:
unhandled exception" (not very helpful an error
message, either) messages.  Even if the error only
lasts a minute, that could easily result in 10,000
warnings being logged.

Do any of the python developers agree that this is a
flaw?  I'm happy to work up a patch for whatever the
desired solution is, if so.


Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-05-31 12:34

Logged In: YES 

You seem to be unwilling to subclass asyncore.dispatcher to
extend its functionality, and the only reason you have given
as to why you are unwilling is "As much as possible a class
should provide sensible methods, so that overriding is kept
to a minimum." (I personally subclass dispatcher and its
async_chat derivative qutie often)

Now, in the case of the other standard socket server and
client framework in the Python standard library, namely the
SocketServer module and its derivatives, you will find
extending the functionality of those classes is via
subclassing and overriding methods as necessary.

To me, when two 'competing' methods of generating socket
servers and clients in the standard library offer the same
method of extension of their base functionality, then
perhaps that is what should be done.  The fact that
basically all of the standard library is subclassable (some
C modules are exceptions to the rule, but should be fixed in
Python 2.5), including types in the base language, further
suggests to me that subclassing is the standard mechanism
for extending the functionality of a class, regardless of
its usefulness in its base state.

In regards to the documentation, it seems to be that
whenever an object has an error, the handle_expt() method is
called (in spending two minutes reading the source). 
Whether or not those errors are rare, perhaps debatable
(I've not seen any in years), but it seems to be
application-specific as to what behavior the socket should
have in the case of an error (you may want to close, I may
want to report the error and reconnect, etc.).


Comment By: Tony Meyer (anadelonbrin)
Date: 2005-05-31 00:42

Logged In: YES 

dispatcher is not at all unusable without subclassing.  You 
can get data with recv() and send it with send() etc.  It can be 
treated as a thin wrapper around socket objects.  Yes, you 
will want to subclass it to get more useful behaviour than you 
can get from a basic socket.  I don't see that this means that 
you should be required to override the handle_expt() function, 
though.  As much as possible a class should provide 
sensible methods, so that overriding is kept to a minimum.

At the very least, this is a documentation error, since the 
documentation states:

handle_expt( ) 

Called when there is out of band (OOB) data for a socket 
connection. This will almost never happen, as OOB is 
tenuously supported and rarely used. 

"Almost never" is completely wrong.


Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-05-31 00:31

Logged In: YES 

I hate to point out the obvious, but dispatcher is wholly
unusable without subclassing.  How would you get data
to/from a connection without replacing handle_read,
handle_write?  How do you handle the case when you want to
connect to someone else or accept connections from someone
else without overloading handle_connect or handle_accept?


Comment By: Tony Meyer (anadelonbrin)
Date: 2005-05-31 00:15

Logged In: YES 

Yes this problem is easily solved by subclassing.  However I 
firmly believe that it is a terrible default behaviour, and that it's 
likely to hit many asyncore users.  A class shouldn't have to 
be subclassed to be usable (ignoring virtual classes and all 
that), and that really is the case here.

The simplest solution would be to change the handler to not 
log the message.  Or log the message once per socket or 


Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-05-31 00:03

Logged In: YES 

Option 1 is not really an option in any case where a large
number of connections are opened (so I don't believe should
be the default).

>From what I understand, certain methods are supposed to be
overridden in a subclass if someone using a class wants
different behavior.

In this case, I believe you can perform either option 2 or 3
in your own code to avoid the thousands of logged lines;
either by creating your own loop() function, or by creating
a subclass of dispatcher and implementing a handle_expt()
method that does as you desire.


You can respond by visiting: 

More information about the Python-bugs-list mailing list