[ python-Bugs-1010098 ] CPU usage shoots up with asyncore

[SourceForge.net]

Bugs item #1010098, was opened at 2004-08-16 11:54
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010098&group_id=5470

Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 6
Submitted By: Thomas (thomasabc)
Assigned to: A.M. Kuchling (akuchling)
Summary: CPU usage shoots up with asyncore

Initial Comment:
The CPU usage of a python application program using 
asyncore.py shoot up to 99% under Python 2.4a1. After 
a comparison, it was found out that the same program 
ran without such high CPU usage under Python 2.3.3

A simple couple of programs to narrow down the problem 
showed that a single loop() was accompanied by a large 
number of poll() with 2.4a1, whereas a single loop() had 
only 3 calls to poll() with 2.3.3.

Please let me know if there is anything else I can do for 
you to analyse. A couple of python scripts are attached 
in a zip file for you as below:

- Summary of the test scripts

With these scripts, the test attempts to identify what 
function calls are called that might explain the high CPU 
usage observed.

asyncore_test.py:
Taken from the Python Library Reference 11.24.2 
asynchat Example, the test program starts a server 
process to simulate client request process.

http_test.py:
This program simply establishes connection to the server 
and closes after 10 seconds.

- Procedure

1. Start asyncore_test.py
2. Start http_test.py
3. Stop asyncore_test.py


----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2004-08-21 19:01

Message:
Logged In: YES 
user_id=31435

I can confirm the primary symptom on WinXP.  For the 10 
seconds http_test.py runs, CPU is nailed to the wall under 
current CVS Python, but is a trickle under 2.3.4.

asyncore appears to be irrelevant.  "The problem" is that the 
socket associated with __main__.http_request_handler 
always comes back in the w set from select.select() under 
current Python, but the select times out under 2.3.4.

More detail:  under current Python, dumping some prints in 
asyncore.poll() delivers a huge number of repetitions of this 
info:

socket map is
{1940: <__main__.http_request_handler connected 
127.0.0.1:3455 at 0x9da800>,
 1948: <__main__.AsyncHTTPServer listening :40004 at 
0x9d7fa8>}
r, w, e passed in is [1940, 1948] [1940, 1948] [1940, 1948]
r, w, e returned is [], [1940], []

Under 2.3.4, there are only a few lines of output total -- 
select() doesn't believe the sockets passed to it are ready to 
read or ready to write.

So the difference is in socket behavior, or in how the layers 
around asyncore here are using sockets.

----------------------------------------------------------------------

Comment By: Thomas (thomasabc)
Date: 2004-08-20 06:05

Message:
Logged In: YES 
user_id=1104845

Sorry, just to let you know that you press Ctrl+C to stop the 
script above at Step 3 to get an output displayed as shown in 
the file asyncore_comparison.txt below.



----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-08-19 13:29

Message:
Logged In: YES 
user_id=80475

Andrew, what this due to your updates or was Martin's loop
counter the cause?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010098&group_id=5470