[Tutor] Performance Issues with Threaded Python Network Server

[Joao Prado Maia]

Hi,

Before even starting with the description of my problem, let me explain my
situation. I'm trying to create a custom Python based NNTP server to serve
as a gateway of sorts to some web message boards. This way people can
connect to the News server and get messages / reply and such.

Please be aware that this is my first real-world python project, so please
be gentil if you see something stupid in my code ;)

Anyway, I read 'Programming Python' a little bit on the Network programming
chapter and decided to use threads (aka SocketServer.ThreadingTCPServer) on
my NNTP server. Everything works great but I have been experiencing some
heavy CPU usage on the server.

A little bit more of information - the heavy CPU load is triggered when the
an user tries to download all 1500 messages / articles of one message board.
The way NNTP works and the way Outlook Express (the newsreader on this case)
works is that it will download all the headers for the articles at once, and
then request the actual body of the articles one by one.

What this means is that the server will write to the 'wfile' file descriptor
to send the response (the headers and bodies of the articles) to the
newsreader.

The problem here is that whenever this happens, the CPU usage of the NNTP
server goes to about 35% and continues increasing slowly while the
newsreader is receiving all the message headers and bodies.

My guess right now is that I'm doing something bad on the routines that spit
the information to the newsreader somehow and because of this the server is
consuming a lot of CPU. Since I'm not an expert in the performance tweaks or
even about possible bottlenecks, I would love if someone could take a look
and maybe get some insight into what could be wrong.

The NNTP server gets its information from a MySQL database (and no, MySQL is
not the bottleneck as far as I know, since 'top' shows the NNTP server
consuming 35% of CPU, not MySQL), formats the output by using string
replacement (aka "%s %s <%s@%s>" % (v,x,z,y)) and writes to the 'wfile' file
descriptor.

The source code for the NNTP server can be found below directly from CVS:
http://cvs.phpbrasil.com/chora/co.php/papercut/papercut.py?r=1.18

The source code for the MySQL backend interface that does most of the
formatting of the information coming from the database can also be found
here:
http://cvs.phpbrasil.com/chora/co.php/papercut/backends/phorum_mysql.py?r=1.
3

Any insights would be very much appreciated.

Joao

--
Joao Prado Maia
Software QA
Bioinformatics Dept.
Lexicon Genetics, Inc.


*************************************************************************** 
 The contents of this communication are intended only for the addressee and
may contain confidential and/or privileged material. If you are not the
intended recipient, please do not read, copy, use or disclose this
communication and notify the sender.  Opinions, conclusions and other
information in this communication that do not relate to the official
business of my company shall be understood as neither given nor endorsed by
it.  
***************************************************************************