[Tutor] Python performance resources & resouce usage hints

[Liam Clarke]

Hi,

I've developed what would be my largest Python app to date. And, going
from the crude Windows Task Manager, it's trying to use as much CPU
time as it can get when it's idle.

This, no doub,t is due to my design.

I have an UDP socket server, a packet cruncher, and a DAO.
Each resides in it's own thread, and each thread is connected to the
next by a Queue.

So it's server ---> cruncher --> DAO.

Each thread's run() method basically looks like this -

        while True:
            try:
                data = self.queue.get(False)
                self.DAO.send_data(data)
            except Empty:
                if self.shutdown:
                    print "\DAO closing"
                    return
                continue


i.e. each thread is looping endlessly until data arrives via the
queue. I can't believe it chews the CPU time the way it does, but I
suppose it's easy to do so.

My query to the list is twofold -

First, if anyone knows of any websites with articles on Python
threading optimisation/pitfalls websites, I'd be greatly appreciative.

I've been reading the Performance tips on the official wiki, so if
there's anything similar I'd be  keen to look at it.

Okay, the 2nd piece of advice I'm seeking, is what would be the most
efficient path here?
My initial thoughts are along three lines:

Either:

A) Increase the queue size of the socket servers
B) Use timer threads to 'pulse' my threads.

Or:

A) Increase the queue size of the socket servers
B) Use blocking queues

Or:

A) Use blocking queues with a timeout
B) Use the socket servers to "wake" processing threads

As Python doesn't have sleeping threads etc, the third option is my
current measure of last resort, as there'll be some substantial
rejigging, and I'm not overly experienced with threads.

On the wiki http://wiki.python.org/moin/PythonSpeed/PerformanceTips#periodic,
I read  that "There is a function in the sys module, setcheckinterval,
which you can call to tell the interpreter how often to perform these
periodic checks."

Should I even think about that? I'm not after performance so much as
less utilisation of system resources...

Much thanks for any guidance offered.

Regards,

Liam Clarke