newbee: Simple Backend Python Script Question
steve at holdenweb.com
Fri Sep 14 17:16:03 CEST 2007
joe shoemaker wrote:
> I need to create python script that is threaded. So the main program
> will run in infinite loop and just retrieving messages and putting them
> in a queue. (Main thread)
> I need child threads from a pool to process the queue. When there is no
> stuff in the queue, they go to the pool and become available but they
> don't terminate. This has to be done continuously.
> Main program need to keep putting stuff in the queue, when there are no
> messages, then it sleeps for short time and check back to see any messages.
> To do this, I guess you don't write joinAll(), so that the main threads
> just don't wait for the child but goes to work.
> am I right?
Pretty much. The way I usually do this is to start a number of threads
reading work items of some sort from a Queue.Queue. The thread will
block automatically if there's nothing on the queue, resuming when
something appears (and if I want orderly termination I use None as a
dummy work unit, and the threads terminate when they receive a None, but
any sentinel value would do).
> Also, child threads (a function that is threaded) will make connecitons
> to the database. I am planning to use threadpool, so that threads reuse
> the connections. So do you close the database connection at the end of
> the function? If not then the connection will be opened forever?
You don't want a thread pool, you want a connection pool, but there's
little advantage to having one that is only shared between your threads.
You really need a system-wide connection pool. Alternatively, you might
find that your database module is thread-safe to the extent that
different threads can use cursors created on the same connection without
If you want each thread to be able to process transactions that are
invisible to the other threads before they are committed you should
ensure that you have a sufficient isolation level, which might imply the
need for a connection-per-thread architecture, in which case you might
expect some benefit from connection pooling.
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Sorry, the dog ate my .sigline
More information about the Python-list