Recommended number of threads? (in CPython)
mk
mrkafk at gmail.com
Thu Oct 29 11:56:42 EDT 2009
Hello everyone,
I wrote run-of-the-mill program for concurrent execution of ssh command
over a large number of hosts. (someone may ask why reinvent the wheel
when there's pssh and shmux around -- I'm not happy with working details
and lack of some options in either program)
The program has a working queue of threads so that no more than
maxthreads number are created and working at particular time.
But this begs the question: what is the recommended number of threads
working concurrently? If it's dependent on task, the task is: open ssh
connection, execute command (then the main thread loops over the queue
and if the thread is finished, it closes ssh connection and does .join()
on the thread)
I found that when using more than several hundred threads causes weird
exceptions to be thrown *sometimes* (rarely actually, but it happens
from time to time). Although that might be dependent on modules used in
threads (I'm using paramiko, which is claimed to be thread safe).
More information about the Python-list
mailing list