how to start thread by group?

[Gabriel Genellina]

En Mon, 06 Oct 2008 11:24:51 -0300, <bieffe62 at gmail.com> escribió:

> On 6 Ott, 15:24, oyster <lepto.pyt... at gmail.com> wrote:
>> my code is not right, can sb give me a hand? thanx
>>
>> for example, I have 1000 urls to be downloaded, but only 5 thread at  
>> one time

> I would restructure my code with someting like this ( WARNING: the
> following code is
> ABSOLUTELY UNTESTED and shall be considered only as pseudo-code to
> express my idea of
> the algorithm (which, also, could be wrong:-) ):

Your code creates one thread per url (but never more than MAX_THREADS  
alive at the same time). Usually it's more efficient to create all the  
MAX_THREADS at once, and continuously feed them with tasks to be done. A  
Queue object is the way to synchronize them; from the documentation:

<code>
 from Queue import Queue
 from threading import Thread

num_worker_threads = 3
list_of_urls = ["http://foo.com", "http://bar.com",
                 "http://baz.com", "http://spam.com",
                 "http://egg.com",
                ]

def do_work(url):
     from time import sleep
     from random import randrange
     from threading import currentThread
     print "%s downloading %s" % (currentThread().getName(), url)
     sleep(randrange(5))
     print "%s done" % currentThread().getName()

# from this point on, copied almost verbatim from the Queue example
# at the end of http://docs.python.org/library/queue.html

def worker():
     while True:
         item = q.get()
         do_work(item)
         q.task_done()

q = Queue()
for i in range(num_worker_threads):
      t = Thread(target=worker)
      t.setDaemon(True)
      t.start()

for item in list_of_urls:
     q.put(item)

q.join()       # block until all tasks are done
print "Finished"
</code>


-- 
Gabriel Genellina