how can a child thread notify a parent thread its status?

[MRAB]

scriptlearner at gmail.com wrote:
> First of all, let me say thank you to all of you.  I have asked many
> questions (some of them are dump questions), and you have kindly
> helped me.  I am not going to reply every message to say thank-you
> since that would be annoying for such group with such high daily
> traffics.  Thank you very much.
> 
> Let's get back to topic of this message.
> Here's how I have implemented it so far, and I am taking the queue of
> work load items approach.
> In my child thread, I will keep checking for available work load item
> until a duration is reached.
> #inside the child#
>         while endTime > time.time():
>             try:
>                  item = self.q.get(True, 3)
>             except Queue.Empty:  #what's wrong?  AttributeError: class
> Queue has no attribute 'Empty'
>                  print 'cant find any work load item, so lets wait and
> try again later'
>                  time.sleep(1) #wait and then check again
>                  continue
>             except:
>                  print "Unexpected error:", sys.exc_info()[0]
>                  raise
>             #do the real work with load item
> 
> In my parent thread, I will initialize X (depending on a cfg file)
> child threads and keep adding load items to a shared q until the
> duration is reached.
> #inside the parent#
>         callCounter = 0
>         workers = [] #a list of child threads
>         totalWorkers = 250
>         endTime = time.time() + duration
>         for i in range(totalWorkers):
>             w = Worker(q, duration, i)
>             w.start() #worker, do your job now!
>             workers.append(w)
> 
>         while endTime > time.time():
>             time.sleep(1)
>                 q.put(getWorkloadItem()) #add workload itmes
>                 callCounter += 1 #actually can we guarantee that the
> call will be sent??
>                                  #should we ask each child to report
> the number of calls they make?
> 
>         for i in range(totalWorkers):
>             workers[i].join()    # Wait for the child threads to
> finish
> 
> 
> Overall, it seems to be working now.  Though, I still have a couple of
> problems to resolve.
> 1. I got the following error for the codes that attempt to catch Empty
> Queue exception.  What's the right way to use it?
>     except Queue.Empty:
> AttributeError: class Queue has no attribute 'Empty'
> 

The exception 'Empty' belongs to the module, not the class. Try
importing as:

     from Queue import Queue, Empty

> 2. What's the best way to have each child thread to report the number
> of requests they send when they are done?  To add the numbers to
> another queue?
> 
Why not? :-)

> 3. I will need to do some logging for response time as well as some
> response contents.  I have two choices, one big log file for all
> threads (both child and parent), and one log file for each thread.
> Given the fact that I may have to log tons of data, I think opening
> and maintaining a bunch of smaller logs may be better than dealing
> with a big one (it may grow very fast).  Is there any best prastice
> for logging in Python?  If I change my mind and go with one big log
> file (pass it to each thread), is there anything I should be aware of
> for multi-thread access (writting) to the same log file?
> 
> Again, thank you.

If you like threads then you could put the log items into a queue and
have another thread writing them to the logfile. :-)

BTW, do you really need 250 threads? Seems like a lot.

I notice that you stop putting items into the queue when endTime is
reached and also the threads terminate when endTime is reached. If items
are put into the queue faster than they're taken out (or an item is put
in just before endTime) then there might still be unprocessed items in
the queue at endTime.