How to stop this race condition more effectively

[David Bolen]

"Adonis" <deltapigz at telocity.com> writes:

> i have these two functions running on seperate threads, but when executed it
> causes a race condition on the system, hogging 100% cpu, i was able to
> alleviate it by inserting a time.sleep(1) function to "slow" it down to
> where its at 0% cpu usage when idle which is what i want, problem solved.
> correct, but i was wondering if it could be optimized more?

I don't really see any race condition here.  A race condition would be
code where two threads can interfere with each other based on their
their sequence of execution - e.g., two code paths are racing to use
or access the same resources, but depending on their execution
sequence the impact on those resources can differ - thus it's
unpredictable in execution, and normally an erroneous result.  It's a
very specific sort of problem in a multi-tasking environment.  But
your threads don't seem to share any data structures.  Neither would a
race condition necessarily have anything to do with CPU usage.

What I do see is that you've written two non-blocking loops.  I'm
assuming (since its not shown) that proc_hndl and proc_clnt run in
different threads.  You've got each thread continually looping and
checking the queue, and even when you pull something from the queue
you do it in a non-blocking mode.  That means that when your queues
are empty, if you don't have the sleep in there your code is
equivalent to:

    def proc_hndl(self):
	while 1:
	    pass

    def proc_clnt(self):
	while 1:
	    pass

Does it make sense now why it eats up all your CPU?  You've created
two infinite loops that will take whatever CPU you have.

Now, adding the sleep is certainly an effective way to minimize the
CPU usage (since it gives the OS an opportunity to dispatch to other
processes/threads).  There are some times when such an approach would
be needed, but in general it's not the most efficient.

It's hard to say without seeing more of the structure of the code, but
in general your goal should be to create a condition whereby your
threads "block" when they have nothing to do.  That means they should
end up waiting for some signal or event before continuing execution.

With queues, the easiest way to block is when trying to retrieve an
element.  For example, at least with the code you showed, this would
be an equivalent implementation of proc_hndl:

    def proc_hndl(self):
        while 1:
            self.xmm_hndl_list_final.append(self.xmm_hndl_list.get())

The key here is that you let the get() method block.  This will put
the thread to sleep (no CPU) until there is something in the queue.
At that point it will return the entry which you will append to the
list, and then immediately try to obtain another entry.  It'll keep
returning entries until the queue is empty again at which point it
will again block.

Now, you don't show in your code how those threads terminate.  A
slight modification to the above would permit you to shut down the
thread (say to exit the program) by posting None to the queue.  You
could actually make it anything you wanted that could uniquely be
identified from proper data - for this example, I'm assuming None
would not itself normally show up in the queue:

    def proc_hndl(self):
        while 1:
	    entry = self.xmm_hndl_list.get()
	    if entry is None:
                break
            self.xmm_hndl_list_final.append(entry
        # Do any cleanup processing here before exiting

But the key in any case is to ensure that you create a condition
whereby your threads will block when they don't have any work to do.
The Python Queue class that you're using is actually ideal for this
sort of stuff.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/