Multiprocessing Queue strange behavior

[Bruno Oliveira]

Hi, thanks for the answer.

I thought about that, but the problem is that I found the problem in code
that *was* using the Queue between processes. This code for example fails
around 60% of the time in one of our linux machines (raising an Empty
exception):

from processing import Queue, Process
import time

def Consume(queue):
    print queue.get_nowait()

if __name__ == '__main__':
    queue = Queue()
    queue.put('x')
    p = Process(target=Consume, args=(queue,))
    p.start()
    p.join()

Again, putting a sleep call before starting the consumer process makes the
code work all time. That is the main reason I believe there is a problem in
the Queue code somewhere.

Cheers,


On Wed, Sep 15, 2010 at 5:47 PM, MRAB <python at mrabarnett.plus.com> wrote:

> On 15/09/2010 21:10, Bruno Oliveira wrote:
>
>> Hi list,
>>
>> I recently found a bug in my company's code because of a strange
>> behavior using multiprocessing.Queue. The following code snippet:
>>
>> from multiprocessing import Queue
>>
>> queue = Queue()
>> queue.put('x')
>> print queue.get_nowait()
>> Fails with:
>>
>> ...
>>   File
>>
>> "E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py",
>> line 153, in getNoWait
>>     return self.get(False)
>>   File
>>
>> "E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py",
>> line 129, in get
>>     raise Empty
>> Queue.Empty
>>
>> Strangely, changing this to:
>>
>> queue = Queue()
>> queue.put('x')
>> time.sleep(0.1) # <<<
>> print queue.get_nowait()
>> Works as expected. Using the original snippet changing the import to
>> threading's Queue also works.
>>
>> It seems like there's a bug in multiprocessing's Queue implementation.
>> Opinions?
>>
>>  I don't think it's a bug as such.
>
> The purpose of the multiprocessing queue is to transfer data between
> different processes, which don't have a shared address space (unlike
> threads, which do).
>
> The transfer involves passing the data between the processes via a
> pipe. This is done in a background thread and takes some time to
> complete, so the data won't appear immediately. It looks like it
> doesn't matter that the putter and the getter happen to be in the same
> process, possibly because no-one expected that someone would use a
> multiprocessing queue within the same process like that, so it doesn't
> check for a shortcut.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100916/3421f3f2/attachment.html>