Proper way to pass Queue to process when using multiprocessing.imap()?
Israel Brewster
ijbrewster at alaska.edu
Wed Sep 4 12:00:05 EDT 2019
>
> On Sep 3, 2019, at 11:09 AM, Israel Brewster <ijbrewster at alaska.edu> wrote:
>
>>
>> On Sep 3, 2019, at 10:49 AM, Peter Otten <__peter__ at web.de> wrote:
>>
>> Israel Brewster wrote:
>>
>>> When using pool.imap to apply a function over a list of values, what is
>>> the proper way to pass additional arguments to the function, specifically
>>> in my case a Queue that the process can use to communicate back to the
>>> main thread (for the purpose of reporting progress)? I have seen
>>> suggestions of using starmap, but this doesn’t appear to have a “lazy”
>>> variant, which I have found to be very beneficial in my use case. The
>>> Queue is the same one for all processes, if that makes a difference.
>>>
>>> I could just make the Queue global, but I have always been told not too.
>>> Perhaps this is an exception?
>>
>> How about wrapping the function into another function that takes only one
>> argument? A concise way is to do that with functools.partial():
>>
>> def f(value, queue): ...
>>
>> pool.imap(partial(f, queue=...), values)
>
> That looks like exactly what I was looking for. I’ll give it a shot. Thanks!
So as it turns out, this doesn’t work after all. I get an error stating that “Queue objects should only be shared between processes through inheritance”. Still a good technique to know though!
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145
>
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell: 907-328-9145
>
>>
>>
>>
>>>
>>> ---
>>> Israel Brewster
>>> Software Engineer
>>> Alaska Volcano Observatory
>>> Geophysical Institute - UAF
>>> 2156 Koyukuk Drive
>>> Fairbanks AK 99775-7320
>>> Work: 907-474-5172
>>> cell: 907-328-9145
>>>
>>
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list <https://mail.python.org/mailman/listinfo/python-list>
More information about the Python-list
mailing list