Proper way to pass Queue to process when using multiprocessing.imap()?

[Rob Gaddi]

On 9/3/19 10:17 AM, Israel Brewster wrote:
> When using pool.imap to apply a function over a list of values, what is the proper way to pass additional arguments to the function, specifically in my case a Queue that the process can use to communicate back to the main thread (for the purpose of reporting progress)? I have seen suggestions of using starmap, but this doesn’t appear to have a “lazy” variant, which I have found to be very beneficial in my use case. The Queue is the same one for all processes, if that makes a difference.
> 
> I could just make the Queue global, but I have always been told not too. Perhaps this is an exception?
>   
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell:  907-328-9145
> 

The first rule is to never use global variables.  The second is to never put too 
much stock in sweeping generalizations.  So long as you can keep that Queue's 
usage pattern fairly well constrained, go ahead and make it global.

One thing to think about that might make this all easier though; have you looked 
at the concurrent.futures module?  I find it does a fantastic job of handling 
this sort of parallelization in a straightforward way.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.