multiprocessing: child process race to answer (forgot to Cc: the list)
William Ray Wing
wrw at mac.com
Sun Nov 3 04:07:42 CET 2013
On Nov 2, 2013, at 11:44 AM, Sherard Hall <smhall05 at gmail.com> wrote:
> Thank you for the response. Processing time is very important so I suspect having to write to disk will take more time than letting the other processes complete without finding the answer. So I did some profiling one process finds the answer in about 250ms, but since I can't stop the other processes, it takes about 800ms before I can use the answer. Do you recommend a global variable flag? Any other suggestions?
> On Nov 2, 2013 8:17 AM, "William Ray Wing" <wrw at mac.com> wrote:
> On Nov 2, 2013, at 1:03 AM, smhall05 <smhall05 at gmail.com> wrote:
> > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote:
> >> On 02/11/2013 02:35, smhall05 wrote:
> >>> I am using a basic multiprocessing snippet I found:
> >>> #-----------------------------------------------------
> >>> from multiprocessing import Pool
> >>> def f(x):
> >>> return x*x
> >>> if __name__ == '__main__':
> >>> pool = Pool(processes=4) # start 4 worker processes
> >>> result = pool.apply_async(f, ) # evaluate "f(10)" asynchronously
> >>> print result.get(timeout=1)
> >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
> >>> #---------------------------------------------------------
> >>> I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
> >>> I have tried:
> >>> sys.exit(0) #this causes the program to hang
> >>> pool.close()
> >>> pool.terminate
> >> Did you actually mean "pool.terminate", or is that a typo for
> >> "pool.terminate()"?
> >>> These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you
> > I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> So, the simplest solution to that situation is to have whichever subprocess that finds the correct answer set a flag which the calling process can check. Depending on your OS, that flag can be anything from setting a lock to something as simple as creating a file which the calling process periodically wakes up and looks for, maybe just a file in which the subprocess has written the answer.
Well, the multiprocessing library provides listeners and clients that wrap BSD style sockets and allow you to send (push) arbitrary python objects to a listener, i.e., the master. There might be something better that was OS specific, but this will keep it pure python. I've not tested it, but there is a simple example here on Stackoverflow:
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-list