run a function in another processor in python

[Peter Otten]

Astan Chee wrote:

> Sorry about that, here is a summary of my complete code. I haven't cleaned
> it up much or anything, but this is what it does:
> 
> import time
> import multiprocessing
> 
> test_constx =0
> test_consty =0
> 
> def functionTester(x):
>   global test_constx

You don't need to declare a variable as global unless you want to rebind 
(assign to) it.

>   global test_consty
>   print "constx " + str(test_constx)
>   print "consty " + str(test_consty)
>   return (test_constx*x[0]-x[1]+test_consty*x[0]+x[2])
> 
> def functionTesterMain(constx,consty):
>   global test_constx
>   global test_consty
>   test_constx = constx
>   test_consty = consty
>   num_args =
> [(61,12,1),(61,12,2),(61,12,3),(61,11,4),(61,12,4),(62,33,4),(7,12,4),
(16,19,4),(35,36,4),(37,38,3),(55,56,3),(57,63,3)]
>   num_processes = multiprocessing.cpu_count()
>   pool = multiprocessing.Pool(num_processes)

I think you need to create the pool outside the function; in the current 
configuration you get three not one Pool instance.

>   rs = []
>   start = time.time()
>   rs = pool.map(functionTester,num_args)
>   end = time.time()
>   elapsed= end - start
>   min = elapsed/60
>   print "Took", elapsed, "seconds to run, which is the same as", min,
> "minutes"
>   pos = 0
>   high = 0
>   n = None
>   for r in rs:
>     if r > high:
>       n = num_args[pos]
>       high = r
>     pos+=1
>   print "high " + str(high)
>   print "n " + str(n)
> 
>   return high,n
> 
> if __name__ == '__main__':
>   for i in range(1,4):
>     a,b = functionTesterMain(i,7)
>     print "-----------"
>     print "a " + str(a)
>     print "b " + str(a)
> 
> 
>   Which doesn't seem to work because the functionTester() needs to be
> simpler and not use global variables.
> I'm using global variables because I'm also trying to pass a few other
> variables and I tried using a class but that just gave me a unpickleable
> error. I tried using zip but I'm confused with how I can get to pass the
> data.

A simple approach would be to pass an index into a list

const_data = zip(range(1, 4), [7]*3)

> I know I can probably combine the data into tuples but that means that
> there is alot of data duplication, especially if the constx and consty are
> large dictionaries (or even custom objects), which might happen later.
> So it seems that map doesn't quite like functions like these. Anyway, I'll
> try and see if threads or something can substitute. I'd appriciate any 
help. Thanks

Your code, slightly modified and cleaned up (yes, four-space indent improves 
readability):

import time
import multiprocessing

const_data = zip(range(1, 4), [7]*3)
num_args = [(61, 12, 1), (61, 12, 2), (61, 12, 3), (61, 11, 4), 
            (61, 12, 4), (62, 33, 4), (7, 12, 4), (16, 19, 4), 
            (35, 36, 4), (37, 38, 3), (55, 56, 3), (57, 63, 3)]

def functionTester(args):
    i, x, y, z = args
    constx, consty = const_data[i]
    
    print "constx", constx
    print "consty", consty
    return constx*x - y + consty*x + z

def functionTesterMain(pool, index):
    start = time.time()
    rs = pool.map(functionTester, (((index, ) + x) for x in num_args))
    end = time.time()
    elapsed = end - start
    min = elapsed/60
    print "Took", elapsed, 
    print "seconds to run, which is the same as", min, "minutes"
    return max(zip(rs, num_args))

if __name__ == '__main__':
    num_processes = multiprocessing.cpu_count()
    pool = multiprocessing.Pool(num_processes)
    for i, _ in enumerate(const_data):
        a, b = functionTesterMain(pool, i)
        print "-----------"
        print "a", a
        print "b", b