multiprocess passing arguments double asterisks
pic8690 at gmail.com
pic8690 at gmail.com
Wed Oct 26 16:44:56 EDT 2016
On Monday, October 24, 2016 at 12:39:47 PM UTC-5, Thomas Nyberg wrote:
> On 10/24/2016 12:45 PM, pic8690 at gmail.com wrote:
> > Thanks for the reply.
> >
> > The code snippet given by Peter is not very clear
> >
> > I would like to multiprocess a function which is written in python of the form bar(**kwargs) which returns a value. This example does not return anything
> >
> > Would you please use my example for the map function?
> >
> > I appreciate your help,
> >
> I'm honestly not totally sure what you want to do. However, say you want
> to do the following (btw this is basically what Dennis said i nhis last
> email, but maybe I can help clarify):
>
> kwargs = {'param1': val1, 'param2': val2})
>
> Then you'd like to have the following two operations performed in
> separate processes:
>
> bar(param1=val1)
> bar(param2=val2)
>
> In that case, I guess I would do something like the following. First
> define bar_wrapper as follows (*I haven't tested any code here!):
>
> def bar_wrapper(pair):
> key, val = pair
> return bar(**{key: val})
>
> Then I would probably do something like
>
> map(bar_wrapper, kwargs.items())
>
> I.e. basically what I'm doing is taking the key-val pairs and producing
> a list of them (as tuples). This is something that you can apply map
> too, but not with the original function. So then the wrapper function
> converts the tuple back to what you want originally.
>
> Hopefully I'm understanding correctly and hopefully this helps.
>
> Cheers,
> Thomas
Thomas,
I have strings & numpy.ndarray as arguments. The wrapper function works great for strings.
Here's what I'm trying to do...
**************************
I have a serial python fn of the form
def bar(**kwargs):
a=kwargs.get("name")
print a
self.a1d=ma.asanyarray(kwargs.get('a1d'), dtype=float)
(****more calculations--takes a long time to compute
returns an object)
I am trying to run this function in parallel.
Here's my Main program
import numpy as np
import numpy.ma as ma
from delegate import parallelize
from hashlib import sha1
a1d=np.zeros((32))
b1d=np.zeros((32))
p=open("/tmp/pdata","rb")
pdata=np.load(p)
for i in range(0,10):
a1d=pdata['t1d']
b1d=pdata['gz1d']
print a1d,b1d
kwargs={'name':'special','a':a1d,'b':b1d}
val=parallelize(bar,kwargs)
***************************************
Error:
line 30, in <module>
val=parallelize(bar,kwargs)
delegate.py", line 311, in process
item, items = items[0], items[1:]
KeyError: 0
**************************************
Error: (with the wrapper function)
val=parallelize(bar_wrapper,kwargs.items())
TypeError: unhashable type: 'numpy.ndarray'
***************************************
How do I pass the string & numpy.ndarray to the function bar(**kwargs)?
Thank you for suggestions & help,
a1d=np.zeros((32))
b1d=np.zeros((32))
p=open("/tmp/pdata","rb")
pdata=np.load(p)
for i in range(0,100):
a1d=pdata['t1d']
b1d=pdata['gz1d']
print a1d,b1d
kwargs={'name':'special','a':a1d,'b':b1d}
val=parallelize(bar,kwargs)
More information about the Python-list
mailing list