[SciPy-user] cow loop_apply method and external modules??

[Leonard J. Reder]

Hello,

I am attempting to use cow and the loop_apply method of machine_clusters 
to execute a simple function within a custom python module.  This same 
example using the math.py module and pow(x,y) function worked fine but 
this one gives the error message:

chimera:/home/soa/dev/users/reder/src/Cow>./test3.py
Slave machine list:  [('inu', 10000), ('chimera', 10001), ('hertz', 10002)]
List of numbers to give to a function [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Instance machine_cluster.
Traceback (most recent call last):
  File "./test3.py", line 31, in ?
    result = cluster.loop_apply(squared.fn,0,(L,))
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/cow.py", 
line 797, in loop_apply
    return self.loop_send_recv(package,loop_data,loop_var)
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/cow.py", 
line 847, in loop_send_recv
    results = self._send_recv(package,addendums)
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/cow.py", 
line 347, in _send_recv
    self.handle_error()
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/cow.py", 
line 408, in handle_error
    raise ClusterError, msg
ClusterError:

The same error occured on all workers:
        exceptions.ImportError: No module named squared########### 
Traceback text from remote machine ###########
Traceback (most recent call last):
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/sync_cluster.py", 
line 549, in handle
    recv_msg = self.recv()
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/sync_cluster.py", 
line 608, in recv
    task = self.packer.funpack(packed_msg)
  File 
"/home/atbe/pkgs/i486-fedora3-linux/stow/python-2.4/lib/python2.4/site-packages/scipy/cow/sync_cluster.py", 
line 78, in funpack
    res = pickle.load(f)
ImportError: No module named squared
################# End remote traceback ##################
chimera:/home/soa/dev/users/reder/src/Cow>

For some reason it cannot find my custom module called squared.  My 
PYTHONPATH variable includes all directories.  Here is the code:

====start of squared.py
#
# Test module for cow
#
def fn(x):
        return x*x

======start of test3.py
#!/home/atbe/pkgs/i486-fedora3-linux/bin/python
#
# Initial test3 of scipy.cow
# Import a custom module squared and see if the apply works ok?
#
import sys
sys.path.append("/home/atbe/pkgs/i486-fedora3-linux/lib")
sys.path.append("/home/soa/dev/users/reder/src/Cow")
sys.path.append(".")
import scipy

slave_list = []
slave_list.append(('inu',10000))
slave_list.append(('chimera',10001))
slave_list.append(('hertz',10002))

print "Slave machine list: ", slave_list

print "List of numbers to give to a function",
L = range(10)
print L

print "Instance machine_cluster."
cluster = scipy.cow.machine_cluster(slave_list)
cluster.start()

cluster.system("setenv PYTHONPATH /home/soa/dev/users/reder/src/Cow")

# process a list of number in parallel applying function fn to them.
import squared
result = cluster.loop_apply(squared.fn,0,(L,))
print "Result of x**2 are ", result

# cluster.info()
print "clean up /tmp/crud* files on all machines."
cluster.system("rm -f /tmp/crud*")

print "stop machine_cluster"
cluster.stop()
=====end of test3.py

All replies to this would be very much appreciated.  From looking at the 
sync_cluster.py I think this should work?

Regards,

Len

-- 
____________________________________________________
Leonard J. Reder
Jet Propulsion Laboratory
Interferometry Systems and Technology Section 383
Email: reder at huey.jpl.nasa.gov
Phone (Voice): 818-354-3639
Phone (FAX): 818-354-4357
Mail Address:
Mail Stop: 171-113
4800 Oak Grove Dr.
Pasadena, CA. 91109
---------------------------------------------------