[IPython-dev] Run IPython Parallel from custom module?
Ryan Nelson
rnelsonchem at gmail.com
Sat Jan 17 14:14:12 EST 2015
Hello everyone,
I'm developing a module for processing some data files. There are several
steps to this process, which can be time consuming, so I would like to
write a helper function in my module that uses IPython.parallel to spread
out the work over several processors. Unfortunately, I'm having trouble
getting things to work properly. Note: I can get something similar to work
when it is not written as a module...
I tried my best to make a simple example that fails with the same error
that I'm seeing in the larger case (below); however, it is still a little
complicated.
After running the test script, I get a "NameError" on the cluster nodes
(traceback below). If I check the nodes manually, the objects are defined.
I can also define new functions from the IPython terminal that act on those
objects, and those work as expected.
Any help you can provide is most appreciated. Sorry if this is a duplicate,
but I couldn't find it anywhere else.
Python 3.4 and 2.7 and IPython 2.3.1
Ryan
________________________________________
Here's the approximate structure of my module:
mymod/
__init__.py
functs.py : helper functions with Parallel support
objects.py : contains my custom processing objects
# functs.py #
-------------
from IPython.parallel import Client
import mymod.objects as objects
def process(files):
client = Client()
dview = client[:]
dview.block = True
# Do this to make module path known
# However, my mod will eventually be installed via pip, so this
# isn't necessary to reproduce error in that case
with dview.sync_imports():
import sys
sys.path.append(['/home/nelson/code/testing/',])
dview['sys.path'] = sys.path
File = objects.File1
dview['File'] = File
result = dview.map_sync(_do_this, files)
return result
def _do_this(fname):
f = File(fname)
f.extra = 'hello'
return f
-------------
# objects.py #
-------------
class File1(object):
def __init__(self, fname):
self.fname = fname
-------------
I also have a test script ("test.py") with the following:
-------------
from mymod.functs import process
result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
[print(i.fname, i.extra) for i in result]
-------------
When I run this script, I get the following errors:
importing sys on engine(s)
Traceback (most recent call last):
File "test2.py", line 3, in <module>
result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
result = dview.map_sync(_do_this, files)
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
366, in map_sync
return self.map(f,*sequences,**kwargs)
File "<string>", line 2, in map
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
66, in sync_results
ret = f(self, *args, **kwargs)
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
624, in map
return pf.map(*sequences)
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 271, in map
ret = self(*sequences)
File "<string>", line 2, in __call__
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 78, in sync_view_results
return f(self, *args, **kwargs)
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 254, in __call__
return r.get()
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
line 118, in get
raise self._exception
File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
line 153, in wait
results = error.collect_exceptions(results, self._fname)
File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py", line
233, in collect_exceptions
raise e
File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py", line
231, in collect_exceptions
raise CompositeError(msg, elist)
IPython.parallel.error.CompositeError: one or more exceptions from call to
method: _do_this
[0:apply]: NameError: name 'File' is not defined
[1:apply]: NameError: name 'File' is not defined
[2:apply]: NameError: name 'File' is not defined
[3:apply]: NameError: name 'File' is not defined
However, if I jump into an interactive Python terminal, these objects are
defined on the cluster nodes.
In [1]: from IPython.parallel import Client
In [2]: client = Client()
In [3]: dview = client[:]
In [4]: dview['File']
Out[4]:
[mymod.objects.File1,
mymod.objects.File1,
mymod.objects.File1,
mymod.objects.File1]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/e3dc9b1a/attachment.html>
More information about the IPython-dev
mailing list