[IPython-dev] Run IPython Parallel from custom module?
MinRK
benjaminrk at gmail.com
Sat Jan 17 15:15:47 EST 2015
This is one of the most common sources of confusion with IPython.parallel.
When you call a function, it’s globals are resolved to its own module. So
when you call _do_this, it is looking for File in mymod, not in the
interactive namespace. If you had defined _do_this interactively (in
__main__), it would have found File. Similarly, if File were defined in the
same module as _do_this, it would also be found.
IPython provides a decorator that lets you define functions in modules that
will be treated as if they were defined in __main__, and will thus have
access to the interactive namespace.
from IPython.parallel import interactive
@interactivedef _do_this(fname):
f = File(fname)
...
-MinRK
On Sat, Jan 17, 2015 at 11:14 AM, Ryan Nelson <rnelsonchem at gmail.com> wrote:
> Hello everyone,
>
> I'm developing a module for processing some data files. There are several
> steps to this process, which can be time consuming, so I would like to
> write a helper function in my module that uses IPython.parallel to spread
> out the work over several processors. Unfortunately, I'm having trouble
> getting things to work properly. Note: I can get something similar to work
> when it is not written as a module...
>
> I tried my best to make a simple example that fails with the same error
> that I'm seeing in the larger case (below); however, it is still a little
> complicated.
>
> After running the test script, I get a "NameError" on the cluster nodes
> (traceback below). If I check the nodes manually, the objects are defined.
> I can also define new functions from the IPython terminal that act on those
> objects, and those work as expected.
>
> Any help you can provide is most appreciated. Sorry if this is a
> duplicate, but I couldn't find it anywhere else.
>
> Python 3.4 and 2.7 and IPython 2.3.1
>
> Ryan
>
> ________________________________________
>
> Here's the approximate structure of my module:
>
> mymod/
> __init__.py
> functs.py : helper functions with Parallel support
> objects.py : contains my custom processing objects
>
> # functs.py #
> -------------
> from IPython.parallel import Client
>
> import mymod.objects as objects
>
> def process(files):
> client = Client()
> dview = client[:]
> dview.block = True
>
> # Do this to make module path known
> # However, my mod will eventually be installed via pip, so this
> # isn't necessary to reproduce error in that case
> with dview.sync_imports():
> import sys
> sys.path.append(['/home/nelson/code/testing/',])
>
> dview['sys.path'] = sys.path
> File = objects.File1
> dview['File'] = File
> result = dview.map_sync(_do_this, files)
> return result
>
> def _do_this(fname):
> f = File(fname)
> f.extra = 'hello'
> return f
> -------------
>
> # objects.py #
> -------------
> class File1(object):
> def __init__(self, fname):
> self.fname = fname
> -------------
>
> I also have a test script ("test.py") with the following:
> -------------
> from mymod.functs import process
> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
> [print(i.fname, i.extra) for i in result]
> -------------
>
> When I run this script, I get the following errors:
>
> importing sys on engine(s)
> Traceback (most recent call last):
> File "test2.py", line 3, in <module>
> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
> File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
> result = dview.map_sync(_do_this, files)
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 366, in map_sync
> return self.map(f,*sequences,**kwargs)
> File "<string>", line 2, in map
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 66, in sync_results
> ret = f(self, *args, **kwargs)
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 624, in map
> return pf.map(*sequences)
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 271, in map
> ret = self(*sequences)
> File "<string>", line 2, in __call__
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 78, in sync_view_results
> return f(self, *args, **kwargs)
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 254, in __call__
> return r.get()
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
> line 118, in get
> raise self._exception
> File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
> line 153, in wait
> results = error.collect_exceptions(results, self._fname)
> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
> line 233, in collect_exceptions
> raise e
> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
> line 231, in collect_exceptions
> raise CompositeError(msg, elist)
> IPython.parallel.error.CompositeError: one or more exceptions from call to
> method: _do_this
> [0:apply]: NameError: name 'File' is not defined
> [1:apply]: NameError: name 'File' is not defined
> [2:apply]: NameError: name 'File' is not defined
> [3:apply]: NameError: name 'File' is not defined
>
> However, if I jump into an interactive Python terminal, these objects are
> defined on the cluster nodes.
> In [1]: from IPython.parallel import Client
>
> In [2]: client = Client()
>
> In [3]: dview = client[:]
>
> In [4]: dview['File']
>
>
> Out[4]:
> [mymod.objects.File1,
> mymod.objects.File1,
> mymod.objects.File1,
> mymod.objects.File1]
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/23cef824/attachment.html>
More information about the IPython-dev
mailing list