[IPython-dev] Run IPython Parallel from custom module?

MinRK benjaminrk at gmail.com
Sat Jan 17 15:15:47 EST 2015


This is one of the most common sources of confusion with IPython.parallel.
When you call a function, it’s globals are resolved to its own module. So
when you call _do_this, it is looking for File in mymod, not in the
interactive namespace. If you had defined _do_this interactively (in
__main__), it would have found File. Similarly, if File were defined in the
same module as _do_this, it would also be found.

IPython provides a decorator that lets you define functions in modules that
will be treated as if they were defined in __main__, and will thus have
access to the interactive namespace.

from IPython.parallel import interactive
@interactivedef _do_this(fname):
    f = File(fname)
    ...

-MinRK
​

On Sat, Jan 17, 2015 at 11:14 AM, Ryan Nelson <rnelsonchem at gmail.com> wrote:

> Hello everyone,
>
> I'm developing a module for processing some data files. There are several
> steps to this process, which can be time consuming, so I would like to
> write a helper function in my module that uses IPython.parallel to spread
> out the work over several processors. Unfortunately, I'm having trouble
> getting things to work properly. Note: I can get something similar to work
> when it is not written as a module...
>
> I tried my best to make a simple example that fails with the same error
> that I'm seeing in the larger case (below); however, it is still a little
> complicated.
>
> After running the test script, I get a "NameError" on the cluster nodes
> (traceback below). If I check the nodes manually, the objects are defined.
> I can also define new functions from the IPython terminal that act on those
> objects, and those work as expected.
>
> Any help you can provide is most appreciated. Sorry if this is a
> duplicate, but I couldn't find it anywhere else.
>
> Python 3.4 and 2.7 and IPython 2.3.1
>
> Ryan
>
> ________________________________________
>
> Here's the approximate structure of my module:
>
> mymod/
>     __init__.py
>     functs.py : helper functions with Parallel support
>     objects.py : contains my custom processing objects
>
> # functs.py #
> -------------
> from IPython.parallel import Client
>
> import mymod.objects as objects
>
> def process(files):
>     client = Client()
>     dview = client[:]
>     dview.block = True
>
>     # Do this to make module path known
>     # However, my mod will eventually be installed via pip, so this
>     # isn't necessary to reproduce error in that case
>     with dview.sync_imports():
>         import sys
>     sys.path.append(['/home/nelson/code/testing/',])
>
>     dview['sys.path'] = sys.path
>     File = objects.File1
>     dview['File'] = File
>     result = dview.map_sync(_do_this, files)
>     return result
>
> def _do_this(fname):
>     f = File(fname)
>     f.extra = 'hello'
>     return f
> -------------
>
> # objects.py #
> -------------
> class File1(object):
>     def __init__(self, fname):
>         self.fname = fname
> -------------
>
> I also have a test script ("test.py") with the following:
> -------------
> from mymod.functs import process
> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
> [print(i.fname, i.extra) for i in result]
> -------------
>
> When I run this script, I get the following errors:
>
> importing sys on engine(s)
> Traceback (most recent call last):
>   File "test2.py", line 3, in <module>
>     result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>   File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
>     result = dview.map_sync(_do_this, files)
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 366, in map_sync
>     return self.map(f,*sequences,**kwargs)
>   File "<string>", line 2, in map
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 66, in sync_results
>     ret = f(self, *args, **kwargs)
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
> 624, in map
>     return pf.map(*sequences)
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 271, in map
>     ret = self(*sequences)
>   File "<string>", line 2, in __call__
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 78, in sync_view_results
>     return f(self, *args, **kwargs)
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
> line 254, in __call__
>     return r.get()
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
> line 118, in get
>     raise self._exception
>   File
> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
> line 153, in wait
>     results = error.collect_exceptions(results, self._fname)
>   File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
> line 233, in collect_exceptions
>     raise e
>   File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
> line 231, in collect_exceptions
>     raise CompositeError(msg, elist)
> IPython.parallel.error.CompositeError: one or more exceptions from call to
> method: _do_this
> [0:apply]: NameError: name 'File' is not defined
> [1:apply]: NameError: name 'File' is not defined
> [2:apply]: NameError: name 'File' is not defined
> [3:apply]: NameError: name 'File' is not defined
>
> However, if I jump into an interactive Python terminal, these objects are
> defined on the cluster nodes.
> In [1]: from IPython.parallel import Client
>
> In [2]: client = Client()
>
> In [3]: dview = client[:]
>
> In [4]: dview['File']
>
>
> Out[4]:
> [mymod.objects.File1,
>  mymod.objects.File1,
>  mymod.objects.File1,
>  mymod.objects.File1]
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/23cef824/attachment.html>


More information about the IPython-dev mailing list