[IPython-dev] Run IPython Parallel from custom module?

Ryan Nelson rnelsonchem at gmail.com
Sat Jan 17 16:17:07 EST 2015


Thanks!

Searching for that decorator brings up a bunch of search hits. However, it
looks like it isn't really documented in the main document yet (?).

I'll try this out.

Ryan



On Sat, Jan 17, 2015 at 3:15 PM, MinRK <benjaminrk at gmail.com> wrote:

> This is one of the most common sources of confusion with IPython.parallel.
> When you call a function, it’s globals are resolved to its own module. So
> when you call _do_this, it is looking for File in mymod, not in the
> interactive namespace. If you had defined _do_this interactively (in
> __main__), it would have found File. Similarly, if File were defined in
> the same module as _do_this, it would also be found.
>
> IPython provides a decorator that lets you define functions in modules
> that will be treated as if they were defined in __main__, and will thus
> have access to the interactive namespace.
>
> from IPython.parallel import interactive
> @interactivedef _do_this(fname):
>     f = File(fname)
>     ...
>
> -MinRK
>>
> On Sat, Jan 17, 2015 at 11:14 AM, Ryan Nelson <rnelsonchem at gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> I'm developing a module for processing some data files. There are several
>> steps to this process, which can be time consuming, so I would like to
>> write a helper function in my module that uses IPython.parallel to spread
>> out the work over several processors. Unfortunately, I'm having trouble
>> getting things to work properly. Note: I can get something similar to work
>> when it is not written as a module...
>>
>> I tried my best to make a simple example that fails with the same error
>> that I'm seeing in the larger case (below); however, it is still a little
>> complicated.
>>
>> After running the test script, I get a "NameError" on the cluster nodes
>> (traceback below). If I check the nodes manually, the objects are defined.
>> I can also define new functions from the IPython terminal that act on those
>> objects, and those work as expected.
>>
>> Any help you can provide is most appreciated. Sorry if this is a
>> duplicate, but I couldn't find it anywhere else.
>>
>> Python 3.4 and 2.7 and IPython 2.3.1
>>
>> Ryan
>>
>> ________________________________________
>>
>> Here's the approximate structure of my module:
>>
>> mymod/
>>     __init__.py
>>     functs.py : helper functions with Parallel support
>>     objects.py : contains my custom processing objects
>>
>> # functs.py #
>> -------------
>> from IPython.parallel import Client
>>
>> import mymod.objects as objects
>>
>> def process(files):
>>     client = Client()
>>     dview = client[:]
>>     dview.block = True
>>
>>     # Do this to make module path known
>>     # However, my mod will eventually be installed via pip, so this
>>     # isn't necessary to reproduce error in that case
>>     with dview.sync_imports():
>>         import sys
>>     sys.path.append(['/home/nelson/code/testing/',])
>>
>>     dview['sys.path'] = sys.path
>>     File = objects.File1
>>     dview['File'] = File
>>     result = dview.map_sync(_do_this, files)
>>     return result
>>
>> def _do_this(fname):
>>     f = File(fname)
>>     f.extra = 'hello'
>>     return f
>> -------------
>>
>> # objects.py #
>> -------------
>> class File1(object):
>>     def __init__(self, fname):
>>         self.fname = fname
>> -------------
>>
>> I also have a test script ("test.py") with the following:
>> -------------
>> from mymod.functs import process
>> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>> [print(i.fname, i.extra) for i in result]
>> -------------
>>
>> When I run this script, I get the following errors:
>>
>> importing sys on engine(s)
>> Traceback (most recent call last):
>>   File "test2.py", line 3, in <module>
>>     result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>>   File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
>>     result = dview.map_sync(_do_this, files)
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 366, in map_sync
>>     return self.map(f,*sequences,**kwargs)
>>   File "<string>", line 2, in map
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 66, in sync_results
>>     ret = f(self, *args, **kwargs)
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 624, in map
>>     return pf.map(*sequences)
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 271, in map
>>     ret = self(*sequences)
>>   File "<string>", line 2, in __call__
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 78, in sync_view_results
>>     return f(self, *args, **kwargs)
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 254, in __call__
>>     return r.get()
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>> line 118, in get
>>     raise self._exception
>>   File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>> line 153, in wait
>>     results = error.collect_exceptions(results, self._fname)
>>   File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>> line 233, in collect_exceptions
>>     raise e
>>   File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>> line 231, in collect_exceptions
>>     raise CompositeError(msg, elist)
>> IPython.parallel.error.CompositeError: one or more exceptions from call
>> to method: _do_this
>> [0:apply]: NameError: name 'File' is not defined
>> [1:apply]: NameError: name 'File' is not defined
>> [2:apply]: NameError: name 'File' is not defined
>> [3:apply]: NameError: name 'File' is not defined
>>
>> However, if I jump into an interactive Python terminal, these objects are
>> defined on the cluster nodes.
>> In [1]: from IPython.parallel import Client
>>
>> In [2]: client = Client()
>>
>> In [3]: dview = client[:]
>>
>> In [4]: dview['File']
>>
>>
>> Out[4]:
>> [mymod.objects.File1,
>>  mymod.objects.File1,
>>  mymod.objects.File1,
>>  mymod.objects.File1]
>>
>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/415828c1/attachment.html>


More information about the IPython-dev mailing list