[IPython-dev] Run IPython Parallel from custom module?
Ryan Nelson
rnelsonchem at gmail.com
Sat Jan 17 16:17:07 EST 2015
Thanks!
Searching for that decorator brings up a bunch of search hits. However, it
looks like it isn't really documented in the main document yet (?).
I'll try this out.
Ryan
On Sat, Jan 17, 2015 at 3:15 PM, MinRK <benjaminrk at gmail.com> wrote:
> This is one of the most common sources of confusion with IPython.parallel.
> When you call a function, it’s globals are resolved to its own module. So
> when you call _do_this, it is looking for File in mymod, not in the
> interactive namespace. If you had defined _do_this interactively (in
> __main__), it would have found File. Similarly, if File were defined in
> the same module as _do_this, it would also be found.
>
> IPython provides a decorator that lets you define functions in modules
> that will be treated as if they were defined in __main__, and will thus
> have access to the interactive namespace.
>
> from IPython.parallel import interactive
> @interactivedef _do_this(fname):
> f = File(fname)
> ...
>
> -MinRK
>
>
> On Sat, Jan 17, 2015 at 11:14 AM, Ryan Nelson <rnelsonchem at gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> I'm developing a module for processing some data files. There are several
>> steps to this process, which can be time consuming, so I would like to
>> write a helper function in my module that uses IPython.parallel to spread
>> out the work over several processors. Unfortunately, I'm having trouble
>> getting things to work properly. Note: I can get something similar to work
>> when it is not written as a module...
>>
>> I tried my best to make a simple example that fails with the same error
>> that I'm seeing in the larger case (below); however, it is still a little
>> complicated.
>>
>> After running the test script, I get a "NameError" on the cluster nodes
>> (traceback below). If I check the nodes manually, the objects are defined.
>> I can also define new functions from the IPython terminal that act on those
>> objects, and those work as expected.
>>
>> Any help you can provide is most appreciated. Sorry if this is a
>> duplicate, but I couldn't find it anywhere else.
>>
>> Python 3.4 and 2.7 and IPython 2.3.1
>>
>> Ryan
>>
>> ________________________________________
>>
>> Here's the approximate structure of my module:
>>
>> mymod/
>> __init__.py
>> functs.py : helper functions with Parallel support
>> objects.py : contains my custom processing objects
>>
>> # functs.py #
>> -------------
>> from IPython.parallel import Client
>>
>> import mymod.objects as objects
>>
>> def process(files):
>> client = Client()
>> dview = client[:]
>> dview.block = True
>>
>> # Do this to make module path known
>> # However, my mod will eventually be installed via pip, so this
>> # isn't necessary to reproduce error in that case
>> with dview.sync_imports():
>> import sys
>> sys.path.append(['/home/nelson/code/testing/',])
>>
>> dview['sys.path'] = sys.path
>> File = objects.File1
>> dview['File'] = File
>> result = dview.map_sync(_do_this, files)
>> return result
>>
>> def _do_this(fname):
>> f = File(fname)
>> f.extra = 'hello'
>> return f
>> -------------
>>
>> # objects.py #
>> -------------
>> class File1(object):
>> def __init__(self, fname):
>> self.fname = fname
>> -------------
>>
>> I also have a test script ("test.py") with the following:
>> -------------
>> from mymod.functs import process
>> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>> [print(i.fname, i.extra) for i in result]
>> -------------
>>
>> When I run this script, I get the following errors:
>>
>> importing sys on engine(s)
>> Traceback (most recent call last):
>> File "test2.py", line 3, in <module>
>> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>> File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
>> result = dview.map_sync(_do_this, files)
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 366, in map_sync
>> return self.map(f,*sequences,**kwargs)
>> File "<string>", line 2, in map
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 66, in sync_results
>> ret = f(self, *args, **kwargs)
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>> 624, in map
>> return pf.map(*sequences)
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 271, in map
>> ret = self(*sequences)
>> File "<string>", line 2, in __call__
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 78, in sync_view_results
>> return f(self, *args, **kwargs)
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>> line 254, in __call__
>> return r.get()
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>> line 118, in get
>> raise self._exception
>> File
>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>> line 153, in wait
>> results = error.collect_exceptions(results, self._fname)
>> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>> line 233, in collect_exceptions
>> raise e
>> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>> line 231, in collect_exceptions
>> raise CompositeError(msg, elist)
>> IPython.parallel.error.CompositeError: one or more exceptions from call
>> to method: _do_this
>> [0:apply]: NameError: name 'File' is not defined
>> [1:apply]: NameError: name 'File' is not defined
>> [2:apply]: NameError: name 'File' is not defined
>> [3:apply]: NameError: name 'File' is not defined
>>
>> However, if I jump into an interactive Python terminal, these objects are
>> defined on the cluster nodes.
>> In [1]: from IPython.parallel import Client
>>
>> In [2]: client = Client()
>>
>> In [3]: dview = client[:]
>>
>> In [4]: dview['File']
>>
>>
>> Out[4]:
>> [mymod.objects.File1,
>> mymod.objects.File1,
>> mymod.objects.File1,
>> mymod.objects.File1]
>>
>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/415828c1/attachment.html>
More information about the IPython-dev
mailing list