[IPython-dev] Confusing NameError in Parallelizing with IPython
MinRK
benjaminrk at gmail.com
Fri Sep 7 19:55:26 EDT 2012
On Fri, Sep 7, 2012 at 2:18 PM, Michael Bilow <michael.k.bilow at gmail.com>wrote:
> Wow, you nailed it, thanks. I was running ipcluster start in a different
> directory from the one my code was in. I still don't understand the
> provenance of that particular error, but I can see how it could cause one.
Here's a mor detailed exploration of what is happening:
import pickle
from wrapper import parallel_run
print pickle.dumps(parallel_run)
gives:
cwrapper
parallel_run
p0
.
In other words, the serialized form of a module function is just the
function name and the module in which it lives.
When the request arrives on the engines, it is deserialized back into its
module namespace like
so<https://github.com/ipython/ipython/blob/master/IPython/utils/pickleutil.py#L98>
:
try:
__import__(self.module)
except ImportError:
pass
else:
g = sys.modules[self.module].__dict__
So what is happening is the import of `wrapper` is failing, but IPython
suppressed that error, and unpacked the function into an empty namespace
instead. It's possible that this error should *not* be suppressed, because
it certainly would have been more informative to you to see `ImportError:
wrapper` rather than a NameError since it came out in the wrong place.
I don't know what case is actually helped by this suppression, so I am
inclined to just cease catching the
error<https://github.com/ipython/ipython/pull/2395>
.
-MinRK
>
> Thanks!
>
> Michael
>
> On Fri, Sep 7, 2012 at 2:04 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>>
>>
>> On Fri, Sep 7, 2012 at 1:48 PM, Michael Bilow <michael.k.bilow at gmail.com>wrote:
>>
>>> Hi!
>>>
>>> I'm new to this list and to IPython, and I started using IPython to
>>> parallelize some code I'd written (complete code:
>>> https://github.com/chuyelchulo/ipython-nameerror, requires numpy and
>>> PIL). I've been encountering a confusing NameError that keeps popping up
>>> when I try to run this program: The function to be mapped (parallel_run) is
>>> imported from a module different from __main__, but the interpreter
>>> complains that the functions and modules used by parallel_run are not part
>>> of the namespace.
>>>
>>> I asked about the error here (
>>> http://stackoverflow.com/questions/12304847/ipython-parallel-computing-namespace-issues),
>>> and was told to come to this list.
>>>
>>
>> Hi again!
>>
>> When I do the following:
>>
>> $> cd /path/to/ipython-nameerror
>> $> ipcluster start -n 4 --daemon
>> $> python wrapper_wrapper.py
>> Time elapsed: 5.85908985138 s
>>
>> It seems to work as expected.
>>
>> A few quick questions:
>>
>> 1. what is the cwd when you run ipcluster? Are your engines local or
>> remote? What do you get from:
>>
>> import os
>> import pprint
>> from IPython import parallel
>>
>> client = parallel.Client()
>> pprint.pprint(client[:].apply_async(os.getcwdu).get_dict())
>>
>> or:
>>
>> %px import wrapper
>>
>> If I start the engines somewhere else such that `wrapper` is not
>> importable, I get the NameErrors you are seeing.
>>
>> -MinRK
>>
>>
>>> Thanks!
>>>
>>> Michael
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120907/db413f9f/attachment.html>
More information about the IPython-dev
mailing list