[IPython-dev] running shell command from engine

[Michael McKerns]

There's a shortcut for a lot of what you did.

Instead of all that stuff below…
After you do:

    rc = Client()

then do

    rc[:].use_dill()

or instead of use_dill, there's use_cloudpickle()… you can try both.

I do know that recent releases of `dill` try to limit the unnecessary
globals when you work within a closure.  `cloudpickle` does the
same, but using a different method than dill does.

What Fernando suggested is a slightly less lazy, but more sure
approach to working with closures.


> Thanks for the reply.
>
> I'm not really being able to use it, I tried:
>         from IPython.parallel import Client
>
>         rc = Client()
>         with rc[:].sync_imports():
>             from subprocess import Popen
>
>         v = rc.load_balanced_view()
>
>         result = v.map(lambda job: Popen([job]), jobs)
>         print list(result)
> but get:
> ValueError: Sorry, cannot pickle functions with closures
>
> Then, I tried using dill as described here:
> http://nbviewer.ipython.org/gist/minrk/5241793, so:
>         from types import FunctionType
>         import dill as pickle
>         from IPython.utils.pickleutil import can_map
>         from IPython.kernel.zmq import serialize
>         from IPython.parallel import Client
>
>         can_map.pop(FunctionType, None)
>         serialize.pickle = pickle
>
>         rc = Client()n
>         with rc[:].sync_imports():
>             from subprocess import Popen
>
>         v = rc.load_balanced_view()
>
>         result = v.map(lambda job: Popen([job]), jobs)
>         print list(result)
> but now it fails with:
>     raise self._exception
> IPython.parallel.error.CompositeError: one or more exceptions from
> call to method: remote_import
> [0:apply]: AttributeError: 'module' object has no attribute '__main__'
> [1:apply]: AttributeError: 'module' object has no attribute '__main__'
> [2:apply]: AttributeError: 'module' object has no attribute '__main__'
> [3:apply]: AttributeError: 'module' object has no attribute '__main__'
>
> Any ideas?
>
> On Thu, Jun 19, 2014 at 12:29 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
>> Sure! Popen won't give you an actual result that's very useful, you'll
>> need
>> to decide whether to capture stdout/err, to use pipes, etc. But
>> subprocess/popen are just python functions like any other, so you can
>> use
>> them all the same...
>>
>> f
>>
>>
>> On Thu, Jun 19, 2014 at 6:59 AM, Antonio González Peña
>> <antgonza at gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> Is it possible to run a shell command from an engine? For example:
>>> result = v.map(lambda job: Popen(['/bin/sh', job]), jobs)
>>>
>>> Thanks
>>>
>>> --
>>> Antonio
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>>
>>
>> --
>> Fernando Perez (@fperez_org; http://fperez.org)
>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>> fernando.perez-at-berkeley: contact me here for any direct mail
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>
>
>
> --
> Antonio
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>


---

Mike McKerns
California Institute of Technology
TEL: (626)395-5773 or (626)590-8470
http://www.its.caltech.edu/~mmckerns
mmckerns at caltech.edu