[IPython-dev] running shell command from engine

[Fernando Perez]

Ah, that's just a limitation of the fact that your lambda is capturing the
enclosing scope.

Just define a normal function with local imports, and you're set.

def mytool():
  import suprocess as sp
  sp.popen(...)

v.map(mytool, ....)

Cheers

f


On Thu, Jun 19, 2014 at 11:42 AM, Antonio González Peña <antgonza at gmail.com>
wrote:

> Thanks for the reply.
>
> I'm not really being able to use it, I tried:
>         from IPython.parallel import Client
>
>         rc = Client()
>         with rc[:].sync_imports():
>             from subprocess import Popen
>
>         v = rc.load_balanced_view()
>
>         result = v.map(lambda job: Popen([job]), jobs)
>         print list(result)
> but get:
> ValueError: Sorry, cannot pickle functions with closures
>
> Then, I tried using dill as described here:
> http://nbviewer.ipython.org/gist/minrk/5241793, so:
>         from types import FunctionType
>         import dill as pickle
>         from IPython.utils.pickleutil import can_map
>         from IPython.kernel.zmq import serialize
>         from IPython.parallel import Client
>
>         can_map.pop(FunctionType, None)
>         serialize.pickle = pickle
>
>         rc = Client()n
>         with rc[:].sync_imports():
>             from subprocess import Popen
>
>         v = rc.load_balanced_view()
>
>         result = v.map(lambda job: Popen([job]), jobs)
>         print list(result)
> but now it fails with:
>     raise self._exception
> IPython.parallel.error.CompositeError: one or more exceptions from
> call to method: remote_import
> [0:apply]: AttributeError: 'module' object has no attribute '__main__'
> [1:apply]: AttributeError: 'module' object has no attribute '__main__'
> [2:apply]: AttributeError: 'module' object has no attribute '__main__'
> [3:apply]: AttributeError: 'module' object has no attribute '__main__'
>
> Any ideas?
>
> On Thu, Jun 19, 2014 at 12:29 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
> > Sure! Popen won't give you an actual result that's very useful, you'll
> need
> > to decide whether to capture stdout/err, to use pipes, etc. But
> > subprocess/popen are just python functions like any other, so you can use
> > them all the same...
> >
> > f
> >
> >
> > On Thu, Jun 19, 2014 at 6:59 AM, Antonio González Peña <
> antgonza at gmail.com>
> > wrote:
> >>
> >> Hello,
> >>
> >> Is it possible to run a shell command from an engine? For example:
> >> result = v.map(lambda job: Popen(['/bin/sh', job]), jobs)
> >>
> >> Thanks
> >>
> >> --
> >> Antonio
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
> >
> >
> >
> > --
> > Fernando Perez (@fperez_org; http://fperez.org)
> > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> > fernando.perez-at-berkeley: contact me here for any direct mail
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
>
>
>
> --
> Antonio
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140619/e2aa3bba/attachment.html>