[IPython-dev] From Pyro to IPython/Jupyter

Michael McKerns mmckerns at caltech.edu
Mon Dec 7 10:16:33 EST 2015


Pyro is good for python to non-python connections, however, if you are
going python to python, a good alternative is to use ParallelPython `pp`,
which has remote python servers. There's an easy-to-install version of it
that's available as `ppft`, which provides better serialization capabilities
than `pp`.  'ppft' uses a lighter weight communication requirement than
anything mentioned already (essentially, it creates a pipe to a connection
across a port and sends by RPC).  This allows you to set up a "remote python
instance", and then ship objects to it across a pipe to be executed
in parallel).

There's also `rpyc`, which essentially does the same thing.
(see: http://www.ibm.com/developerworks/linux/library/l-rpyc)
Classic mode in `rpyc` is especially powerful, if you unequivocally
trust your users -- if not, don't use 'Classic', and register all your
functions that you want to make available on the remote machine.

I have used both `rpyc` and `ppft` inside of ssh-tunneled connections,
which can be established automatically via `paramiko` or `pathos`.
For flexible single connections where I want to do something complex,
I often use `rpyc` inside a combination of `paramiko` and `pathos`
(depending on what needs to be done).  For parallel computing on a remote
machine, I use `pathos` plus `ppft`.  Both mechanism have been used
successfully to tunnel into remote machines at the DOE labs for over
10 years, including making remote connections to the ARCS spectrometer
at ORNL, and for creating more complex ad-hoc graphs of distributed
workers (that don't require a master-controller relationship) across
several DOE leadership class machines.  Any of the above is very simple
solution, as it doesn't try to do too much… which is why it works.

In this particular case, the Jupyter/Ipython environment for distributed
computing doesn't seem to me to be quite what you are looking for, at least
not just yet.  However, it would be pretty sweet if Jupyter/Ipython went
that way
eventually, as it can handle much more complexity across the pipe and in what
it can directly hook into than what I'm suggesting.

> Le 02/12/2015, Raymond a écrit :
>> We have been testing the use of Pyro4 as a method of loading remote
>> NeXus files within NeXpy, which also uses a Jupyter shell
>> (http://nexpy.github.io/nexpy/). It’s not part of the official
>> distribution yet, but I would be happy to discuss it with you outside
>> this list. As a fellow neutron scatterer, it might be relevant.
> Great!
> I will surely come back to you, but I first need to understand all
> concepts behind IPython 4 and Jupyter...
> BTW, NeXus is also on our TODO list of things to learn ;o)
> --
>     Frédéric MANTEGAZZA             CEA-Grenoble
>     Tel.     : 33 (0) 476 207 617   INAC/SPSMS/MDN
>     Fax      : 33 (0) 476 483 906   17, rue des Martyrs
>     Courriel : mantegazza at ill.fr    F-38054 Grenoble Cedex 09
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> https://mail.scipy.org/mailman/listinfo/ipython-dev


Mike McKerns
California Institute of Technology

More information about the IPython-dev mailing list