[IPython-dev] From Pyro to IPython/Jupyter
Frédéric Mantegazza
mantegazza at ill.fr
Tue Dec 8 01:28:08 EST 2015
Le 07/12/2015, Michael a écrit :
> Pyro is good for python to non-python connections
Really ? AFAK, Pyro can only be used from Pythons scripts...
> however, if you are going python to python, a good alternative is to use
> ParallelPython `pp`, which has remote python servers. There's an
> easy-to-install version of it that's available as `ppft`, which provides
> better serialization capabilities than `pp`. 'ppft' uses a lighter
> weight communication requirement than anything mentioned already
> (essentially, it creates a pipe to a connection across a port and sends
> by RPC). This allows you to set up a "remote python instance", and then
> ship objects to it across a pipe to be executed (potentially
> in parallel).
>
> There's also `rpyc`, which essentially does the same thing.
> (see: http://www.ibm.com/developerworks/linux/library/l-rpyc)
> Classic mode in `rpyc` is especially powerful, if you unequivocally
> trust your users -- if not, don't use 'Classic', and register all your
> functions that you want to make available on the remote machine.
I will have a look at these framworks...
> I have used both `rpyc` and `ppft` inside of ssh-tunneled connections,
> which can be established automatically via `paramiko` or `pathos`.
> For flexible single connections where I want to do something complex,
> I often use `rpyc` inside a combination of `paramiko` and `pathos`
> (depending on what needs to be done). For parallel computing on a remote
> machine, I use `pathos` plus `ppft`. Both mechanism have been used
> successfully to tunnel into remote machines at the DOE labs for over
> 10 years, including making remote connections to the ARCS spectrometer
> at ORNL, and for creating more complex ad-hoc graphs of distributed
> workers (that don't require a master-controller relationship) across
> several DOE leadership class machines. Any of the above is very simple
> solution, as it doesn't try to do too much… which is why it works.
Thanks for your feedback. It's important to know how reliable solutions
are.
> In this particular case, the Jupyter/Ipython environment for distributed
> computing doesn't seem to me to be quite what you are looking for, at
> least not just yet.
Yes, I agree. For now, we will keep our architecture, and use IPython as
embedded shell, as we did before.
Maybe replace Pyro with one of the suggested framework.
--
Frédéric MANTEGAZZA CEA-Grenoble
Tel. : 33 (0) 476 207 617 INAC/SPSMS/MDN
Fax : 33 (0) 476 483 906 17, rue des Martyrs
Courriel : mantegazza at ill.fr F-38054 Grenoble Cedex 09
More information about the IPython-dev
mailing list