[IPython-dev] From Pyro to IPython/Jupyter

[Frédéric Mantegazza]

Le 07/12/2015, Michael a écrit :

> Pyro is good for python to non-python connections

Really ? AFAK, Pyro can only be used from Pythons scripts...

> however, if you are going python to python, a good alternative is to use
> ParallelPython `pp`, which has remote python servers. There's an
> easy-to-install version of it that's available as `ppft`, which provides
> better serialization capabilities than `pp`.  'ppft' uses a lighter
> weight communication requirement than anything mentioned already
> (essentially, it creates a pipe to a connection across a port and sends
> by RPC).  This allows you to set up a "remote python instance", and then
> ship objects to it across a pipe to be executed (potentially
> in parallel).
> 
> There's also `rpyc`, which essentially does the same thing.
> (see: http://www.ibm.com/developerworks/linux/library/l-rpyc)
> Classic mode in `rpyc` is especially powerful, if you unequivocally
> trust your users -- if not, don't use 'Classic', and register all your
> functions that you want to make available on the remote machine.

I will have a look at these framworks...

> I have used both `rpyc` and `ppft` inside of ssh-tunneled connections,
> which can be established automatically via `paramiko` or `pathos`.
> For flexible single connections where I want to do something complex,
> I often use `rpyc` inside a combination of `paramiko` and `pathos`
> (depending on what needs to be done).  For parallel computing on a remote
> machine, I use `pathos` plus `ppft`.  Both mechanism have been used
> successfully to tunnel into remote machines at the DOE labs for over
> 10 years, including making remote connections to the ARCS spectrometer
> at ORNL, and for creating more complex ad-hoc graphs of distributed
> workers (that don't require a master-controller relationship) across
> several DOE leadership class machines.  Any of the above is very simple
> solution, as it doesn't try to do too much… which is why it works.

Thanks for your feedback. It's important to know how reliable solutions
are.

> In this particular case, the Jupyter/Ipython environment for distributed
> computing doesn't seem to me to be quite what you are looking for, at
> least not just yet.

Yes, I agree. For now, we will keep our architecture, and use IPython as
embedded shell, as we did before.

Maybe replace Pyro with one of the suggested framework.

-- 
    Frédéric MANTEGAZZA             CEA-Grenoble
    Tel.     : 33 (0) 476 207 617   INAC/SPSMS/MDN
    Fax      : 33 (0) 476 483 906   17, rue des Martyrs
    Courriel : mantegazza at ill.fr    F-38054 Grenoble Cedex 09