[IPython-dev] Kernels and parallel support?

MinRK benjaminrk at gmail.com
Sat Mar 22 13:59:50 EDT 2014


On Sat, Mar 22, 2014 at 6:47 AM, Doug Blank <doug.blank at gmail.com> wrote:

> On Fri, Mar 21, 2014 at 12:10 PM, Min RK <benjaminrk at gmail.com> wrote:
> >
> >> On Mar 21, 2014, at 5:36, Doug Blank <doug.blank at gmail.com> wrote:
> >>
> >>> On Thu, Mar 20, 2014 at 4:06 PM, Doug Blank <doug.blank at gmail.com>
> wrote:
> >>>> On Thu, Mar 20, 2014 at 2:19 PM, MinRK <benjaminrk at gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On Thu, Mar 20, 2014 at 10:58 AM, Doug Blank <doug.blank at gmail.com>
> wrote:
> >>>>>
> >>>>> After reading:
> >>>>>
> >>>>> http://ipython.org/ipython-doc/dev/parallel/parallel_intro.html
> >>>>>
> >>>>> and related pages, I am unsure what the relationship between parallel
> >>>>> processing and kernels are. Can external kernels be used in a
> parallel
> >>>>> environment? Or is this part of IPython python-kernel specific? Does
> >>>>> it use ZMQ messages?
> >>>>
> >>>>
> >>>> Depends what you mean by 'external'. You can start engines at any
> time on
> >>>> any machine, and they can join a cluster.  Yes, IPython.parallel uses
> zmq.
> >>>
> >>> If I wanted to explore using a cluster of 3rd-party kernels, such as
> >>> IHaskell or ICalico, in a parallel environment, where would I begin?
> >>> Has anyone done this before?
> >>>
> >>> I see here:
> >>>
> >>>
> http://ipython.org/ipython-doc/dev/parallel/magics.html#engines-as-kernels
> >>>
> >>> that kernels connect to a controller to create an engine. If the
> >>> standard ipcontrollers can start up external kernels, I guess a place
> >>> to begin might be implementing what the %px magic does?
> >>
> >> Along with those general questions, a specific one:
> >>
> >> I would have thought that "ipcluster start --profile calico" would
> >> have used my c.KernelManager.kernel_cmd to start my kernels, but I
> >> can't get ipcluster to use it. Likewise, if I use the "Cluster" tab in
> >> the notebook to start a cluster, I also don't get my kernel.
> >>
> >> How to start a cluster of third-party kernels?
> >
> > It's not that simple. KernelManagers are not used in ipcluster.
> Normally, all engines are started manually (ipcluster is basically: for i
> in range(n): ipengine).  If you want to turn your kernel into an engine,
> you are going to have to teach it about the ipcontroller connection files,
> and connecting instead of binding.
>
> First, the idea of allowing non-Python kernels to be able to easily
> become parallelized is a fantastic possibility! There are many
> languages/systems that could benefit from this.
>
> I have gone through the anatomy of the communication between a hub,
> engines, and a notebook here:
>
> http://wiki.roboteducation.org/IPython_Parallel_API
>
> (corrections of misunderstandings welcome). Two questions:
>
> 1) In addition to adding the ipcontroller communications needed, it
> appears that pickling and eval are used to execute remote code (even
> though pack/unpack settings in ipcontroller_*.py files are marked as
> "json"). Of course, this won't work for non-Python kernels. Could a
> real json representation be used for values, getting rid of the need
> for pickle?
>

JSON *is* used for message serialization. However, the main message used in
IPython.parallel is `apply`, which sends native functions and data, and
thus uses native serialization (pickle on Python). There are no plans to
change this. Code-as-text execution (%px, view.execute), however, uses the
same execute_request message that is used for execution in the notebook,
etc., and should work fine on a non-Python kernel.


>
> 2) Knowing that "Full, integrated support for non-Python kernels" is a
> line item in the 3.0 roadmap [1], and this is mentioned in
> IPython/parallel/engine/engine.py:
>
> # FIXME: This is a hack until IPKernelApp and IPEngineApp can be fully
> merged
> app = IPKernelApp(parent=self, shell=self.kernel.shell,
> kernel=self.kernel, log=self.log)
>
> is there a chance that Kernels and Engines will be merged this year
> so that parallel processing for non-Python kernels would be possible?
>

Yes, this is my hope.


>
> Thanks for any additional information!
>
> -Doug
>
> [1] -
> https://hackpad.com/IPython-Winter-2014-Development-Meeting-fKrExqKCWmC
>
> >
> >>
> >> -Doug
> >>
> >>> -Doug
> >>>
> >>>> -MinRK
> >>>>
> >>>>>
> >>>>>
> >>>>> Thanks for any additional data,
> >>>>>
> >>>>> -Doug
> >>>>> _______________________________________________
> >>>>> IPython-dev mailing list
> >>>>> IPython-dev at scipy.org
> >>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> IPython-dev mailing list
> >>>> IPython-dev at scipy.org
> >>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>>>
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140322/b6e7a6b4/attachment.html>


More information about the IPython-dev mailing list