[IPython-dev] Kernels and parallel support?

Min RK benjaminrk at gmail.com
Fri Mar 21 12:10:43 EDT 2014


> On Mar 21, 2014, at 5:36, Doug Blank <doug.blank at gmail.com> wrote:
> 
>> On Thu, Mar 20, 2014 at 4:06 PM, Doug Blank <doug.blank at gmail.com> wrote:
>>> On Thu, Mar 20, 2014 at 2:19 PM, MinRK <benjaminrk at gmail.com> wrote:
>>> 
>>> 
>>> 
>>>> On Thu, Mar 20, 2014 at 10:58 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>> 
>>>> After reading:
>>>> 
>>>> http://ipython.org/ipython-doc/dev/parallel/parallel_intro.html
>>>> 
>>>> and related pages, I am unsure what the relationship between parallel
>>>> processing and kernels are. Can external kernels be used in a parallel
>>>> environment? Or is this part of IPython python-kernel specific? Does
>>>> it use ZMQ messages?
>>> 
>>> 
>>> Depends what you mean by 'external'. You can start engines at any time on
>>> any machine, and they can join a cluster.  Yes, IPython.parallel uses zmq.
>> 
>> If I wanted to explore using a cluster of 3rd-party kernels, such as
>> IHaskell or ICalico, in a parallel environment, where would I begin?
>> Has anyone done this before?
>> 
>> I see here:
>> 
>> http://ipython.org/ipython-doc/dev/parallel/magics.html#engines-as-kernels
>> 
>> that kernels connect to a controller to create an engine. If the
>> standard ipcontrollers can start up external kernels, I guess a place
>> to begin might be implementing what the %px magic does?
> 
> Along with those general questions, a specific one:
> 
> I would have thought that "ipcluster start --profile calico" would
> have used my c.KernelManager.kernel_cmd to start my kernels, but I
> can't get ipcluster to use it. Likewise, if I use the "Cluster" tab in
> the notebook to start a cluster, I also don't get my kernel.
> 
> How to start a cluster of third-party kernels?

It's not that simple. KernelManagers are not used in ipcluster. Normally, all engines are started manually (ipcluster is basically: for i in range(n): ipengine).  If you want to turn your kernel into an engine, you are going to have to teach it about the ipcontroller connection files, and connecting instead of binding.


> 
> -Doug
> 
>> -Doug
>> 
>>> -MinRK
>>> 
>>>> 
>>>> 
>>>> Thanks for any additional data,
>>>> 
>>>> -Doug
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



More information about the IPython-dev mailing list