[IPython-dev] Kernels and parallel support?
Min RK
benjaminrk at gmail.com
Fri Mar 21 12:10:43 EDT 2014
> On Mar 21, 2014, at 5:36, Doug Blank <doug.blank at gmail.com> wrote:
>
>> On Thu, Mar 20, 2014 at 4:06 PM, Doug Blank <doug.blank at gmail.com> wrote:
>>> On Thu, Mar 20, 2014 at 2:19 PM, MinRK <benjaminrk at gmail.com> wrote:
>>>
>>>
>>>
>>>> On Thu, Mar 20, 2014 at 10:58 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>>
>>>> After reading:
>>>>
>>>> http://ipython.org/ipython-doc/dev/parallel/parallel_intro.html
>>>>
>>>> and related pages, I am unsure what the relationship between parallel
>>>> processing and kernels are. Can external kernels be used in a parallel
>>>> environment? Or is this part of IPython python-kernel specific? Does
>>>> it use ZMQ messages?
>>>
>>>
>>> Depends what you mean by 'external'. You can start engines at any time on
>>> any machine, and they can join a cluster. Yes, IPython.parallel uses zmq.
>>
>> If I wanted to explore using a cluster of 3rd-party kernels, such as
>> IHaskell or ICalico, in a parallel environment, where would I begin?
>> Has anyone done this before?
>>
>> I see here:
>>
>> http://ipython.org/ipython-doc/dev/parallel/magics.html#engines-as-kernels
>>
>> that kernels connect to a controller to create an engine. If the
>> standard ipcontrollers can start up external kernels, I guess a place
>> to begin might be implementing what the %px magic does?
>
> Along with those general questions, a specific one:
>
> I would have thought that "ipcluster start --profile calico" would
> have used my c.KernelManager.kernel_cmd to start my kernels, but I
> can't get ipcluster to use it. Likewise, if I use the "Cluster" tab in
> the notebook to start a cluster, I also don't get my kernel.
>
> How to start a cluster of third-party kernels?
It's not that simple. KernelManagers are not used in ipcluster. Normally, all engines are started manually (ipcluster is basically: for i in range(n): ipengine). If you want to turn your kernel into an engine, you are going to have to teach it about the ipcontroller connection files, and connecting instead of binding.
>
> -Doug
>
>> -Doug
>>
>>> -MinRK
>>>
>>>>
>>>>
>>>> Thanks for any additional data,
>>>>
>>>> -Doug
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
More information about the IPython-dev
mailing list