[IPython-dev] Kernels and parallel support?

[Doug Blank]

On Fri, Mar 21, 2014 at 12:10 PM, Min RK <benjaminrk at gmail.com> wrote:
>
>> On Mar 21, 2014, at 5:36, Doug Blank <doug.blank at gmail.com> wrote:
>>
>>> On Thu, Mar 20, 2014 at 4:06 PM, Doug Blank <doug.blank at gmail.com> wrote:
>>>> On Thu, Mar 20, 2014 at 2:19 PM, MinRK <benjaminrk at gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>> On Thu, Mar 20, 2014 at 10:58 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>>>
>>>>> After reading:
>>>>>
>>>>> http://ipython.org/ipython-doc/dev/parallel/parallel_intro.html
>>>>>
>>>>> and related pages, I am unsure what the relationship between parallel
>>>>> processing and kernels are. Can external kernels be used in a parallel
>>>>> environment? Or is this part of IPython python-kernel specific? Does
>>>>> it use ZMQ messages?
>>>>
>>>>
>>>> Depends what you mean by 'external'. You can start engines at any time on
>>>> any machine, and they can join a cluster.  Yes, IPython.parallel uses zmq.
>>>
>>> If I wanted to explore using a cluster of 3rd-party kernels, such as
>>> IHaskell or ICalico, in a parallel environment, where would I begin?
>>> Has anyone done this before?
>>>
>>> I see here:
>>>
>>> http://ipython.org/ipython-doc/dev/parallel/magics.html#engines-as-kernels
>>>
>>> that kernels connect to a controller to create an engine. If the
>>> standard ipcontrollers can start up external kernels, I guess a place
>>> to begin might be implementing what the %px magic does?
>>
>> Along with those general questions, a specific one:
>>
>> I would have thought that "ipcluster start --profile calico" would
>> have used my c.KernelManager.kernel_cmd to start my kernels, but I
>> can't get ipcluster to use it. Likewise, if I use the "Cluster" tab in
>> the notebook to start a cluster, I also don't get my kernel.
>>
>> How to start a cluster of third-party kernels?
>
> It's not that simple. KernelManagers are not used in ipcluster. Normally, all engines are started manually (ipcluster is basically: for i in range(n): ipengine).  If you want to turn your kernel into an engine, you are going to have to teach it about the ipcontroller connection files, and connecting instead of binding.

First, the idea of allowing non-Python kernels to be able to easily
become parallelized is a fantastic possibility! There are many
languages/systems that could benefit from this.

I have gone through the anatomy of the communication between a hub,
engines, and a notebook here:

http://wiki.roboteducation.org/IPython_Parallel_API

(corrections of misunderstandings welcome). Two questions:

1) In addition to adding the ipcontroller communications needed, it
appears that pickling and eval are used to execute remote code (even
though pack/unpack settings in ipcontroller_*.py files are marked as
"json"). Of course, this won't work for non-Python kernels. Could a
real json representation be used for values, getting rid of the need
for pickle?

2) Knowing that "Full, integrated support for non-Python kernels" is a
line item in the 3.0 roadmap [1], and this is mentioned in
IPython/parallel/engine/engine.py:

# FIXME: This is a hack until IPKernelApp and IPEngineApp can be fully merged
app = IPKernelApp(parent=self, shell=self.kernel.shell,
kernel=self.kernel, log=self.log)

is there a chance that Kernels and Engines will be merged this years
so that parallel processing for non-Python kernels would be possible?

Thanks for any additional information!

-Doug

[1] - https://hackpad.com/IPython-Winter-2014-Development-Meeting-fKrExqKCWmC

>
>>
>> -Doug
>>
>>> -Doug
>>>
>>>> -MinRK
>>>>
>>>>>
>>>>>
>>>>> Thanks for any additional data,
>>>>>
>>>>> -Doug
>>>>> _______________________________________________
>>>>> IPython-dev mailing list
>>>>> IPython-dev at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev