[IPython-dev] Parallel support for external wrapperkernels in master?

Doug Blank doug.blank at gmail.com
Tue Sep 9 22:41:35 EDT 2014


On Tue, Sep 9, 2014 at 5:05 PM, Doug Blank <doug.blank at gmail.com> wrote:

> On Tue, Sep 9, 2014 at 3:05 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>> There aren't currently plans to make IPython.parallel language agnostic
>> (it will remain an IPython project, not a Jupyter one), because it doesn't
>> make much sense if the Client and Kernel aren't in the same language, so
>> making a kernel available from a new language should also mean writing a
>> new Client object in order for it to be useful.  This is distinctly
>> different from the regular kernel-client relationship, which mostly talk
>> code-as-text, which doesn't require the frontend to know much about the
>> kernel. IPython.parallel talks code-as-code (pickle), because it's meant
>> for interacting with actual results, rather than just their representation.
>>
>
> Ok, I understand. Thinking about this a bit, it should be possible to wrap
> up the Client so that we can start our Python-based kernels on each node in
> the cluster, and then move values back and forth through some mechanism
> (maybe just strings, or maybe a serialize/unserialize process). That way,
> we can still use the IPython infrastructure, and our kernels as
> well---without having to reinvent too much. Will report back...
>

With just a small bit of code [1], we can start up a wrapperkernel-based
kernel on each node of the cluster, and call it directly (without an
additional process) to evaluate code. Some of these languages use Python
types directly (like our Prolog and Scheme), and so it fits right in with
the language with no additional work. Other kernels (like Octave) may take
a little more work. In any event, I was able to run Scheme code tonight in
parallel, interactively... a longtime goal!

With some more refinement, this will be part of a collection of "magics"...
metacommands for Python-based 3rd-party kernels. Any of those languages
should be able to be run in parallel, at least in a rudimentary fashion.
Will announce here soon, as soon as we finish up some loose ends of the
project, currently called jupyter_kernel.

Thanks again for the hints and feedback,

-Doug

 [1]
https://github.com/blink1073/jupyter_kernel/blob/master/jupyter_kernel/magics/parallel_magic.py


> -Doug
>
>
>>
>> -MinRK
>>
>> On Tue, Sep 9, 2014 at 11:47 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>>> On Tue, Sep 9, 2014 at 2:25 PM, Fernando Perez <fperez.net at gmail.com>
>>> wrote:
>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 11:23 AM, Doug Blank <doug.blank at gmail.com>
>>>> wrote:
>>>>
>>>>> Just wondering if parallel support for external kernels (based on
>>>>> wrapperkernel) is working in master yet, or planned to work soon?
>>>>>
>>>>
>>>> You always doing the crazy stuff ;)
>>>>
>>>> No, we haven't really looked into the intersection of the parallel
>>>> machinery and alternate kernels.  So go forward and blaze trail! :)
>>>>
>>>
>>> This isn't too crazy, now that y'all have the wrapperkernel nicely
>>> factored out :)
>>>
>>> This must be pretty close to being able to be made to work, at least for
>>> those kernels that are Python-based, right?
>>>
>>> Just can't wait to run the chatbot_kernel in parallel :)
>>>
>>> -Doug
>>>
>>>
>>>>
>>>> Cheers,
>>>>
>>>> f
>>>> --
>>>> Fernando Perez (@fperez_org; http://fperez.org)
>>>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>>>> fernando.perez-at-berkeley: contact me here for any direct mail
>>>>
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140909/f4a233e4/attachment.html>


More information about the IPython-dev mailing list