[IPython-dev] Enhancement proposal for kernel messaging protocol

Takafumi Arakaki aka.tkf at gmail.com
Fri Dec 7 06:00:25 EST 2012


Thanks for the comment.  I opened a pull request for protocol
versioning in github sometime ago:
https://github.com/ipython/ipython/pull/2649

For history_request enhancement, I already made the patches but
I will wait for the relevant PR to be merged:
https://github.com/ipython/ipython/pull/2609


On Wed, Nov 28, 2012 at 4:35 AM, Brian Granger <ellisonbg at gmail.com> wrote:
>> I wrote a PR for adding new option to messaging protocol [1]_.
>> Bussonnier Matthias and Thomas Kluyver suggested to discuss about
>> possible extensions for current messaging protocol, and then make an
>> IPEP before working on these extensions.  So, let me start the
>> discussion here.
>
> Great, thanks for getting this started.
>
>>
>> 1. Protocol introspection.
>>
>> Currently, messaging protocol has no introspection mechanism.  It is
>> not a problem when the messaging protocol is used only for internal
>> communication.  However, there are various software using IPython via
>> messaging protocol.  Also, IPython version of the kernel and client
>> can be different, for example when used over ssh connection.
>> Therefore, we need some introspection mechanism to inform messaging
>> protocol supported by connecting kernel.
>
> I agree with this assessment.
>
>> One way to inform supported messaging protocol is to add version
>> number to the messaging protocol.  There are several options for
>> version numbering:
>>
>> a. Protocol version = IPython version.
>>
>>    This may be the simplest way to do it.  But in development version,
>>    supported protocol will be unclear.
>
> The problem with this is that the message protocol is quite stable
> over the ipython versions.  This is why the notebook format has its
> own version numbers.
>
>> b. Semantic Versioning [2]_.
>>
>>    This is some what close to how notebook version works (right?).
>>    Simple explanation: It takes the form of `x.y.z`.  Patch version
>>    `z` is incremented for bug fix.  Minor version `y` is incremented
>>    for backward compatible changes.  Major version `x` is incremented
>>    for backwards incompatible changes.
>>
>>    For the messaging protocol, the next version number will be 1.1.0
>>    if we think the current protocol is public.  It will be 0.2.0 if we
>>    think the current protocol is still in development.
>>
>> There are several ways to let client know the version number.
>
> I would use the exact same convention that we do in the notebook format.  X.Y:
>
> X = major versions, incompatible
> Y = minor versions, compatible
>
>> c. Add `version_request` / `version_reply` messaging protocol.
>
> Yes, this is what I would prefer.
>
>> d. Add `version` slot to the message format.  It can be in
>>    the top level, `header` or `metadata`.
>> e. Send version number during "hand shake" (but there is no
>>    such stage for the current messaging protocol, I suppose?).
>>
>>
>> 2. New optional `unique` key to `history_request`.
>>
>> I propose [1]_ to add this new boolean key `unique` to
>> `history_request` [3]_.  When this key is specified and true,
>> `history_reply` contains only entries of unique input.  This is useful
>> for searching history using `hist_access_type: 'search'`.  This
>> operation can be done in client side.  However, sending large number
>> of entries over messaging protocol is time consuming.  One case this
>> `unique` key can be critical is when using `history_request` in an
>> interactive UI to search IPython history in as-you-type manner
>> ("QuickSilver style"; for example, my Emacs client EIN [4]_ has such
>> UI).  For such kind of UI speed is critical and it is preferred to be
>> done in the kernel side.
>>
>> (As you can see, this is the reason why I made the PR in the first
>> place!)
>
> Yes, I can see that this would be a good thing to have.
>
>>
>> 3. Messaging protocol to pass structured data to kernel.
>>
>> Currently it is not possible to pass structured data such as dict from
>> client to kernel.  You can generate Python code and use
>> `execute_request`, but it is difficult to do this properly in
>> non-Python client such as Javascript client.  Note that it is
>> already possible to pass structured data to client from kernel
>> by using JSON repr.
>>
>> (I think I saw some argument related to this issue tracker, but
>> I can't find it now.  Please post the link if you know.)
>>
>> There are several ways to solve this problem:
>>
>> a. Add RPC-like message `call_request` / `call_reply`.  Client can
>>    execute a short code to register "methods" in kernel to use this
>>    RPC.  For example `__import__('mylib.ipyutils').register()`.
>>
>> b. Add `override_variables` slot to `execute_request`.  This slot
>>    takes a dict and the value in the dict will override the value
>>    correspond to its key.  For example, client can execute `func(x)`
>>    with `override_variables: {'x': 1}`.
>>
>>    The problem is that it contaminates user's namespace.  To solve this
>>    problem, we can add another `namespace` slot to `execute_request`.
>>    This way, client can import its supporting library without
>>    contaminating user's namespace and execute its functions using
>>    `override_variables`.  For example, client can issue `execute_request`
>>    like this::
>>
>>      {'code' "import mylib; mylib.ipyutils.func(x)",
>>       'namespace': 'mylib',
>>       'override_variables': {'x': 1}}
>>
>>    Merit of this approach comparing to the other one (`call_request`)
>>    is that `namespace` can be used for another purpose.  For example,
>>    notebook client can have separated namespace for each worksheet,
>>    to avoid namespace collision.  Sharing data between namespaces can
>>    be easily done by using singleton object.  Other possible application
>>    is editor integration.  For example, you have associate one namespace
>>    to each file (= python module) opened in the editor.
>
> Even though we will eventually need this, I don't think we are ready
> to tackle it yet.  This will happen as part of the work on interactive
> widgets that will hopefully get started in Jan.
>
> But, for the first two of these enhancements, I think they sound great
> and don't need an IPEP.  A separate PR for each would be great :)
>
> Cheers,
>
> Brian
>
>> .. [1] https://github.com/ipython/ipython/pull/2609
>> .. [2] http://semver.org/
>> .. [3] http://ipython.org/ipython-doc/dev/development/messaging.html#history
>> .. [4] http://github.com/tkf/emacs-ipython-notebook
>>
>> ---
>> Takafumi Arakaki
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> --
> Brian E. Granger
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu and ellisonbg at gmail.com
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



More information about the IPython-dev mailing list