[IPython-dev] Enhancement proposal for kernel messaging protocol
Takafumi Arakaki
aka.tkf at gmail.com
Fri Dec 7 06:00:25 EST 2012
Thanks for the comment. I opened a pull request for protocol
versioning in github sometime ago:
https://github.com/ipython/ipython/pull/2649
For history_request enhancement, I already made the patches but
I will wait for the relevant PR to be merged:
https://github.com/ipython/ipython/pull/2609
On Wed, Nov 28, 2012 at 4:35 AM, Brian Granger <ellisonbg at gmail.com> wrote:
>> I wrote a PR for adding new option to messaging protocol [1]_.
>> Bussonnier Matthias and Thomas Kluyver suggested to discuss about
>> possible extensions for current messaging protocol, and then make an
>> IPEP before working on these extensions. So, let me start the
>> discussion here.
>
> Great, thanks for getting this started.
>
>>
>> 1. Protocol introspection.
>>
>> Currently, messaging protocol has no introspection mechanism. It is
>> not a problem when the messaging protocol is used only for internal
>> communication. However, there are various software using IPython via
>> messaging protocol. Also, IPython version of the kernel and client
>> can be different, for example when used over ssh connection.
>> Therefore, we need some introspection mechanism to inform messaging
>> protocol supported by connecting kernel.
>
> I agree with this assessment.
>
>> One way to inform supported messaging protocol is to add version
>> number to the messaging protocol. There are several options for
>> version numbering:
>>
>> a. Protocol version = IPython version.
>>
>> This may be the simplest way to do it. But in development version,
>> supported protocol will be unclear.
>
> The problem with this is that the message protocol is quite stable
> over the ipython versions. This is why the notebook format has its
> own version numbers.
>
>> b. Semantic Versioning [2]_.
>>
>> This is some what close to how notebook version works (right?).
>> Simple explanation: It takes the form of `x.y.z`. Patch version
>> `z` is incremented for bug fix. Minor version `y` is incremented
>> for backward compatible changes. Major version `x` is incremented
>> for backwards incompatible changes.
>>
>> For the messaging protocol, the next version number will be 1.1.0
>> if we think the current protocol is public. It will be 0.2.0 if we
>> think the current protocol is still in development.
>>
>> There are several ways to let client know the version number.
>
> I would use the exact same convention that we do in the notebook format. X.Y:
>
> X = major versions, incompatible
> Y = minor versions, compatible
>
>> c. Add `version_request` / `version_reply` messaging protocol.
>
> Yes, this is what I would prefer.
>
>> d. Add `version` slot to the message format. It can be in
>> the top level, `header` or `metadata`.
>> e. Send version number during "hand shake" (but there is no
>> such stage for the current messaging protocol, I suppose?).
>>
>>
>> 2. New optional `unique` key to `history_request`.
>>
>> I propose [1]_ to add this new boolean key `unique` to
>> `history_request` [3]_. When this key is specified and true,
>> `history_reply` contains only entries of unique input. This is useful
>> for searching history using `hist_access_type: 'search'`. This
>> operation can be done in client side. However, sending large number
>> of entries over messaging protocol is time consuming. One case this
>> `unique` key can be critical is when using `history_request` in an
>> interactive UI to search IPython history in as-you-type manner
>> ("QuickSilver style"; for example, my Emacs client EIN [4]_ has such
>> UI). For such kind of UI speed is critical and it is preferred to be
>> done in the kernel side.
>>
>> (As you can see, this is the reason why I made the PR in the first
>> place!)
>
> Yes, I can see that this would be a good thing to have.
>
>>
>> 3. Messaging protocol to pass structured data to kernel.
>>
>> Currently it is not possible to pass structured data such as dict from
>> client to kernel. You can generate Python code and use
>> `execute_request`, but it is difficult to do this properly in
>> non-Python client such as Javascript client. Note that it is
>> already possible to pass structured data to client from kernel
>> by using JSON repr.
>>
>> (I think I saw some argument related to this issue tracker, but
>> I can't find it now. Please post the link if you know.)
>>
>> There are several ways to solve this problem:
>>
>> a. Add RPC-like message `call_request` / `call_reply`. Client can
>> execute a short code to register "methods" in kernel to use this
>> RPC. For example `__import__('mylib.ipyutils').register()`.
>>
>> b. Add `override_variables` slot to `execute_request`. This slot
>> takes a dict and the value in the dict will override the value
>> correspond to its key. For example, client can execute `func(x)`
>> with `override_variables: {'x': 1}`.
>>
>> The problem is that it contaminates user's namespace. To solve this
>> problem, we can add another `namespace` slot to `execute_request`.
>> This way, client can import its supporting library without
>> contaminating user's namespace and execute its functions using
>> `override_variables`. For example, client can issue `execute_request`
>> like this::
>>
>> {'code' "import mylib; mylib.ipyutils.func(x)",
>> 'namespace': 'mylib',
>> 'override_variables': {'x': 1}}
>>
>> Merit of this approach comparing to the other one (`call_request`)
>> is that `namespace` can be used for another purpose. For example,
>> notebook client can have separated namespace for each worksheet,
>> to avoid namespace collision. Sharing data between namespaces can
>> be easily done by using singleton object. Other possible application
>> is editor integration. For example, you have associate one namespace
>> to each file (= python module) opened in the editor.
>
> Even though we will eventually need this, I don't think we are ready
> to tackle it yet. This will happen as part of the work on interactive
> widgets that will hopefully get started in Jan.
>
> But, for the first two of these enhancements, I think they sound great
> and don't need an IPEP. A separate PR for each would be great :)
>
> Cheers,
>
> Brian
>
>> .. [1] https://github.com/ipython/ipython/pull/2609
>> .. [2] http://semver.org/
>> .. [3] http://ipython.org/ipython-doc/dev/development/messaging.html#history
>> .. [4] http://github.com/tkf/emacs-ipython-notebook
>>
>> ---
>> Takafumi Arakaki
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> --
> Brian E. Granger
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu and ellisonbg at gmail.com
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
More information about the IPython-dev
mailing list