[IPython-dev] Enhancement proposal for kernel messaging protocol

[Takafumi Arakaki]

Dear IPython developers,

I wrote a PR for adding new option to messaging protocol [1]_.
Bussonnier Matthias and Thomas Kluyver suggested to discuss about
possible extensions for current messaging protocol, and then make an
IPEP before working on these extensions.  So, let me start the
discussion here.


1. Protocol introspection.

Currently, messaging protocol has no introspection mechanism.  It is
not a problem when the messaging protocol is used only for internal
communication.  However, there are various software using IPython via
messaging protocol.  Also, IPython version of the kernel and client
can be different, for example when used over ssh connection.
Therefore, we need some introspection mechanism to inform messaging
protocol supported by connecting kernel.

One way to inform supported messaging protocol is to add version
number to the messaging protocol.  There are several options for
version numbering:

a. Protocol version = IPython version.

   This may be the simplest way to do it.  But in development version,
   supported protocol will be unclear.

b. Semantic Versioning [2]_.

   This is some what close to how notebook version works (right?).
   Simple explanation: It takes the form of `x.y.z`.  Patch version
   `z` is incremented for bug fix.  Minor version `y` is incremented
   for backward compatible changes.  Major version `x` is incremented
   for backwards incompatible changes.

   For the messaging protocol, the next version number will be 1.1.0
   if we think the current protocol is public.  It will be 0.2.0 if we
   think the current protocol is still in development.

There are several ways to let client know the version number.

c. Add `version_request` / `version_reply` messaging protocol.
d. Add `version` slot to the message format.  It can be in
   the top level, `header` or `metadata`.
e. Send version number during "hand shake" (but there is no
   such stage for the current messaging protocol, I suppose?).


2. New optional `unique` key to `history_request`.

I propose [1]_ to add this new boolean key `unique` to
`history_request` [3]_.  When this key is specified and true,
`history_reply` contains only entries of unique input.  This is useful
for searching history using `hist_access_type: 'search'`.  This
operation can be done in client side.  However, sending large number
of entries over messaging protocol is time consuming.  One case this
`unique` key can be critical is when using `history_request` in an
interactive UI to search IPython history in as-you-type manner
("QuickSilver style"; for example, my Emacs client EIN [4]_ has such
UI).  For such kind of UI speed is critical and it is preferred to be
done in the kernel side.

(As you can see, this is the reason why I made the PR in the first
place!)


3. Messaging protocol to pass structured data to kernel.

Currently it is not possible to pass structured data such as dict from
client to kernel.  You can generate Python code and use
`execute_request`, but it is difficult to do this properly in
non-Python client such as Javascript client.  Note that it is
already possible to pass structured data to client from kernel
by using JSON repr.

(I think I saw some argument related to this issue tracker, but
I can't find it now.  Please post the link if you know.)

There are several ways to solve this problem:

a. Add RPC-like message `call_request` / `call_reply`.  Client can
   execute a short code to register "methods" in kernel to use this
   RPC.  For example `__import__('mylib.ipyutils').register()`.

b. Add `override_variables` slot to `execute_request`.  This slot
   takes a dict and the value in the dict will override the value
   correspond to its key.  For example, client can execute `func(x)`
   with `override_variables: {'x': 1}`.

   The problem is that it contaminates user's namespace.  To solve this
   problem, we can add another `namespace` slot to `execute_request`.
   This way, client can import its supporting library without
   contaminating user's namespace and execute its functions using
   `override_variables`.  For example, client can issue `execute_request`
   like this::

     {'code' "import mylib; mylib.ipyutils.func(x)",
      'namespace': 'mylib',
      'override_variables': {'x': 1}}

   Merit of this approach comparing to the other one (`call_request`)
   is that `namespace` can be used for another purpose.  For example,
   notebook client can have separated namespace for each worksheet,
   to avoid namespace collision.  Sharing data between namespaces can
   be easily done by using singleton object.  Other possible application
   is editor integration.  For example, you have associate one namespace
   to each file (= python module) opened in the editor.

.. [1] https://github.com/ipython/ipython/pull/2609
.. [2] http://semver.org/
.. [3] http://ipython.org/ipython-doc/dev/development/messaging.html#history
.. [4] http://github.com/tkf/emacs-ipython-notebook

---
Takafumi Arakaki