[IPython-dev] Implementing inline images in a kernel

Thomas Kluyver takowl at gmail.com
Tue Jan 28 20:33:50 EST 2014


Hi Doug,

On 28 January 2014 16:58, Doug Blank <doug.blank at gmail.com> wrote:

> In implementing a new kernel, I can appropriately turn returned values
> into their rich representation messages. But I'm wrestling with the idea
> that the kernel could somehow, in the middle of computing, send such rich
> representations to be displayed.
>
> I've seen bits of code that appear to examine strings to see if they begin
> with certain signatures that could detect PNG or JPEG encodings. Is that
> how it would work... you just send a 64-bit encoded string to the front
> end, and it automatically detects it as a rich representation? Or is there
> a general standard for marking strings automatically as rich
> representations? Or must the kernel construct a message to get any rich
> reps? I could make the kernel monitor for special strings (sent from lower
> levels), intercept them, and then send the special rich message.
>
> If this question is confusing, perhaps it is because my kernel interfaces
> with many subsystems that can generate strings to the output, but hooking
> up the lower-level to the rich message system would be hard. I suspect that
> this is similar to making an external language, like R, generate images
> inline. How does it do it?
>

The code in question does have to know something about the message spec; we
don't try to detect things like images from a simple stream. The code needs
to builds a dictionary keyed by the mimetypes of the outputs it offers, and
send that to the frontend in a display_data message:
http://ipython.org/ipython-doc/dev/development/messaging.html#display-data

So a simple function to display a PNG might look like this:

 def display(pngdata):

shell_channel.send_msg(type='display_data',

content={'source': 'my display func', #Not sure what this is used for

'data': {'image/png': base64encode(pngdata)},

'metadata': {'image/png': {'width':640, 'height':480}},

},

parent=last_execute_request

)

(this is abstract, not written against any real kernel APIs)

For Python, we've extended Python's concept of a __repr__ magic method to
support other formats, e.g. _repr_html_. Then we have a convenient
function, IPython.display.display(), which builds the dictionary of output
formats and takes care of sending it.

The R magic is a bit of a special case: it gets R to save the images it
generates to files in a temporary directory, and after the R code has
finished, it reads all the files from that directory back and sends
display_data messages for them. That's only practical because of the way
R's plotting backends work, and it still leads to problems on Windows.

Your kernel could intercept strings written to stdout, spot certain
patterns, and turn them into display messages, but it has the feel of a bad
idea - what if you actually want to print the first hundred bytes of a PNG,
for instance.

I hope that helps,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140128/47f7953e/attachment.html>


More information about the IPython-dev mailing list