[IPython-dev] Implementing inline images in a kernel

Doug Blank doug.blank at gmail.com
Tue Jan 28 21:02:11 EST 2014


On Tue, Jan 28, 2014 at 8:33 PM, Thomas Kluyver <takowl at gmail.com> wrote:

> Hi Doug,
>
> On 28 January 2014 16:58, Doug Blank <doug.blank at gmail.com> wrote:
>
>> In implementing a new kernel, I can appropriately turn returned values
>> into their rich representation messages. But I'm wrestling with the idea
>> that the kernel could somehow, in the middle of computing, send such rich
>> representations to be displayed.
>>
>> I've seen bits of code that appear to examine strings to see if they
>> begin with certain signatures that could detect PNG or JPEG encodings. Is
>> that how it would work... you just send a 64-bit encoded string to the
>> front end, and it automatically detects it as a rich representation? Or is
>> there a general standard for marking strings automatically as rich
>> representations? Or must the kernel construct a message to get any rich
>> reps? I could make the kernel monitor for special strings (sent from lower
>> levels), intercept them, and then send the special rich message.
>>
>> If this question is confusing, perhaps it is because my kernel interfaces
>> with many subsystems that can generate strings to the output, but hooking
>> up the lower-level to the rich message system would be hard. I suspect that
>> this is similar to making an external language, like R, generate images
>> inline. How does it do it?
>>
>
> The code in question does have to know something about the message spec;
> we don't try to detect things like images from a simple stream. The code
> needs to builds a dictionary keyed by the mimetypes of the outputs it
> offers, and send that to the frontend in a display_data message:
> http://ipython.org/ipython-doc/dev/development/messaging.html#display-data
>
> So a simple function to display a PNG might look like this:
>
> def display(pngdata):
>
> shell_channel.send_msg(type='display_data',
>
> content={'source': 'my display func', #Not sure what this is used for
>
> 'data': {'image/png': base64encode(pngdata)},
>
> 'metadata': {'image/png': {'width':640, 'height':480}},
>
> },
>
> parent=last_execute_request
>
> )
>
> (this is abstract, not written against any real kernel APIs)
>
>
Thanks, yes, I have pretty much that in the high-level kernel. I ask the
lower parts to eval the notebook cell code, and I can convert that as you
show.


> For Python, we've extended Python's concept of a __repr__ magic method to
> support other formats, e.g. _repr_html_. Then we have a convenient
> function, IPython.display.display(), which builds the dictionary of output
> formats and takes care of sending it.
>
>
Right, I could do that, but would require doing it for each language at the
lower level. Looking for a solution at the higher level where everything
must go through on the way to the frontend.


> The R magic is a bit of a special case: it gets R to save the images it
> generates to files in a temporary directory, and after the R code has
> finished, it reads all the files from that directory back and sends
> display_data messages for them. That's only practical because of the way
> R's plotting backends work, and it still leads to problems on Windows.
>
>
Interesting, but I think I can do better than that with this kernel.


> Your kernel could intercept strings written to stdout, spot certain
> patterns, and turn them into display messages, but it has the feel of a bad
> idea - what if you actually want to print the first hundred bytes of a PNG,
> for instance.
>

Perhaps I can make it smell better :) Yes, this method would be embedding
rich text in a string, but in such a manner that wouldn't be confused with
real strings.

But maybe you are correct... maybe I can make available to the low level a
display() function... I'll try that first.

Thanks!

-Doug


>
> I hope that helps,
> Thomas
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140128/ecaccd66/attachment.html>


More information about the IPython-dev mailing list