<br><br><div class="gmail_quote">On Tue, Jun 19, 2012 at 4:20 PM, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
When the metadata PR come up, I was originally going to vote -1 on it<br>
because of this issue. I sat on it for a while and in the end decided<br>
that it was OK because I think the need for metadata is already upon<br>
us even though we don't have an actual usage case in our own code base<br>
(for example, we don't have a metadata UI in the notebook web app).<br>
<br>
There is a fine line to walk here. On one hand, I completely agree<br>
with you that we should try to future-proof the notebook format to<br>
minimize disruptive format changes. On the other hand, adding things<br>
too soon leads to even more potential disruption for the following<br>
reason. As I developed the notebook format and notebook UI last<br>
summer, there were multiple situations where I added something to the<br>
notebook format before I actually used it in the UI. In many of these<br>
cases, when I did get around to developing the UI for it, I realized<br>
that my original thoughts on that element were incomplete. It wasn't<br>
until I wrote the UI that used the data that I realized exactly what<br>
the format of that data needed to be. As a result, I had to go back<br>
and modify the notebook format. After a few iterations of this, I<br>
realized that this approach was broken and started to enforce the<br>
following simple rule on myself: don't add it to the notebook format<br>
until I am ready to write the UI code that uses it. That rule served<br>
me very well last summer.<br>
<br>
This is why for example the notebook and cells do not currently have<br>
any timestamp information (even though I think we will eventually want<br>
it). The one notebook feature (which I regret adding to the format)<br>
that doesn't have a UI is the multiple worksheets. We absolutely want<br>
that as a feature, I just wish I had waited to add it to the notebook<br>
format. When we do implement the mulitple worksheet UI, it is likely<br>
we will want to go back and make changes to the notebook format to<br>
better reflect the UI (for example, we will probably want to persist<br>
which worksheet is active/open).<br></blockquote><div><br></div><div>I couldn't agree less. There is simply no reason that adding support for multiple worksheets in future versions of IPython should render single-sheet notebooks unreadable in 0.13, just like adding new metadata should not make the notebook artificially unreadable.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
For the cell and worksheet metadata, I knew we would eventually need<br>
it and I didn't want to hold up the beta release any longer. But<br>
there are still unanswered questions related to it:<br>
<br>
* What types of things go in the metadata?<br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">* Is this an area for us to write data to, or for advanced users to<br>
write data to?<br>
* Is it entirely unstructured, or will we require a discussion for<br>
each new key/value entry into it. </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
It is not at all clean that the current metadata design will hold up<br>
to our answers of these questions. But in the end, I sort of wanted<br>
to add the metadata as it is now, so we could being to see how we and<br>
others start to use it. But just because we added the metadata to the<br>
notebook format definitely doesn't mean that future-proofs this part<br>
of the notebook format.<br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hope this clarifies things a bit.<br></blockquote><div><br></div><div>Sure, while it is extremely clear that we need cell metadata, we cannot be 100% certain that</div><div>a simple dict will solve 100% of the cases we encounter. But adding it now means that we have at least a *chance*</div>
<div>of making a release that is not backwards-incompatible.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Back to the question of output-level metadata. When a bit of code<br>
remains unused for almost a year, I start to question whether we<br>
really need it. I not convinced we don't need it, I am not sure. In<br>
light of this, I don't think that adding it to the notebook format<br>
makes sense. When one of us finds a good purpose for this metadata,<br>
let's add it to the nbformat them.<br></blockquote><div><br></div><div>I believe the only current use is in the parallel display republishing, where the engine ID is added to the display data</div><div>so that frontends could theoretically draw display data differently based on which engine it came from.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The other philosophical line of reasoning that I am being guided by<br>
here is simplicity. It would be very easy to over design the notebook<br>
format and add all sorts of feature that we might need. I think this<br>
is a wrong direction to go. We want a notebook format that is as<br>
compact and minimal as possible, where each and every bit of data is<br>
there for a well-defined and justified reason.<br></blockquote><div><br></div><div>I think it's simple: We have had ideas over and over and over again for features requiring metadata attached to cells (hashes, links, timestamps, etc.), so this is clearly a feature we have a need for right now. It would be totally silly for adding timestamps to require updating the nbformat in a backward-incompatible way. And the biggest advantage of using json is that adding keys has no effect on backwards *readability*. It's only adding values/types that can cause problems, and should force new versions (e.g. changing worsheet to worksheets, or adding new cell types).</div>
<div><br></div><div>-MinRK</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
<br>
Brian<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On Tue, Jun 19, 2012 at 3:25 PM, MinRK <<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>> wrote:<br>
><br>
><br>
> On Tue, Jun 19, 2012 at 3:23 PM, Brian Granger <<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a>> wrote:<br>
>><br>
>> On Tue, Jun 19, 2012 at 3:19 PM, MinRK <<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>> wrote:<br>
>> ><br>
>> ><br>
>> > On Tue, Jun 19, 2012 at 3:18 PM, Brian Granger <<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> On Tue, Jun 19, 2012 at 2:59 PM, Fernando Perez <<a href="mailto:fperez.net@gmail.com">fperez.net@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > On Tue, Jun 19, 2012 at 1:17 PM, MinRK <<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>> wrote:<br>
>> >> >> Yes - we put metadata on outputs for a reason, presumably. If this<br>
>> >> >> shouldn't be saved, it should probably be removed from the API.<br>
>> >> ><br>
>> >> > I can't recall precisely what we had in mind when we put it in, but<br>
>> >> > something that springs to mind as potentially useful, for example,<br>
>> >> > would be to specify a desired priority order for the various types of<br>
>> >> > outputs. Right now when a client can display several kinds of output<br>
>> >> > it just makes a choice, but we could let objects provide a hint of<br>
>> >> > the<br>
>> >> > preferred order, based on what they know about the relative quality<br>
>> >> > of<br>
>> >> > each.<br>
>> >><br>
>> >> I originally put it there to allow objects to provide hints to the<br>
>> >> frontend on how it should display a representation. This is similar<br>
>> >> to how the payloads can indicate where it came from.<br>
>> >><br>
>> >> > So I'd vote for not removing this, as it may prove useful...<br>
>> >><br>
>> >> I also think it could be useful, although it seems a bit excessive to<br>
>> >> store metadata for each output. Here is what I propose. We simply<br>
>> >> leave it alone until we have an actual use case that will help us<br>
>> >> figure out exactly what this should look like. Without a concrete<br>
>> >> usage case, it is difficult to know what is needed.<br>
>> ><br>
>> ><br>
>> > But this doesn't answer the immediate question: Should this metadata<br>
>> > dict be<br>
>> > included in the nbformat<br>
>><br>
>> I would vote no - not until we have a real usage case. I don't like<br>
>> to add things to the notebook format until we are actually using them.<br>
><br>
><br>
> Then should we remove all of the metadata stuff we just added? The whole<br>
> point was to prepare the nbformat for future changes to we don't have to<br>
> update the nbformat, which is incredibly painful and should be done as<br>
> rarely as possible.<br>
><br>
> -MinRK<br>
><br>
>><br>
>><br>
>> >><br>
>> >><br>
>> >> > f<br>
>> >> > _______________________________________________<br>
>> >> > IPython-dev mailing list<br>
>> >> > <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> >> > <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> Brian E. Granger<br>
>> >> Cal Poly State University, San Luis Obispo<br>
>> >> <a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a> and <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
>> >> _______________________________________________<br>
>> >> IPython-dev mailing list<br>
>> >> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> >> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > IPython-dev mailing list<br>
>> > <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> > <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Brian E. Granger<br>
>> Cal Poly State University, San Luis Obispo<br>
>> <a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a> and <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
>> _______________________________________________<br>
>> IPython-dev mailing list<br>
>> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> IPython-dev mailing list<br>
> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
<br>
<br>
<br>
--<br>
Brian E. Granger<br>
Cal Poly State University, San Luis Obispo<br>
<a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a> and <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
</div></div></blockquote></div><br>