<div dir="ltr"><div>Brian et al,<br></div><br>Brian I hope your move/travel/etc was as pleasant as such things can be.<br><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Jul 12, 2013 at 9:21 AM, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
Gabriel,<br>
</div><snip><br><div><div>
<br>
</div>Great, let's talk in Sept. to figure out a time that would work.<br></div></blockquote><br></div><div class="gmail_quote">I'm still quite interested in meeting with you guys. Somewhere near the end of the month would be best for me, but I'm pretty flexible.<br>
</div><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><snip><br>
<br>
</div><div>
> Branching/DAG notebooks allow a single document to encompass the research<br>
> you did, while providing easy access to various views corresponding to the<br>
> generation of intermediate, alternative, and final results.<br>
><br>
> These more complex notebooks allow the viewer to ask and answer important<br>
> questions such as "What else did (s)he try here?" and potentially even "Why<br>
> did (s)he choose this particular analysis strategy?". These questions can be<br>
> answered in the text or external supplementary materials in a linear<br>
> notebook, but this is a significant barrier to reproducibility of the<br>
> research process (as opposed to the analysis results).<br>
<br>
</div>I can see that, however, I think the pure alt cells lack a critical<br>
feature. They treat all branches as being equally important. In<br>
reality, the branch that is chosen as the "best" one will likely<br>
require further analysis and discussion that that other branches<br>
don't. Putting the different branches side by side makes it a little<br>
like "choose your own adventure" - when in reality, the author of the<br>
research want to steer the reader along a very particular path. The<br>
alternative paths maybe useful to have around, but they should be be<br>
given equal weight as the "best" one. But, maybe it is just<br>
presentation and can be accounted for in descriptive text.<br></blockquote><div><br></div><div>This is very true. My current thinking calls for both a "default" designation and a "most recently selected/run" designation, which I believe deals with the valid concern you raise above.<br>
<br></div><div>There are also other important designations for "branch types". The most notable/easily explained of these is the concept of a "terminal" branch, which is a branch that records important computations (and prose), and which a viewer of the notebook (be it the original author, a reviewer, a student, or someone looking to extend the work) may want to look at or run, but whose output is not compatible with the subsequent computations. This arises most commonly when one analysis strategy is implemented and pursued, but ultimately abandoned (hopefully for good reasons, and with this we can check!) in favor of a different final strategy which produces incompatible output. The subsequent code then makes assumptions about the output which are compatible with the final strategy computations, but not the original strategy ones. A way to gracefully deal with this case is important for any document/processing/rendering system attempting to pursue these concepts.<br>
<br></div><div>There are other cases that arise with these documents, but I will omit a detailed discussion of them and what I think should be done to support them here, as that would make this mail burdensomely long and it is not my primary message.<br>
</div><div> <br></div><div>I will note, though, that while I agree that the final/core/whathaveyou and secondary/informative/archival branches should not be indistinguishable, it is important for my usecase that they be easily accessible when the reader wants to in both interactive (notebook) and headless (nbconvert) modes.<br>
</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br></div>
<snip><br><div>
<div><br>
> As a practical/UI standpoint unselected branches can be hidden almost<br>
> entirely (in theory, not currently in my PoC :p), resulting in a view<br>
> equivalent to (any) the only view offered by a linear notebook. This means<br>
> that from a viewer (and author since a straight line IS a DAG and nesting<br>
> isn't forced) standpoint, what I'm describing is in essense a strict<br>
> extension of what the notebook does now, rather than a change.<br>
<br>
</div>I would be *more* interested in alt-cell approaches that present the<br>
notebook as a linear entity in all cases, but that has the alt-cell<br>
logic underneath. For example, what about the following:<br>
<br>
* A user writes the different N alt cells in linear sequence<br>
* The result is a purely linear notebook where one of the N cells should be run.<br>
* We write a JavaScript plugin for the notebook that does a couple of things:<br>
<br>
1. It provides a cell toolbar for marking those cells as members of an<br>
alt-set. This would simple modify the cell level metadata and allow<br>
the author to provide titles of each alt-member.<br></div></blockquote><div><br></div><div>What about branching that is 2 or more levels deep? That happens naturally with my approach but sounds difficult/annoying to keep track of in the one you are describing.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
2. It provides the logic for building a UI for viewing one of the<br>
alt-set members at a time. It could be as simple as injecting a drop<br>
down menu that shows one and hides the rest.<br></div></blockquote><div><br></div><div>I have an ugly but functional version of this now in my implementation.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<br>
* This plugin could simple walk the notebook cells and find all the<br>
alt-cell sets and build this supplementary UI.<br>
* This plugin could also have settings that allow the author to select<br>
the "best" member of the alt-set.<br>
* nbconvert Transformers could use the cell level metadata to export<br>
the notebook in different formats.<br>
<br>
As I write about this - I think this would be extremely nice, and it<br>
would not be difficult to write at all. Because of how our JavaScript<br>
plugins work, it could be developed outside IPython initially. The<br>
question of inclusion in the official code base could be handled<br>
later. Honestly, this approach should be much easier than the work<br>
you have already done.<br></div></blockquote><div><br></div><div>Well, editing the notebook once it exists in this form seems like it would be much less fun, in terms of adding new cells.<br><br></div><div>What you're describing is also much more onerous for the author. With what I have now, you declare a cell to be an altset or task and everything just sort of works. New cells are inserted in the right places, cells trivially know who their parents are, etc. <br>
<br>If I understand you correctly, the author would have to write all the alternatives in a big linear document (not fun or easy to test, see discussion below) and then click a bunch of buttons to manually select what cells go in which alternate. That is a much larger cognitive burden on the author (as well as probably being really annoying...).<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
Best of all the resulting notebooks would remain standard linear<br>
notebooks that could be shared today on nbviewer, etc. It would just<br>
work.<br></div></blockquote><div><br></div><div>Respectfully, this is actually the fatal flaw of this approach IMO, both in this case and in other cases where a JS plugin/extension uses the metadata approach to fundamentally modify <i>behavior</i> (as opposed to aestethics/UI) of the IPython Notebook.<br>
<div><br></div>The issue, stated in the context of the
nesting/alts/etc cells extension, is that a notebook that has
branching/alternates *requires* that they be understood as such, rather
than simply benefiting from it.<br><br></div><div>The ability to distribute notebooks I write and have them work properly is entirely core to my usecase for IPython. If I can't do so, what I personally can get IPython or IPython notebooks to do on my own machine is not something I have any real interest in. Now you may be thinking to yourself "But Gabe, no one is using your fork so you can't do that now with your implementation anyway". That is true, but if someone without my fork installed manages to get their hands on a notebook which uses the nesting features, it will break when they try to load it.<br>
</div><div><br></div><div>If I create an extension as you are describing, create a complex notebook using it, and someone without the plugin installed finds it, downloads it, and runs it, it will <i>run fine and happily give them incorrect results without even noticing the extra bits I stuck in the metadata</i>.<br>
<br></div><div>The core issue here is that running a notebook with branching as a linear notebook by executing each of the branches in sequence is actually erroneous and will produce undefined, untrustworthy, and likely incorrect, behavior and output. The reason for this is that branches/alternatives are assumed to be mutually exclusive by the computational model, and can alter objects in-place in manners that can have unintended cumulative effects.<br>
<br></div><div>As a very simple example consider branches which handle outliers in a certain variable by modifying the variable in-place and trimming its values by .1, 1, 5, and 10%, respectively, using quantiles and then consider what would happen if these branches were all run in an arbitrary order. <br>
<br>It is easy to see that the outcome from running all the branches (which is what will silently happen if the notebook is treated as a standard linear notebook because the plugin is not being used) does not reflect any of the choices intended by the author and more complex situations could be difficult to predict at all without sitting down and thinking about it.<br>
<br></div><div>As such, I would not be comfortable distributing branching notebooks using the extension mechanism as I understand it to exist now because a) I feel it indirectly damages the type of scientific reprodicibility and result trustworthiness I seek to advance, and b) I don't want to spend all my time fielding angry emails/bugreports from notebook authors who sent their notebooks to collaborators who didn't have the plugin installed.<br>
</div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
<div><div><snip><br></div>
</div><div><br>
> Consider the example of classifying new data based on a training set via<br>
> KNN, SVM, and GLM approaches. These approaches all need different sets of<br>
> parameters, return different types of objects as the output of the fitting<br>
> function, may have subtley different behaviour when being used for<br>
> prediction, etc.<br>
<br>
</div>Yep, that is the big challenge with the branching idea in general. It<br>
is not always true that the members of the alt sets can be swapped<br>
out.<br></div></blockquote><div><br></div><div>And under the model I am envisioning, that is actually an informative and queriable feature, rather than a drawback. See my discussion above regarding terminal branches.<br>
</div><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div><br></div><div>
<snip><br>
<br>
</div>I hope you can see that I really like the general idea and think the<br>
usage cases you are describing are really important. I think I can<br>
speak for the project in saying that we want the notebook to be useful<br>
for things like this. But I think our abstractions are important<br>
enough that we make every attempt to see how we can do these while<br>
leveraging our existing abstractions. This is partially a question<br>
about implementation, but also partly a question about how the new<br>
features are thought about. The reason we don't like to break<br>
abstractions for new features is that we have found an interesting<br>
relationship between abstraction breaking and new features. We have<br>
found that when a new feature/idea breaks a core abstraction that we<br>
have thought about very carefully, it is usually because the feature<br>
has not been fully understood. Time and time again, we have found<br>
that when we take the time to fully understand the feature, it usually<br>
fits within our abstractions beautifully and is even much better that<br>
we ever imagined it could be.<br>
<br>
The plugin idea above is a perfect example of this. By preserving the<br>
abstractions the new feature itself a multiplication of even new<br>
functionality:<br>
<br>
* The resulting notebooks can still be version controlled. This means<br>
that the different alt-cell can be thrown into git and when we develop<br>
a visual diff tool for notebooks, they will *just work*.<br></div></blockquote><div><br></div><div>I don't really understand this point. I have numerous fork-based non-linear notebooks under version control.<br><br>Also, when you have a visual diff tool, it will successfully do <i>something</i> when given a linear+metadata branching notebook, but whether that something would be to deliver the information required to understand changes to non-linear notebooks is less clear (and seems somewhat unlikely).<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
* The notebooks can immediately leverage the abstractions we have put<br>
into place for converting notebooks to different formats. You could<br>
write custom transformers to present the notebook in a reveal.js<br>
giving alt-cells special treatment. <br></div></blockquote><div><br><br></div><div>I could write custom transformers, this is true, but the default behavior would treat the notebook as if it actually were linear (instead of just being stored that way) which is problematic.<br>
</div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
* All of this can be done, and into the hands of user, without going<br>
through those overly conservative IPython developers ;-)<br>
* It will just work with nbviewer as well.<br></div></blockquote><div> </div><div>Again, I disagree. It would *display* in nbviewer, but not work, in that the display would be actively misleading regarding what the notebook would do when executed properly.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
* It provides a cleanly abstracted foundation for other people to build upon<br></div></blockquote><div><br></div><div>I agree that this is important, but it is not clear to me that it would be more true in the case that I created the extension via custom JS than it would if nesting were supported in the actual ipynb format and core notebook mechanisms.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
In summary, we are trying to build an architecture that allows a few<br>
simple abstractions (we actually don't have that many!) to combine in<br>
boundless ways to create features we never planned on, but that "just<br>
work".<br></div></blockquote><div><br></div><div>I agree that the customjs + metadata extensions approach is very powerful and almost infinitely versatile. I think it is great for extensions which change appearance/rendering/UI details of how the notebook behaves.<br>
<br></div><div>As far as I can see, however, it has some signficant problems with regard to extensions which fundamentally change non-rendering behavior of notebooks (please correct me if I'm wrong), namely:<br><ul><li>
There is no guarantee that notebooks authored using an extension which alters fundamental behaviors will work or visibly fail in the absence of that extension<br></li><li>There is no way for an individual notebook to require a particular extension</li>
<li>There is no way to ensure that two extensions are compatible with each-other</li><li>There is no standard/unified way for end-users to install extensions</li><li>There is no way for users to determine which extensions they have</li>
</ul><p>The first point is not true of extensions which exclusively affect rendering and UI, making the rest of the points minor nuisances rather than critical issues.<br></p></div>Looking forward to hearing your (further) thoughts about this stuff and hopefully meeting you in person soon.<br>
<br></div><div class="gmail_quote">~G<br clear="all"></div><br>-- <br>Gabriel Becker<br>Graduate Student<br>Statistics Department<br>University of California, Davis<br>
</div></div></div></div>