[IPython-dev] Project idea: Automatic lab notebook for iPython

[Peter Macko]

Hi iPython developers,

Here is a new project idea: automatic lab notebook for iPython and 
iPython Notebook, which would keep track of how each of your output 
files was produced, linking this "history" (or a "lineage") of an object 
across different iPython sessions and different iPython notebooks, and 
storing it persistently. This is frequently referred to in the Computer 
Science literature as "provenance."

It will enable you to ask questions like "what did I do to produce this 
plot?" - and for example, it will tell you that you downloaded the input 
data set on Monday from such and such website, you ran all these 
commands to process the data on Tuesday, and then produced this plot on 
Thursday from a different iPython session. Note that this goes beyond 
(and is complementary in purpose to) iPython Notebook, since the history 
of a file is tracked across different sessions and Notebooks, and when 
you ask a question, you will get only the relevant information, 
suppressing any additional things that you did that are unrelated to the 
file in which you are interested.

We are in touch with computational scientists all the way from 
bioinformatics to physics that are very interested in this feature! We 
met their needs partially by developing a cross-platform, multi-lingual 
library (https://code.google.com/p/core-provenance-library/) that they 
can use to annotate their Python (and non-Python) scripts in order to 
track the lineage of their objects.

Our vision is that this will be all done fully automatically, without 
requiring the users to manually annotate their scripts. But 
unfortunately neither of us who are involved in this project has the 
resources or the knowledge of the iPython code-base to tackle this 
challenge. We need your help to make this happen! We have some ideas 
about how we might go about this, but we need someone who knows more 
about iPython to talk them over and to spearhead the actual development. 
Please let us know if you can help!

Thank you,

Peter Macko

Harvard School of Engineering and Applied Sciences
33 Oxford St.
Cambridge, MA 02138