[IPython-dev] Project idea: Automatic lab notebook for iPython
Peter Macko
pmacko at eecs.harvard.edu
Tue Apr 16 19:38:58 EDT 2013
Hi iPython developers,
Here is a new project idea: automatic lab notebook for iPython and
iPython Notebook, which would keep track of how each of your output
files was produced, linking this "history" (or a "lineage") of an object
across different iPython sessions and different iPython notebooks, and
storing it persistently. This is frequently referred to in the Computer
Science literature as "provenance."
It will enable you to ask questions like "what did I do to produce this
plot?" - and for example, it will tell you that you downloaded the input
data set on Monday from such and such website, you ran all these
commands to process the data on Tuesday, and then produced this plot on
Thursday from a different iPython session. Note that this goes beyond
(and is complementary in purpose to) iPython Notebook, since the history
of a file is tracked across different sessions and Notebooks, and when
you ask a question, you will get only the relevant information,
suppressing any additional things that you did that are unrelated to the
file in which you are interested.
We are in touch with computational scientists all the way from
bioinformatics to physics that are very interested in this feature! We
met their needs partially by developing a cross-platform, multi-lingual
library (https://code.google.com/p/core-provenance-library/) that they
can use to annotate their Python (and non-Python) scripts in order to
track the lineage of their objects.
Our vision is that this will be all done fully automatically, without
requiring the users to manually annotate their scripts. But
unfortunately neither of us who are involved in this project has the
resources or the knowledge of the iPython code-base to tackle this
challenge. We need your help to make this happen! We have some ideas
about how we might go about this, but we need someone who knows more
about iPython to talk them over and to spearhead the actual development.
Please let us know if you can help!
Thank you,
Peter Macko
Harvard School of Engineering and Applied Sciences
33 Oxford St.
Cambridge, MA 02138
More information about the IPython-dev
mailing list