[IPython-dev] Project idea: Automatic lab notebook for iPython

[Maximilian Albert]

Hi Peter,

I wasn't aware of the Core Provenance Library but it looks like a very
interesting project. I'll definitely check it out.

On a related note, in case people haven't seen it I just wanted to throw
out a link to the Sumatra  [1] project, which has a similar goal. However,
as far as I could gather from its website, the Core Provenance Library
seems to take a very different approach (by trying to be as unobtrusive for
the workflow as possible, IIUC?). So I guess the link is not that useful
for you, Peter, but I thought it might be of general interest. If anybody
knows any other provenance tracking tools, I'd be interested to hear about
them.

Best regards,
Max

[1] http://pythonhosted.org/Sumatra/


2013/4/17 Peter Macko <pmacko at eecs.harvard.edu>

> Hi iPython developers,
>
> Here is a new project idea: automatic lab notebook for iPython and
> iPython Notebook, which would keep track of how each of your output
> files was produced, linking this "history" (or a "lineage") of an object
> across different iPython sessions and different iPython notebooks, and
> storing it persistently. This is frequently referred to in the Computer
> Science literature as "provenance."
>
> It will enable you to ask questions like "what did I do to produce this
> plot?" - and for example, it will tell you that you downloaded the input
> data set on Monday from such and such website, you ran all these
> commands to process the data on Tuesday, and then produced this plot on
> Thursday from a different iPython session. Note that this goes beyond
> (and is complementary in purpose to) iPython Notebook, since the history
> of a file is tracked across different sessions and Notebooks, and when
> you ask a question, you will get only the relevant information,
> suppressing any additional things that you did that are unrelated to the
> file in which you are interested.
>
> We are in touch with computational scientists all the way from
> bioinformatics to physics that are very interested in this feature! We
> met their needs partially by developing a cross-platform, multi-lingual
> library (https://code.google.com/p/core-provenance-library/) that they
> can use to annotate their Python (and non-Python) scripts in order to
> track the lineage of their objects.
>
> Our vision is that this will be all done fully automatically, without
> requiring the users to manually annotate their scripts. But
> unfortunately neither of us who are involved in this project has the
> resources or the knowledge of the iPython code-base to tackle this
> challenge. We need your help to make this happen! We have some ideas
> about how we might go about this, but we need someone who knows more
> about iPython to talk them over and to spearhead the actual development.
> Please let us know if you can help!
>
> Thank you,
>
> Peter Macko
>
> Harvard School of Engineering and Applied Sciences
> 33 Oxford St.
> Cambridge, MA 02138
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130419/a334ad79/attachment.html>