[IPython-dev] Project idea: Automatic lab notebook for iPython

[Peter Macko]

Hi Max,

Thanks for the link.

You're correct - one of the main goals of the CPL is to be as unobtrusive as possible: It takes some effort for an application developer to put it into their project, but once it's done, it is completely unobtrusive. The second goal is to make it easier to integrate provenance across different programs, regardless of the language in which they are written.

Best,
-Peter


On Apr 19, 2013, at 5:28 AM, Maximilian Albert wrote:

> Hi Peter,
> 
> I wasn't aware of the Core Provenance Library but it looks like a very interesting project. I'll definitely check it out.
> 
> On a related note, in case people haven't seen it I just wanted to throw out a link to the Sumatra  [1] project, which has a similar goal. However, as far as I could gather from its website, the Core Provenance Library seems to take a very different approach (by trying to be as unobtrusive for the workflow as possible, IIUC?). So I guess the link is not that useful for you, Peter, but I thought it might be of general interest. If anybody knows any other provenance tracking tools, I'd be interested to hear about them.
> 
> Best regards,
> Max
> 
> [1] http://pythonhosted.org/Sumatra/
> 
> 
> 2013/4/17 Peter Macko <pmacko at eecs.harvard.edu>
> Hi iPython developers,
> 
> Here is a new project idea: automatic lab notebook for iPython and
> iPython Notebook, which would keep track of how each of your output
> files was produced, linking this "history" (or a "lineage") of an object
> across different iPython sessions and different iPython notebooks, and
> storing it persistently. This is frequently referred to in the Computer
> Science literature as "provenance."
> 
> It will enable you to ask questions like "what did I do to produce this
> plot?" - and for example, it will tell you that you downloaded the input
> data set on Monday from such and such website, you ran all these
> commands to process the data on Tuesday, and then produced this plot on
> Thursday from a different iPython session. Note that this goes beyond
> (and is complementary in purpose to) iPython Notebook, since the history
> of a file is tracked across different sessions and Notebooks, and when
> you ask a question, you will get only the relevant information,
> suppressing any additional things that you did that are unrelated to the
> file in which you are interested.
> 
> We are in touch with computational scientists all the way from
> bioinformatics to physics that are very interested in this feature! We
> met their needs partially by developing a cross-platform, multi-lingual
> library (https://code.google.com/p/core-provenance-library/) that they
> can use to annotate their Python (and non-Python) scripts in order to
> track the lineage of their objects.
> 
> Our vision is that this will be all done fully automatically, without
> requiring the users to manually annotate their scripts. But
> unfortunately neither of us who are involved in this project has the
> resources or the knowledge of the iPython code-base to tackle this
> challenge. We need your help to make this happen! We have some ideas
> about how we might go about this, but we need someone who knows more
> about iPython to talk them over and to spearhead the actual development.
> Please let us know if you can help!
> 
> Thank you,
> 
> Peter Macko
> 
> Harvard School of Engineering and Applied Sciences
> 33 Oxford St.
> Cambridge, MA 02138
> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130419/e533befa/attachment.html>