Request comments/vote on use of notebooks in yt docs
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.com wrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.com wrote: Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum <nathan12343@gmail.com<javascript:_e({}, 'cvml', 'nathan12343@gmail.com');>> wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels <chummels@gmail.com<javascript:_e({}, 'cvml', 'chummels@gmail.com');>
wrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org <javascript:_e({}, 'cvml', 'yt-dev@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org <javascript:_e({}, 'cvml', 'yt-dev@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
So I also thought there was a way to do this, but it looks like there isn't: https://github.com/ipython/nbconvert/issues/13
The button for it is to evaluate all the cells. Then you would need to save that ipynb, and run nbconvert on the completely evaluated notebook, as I understand.
Chris
On Tue, Oct 22, 2013 at 5:29 PM, Anthony Scopatz scopatz@gmail.com wrote:
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Just e-mailed ipython-dev about this: http://mail.scipy.org/pipermail/ipython-dev/2013-October/012501.html
On Tue, Oct 22, 2013 at 5:38 PM, Chris Malone chris.m.malone@gmail.comwrote:
So I also thought there was a way to do this, but it looks like there isn't: https://github.com/ipython/nbconvert/issues/13
The button for it is to evaluate all the cells. Then you would need to save that ipynb, and run nbconvert on the completely evaluated notebook, as I understand.
Chris
On Tue, Oct 22, 2013 at 5:29 PM, Anthony Scopatz scopatz@gmail.comwrote:
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
On Tue, Oct 22, 2013 at 7:38 PM, Chris Malone chris.m.malone@gmail.comwrote:
So I also thought there was a way to do this, but it looks like there isn't: https://github.com/ipython/nbconvert/issues/13
I agree with pi here. Executing all while saving or converting is a bad idea.
However, a command line interface or a way from python to execute all cells is really useful to have.
Be Well Anthony
The button for it is to evaluate all the cells. Then you would need to save that ipynb, and run nbconvert on the completely evaluated notebook, as I understand.
Chris
On Tue, Oct 22, 2013 at 5:29 PM, Anthony Scopatz scopatz@gmail.comwrote:
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
The issue you pasted led here: https://github.com/ipython/ipython/issues/1675
Which led to here in the end: https://github.com/paulgb/runipy
I'm not sure if there is an "official" way to do it, but runipy seems like it's worth looking into.
On Tue, Oct 22, 2013 at 5:46 PM, Anthony Scopatz scopatz@gmail.com wrote:
On Tue, Oct 22, 2013 at 7:38 PM, Chris Malone chris.m.malone@gmail.comwrote:
So I also thought there was a way to do this, but it looks like there isn't: https://github.com/ipython/nbconvert/issues/13
I agree with pi here. Executing all while saving or converting is a bad idea.
However, a command line interface or a way from python to execute all cells is really useful to have.
Be Well Anthony
The button for it is to evaluate all the cells. Then you would need to save that ipynb, and run nbconvert on the completely evaluated notebook, as I understand.
Chris
On Tue, Oct 22, 2013 at 5:29 PM, Anthony Scopatz scopatz@gmail.comwrote:
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Ah, nice catch - I missed that link somehow. runipy looks interesting...
Hopefully your message to IPython-dev will trigger some more "official" discussion.
On Tue, Oct 22, 2013 at 5:52 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
The issue you pasted led here: https://github.com/ipython/ipython/issues/1675
Which led to here in the end: https://github.com/paulgb/runipy
I'm not sure if there is an "official" way to do it, but runipy seems like it's worth looking into.
On Tue, Oct 22, 2013 at 5:46 PM, Anthony Scopatz scopatz@gmail.comwrote:
On Tue, Oct 22, 2013 at 7:38 PM, Chris Malone chris.m.malone@gmail.comwrote:
So I also thought there was a way to do this, but it looks like there isn't: https://github.com/ipython/nbconvert/issues/13
I agree with pi here. Executing all while saving or converting is a bad idea.
However, a command line interface or a way from python to execute all cells is really useful to have.
Be Well Anthony
The button for it is to evaluate all the cells. Then you would need to save that ipynb, and run nbconvert on the completely evaluated notebook, as I understand.
Chris
On Tue, Oct 22, 2013 at 5:29 PM, Anthony Scopatz scopatz@gmail.comwrote:
On Tue, Oct 22, 2013 at 7:24 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
If we keep the evaluated notebooks in the repo, we need to store the images since they're inlined as strings in the ipynb file.
I'm not sure if it's possible to programatically evaluate all the cells in a notebook. If we can do that, that's probably the way to go.
This has to be possible. There is a button for it =). if not we should complain / issue a PR until it is there.
Be Well Anthony
Nathan
On Tuesday, October 22, 2013, Chris Malone wrote:
I'm not by any means an expert on this, but can't we do something similar where each commit runs nbconvert on the ipynb to create static HTML and images that are then pushed to the web? I'm not sure I understand the need to actually keep the images within the repo. What am I missing?
On Oct 22, 2013, at 4:24 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I know I originally suggested moving to notebooks, but upon further reflection I think it might be too much of a pain for us to manage. Given that the notebooks would be pretty useless if they came down un-evaluated, we would need to store many images in the docs repo.
I think our current solution of static scripts along with a sphinx plugin that gets executed in a docs build for every commit to the docs repo is a good one.
On Tue, Oct 22, 2013 at 1:13 PM, Cameron Hummels chummels@gmail.comwrote:
> Hey everyone, > > The documentation sprint is next Monday and Tuesday for those of you > who want to participate. I'll send out another email regarding that in the > next day or so. > > In preparation for that, though, I wanted to request input from the > developer community on something related to the docs. > > Right now, the cookbook page contains a lot of recipes for doing > various things, and I think it is hugely beneficial to the community to > maintain this (I personally use this page a lot too!). However, with the > advent of ipython notebooks over the last year, we are faced with a > question: should we move toward incorporating more notebooks into our > documentation, and specifically, do you we want to transfer the existing > cookbook to a series of notebooks for each task? > > Benefits: > --Portability: users can download an entire notebook for both > viewing how it should work as well as being able to execute it locally on > their own datasets > --Illustrative: Interim steps in a cookbook can produce output that > can show up inside the notebook, instead of being a single script which > generates an image/output at the end (as is the case in the current > paradigm) > --Narrative: notebooks provide more space for narrating each step, > instead of confining any narrative to comments in the recipe itself > > Disadvantages: > --Work: it is going to take a decent amount of work to move all of > the recipes over from the existing cookbook to individual notebooks > --Bulking of repo: In the current paradigm, images associated with > each recipe are generated dynamically on the server by executing each > script, thereby minimizing the number of files that need to be tracked by > mercurial. By moving to a notebook with images that are embedded in each > notebook, we'd potentially increase the footprint of the repository > substantially, especially if there were frequent updates of individual > recipes. > > I also like the yt bootcamp notebooks that Matt put together a year > ago. I think they are great for getting new users up to speed on how to > use various aspects of the code. Perhaps this notebook could make its way > into the beginning of the cookbook for a more streamlined approach to the > documentation? > > So now is your chance to vote: > > Move cookbook to ipython notebooks? +/- 0-1? > > Move yt bootcamp to cookbook? +/- 0-1? > > Comments? Suggestions? > > Cameron > > -- > Cameron Hummels > Postdoctoral Researcher > Steward Observatory > University of Arizona > http://chummels.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.com wrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
After following this discussion, I am kind of in agreement with Brian. I think having notebooks would ease our job, but would also make it a bit harder for people reading the docs to get at little snippets, particularly if they are broken up with non-commented lines of text. Perhaps what would be better would be organizing the cookbooks somewhat nicer, and moving longer-form items to notebooks.
As long as we're on the topic, I have been working a little bit on docs for yt-3.0. I think we may want to consider the following things for the 3.0 docs:
* Removing the orientation and just providing the bootcamp (which will be extended), sticking the helpful links in other sections * Removing the workshop materials, as the are not 100% relevant anymore (and indicating what is and is not relevant is tricky) * Reworking the field list, although I don't yet know in what way. * Moving "loading data" to the top level. (I have done this already, as I think it needs to be a lot more prominent.)
-Matt
On Wed, Oct 23, 2013 at 6:23 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
After the discussion pointing to using the runipy utility, which solves all of the server-side problems, it seemed to me that this was a slam dunk on switching to notebooks. I still think it is easy with a cookbook made of notebooks to grab a snippet of code and use it on the client side, just using copy/paste (as most people do with the existing code snippets). To replicate the existing format of the code snippets would be trivial as well. When we're generating the notebooks, you have the option of saving as a notebook, or saving it as straightup python code. We could save both versions of the code. Then on the cookbook, we display our series of recipes showing the notebook inline, but then presenting a link to each as a notebook as well as the raw python code. It seems like that covers all of our bases, as long as we can display the notebooks inline in sphinx. It was my understanding that this was possible with runipy and a sphinx plugin. That said, i'm not tied to this idea, it just seemed to have so many advantages to the user base and with the discovery of runipy, almost no disadvantages.
I'm -1 on having some items in the cookbook be for notebooks and some be for code snippets. I think that kind of heterogeneity is confusing.
On Wed, Oct 23, 2013 at 6:17 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
After following this discussion, I am kind of in agreement with Brian. I think having notebooks would ease our job, but would also make it a bit harder for people reading the docs to get at little snippets, particularly if they are broken up with non-commented lines of text. Perhaps what would be better would be organizing the cookbooks somewhat nicer, and moving longer-form items to notebooks.
As long as we're on the topic, I have been working a little bit on docs for yt-3.0. I think we may want to consider the following things for the 3.0 docs:
- Removing the orientation and just providing the bootcamp (which will be
extended), sticking the helpful links in other sections
- Removing the workshop materials, as the are not 100% relevant anymore
(and indicating what is and is not relevant is tricky)
- Reworking the field list, although I don't yet know in what way.
- Moving "loading data" to the top level. (I have done this already, as
I think it needs to be a lot more prominent.)
-Matt
On Wed, Oct 23, 2013 at 6:23 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility
- it's easy enough for me to download the script and load it into a
notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
It should be straightforward to generate downloadable .py scripts using nbconvert on a recipe notebook. The way I see it, on each page we will present an evaluated notebook (sort of like the pyne documentation, e.g.: http://pynesim.org/gallery/data_sources.html), along with links to evaluated and unevaluated versions of the notebook and an nbconverted python script version of the notebook.
Brian and Matt, does that satisfy your concerns?
Another big win with this is that it becomes straightforward to version other notebooks we track. For example, instead of hosting the bootcamp notebooks on the yt hub, they could become part of the documentation repo and could be run with every documentation commit (possibly every commit to the yt repo?). This way we could ensure that the bootcamp notebooks are always up to date.
In the past there was also some discussion of hosting 3.0 example notebooks that update whenever we commit to the 3.0 repo as a lightweight form of answer testing. Again, it would be straightforward to automate this using these new tools.
I'm going to try to come up with a proof of concept for a sphinx plugin that does all of this sometime this weekend. Even if we decide not to use it for the cookbook, I think it will have utility elsewhere.
-Nathan
On Wed, Oct 23, 2013 at 7:52 AM, Cameron Hummels chummels@gmail.com wrote:
After the discussion pointing to using the runipy utility, which solves all of the server-side problems, it seemed to me that this was a slam dunk on switching to notebooks. I still think it is easy with a cookbook made of notebooks to grab a snippet of code and use it on the client side, just using copy/paste (as most people do with the existing code snippets). To replicate the existing format of the code snippets would be trivial as well. When we're generating the notebooks, you have the option of saving as a notebook, or saving it as straightup python code. We could save both versions of the code. Then on the cookbook, we display our series of recipes showing the notebook inline, but then presenting a link to each as a notebook as well as the raw python code. It seems like that covers all of our bases, as long as we can display the notebooks inline in sphinx. It was my understanding that this was possible with runipy and a sphinx plugin. That said, i'm not tied to this idea, it just seemed to have so many advantages to the user base and with the discovery of runipy, almost no disadvantages.
I'm -1 on having some items in the cookbook be for notebooks and some be for code snippets. I think that kind of heterogeneity is confusing.
On Wed, Oct 23, 2013 at 6:17 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
After following this discussion, I am kind of in agreement with Brian. I think having notebooks would ease our job, but would also make it a bit harder for people reading the docs to get at little snippets, particularly if they are broken up with non-commented lines of text. Perhaps what would be better would be organizing the cookbooks somewhat nicer, and moving longer-form items to notebooks.
As long as we're on the topic, I have been working a little bit on docs for yt-3.0. I think we may want to consider the following things for the 3.0 docs:
- Removing the orientation and just providing the bootcamp (which will
be extended), sticking the helpful links in other sections
- Removing the workshop materials, as the are not 100% relevant anymore
(and indicating what is and is not relevant is tricky)
- Reworking the field list, although I don't yet know in what way.
- Moving "loading data" to the top level. (I have done this already,
as I think it needs to be a lot more prominent.)
-Matt
On Wed, Oct 23, 2013 at 6:23 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility
- it's easy enough for me to download the script and load it into a
notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.com wrote: Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
1) Write the cookbook examples in a notebook, annotating it with comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with comments
and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility
- it's easy enough for me to download the script and load it into a
notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.comwrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with comments
and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility
- it's easy enough for me to download the script and load it into a
notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.comwrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.comwrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility
- it's easy enough for me to download the script and load it into a
notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I'm personally +1 on Brian's email on this subject
On Sat, 26 Oct 2013 16:08:45 -0400, Brian O'Shea wrote:
Hi folks,
I'm sorry
for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do
data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool,
and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian
(though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith
wrote:
There seems to be enough utility in having both the
notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM,
Chris Malone wrote:
I'm +1 on this, with the possibility of
providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at
12:28 PM, Sam Skillman wrote:
Hi all, I think if we
don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
1) Write the cookbook examples in a notebook, annotating it with comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves.
- Display the notebooks in the
docs
- Allow for download of both a stripped down (no images
included) .ipynb and nbconverted script.
The neat thing is that
now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers,
Sam
On Wed, Oct 23, 2013 at 10:02 PM,
Elizabeth Tasker wrote:
Hi everyone, I agree with
Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have
only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The
notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM,
Brian O'Shea wrote:
Hi Cameron,
For what
it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron
Hummels wrote:
> Hey everyone, > The documentation
sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
> In preparation for that, though, I wanted to request
input from the developer community on something related to the docs.
> Right now, the cookbook page contains a lot of recipes for
doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
> Benefits: > --Portability: users can download an entire
notebook for both viewing how it should work as well as being able to execute it locally on their own datasets
> --Illustrative:
Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm)
> --Narrative: notebooks provide more space for narrating each
step, instead of confining any narrative to comments in the recipe itself
> Disadvantages: > --Work: it is going to take a
decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks
> --Bulking of repo: In the
current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
> I also like the yt bootcamp notebooks that Matt put together a
year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
> So now is your chance to vote:
> Move cookbook to ipython notebooks? +/- 0-1? > Move yt
bootcamp to cookbook? +/- 0-1?
> Comments? Suggestions? >
Cameron
> -- > Cameron Hummels > Postdoctoral
Researcher
> Steward Observatory > University of Arizona
> http://chummels.org [1] >
_______________________________________________
> yt-dev mailing
list
> yt-dev@lists.spacepope.org [2] >
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [3]
_______________________________________________
yt-dev mailing
list
yt-dev@lists.spacepope.org [5]
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [6]
yt-dev
mailing list
yt-dev@lists.spacepope.org [8]
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [9]
yt-dev
mailing list
yt-dev@lists.spacepope.org [11]
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [12]
yt-dev mailing
list
yt-dev@lists.spacepope.org [14]
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [15]
_______________________________________________
yt-dev mailing
list
yt-dev@lists.spacepope.org [17]
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [18]
Links: ------ [1] http://chummels.org/ [2] mailto:yt-dev@lists.spacepope.org [3] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [4] mailto:chummels@gmail.com [5] mailto:yt-dev@lists.spacepope.org [6] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [7] mailto:bwoshea@gmail.com [8] mailto:yt-dev@lists.spacepope.org [9] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [10] mailto:tasker@astro1.sci.hokudai.ac.jp [11] mailto:yt-dev@lists.spacepope.org [12] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [13] mailto:samskillman@gmail.com [14] mailto:yt-dev@lists.spacepope.org [15] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [16] mailto:chris.m.malone@gmail.com [17] mailto:yt-dev@lists.spacepope.org [18] http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org [19] mailto:brittonsmith@gmail.com
I'll see if I can put together a template of what I originally envisioned for this, so that people don't think I'm doing away with the current functionality of the cookbook. But for now here is a description.
What I sought and seemingly failed to convey was for the cookbook to still have the "scripts" displayed inline as they currently are. But when I say scripts, I mean sphinx'ified ipython notebooks, where instead of showing the images at the end of the script as is currently done, it would show the images that were generated in the script after the line in the text that generated them along with comments included inline in the notebook. So one could still scroll down the webpage looking at demonstrations of a variety of functions in yt the way they currently can. The difference, however, would be that one could download those notebooks (as well as just the python scripts themselves) so that a user could execute these notebooks (or python scripts) locally. In my mind, we are not losing any functionality from the old page at all, and we're merely moving it to a slightly different format that provides additional benefits, while still retaining the ability to display and download python scripts as one can currently do. In essence, this is what Brian is recommending at the end of his email.
Brian, I don't think your statement that "using notebooks would make the cookbook much longer" is true. You mean from including comments in the scripts? Maybe it will increase its length by 10% but not much, and I think comments are ideal for helping out both new and experienced users alike to interpret what these scripts do. However, the cookbook *will* get longer because we're going to add a bunch of additional scripts in the coming few days, but that's good, and that's going to happen regardless of the format switch.
Cameron
On Sat, Oct 26, 2013 at 1:16 PM, Butler,Michael J butler85@astro.ufl.eduwrote:
**
I'm personally +1 on Brian's email on this subject
On Sat, 26 Oct 2013 16:08:45 -0400, Brian O'Shea wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.comwrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone. Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.comwrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks. Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all, I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script. The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion. Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone, I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples. Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook. Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs. The notebooks are also not quite as easy to use as a downloaded code snippet. Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone, The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so. In preparation for that, though, I wanted to request input from the developer community on something related to the docs. Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task? Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes. I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation? So now is your chance to vote: Move cookbook to ipython notebooks? +/- 0-1? Move yt bootcamp to cookbook? +/- 0-1? Comments? Suggestions? Cameron -- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded?
I'm +1 on this idea. When I work with my students during our meeting, we use python notebooks to explore the data, but when they or I do serious analysis, we use scripts.
Given this is the main way of using yt, I think there's a lot of benefit in demonstrating it as the standard and keeping the python notebooks as a teaching tool that one could click on to view. (Rather like the 'hint' system in online homework systems!).
Elizabeth
On Oct 27, 2013, at 5:08 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.com wrote: There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.com wrote: I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.com wrote: Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves.
- Display the notebooks in the docs
- Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker tasker@astro1.sci.hokudai.ac.jp wrote: Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.com wrote: Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
I've finished up a prototpye for the sphinx directive and issued a PR about this: https://bitbucket.org/yt_analysis/yt-doc/pull-request/109/new-notebook-sphin...
To get a taste for what the new embedded notebooks will look like, take a look at the docs build I've put on my webserver:
http://ngoldbaum.net/docs_build/bootcamp/simple_visualization.html
After creating the extension and playing around a bit with the existing cookbook recipes, I think I've come around more to Elizabeth, Brian, and Matt's point of view. Many of the example scripts show off functionality that is most useful in the context of a script. If one were trying to do the same thing in a notebook, they would most likely use similar, but somewhat different code.
I think instead of wholesale converting all the example scripts, we should include example notebooks in addition to the scripts. With the new sphinx plugin I've written, these example notebooks will be directly embedded in the docs and should flow nicely with the example scripts. Some of the scripts probably work better as notebooks and we probably should think about converting them as they come up.
One place where I could see the notebooks working very nicely is in the narrative docs. Since the notebook directive produces output that's indistinguishable from the output generated from normal sphinx .rst files, we could adapt the content of the existing narrative docs to be notebooks. The major benefit here is that output will be embedded in the docs and that docs will be tested every time they are built.
I've got a couple of example notebooks to show off movie-making and parallelism under IPython's parallelism that I'd like to add in the next couple of days. If any of you have some notebooks that you think would work well in the documentation, please consider adding them!
-Nathan
On Sat, Oct 26, 2013 at 5:58 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded?
I'm +1 on this idea. When I work with my students during our meeting, we use python notebooks to explore the data, but when they or I do serious analysis, we use scripts.
Given this is the main way of using yt, I think there's a lot of benefit in demonstrating it as the standard and keeping the python notebooks as a teaching tool that one could click on to view. (Rather like the 'hint' system in online homework systems!).
Elizabeth
On Oct 27, 2013, at 5:08 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.comwrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.comwrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
Hey everyone,
The documentation sprint is next Monday and Tuesday for those of you who want to participate. I'll send out another email regarding that in the next day or so.
In preparation for that, though, I wanted to request input from the developer community on something related to the docs.
Right now, the cookbook page contains a lot of recipes for doing various things, and I think it is hugely beneficial to the community to maintain this (I personally use this page a lot too!). However, with the advent of ipython notebooks over the last year, we are faced with a question: should we move toward incorporating more notebooks into our documentation, and specifically, do you we want to transfer the existing cookbook to a series of notebooks for each task?
Benefits: --Portability: users can download an entire notebook for both viewing how it should work as well as being able to execute it locally on their own datasets --Illustrative: Interim steps in a cookbook can produce output that can show up inside the notebook, instead of being a single script which generates an image/output at the end (as is the case in the current paradigm) --Narrative: notebooks provide more space for narrating each step, instead of confining any narrative to comments in the recipe itself
Disadvantages: --Work: it is going to take a decent amount of work to move all of the recipes over from the existing cookbook to individual notebooks --Bulking of repo: In the current paradigm, images associated with each recipe are generated dynamically on the server by executing each script, thereby minimizing the number of files that need to be tracked by mercurial. By moving to a notebook with images that are embedded in each notebook, we'd potentially increase the footprint of the repository substantially, especially if there were frequent updates of individual recipes.
I also like the yt bootcamp notebooks that Matt put together a year ago. I think they are great for getting new users up to speed on how to use various aspects of the code. Perhaps this notebook could make its way into the beginning of the cookbook for a more streamlined approach to the documentation?
So now is your chance to vote:
Move cookbook to ipython notebooks? +/- 0-1?
Move yt bootcamp to cookbook? +/- 0-1?
Comments? Suggestions?
Cameron
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
On Mon, Oct 28, 2013 at 12:20 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hi all,
I've finished up a prototpye for the sphinx directive and issued a PR about this: https://bitbucket.org/yt_analysis/yt-doc/pull-request/109/new-notebook-sphin...
To get a taste for what the new embedded notebooks will look like, take a look at the docs build I've put on my webserver:
http://ngoldbaum.net/docs_build/bootcamp/simple_visualization.html
I am very impressed. I like this a lot. In fact, I think at this point what we may want to do is begin to move the narrative documentation wholesale into this system.
After creating the extension and playing around a bit with the existing cookbook recipes, I think I've come around more to Elizabeth, Brian, and Matt's point of view. Many of the example scripts show off functionality that is most useful in the context of a script. If one were trying to do the same thing in a notebook, they would most likely use similar, but somewhat different code.
After a few days of listening to the various opinions on this, I think this is where I currently stand --
* Using the notebook has some subtleties that make it somewhat difficult to use the exact same code in scripting vs notebook. (.show(), imods, etc) * I'd prefer something simple like the notebook from the *developer* perspective as it's easier from CI and authorship perspectives * Narrative docs seem *extremely* well-suited to this
I guess what it comes down to is, do we want the cookbook to be *both* small, self-contained snippets designed to be dropped into other scripts, or do we want it to be long examples? If the former, scripts seem like the way to go. If the latter, notebooks.
And, actually, maybe there's a place for both? "Cookbook" => short little snippets. "Tutorials" or "Examples" => longer, narrative docs made from notebooks? Like the bootcamp?
I think instead of wholesale converting all the example scripts, we should include example notebooks in addition to the scripts. With the new sphinx plugin I've written, these example notebooks will be directly embedded in the docs and should flow nicely with the example scripts. Some of the scripts probably work better as notebooks and we probably should think about converting them as they come up.
+1!
One place where I could see the notebooks working very nicely is in the narrative docs. Since the notebook directive produces output that's indistinguishable from the output generated from normal sphinx .rst files, we could adapt the content of the existing narrative docs to be notebooks. The major benefit here is that output will be embedded in the docs and that docs will be tested every time they are built.
I'm 100% on board with this!
And as a sidenote, I wanted to bring this up here before the sprint today -- what about getting rid of the Orientation and the workshop materials in the 2.6 docs? I think they're both well-covered by other items, or out of date.
I've got a couple of example notebooks to show off movie-making and parallelism under IPython's parallelism that I'd like to add in the next couple of days. If any of you have some notebooks that you think would work well in the documentation, please consider adding them!
Awesome. Really great work, Nathan. Thank you.
-Nathan
On Sat, Oct 26, 2013 at 5:58 PM, Elizabeth Tasker tasker@astro1.sci.hokudai.ac.jp wrote:
why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded?
I'm +1 on this idea. When I work with my students during our meeting, we use python notebooks to explore the data, but when they or I do serious analysis, we use scripts.
Given this is the main way of using yt, I think there's a lot of benefit in demonstrating it as the standard and keeping the python notebooks as a teaching tool that one could click on to view. (Rather like the 'hint' system in online homework systems!).
Elizabeth
On Oct 27, 2013, at 5:08 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.com wrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.com wrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.com wrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker tasker@astro1.sci.hokudai.ac.jp wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.com wrote: > > Hey everyone, > > The documentation sprint is next Monday and Tuesday for those of you > who want to participate. I'll send out another email regarding that in the > next day or so. > > In preparation for that, though, I wanted to request input from the > developer community on something related to the docs. > > Right now, the cookbook page contains a lot of recipes for doing > various things, and I think it is hugely beneficial to the community to > maintain this (I personally use this page a lot too!). However, with the > advent of ipython notebooks over the last year, we are faced with a > question: should we move toward incorporating more notebooks into our > documentation, and specifically, do you we want to transfer the existing > cookbook to a series of notebooks for each task? > > Benefits: > --Portability: users can download an entire notebook for both viewing > how it should work as well as being able to execute it locally on their own > datasets > --Illustrative: Interim steps in a cookbook can produce output that > can show up inside the notebook, instead of being a single script which > generates an image/output at the end (as is the case in the current > paradigm) > --Narrative: notebooks provide more space for narrating each step, > instead of confining any narrative to comments in the recipe itself > > Disadvantages: > --Work: it is going to take a decent amount of work to move all of > the recipes over from the existing cookbook to individual notebooks > --Bulking of repo: In the current paradigm, images associated with > each recipe are generated dynamically on the server by executing each > script, thereby minimizing the number of files that need to be tracked by > mercurial. By moving to a notebook with images that are embedded in each > notebook, we'd potentially increase the footprint of the repository > substantially, especially if there were frequent updates of individual > recipes. > > I also like the yt bootcamp notebooks that Matt put together a year > ago. I think they are great for getting new users up to speed on how to use > various aspects of the code. Perhaps this notebook could make its way into > the beginning of the cookbook for a more streamlined approach to the > documentation? > > So now is your chance to vote: > > Move cookbook to ipython notebooks? +/- 0-1? > > Move yt bootcamp to cookbook? +/- 0-1? > > Comments? Suggestions? > > Cameron > > -- > Cameron Hummels > Postdoctoral Researcher > Steward Observatory > University of Arizona > http://chummels.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
And as a sidenote, I wanted to bring this up here before the sprint today -- what about getting rid of the Orientation and the workshop materials in the 2.6 docs? I think they're both well-covered by other items, or out of date.
I'm +1 on this. I think they're somewhat outdated at this point, and I don't imagine too many people continue (or should continue) to use them.
Cameron
-Nathan
On Sat, Oct 26, 2013 at 5:58 PM, Elizabeth Tasker tasker@astro1.sci.hokudai.ac.jp wrote:
why not flip it around and have the script be the standard, compact way
of
viewing the data, but one can click on a link to take the user to a
notebook
that they can look at, and which can be downloaded?
I'm +1 on this idea. When I work with my students during our meeting, we use python notebooks to explore the data, but when they or I do serious analysis, we use scripts.
Given this is the main way of using yt, I think there's a lot of benefit in demonstrating it as the standard and keeping the python notebooks as
a
teaching tool that one could click on to view. (Rather like the 'hint' system in online homework systems!).
Elizabeth
On Oct 27, 2013, at 5:08 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a
couple
of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people
are
doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are
new
to yt, it makes sense to present the cookbook code in the manner that
most
closely approximates how a new user will actually use it, so presenting
the
code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's
easy
to scroll through and extract what one needs, which is useful for the
OTHER
use of the cookbook, which is for more advanced users to quickly zip
through
and remind themselves of how the heck a piece of functionality works.
Using
iPython notebooks would make the cookbook much longer, or force us to
break
it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about
are
embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester.
It's
also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the
standard,
compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded?
That
allows the cookbook to be relatively easy to parse, and also lets
people who
want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith <brittonsmith@gmail.com
wrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand,
the
notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the
other, I
think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds
for new
users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script
nearby,
would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone <
chris.m.malone@gmail.com>
wrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman <samskillman@gmail.com
wrote:
Hi all,
I think if we don't take this opportunity to use notebooks during
this
major update to the documentation, we'll be kicking ourselves in
about a
year for missing it. I'd like to voice my support for the following
layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of
conventions for
data loading so that with minimal work users could change the path
to the
data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the
examples
after pointing to the data locations. This would be really really
nice in
my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker tasker@astro1.sci.hokudai.ac.jp wrote: > > Hi everyone, > > I agree with Brian. I've only recently discovered python notebooks
and
> I love them for my own analysis and when I'm looking at data
together with
> my students and we're exploring data sets, but I feel they are
unnecessary
> cumbersome for code examples. > > Most of the cookbook snippets have only one simple product (image, > plot etc), so you don't gain a great deal by showing the results
from each
> line in the notebook. > > Additionally, however great notebooks are, they're not as handy for > tasks that you need to perform repeatedly or for writing full length > analysis scripts. In my opinion, using yt as a front end to more
detailed
> analysis is one of its major strengths. If you take away the
cookbook
> scripts, we lack examples of yt in python programs. > > The notebooks are also not quite as easy to use as a downloaded code > snippet. > > Elizabeth > > > > On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com
wrote:
> > Hi Cameron, > > For what it's worth, as a user of yt I find the current cookbook > format to be incredibly useful. I don't think notebooks would add
to the
> utility - it's easy enough for me to download the script and load
it into a
> notebook on my own machine if that's what I want to do. It
definitely seems
> that the challenges (and possible downsides) substantially outweigh
the
> benefits, at least for me and my usage patterns. > > --Brian > > > > On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels <
chummels@gmail.com>
> wrote: >> >> Hey everyone, >> >> The documentation sprint is next Monday and Tuesday for those of
you
>> who want to participate. I'll send out another email regarding
that in the
>> next day or so. >> >> In preparation for that, though, I wanted to request input from the >> developer community on something related to the docs. >> >> Right now, the cookbook page contains a lot of recipes for doing >> various things, and I think it is hugely beneficial to the
community to
>> maintain this (I personally use this page a lot too!). However,
with the
>> advent of ipython notebooks over the last year, we are faced with a >> question: should we move toward incorporating more notebooks into
our
>> documentation, and specifically, do you we want to transfer the
existing
>> cookbook to a series of notebooks for each task? >> >> Benefits: >> --Portability: users can download an entire notebook for both
viewing
>> how it should work as well as being able to execute it locally on
their own
>> datasets >> --Illustrative: Interim steps in a cookbook can produce output that >> can show up inside the notebook, instead of being a single script
which
>> generates an image/output at the end (as is the case in the current >> paradigm) >> --Narrative: notebooks provide more space for narrating each step, >> instead of confining any narrative to comments in the recipe itself >> >> Disadvantages: >> --Work: it is going to take a decent amount of work to move all of >> the recipes over from the existing cookbook to individual notebooks >> --Bulking of repo: In the current paradigm, images associated with >> each recipe are generated dynamically on the server by executing
each
>> script, thereby minimizing the number of files that need to be
tracked by
>> mercurial. By moving to a notebook with images that are embedded
in each
>> notebook, we'd potentially increase the footprint of the repository >> substantially, especially if there were frequent updates of
individual
>> recipes. >> >> I also like the yt bootcamp notebooks that Matt put together a year >> ago. I think they are great for getting new users up to speed on
how to use
>> various aspects of the code. Perhaps this notebook could make its
way into
>> the beginning of the cookbook for a more streamlined approach to
the
>> documentation? >> >> So now is your chance to vote: >> >> Move cookbook to ipython notebooks? +/- 0-1? >> >> Move yt bootcamp to cookbook? +/- 0-1? >> >> Comments? Suggestions? >> >> Cameron >> >> -- >> Cameron Hummels >> Postdoctoral Researcher >> Steward Observatory >> University of Arizona >> http://chummels.org >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Nathan,
First of all, let me thank you for putting together this functionality. It looks great!
Secondly, I'm confused about something. Notebooks can be as simple as we want to make them. I don't see why we cannot just convert the existing scripts into notebooks without interspersing them with lots of comments and intermediate images. That way we retain the current short scripts that everyone (including me) is pleased with, while having the ability to make more advanced topics into longer or more detailed notebooks. Then it's all in your proposed new infrastructure as well. Am I missing something here?
Cameron
On Sun, Oct 27, 2013 at 9:20 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Hi all,
I've finished up a prototpye for the sphinx directive and issued a PR about this: https://bitbucket.org/yt_analysis/yt-doc/pull-request/109/new-notebook-sphin...
To get a taste for what the new embedded notebooks will look like, take a look at the docs build I've put on my webserver:
http://ngoldbaum.net/docs_build/bootcamp/simple_visualization.html
After creating the extension and playing around a bit with the existing cookbook recipes, I think I've come around more to Elizabeth, Brian, and Matt's point of view. Many of the example scripts show off functionality that is most useful in the context of a script. If one were trying to do the same thing in a notebook, they would most likely use similar, but somewhat different code.
I think instead of wholesale converting all the example scripts, we should include example notebooks in addition to the scripts. With the new sphinx plugin I've written, these example notebooks will be directly embedded in the docs and should flow nicely with the example scripts. Some of the scripts probably work better as notebooks and we probably should think about converting them as they come up.
One place where I could see the notebooks working very nicely is in the narrative docs. Since the notebook directive produces output that's indistinguishable from the output generated from normal sphinx .rst files, we could adapt the content of the existing narrative docs to be notebooks. The major benefit here is that output will be embedded in the docs and that docs will be tested every time they are built.
I've got a couple of example notebooks to show off movie-making and parallelism under IPython's parallelism that I'd like to add in the next couple of days. If any of you have some notebooks that you think would work well in the documentation, please consider adding them!
-Nathan
On Sat, Oct 26, 2013 at 5:58 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded?
I'm +1 on this idea. When I work with my students during our meeting, we use python notebooks to explore the data, but when they or I do serious analysis, we use scripts.
Given this is the main way of using yt, I think there's a lot of benefit in demonstrating it as the standard and keeping the python notebooks as a teaching tool that one could click on to view. (Rather like the 'hint' system in online homework systems!).
Elizabeth
On Oct 27, 2013, at 5:08 AM, Brian O'Shea bwoshea@gmail.com wrote:
Hi folks,
I'm sorry for not responding to Nathan and Cameron's emails from a couple of days ago in a timely fashion.
I believe that people who use yt to do data analysis in a production setting are primarily using scripts, as Britton said. This is certainly true in my own research group, and I think generally true when people are doing batch analysis of large datasets (and/or lots of datasets) on supercomputers. If the cookbook is truly meant to help users that are new to yt, it makes sense to present the cookbook code in the manner that most closely approximates how a new user will actually use it, so presenting the code as scripts in the cookbook makes the most sense to me. Also, the current implementation of the cookbook is relatively compact, so it's easy to scroll through and extract what one needs, which is useful for the OTHER use of the cookbook, which is for more advanced users to quickly zip through and remind themselves of how the heck a piece of functionality works. Using iPython notebooks would make the cookbook much longer, or force us to break it up into a bunch of segments. Either way, it would also be harder to quickly parse through because the actual commands the user cares about are embedded inside quite a bit of extraneous (for this purpose) text. I'm concerned that this mailing list's enthusiasm for the new, shiny thing, while commendable in many ways, is driving us away from the original purposes of the cookbook, at least as I understand them.
That said, I do think that iPython notebooks are very cool, and I intend to use them as a teaching tool in a class I'm teaching next semester. It's also pretty clear which way the wind is blowing on this mailing list, so putting iPython notebooks into the cookbook in some way appears to be inevitable. So, why not flip it around and have the script be the standard, compact way of viewing the data, but one can click on a link to take the user to a notebook that they can look at, and which can be downloaded? That allows the cookbook to be relatively easy to parse, and also lets people who want to see the notebooks do so.
--Brian (though possibly "Grampa Brian" is a more appropriate signoff, given the tone of my email)
On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith brittonsmith@gmail.comwrote:
There seems to be enough utility in having both the notebooks and at least some way of downloading pure python scripts. On the one hand, the notebooks are illustrative for both yt usage and for learning ipython notebooks themselves. As a teaching tool, they seem great. On the other, I think the primary usage of yt for analysis is through scripting and we should try to preserve that notion and make these available as seeds for new users looking to build their own scripts. I think that presenting the recipes with the notebook, but with a button to download as a script nearby, would be an option that satisfies everyone.
Britton
On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone chris.m.malone@gmail.comwrote:
I'm +1 on this, with the possibility of providing also just the .py files from nbconvert for those that won't/can't use notebooks.
Chris
On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman samskillman@gmail.comwrote:
Hi all,
I think if we don't take this opportunity to use notebooks during this major update to the documentation, we'll be kicking ourselves in about a year for missing it. I'd like to voice my support for the following layout:
- Write the cookbook examples in a notebook, annotating it with
comments and reasoning in markdown cells. Use some amount of conventions for data loading so that with minimal work users could change the path to the data and run themselves. 2) Display the notebooks in the docs 3) Allow for download of both a stripped down (no images included) .ipynb and nbconverted script.
The neat thing is that now you have all these .ipynb files in the doc repo. It would be stupid simple to then show people how to go to that folder, launch yt notebook, you can then interactively execute the examples after pointing to the data locations. This would be really really nice in my opinion.
Cheers, Sam
On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker < tasker@astro1.sci.hokudai.ac.jp> wrote:
Hi everyone,
I agree with Brian. I've only recently discovered python notebooks and I love them for my own analysis and when I'm looking at data together with my students and we're exploring data sets, but I feel they are unnecessary cumbersome for code examples.
Most of the cookbook snippets have only one simple product (image, plot etc), so you don't gain a great deal by showing the results from each line in the notebook.
Additionally, however great notebooks are, they're not as handy for tasks that you need to perform repeatedly or for writing full length analysis scripts. In my opinion, using yt as a front end to more detailed analysis is one of its major strengths. If you take away the cookbook scripts, we lack examples of yt in python programs.
The notebooks are also not quite as easy to use as a downloaded code snippet.
Elizabeth
On Oct 23, 2013, at 7:23 PM, Brian O'Shea bwoshea@gmail.com wrote:
Hi Cameron,
For what it's worth, as a user of yt I find the current cookbook format to be incredibly useful. I don't think notebooks would add to the utility - it's easy enough for me to download the script and load it into a notebook on my own machine if that's what I want to do. It definitely seems that the challenges (and possible downsides) substantially outweigh the benefits, at least for me and my usage patterns.
--Brian
On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels chummels@gmail.comwrote:
> Hey everyone, > > The documentation sprint is next Monday and Tuesday for those of you > who want to participate. I'll send out another email regarding that in the > next day or so. > > In preparation for that, though, I wanted to request input from the > developer community on something related to the docs. > > Right now, the cookbook page contains a lot of recipes for doing > various things, and I think it is hugely beneficial to the community to > maintain this (I personally use this page a lot too!). However, with the > advent of ipython notebooks over the last year, we are faced with a > question: should we move toward incorporating more notebooks into our > documentation, and specifically, do you we want to transfer the existing > cookbook to a series of notebooks for each task? > > Benefits: > --Portability: users can download an entire notebook for both > viewing how it should work as well as being able to execute it locally on > their own datasets > --Illustrative: Interim steps in a cookbook can produce output that > can show up inside the notebook, instead of being a single script which > generates an image/output at the end (as is the case in the current > paradigm) > --Narrative: notebooks provide more space for narrating each step, > instead of confining any narrative to comments in the recipe itself > > Disadvantages: > --Work: it is going to take a decent amount of work to move all of > the recipes over from the existing cookbook to individual notebooks > --Bulking of repo: In the current paradigm, images associated with > each recipe are generated dynamically on the server by executing each > script, thereby minimizing the number of files that need to be tracked by > mercurial. By moving to a notebook with images that are embedded in each > notebook, we'd potentially increase the footprint of the repository > substantially, especially if there were frequent updates of individual > recipes. > > I also like the yt bootcamp notebooks that Matt put together a year > ago. I think they are great for getting new users up to speed on how to > use various aspects of the code. Perhaps this notebook could make its way > into the beginning of the cookbook for a more streamlined approach to the > documentation? > > So now is your chance to vote: > > Move cookbook to ipython notebooks? +/- 0-1? > > Move yt bootcamp to cookbook? +/- 0-1? > > Comments? Suggestions? > > Cameron > > -- > Cameron Hummels > Postdoctoral Researcher > Steward Observatory > University of Arizona > http://chummels.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (10)
-
Anthony Scopatz -
Brian O'Shea -
Britton Smith -
Butler,Michael J -
Cameron Hummels -
Chris Malone -
Elizabeth Tasker -
Matthew Turk -
Nathan Goldbaum -
Sam Skillman