Re: [yt-dev] 2012 Community Survey Summary

Hi all. Just a few quick comments.

yt-isms, which I'd like to see disappear -- making a "collection" of a single image, "c" implicitly indicating domain center, how do you know which fields are available, blah blah.

This is the most important thing to me. Some broad refactor for this would really help the code quality and maintainability. I think it would really clean up my yt scripts as well. I'd like it if I could make a plot with nice-looking tick marks, labels,

and colorbar by manipulating a plotting window and without writing a line of matplotlib code or python code.

While I don't exactly like the matplotlib API, I think it could get worse if we try to wrap it all up. I have found that I would rather have full control, even if it gets messy. If I really want to tinker with a plot, I would rather use matplotlib directly. Right now, random API references tend to appear at the top of the search

results while the narrative docs appear at the bottom which is very confusing.

Completely agree. This is one of the main reasons I stopped using sphinx. Even for the python docs, I have found it easier to google the site than to use the sphinx search. Best, Casey On Tue, May 29, 2012 at 7:59 AM, Matthew Turk wrote:

Hi Nathan and Elizabeth (and everyone else),

Thanks very much for your comments. My delay in replying was to allow ample time for anyone else interested to chime in. (And anyone still thinking of doing so, please do!)

...

Hi all, just a few comments:

I think the main issue is that yt is at its heart an extension of the

On Wed, May 23, 2012 at 3:00 AM, Nathan Goldbaum wrote: python language. If a new user is not familiar with python initially then they will have a much more difficult time following the docs and learning how to control yt. At the same time, the fact that yt is embedded in a high level programming language means that it is very powerful and easily extensible to someone familiar with python and willing to wade through the decently-sized codebase.

I think that is definitely true. And in the past, we've tried to ease the process of getting people on board with Python -- through the short orientation document, links to other places, and so on, but I think trying to address this on our own may be a losing battle. Python is rapidly gaining mindshare in astronomy and so perhaps this will be less and less of a problem as time goes on.

This brings up Elizabeth's point, though, that there *are* easier introductions to yt than strictly through the Python interface. A week and a half ago I gave a morning lecture in Sapporo, with Britton, John and Elizabeth, on yt to a room full of people who hadn't done much or any Python programming. (There were a handful of Ruby programmers, though, which was interesting.) During that lecture, I remembered just how weird some things are. If you're not used to Python, or even if you are but aren't used to yt, the following couple of lines can be really daunting:

from yt.mods import * pf = load("DD0076/DD0076") pc = PlotCollection(pf, "c") pc.add_slice("Density", "x") pc.save()

Yes, each line makes sense by itself, but taken as a whole to swallow with no exposure to the idioms or syntax of Python this can be frustrating. (And, let's not forget that this includes a lot of organically developed yt-isms, which I'd like to see disappear -- making a "collection" of a single image, "c" implicitly indicating domain center, how do you know which fields are available, blah blah.)

...

Elizabeth's point about Reason in the docs is worth thinking about. I

don't think Reason is quite there yet, but if we can point to the gui in the docs before we even mention a python tutorial, I think far fewer people will be scared off in the same way as the negative commenter.

I agree. I spent some time last week in a jetlagged stupor evaluating where I thought Reason was weak, inspecting the code, and playing with it a little bit. I've come up with a plan, but I don't see it being implementable in the next couple weeks, but rather sometime over the next couple months.

* Add a file open dialog. I did this last week, and I'm actually quite pleased with how it turned out. Now you can specify files on the command line, open from a window, or use -f to load parameter files in Reason. * Refactor the javascript code. This has no direct implications for the user, but it will provide much more maintainable software. This includes: * Migrate to ExtJS4 * Move to using classes provided by ExtJS for widgets, etc etc (i.e., make creating a new widget *way* easier) * Reorganize .js files * Make ExtJS get loaded from a single .zip file rather than an unzipped directory in $YT_DEST/src/ * Add a few more widgets for 3D viz and improve the existing ones * Change the view to be parameter-file centric, rather than notebook centric. This should make choosing fields, making new widgets, etc etc much easier. The notebook will still be available, but rather than having the user right-click on the pf, present a pf widget that has all the available widgets as well as stats and whatnot about the parameter file. * Test out linked view widgets (we used to have this in Reason v1). See http://arxiv.org/abs/1205.4747 for more details.

Right now it's all a bit widget-centric, but I think moving to more parameter file -centric would improve things. This would then nicely migrate into time series-centric analysis... Way down the line we can consider using the IPython infrastructure. I thought we could try to do that on a much shorter term, but I was wrong.

Jeff has a set of things he'd like to polish about reason, but I'm not going to try to summarize or list them here. I think it just needs lots of polish.

...

For that to happen It should be much easier to write new analysis

modules for reason. If there were an API to add new functionality in the same way as we can add new derived quantities or time series analysis tasks in pure python, I think reason would get much more attention.

I like this idea. It would be great if we could have the main widgets and then a set of contributed analysis modules. Like, imagine we had the usual slices and plots and stuff available, and then made it easy to add on a new "canned analysis" that prompted the user for some inputs, applied itself, and then returned either a display or a data object back. This could probably be done with introspection and annotations. I took a stab at something like this a while back:

http://bitbucket.org/MatthewTurk/yt.web_interactors

but that was in a standanlone environment. You decorated your function, and then yt figured out how to create widgets it showed to the user in a web browser. You set values, clicked "Go" and it returned a new value back to you.

...

One option that may be worth thinking about is a gui plotting interface

separate from reason. I know we want the new plotting interface to expose the matplotlib backend more readily - something I am completely for - but I think matplotlib can be quite confusing for newcomers.

Sigh ... yeah, I see where you're coming from with wanting a native interface. We have a simple interface to matplotlib that can do the zooms and whatnot. But I'm not sure what to do here, as even thinking about dealing with GUI toolkits gives me headaches. Can you elaborate a bit on what you'd want to see?

...

I'd like it if I could make a plot with nice-looking tick marks, labels,

and colorbar by manipulating a plotting window and without writing a line of matplotlib code or python code. It would also be cool to set up a plot interactively and save a script that could reproduce that plot or press a button to interactively make the exact same plot with a different dataset. This is all stuff that visit does and I think there are some lessons to be learned with how visit handles data, even if the plotting and scripting is sometimes pretty awful (sorry visit developers…).

There's definitely a long list of things we could learn from visit. I see what you mean about making simple plots.

Perhaps we should envision an interface where we expose the matplotlib interface readily, but we also provide a number of commonly done idioms, named with plaintext names. For instance, it's pretty common to want to stagger the ticks every other. So, make that a recipe that can be applied to the matplotlib interface.

...

One other thing that I've mentioned before that would make the docs much

easier to use for a newcomer would be an improved search. Right now, random API references tend to appear at the top of the search results while the narrative docs appear at the bottom which is very confusing.

I'm going to try to improve this. Britton and I went over some of the docs, and I think we're going to try to improve the examples/cookbook, and also remove the API docs from the search function. This is a common complaint.

...

Thanks for posting the results of the survey Matt. I'm looking forward

to working on this stuff this summer :)

...

Cheers,

Nathan

On May 22, 2012, at 9:20 PM, Elizabeth Tasker wrote:

...

Hi everyone,

I have a couple of comments related to the negative comment on the yt

...

...

The comment about yt being heavily cosmology based I do not agree with.

As a non-cosmo simulator --one who still intends to torch a fellow developer's house for declaring all my research 'test problems' at a user meeting-- I use yt for all my simulation analysis and use a wide range of its capabilities. In fact, the only part I can think that I do not use is

survey. In particular, there is one part I agree with and one I do not: the halo finder, but I do use connected sets which is a similar idea for gas.

...

...

I therefore feel this comment is not part of the general consensus and

can be ignored.

:-)

...

...

However, the comment about the documentation being from the user's

rather than the programmer's, point of view is interesting and think worth some thought. For a while, I skirted around yt, aware that it was powerful but baulking at the effort needed to use it comfortably. Last summer, I had to write an entirely new analysis program for my simulations and I bit the bullet and starting working extensively with yt and python. I've never picked up an IDL script since, but there was definitely an energy bump. I also think that I do find the documentation **significantly** clearer now I approach it from a programmer's perspective.

...

I have a couple of suggestions of what I think might help with this:

(1) The quickest option is to keep everything the same but add more

actual examples. For instance, the routine call, e.g.

...

contour(self, field, ncont=5, factor=4, take_log=False, clim=None,

plot_args=None):

...

is confusion for a non-programmer. When I first came to yt, I spent a

while with thoughts like:

...

What is 'self'? Do I need to pass it like in C++ with Grid-> ?

Although it's slightly painful, I think it would be much improved if

there was an actual call example for each one of the callbacks and other tools. So we'd have:

...

contour(self, field, ncont=5, factor=4, take_log=False, clim=None,

...

(This is a proxy for ContourCallback.) pc.modify["contours"]("Density", ncont=1, take_log=False) Add contours in field to the plot. ncont governs the number of contours generated, factor governs the number of points used in the interpolation, take_log governs how it is contoured and clim gives the (upper, lower)

plot_args=None): limits for contouring.

This is a great suggestion, and I will work to implement this. The organization also needs some work, but I'm the wrong person for that job.

I'll also see if we can have example images below each of these.

...

...

(2) My second suggestion involves the gui and concerns the way yt is

presented. Currently, we talk about scripts and then mention there is a gui. How about we do it the other way around? So a new user's first port of call should be the gui. Then, we get to the section "beyond the gui" and talk about scripting and cookbook-ing and then finally onto full simulation analysis, i.e. writing whatever analysis package you like in python, but making use of the routines yt has.

...

For instance, in the workshop we just had at Hokudai, we could have

stood up and brought up the yt gui before anything else. *Then* gone onto scripts.

...

The docs could be rearranged in a similar fashion.

This is an interesting idea that had not occurred to me. While Nathan brings up the notion that Reason is not ready, I think it is perhaps worthwhile to figure out in what way it isn't, address those, and then move forward with making this the standard method of interfacing with yt for newcomers. I'm going to be preparing a screencast of "yt for beginners", just a short little demo, and I'll do it using Reason first. I think it would ease the transition, and also help those people new to Python to avoid having the big daunting task of their first experience being to write a script. While of course that's where (in an ideal world) they'd end up, maybe it shouldn't be where they start.

...

...

I think this might provide a much faster way for new users to start

getting into yt and make the gui of much more interest. Currently, people come to it after they learn to script at which point, its value is much less because it's inevitably more limited.

Good point.

Again, thanks to everybody for their feedback, and any other comments would be greatly appreciated.

Best,

Matt

...

...

Elizabeth

PS today's google-doodle (or possible tomorrow's for the USA) is

awesome.

...

!DSPAM:10175,4fbc659f1748213191834! _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

!DSPAM:10175,4fbc659f1748213191834!

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org