I've closed up the 2012 community survey. We had a fair number of responses (24), which I think is a good cross-section of the community. (There are 134 people on yt-users, so this is just under 20% of the list population, many of whom may not be actual or active users.)
For the most part, it was very, very positive. We're definitely doing some things very right. However, we had one particularly negative response (explored in detail below), and a good amount of constructive criticism scattered through the positive responses. I'll detail the particularly negative response, and then discuss the rest in bulk. At the end I've included the set of free-form comments left in the survey. I think the big takeaways there are pretty clear, and mostly are on the roadmap!
I'm happy to make available the original responses, but I've collated most of them here. I have no doubt there is a huge selection effect, and I'm really grateful that the strongly negative response was left at all, so that we could get an insight into what went wrong!
Finally, I'd encourage everyone reading this to reply if you have comments, suggestions, or any feedback that did not make it into the survey. I don't want this to be the end of the conversation, but rather the start of the feedback process.
= Negative =
The negative response was, indeed, quite negative. (And, it was difficult to read, to be honest.) Here are the comments, unedited.
"I want more narrative documentation, It should be written from the user's rather than the programmer's point of view. Start with simple tasks, and then build up to scripting and modification only slowly."
"Unreliability -- both times I've tried to use it, I had to invoke developer support to even get basic work done. Terribly written documentation -- virtually incomprehensible if you aren't both a cosmologist and a programmer."
"The concepts are powerful, but it should either be stated to be specifically for cosmological visualization only, or its general purpose use should be better supported."
"Helpful and friendly, but rather too convinced of yt's usefulness, so that people get sucked in for whom the tool is not at all ready yet."
"It is far too focused on the cosmology use case to be useful as a general purpose visualization tool. This impacts the documentation, which only makes sense if you are in the mindset of halo-finding and object property measurement, and the testing, which relies on users to find such basic things as support for 2D model visualizations (the most basic test problems) or non-square grids in nominally supported
Basically, my takeaways from this remarkably well-written, if blunt, takedown were:
I am a bit sorry that this submitter did not include contact information and instead chose to submit anonymously. A followup discussion would have been very productive, particularly because I think this response will spur a number of changes. I've shared this response with a few other people on this list, and all of them have been eager to make changes in response to the comments. It would have been nice to be able to discuss them with this person, and to start an ongoing discussion.
However, regardless, this should act as a wakeup call. I will endeavor to ensure that we support reduced dimensionality test problems, and during the next iteration of the documentation I hope we can iterate on the tutorial, the narrative docs, and so on. I suppose in the past we have neglected this population of computational astrophysicists that are not as at home with programming, so perhaps either adding better educational materials or identifying existing resources that exist (and linking to them) would be useful.
One plan I'd had had been to add screencasts. I think organizing these as how to install, then moving on to how to get going with each simulation code type (and providing explicit sample scripts and example data) independently would be a good strategy.
Does anyone have any additional thoughts on this? Furthermore, does anyone else want to pile on?
= All Results =
We had 24 replies. Of these people:
-- 20 (80%) said yt was easy to use 3 (12%) said yt was difficult to use
I think this is great. And, I think it's a huge compliment to the work we've all put in to make the code easier to use.
-- 16 (64%) said they use yt all the time (!) 7 (28%) use it once in a while
Again, wow. While it's difficult to read too much into this because of the selection bias, it's still great. And the final response is valuable to put into context the overall survey results.
-- The GUI Reason:
2 (8%) Use it often 10 (40%) Use it once in a while 11 (44%) Don't use it
This is interesting, but I'd put it out to the rest of you to interpret. Sam, Jeff, thoughts?
-- The thing yt needs more than anything else:
2 (8%) More supported codes 3 (12%) Speed or parallelism improvements 6 (24%) More canned tasks for astrophysical analysis 1 (4%) Better/easier visualization not destined for publication 5 (20%) Better/easier visualization for publication 0 (0%) Improvements to the GUI 3 (12%) API documentation 3 (12%) Narrative documentation and tutorials 0 (0%) A more friendly development process or community 0 (0%) More reliability or error handling 0 (0%) Ability to conduct time series analysis
It seems like the top two priorities are better/easier publication quality viz, and canned astrophysical tasks. This is excellent, as they are both well within our wheelhouse. One thing we should start pushing harder on is encouraging users to submit analysis modules upstream -- and to create uniform interfaces for analysis modules, particularly ones that work on time series. Britton and I have talked about smoothing up the various interfaces, making things more "Dataflow" or "Pipeline" oriented, and this is a good point of focus for that.
One sad bit is that I really want to improve the GUI (and I think John and Sam have expressed a similar desire, and I know Jeff has) but it looks like either due to indifference or its capabilities, this isn't a priority for others.
-- What should be changed about the documentation?
0 (0%) There's too much -- slim it down! 7 (32%) I want more narrative documentation 17 (77%) I want more example scripts 6 (27%) The organization should be improved 2 (9%) I'd like more screencasts
More examples! We have let the cookbook falter a bit. That should be reorganized and made more extensive.
Of the 24 responses, 14 had watched one or more videos from the workshop. This was the most surprising response.
-- Which of these pieces of functionality have you used:
10 (48%) yt pastebin (the yt pastebin) 6 (29%) Clump finder 11 (52%) Halo finder 17 (81%) Volume renderer 9 (43%) IRC for yt help
I found this the most interesting. More people have used the volume renderer than have used the pastebin! I guess it's a hit, and I would guess it's also a big draw.
-- Volume rendering in yt is ...
15 (75%) Attractive 13 (65%) Hard to get right 2 (10%) Slow
This completely jives with my experience. VR is super hard to make look nice, easy to make look terrible, and gives great results one you've set it up. I think coming up with new methods for rapid iteration, pre-visualization, and so on are all going to be important in the near future.
= Specific Comments =
Have you found anything in yt to be particularly annoying?
If there's one message about yt you want to send, what is it?
yt is elegant.
How do you find the yt community to be?
Very welcoming - no complaints!
Have you ever contributed code to yt? If so, what was good or bad about that experience?
Anything to add about deficiencies in yt?