Mailman 3 November 2011 - yt-dev

Timeline and Issues for 2.3 Release
by Sam Skillman Nov. 16, 2011

Nov. 16, 2011

Hi all, After the totally awesome G+ hangout, we've made a few decisions on the upcoming 2.3 release. First of all, we have set a deadline of December 15th. Out of the "new stuff since 2.2", shown below, I have created documentation issues https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… for each feature (except the bug fixes). In order for the feature to be included in the 2.3 release, the corresponding documentation issue must be closed. Stay tuned for Jeff's email describing what is needed for a documentation issue to be closed. If on the 15th a given feature has not been passed through this process, it will be bumped to 2.4. I have assigned many of the tasks to individual people, but there are a few that have not yet been assigned. If you see your feature on that list, assign it to yourself and start documenting! New stuff since 2.2: * Multi-level parallelism * Real, extensive answer tests * Boolean data regions * Isocontours / flux calculations * "deliberate_field" branch merge (pending PR) * PHOP memory improvements * Bug fixes for tests * Parallel data loading for RAMSES, along with other speedups and improvements there * Performance improvements for volume rendering * (beta) Rockstar interoperability * Adaptive HEALpix support (with a few bugs left) * Column density calculations * Massive speedup for 1D profiles * Lots more, bug fixes etc. (Including a bug fix for the tests!) * Re-organization of Docs/Website Your Diamond Manifold Overlord, Sam

2 2

yt release discussion
by Sam Skillman Nov. 15, 2011

Nov. 15, 2011

Hi all, We're coming up on a lot of features without another release, so a few of us were thinking we'd have a release discussion for 2.3. There are a few things we should talk about. One is to cover the outstanding issues and assign/solve/push them off: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&sort=mil… Another big area of discussion will be on documentation. Some people have been quite good about keeping up to date (e.g. Stephen), but a few of us (including but not limited to me!) have fallen behind on some of the new features like the new parallelism and testing. Additionally, Cameron and Jeff have been working quite a lot on rewriting/reorganizing the documentation. If we go back and take a look at a discussion of the roadmap: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2011-May/001574… it looks like we have gone pretty far away from the original outline of a 2.3 release aka "Brick Trombone", so it is hereby reborn as "Diamond Manifold". "Lime Trumpet" (3.0) is slated to conquer non-cartesian datasets and possibly some new IO subsystems. Personally I think "Lime Tuba" is a more fitting name, but that can also be discussed. New stuff since 2.2: * Multi-level parallelism * Real, extensive answer tests * Boolean data regions * Isocontours / flux calculations * "deliberate_field" branch merge (pending PR) * PHOP memory improvements * Bug fixes for tests * Parallel data loading for RAMSES, along with other speedups and improvements there * Performance improvements for volume rendering * (beta) Rockstar interoperability * Adaptive HEALpix support (with a few bugs left) * Column density calculations * Massive speedup for 1D profiles * Lots more, bug fixes etc. (Including a bug fix for the tests!) In any case, if anyone wants to discuss what needs to be done and when we want to roll 2.3 out by, I suggest we have a G+ hangout at the yt page ( https://plus.google.com/107728486871834552760). I've set up a doodle -- http://www.doodle.com/95f9rccyreundr65 -- if you're interested fill it out and we'll figure out a time. Let me know if none of the times are possible. Sam

3 4

Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch
by Matthew Turk Nov. 11, 2011

Nov. 11, 2011

Hi all, Over the last couple months, Casey and I have been working -- on and off! -- on a new branch of the code called "deliberate_fields." This branch will change, in a substantial but easy-to-update way, how fields are handled in yt. I recognize this email is long. But if you use non-standard fields, a bunch of derived fields, unit modifications, any of that, it may affect you. So I *please* ask that you read it and, if you like, contribute back to the discussion. This is one of the items I really want to have done for a hypothetical 2.3 release. = Background = The way fields work currently was designed a bit haphazardly. They use FieldInfoContainers, objects which share state and which contain unions of the known derived fields and the known IO-based fields. One of the problems with this is that the only thing that separates a derived field from a known field is that function that generates the field: the IO-based fields all use a lambda which returns None, and the non-IO based fields return actual fields. This is pretty sub-optimal, and it actually lands us in trouble when (for instance) we have fields wandering around named things like "Thermal_Energy" and "ThermalEnergy"; the mechanism by which one is selected and the other not is problematic, and to get around infinite recursion, hacks have had to be applied. As it stands, to find a field, the shared-state "field info" on a parameter file is queried; this then will try to check universal fields. But because of how the fields are stored, the field info cascade can also operate in reverse. The big problem is that the field selection mechanism doesn't seem to have a bus factor >= 1.0. And, it has a number of hacks to make it work with conflicting field definitiosn and the like. Unfortunately, layering these hacks on top of each other makes it much harder for other codes to be supported; translations are not reliable, and sometimes cause too many levels of recursion to be added. Something simpler is necessary. = What this does = Essentially, this creates multi-level, explicit fallbacks. The field info container, which was a bloated, weird shared state object, is now simply a dictionary subclass with a "fallback" option. When you create them, you can either create it in isolation (with no fallback) or with a fallback. When you query it, if it does not have a field, it checks its fallback. There are, additionally, two new functions for IO: the translation function and the null function. The first is to translate, for instance, "density" to "Density" and the second is to indicate that a field is expected to be found in an output from the simulation code. There are now affiliated with each simulation code two field info objects: the "known" fields, which may appear in files, and the non-known (i.e., code-specific derived) fields. These live as the attributes _fieldinfo_fallback and _fieldinfo_known on the StaticOutput sublcass corresponding to a simulation code. When the Hierarchy (not static output) is instantiated, the first step is to create a new field_info object. This has, as a fallback, the _fieldinfo_fallback, which itself has as a fallback the universally-known derived fields. The hierarchy then queries the output file for which fields are available. This process then looks for a corresponding field in fieldinfo_known, and if it finds it, it adds it to the field_info object, *overriding* any possible derived fields. (In this manner, for instance, yt will not recalculate a "CoolingTime" field if one exists in the output.) = What it aims to do in the future = This will be utilized in three main ways: 1) Making it more clear which fields belong to which code, and which come from disk and which are derived 2) Help move IO into fields, to optimize for geometries and data containers 3) Make units more clear and specific 4) This is all designed around better supporting the GDF. = Where from here? = It would be hugely beneficial if you could test this and report back. I have created a pull request: https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-util… This is by no means a settled matter; I think we need to have testing on this, buy-in from developers and users, and to make sure that old code doesn't beak. The test cases all pass for me for Enzo. Before this can be merged, I would hope we can get some testing from: * Enzo * Nyx * FLASH * Orion and any other codes that can hear me. Thanks very much for your time; please let me know if you have any questions, concerns, jokes, comments, improvements, CDs of your band, suggestions, and so on. For this major of a change I'd like to keep discussion on list, so the record of this is a bit more prominent. Best, Matt

5 11

proposal: turning off parameter file storage by default
by Sam Skillman Nov. 10, 2011

Nov. 10, 2011

Hi all, There have been some recent issues that have popped up when parameter file storage is used when accessing different parameter files at the same time in a parallel job. While the long term solution is to move to something thread-safe like sqlite, I am of the opinion that for now we should turn off this capability by default. We would leave the capability in there, and it would be turned on with: from yt.config import ytcfg; ytcfg["yt", "storeparameterfiles"] = "True" at the very top of a yt script. Since this could disrupt some people's workflows, I think a +/-1 would be helpful, along with any strong opinions. Thanks, Sam

3 3

Written interview floss4science.com
by FLOSS FORSCIENCE Nov. 7, 2011

Nov. 7, 2011

Hi. My name is Edwood Ocasio and I am the founder of http://www.floss4science.com, a site about open source software in science and engineering. I will like to publish a written interview about the yt Project as a featured post in floss4science.com I worked for 20 years as a professor of computer science, mathematics and statistics in different universities in Puerto Rico. You can easily find traces of me on the web. I also have a group for an Spanish speaking audience interested in open source software for scientific computing ( http://groups.google.com/group/compcilibre). * The interview publishing process * 1. I will send you a private link to drafts of the interview before publishing it to allow you to make corrections. 2. Your photo and/or screenshots of your software can be included. The photo is not a requirement. 3. Interviews will be queued for weekly publishing. I will notify you the expected publishing date. * Interview license * This interview, as all content in floss4science.com, will be distributed with a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License (http://creativecommons.org/licenses/by-nc-nd/3.0/) Below are the questions. You can answer them right here and just reply to this email. You can leave questions unanswered if they do not seem to apply to you or the yt Project or cannot answer for whatever reason. Again, thanks. ====== INTERVIEW QUESTIONS ====== 1. Please, give us a brief introduction about yourself, i.e. profession, workplace, projects, etc. 2. What is the yt Project? 3. Why and when did the yt Project came to be? 4. In which [language(s)|platform(s)] is the project developed? 5. Does the yt Project have sponsors (private, academic, other)? 6. How are the sponsors supporting the project? 7. How many users you estimate the yt Project have? 8. Do you know where is the yt Project used (corporations, universities, other)? 9. How many team members does the project have? 10. In what areas of the yt Project development do you currently need help? 11. How can people get involved with the project? 12. What features are in the roadmap? 13. Which projects, blogs or sites related to open source software for science can you recommend? 14. Why do you consider free/libre open source software important for the advancement of your field? 15. Is there any other topic you would like our readers to know about? 16. Where people can contact you and learn more about the yt Project (website, blog, email, twitter, identi.ca, Facebook, etc.)? Thank you for sharing with us more about you and the yt Project. =========== -- Cordially, Edwood Ocasio Founder floss4science.com E: info(a)floss4science.com W: http://www.floss4science.com Identi.ca: http://identi.ca/floss4science Twitter: http://twitter.com/#!/floss4science

2 1

Releases, pull requests, "governance" ...
by Matthew Turk Nov. 2, 2011

Nov. 2, 2011

Hi all, There's been some discussion on the Enzo mailing list: https://groups.google.com/forum/#!topic/enzo-users/vtlCedbJILU about the stable/dev branch of yt. The takeaway is that bugs get fixed in the dev branch, but since our releases are much less frequent than that, they don't necessarily get propagated very rapidly to stable -- in large part because we also weave lots of new development in and out. Enzo deals with this by doing all development in forks, mandating pull requests, and being careful about which items to merge and when. It's not clear to me that we could implement that model, even if we wanted to. We've had recent success with developing in forks, issuing pull requests and the like, but there's no codified set of "when to develop in a fork" guidelines or anything like that. There've been some rumblings of "governance" discussions about yt, but nothing ever really formalized. So I guess what it comes down to -- do we want to start being aggressive about putting out patch (i.e., 2.2.X) releases? Do we need a set of release managers or curators, like there are in Enzo? When do we develop in a fork, when do we feel comfortable pushing to the main repo? The problem, as I see it, is that in some respects yt suffers from the same *problems* a project like Enzo does, in that some pieces of functionality are critical to a few applications, but it does not have the level of support or engagement as Enzo, so we don't have as much ability to spread workload around. Any thoughts on this? -Matt

3 2

undefined variables in amr_kdtree.py
by Britton Smith Nov. 1, 2011

Nov. 1, 2011

Hi everyone, I'm coming across instances of the variable 'nprocs' being undefined in save_kd_bricks and load_kd_bricks in amr_kdtree.py. I can't quite tell what they should be instead or where they should be defined. Does anyone have any ideas? Britton

2 1