Mailman 3 November 2011 - yt-dev

QC for 2.3 release
by j s oishi Dec. 1, 2011

Dec. 1, 2011

Hi All, Per today's developer meeting, we decided on a 15 Dec 2011 release date for 2.3. The only remaining outstanding tickets concern documentation, and we want to ensure that the documentation undergoes some kind of quality control before the release. Unlike the code itself, which can (and will) be submitted to automated answer testing, the documentation will need to be tested by actual humans. We decided the best way to go about this would be to have a potential pull request acceptor first test the documentation by reading it and attempting to use the new feature before accepting. This shouldn't take too long, and should ideally be done using the acceptor's own data (to ensure that at least some modification of the script is taking place). If there are any questions or comments about this, let me know. If people not in the meeting this morning are satisfied with this, we'll just adopt it as we go to close the tickets sam has added here: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… thanks, j

2 2

Contributing to the documentation
by Christopher Moody Nov. 24, 2011

Nov. 24, 2011

Hi folks, It's been a while now that I've been using the Sunrise exporter that Matt wrote a long time ago, and I've written up a bit of documentation (and troubleshooting) on how to actually go about using it. How do I post this to the yt documentation website? What section does it go in, analysis modules? Cookbook? Thanks! chris

3 2

hg clarification
by Geoffrey So Nov. 23, 2011

Nov. 23, 2011

Since I screwed up forking yt last time (wanted dev but got stable), I didn't want to repeat my mistake, so I need someone to check my findings. I went to http://yt-project.org/ and clicked on "Fork us on BitBucket!" and it brought me to BitBucket page with a name "yt" and a specific tip of 4ec73cabefb6 But it doesn't say anything about if it is the stable or dev branch. I then went and looked at the two install scripts available, and found that the variable $(BRANCH) is "stable" or "yt", so I am guessing after I forked on BitBucket, I need to do the following # update the local branch to dev branch named "yt" hg up -C yt # commit the change on local machine hg commit -m "" # push the change of branch up to the fork in BitBucket hg push to switch the yt I just forked from whatever branch it is in right now into the dev branch of yt named "yt" Is that correct? If that is the case, I think last time I did the fork but didn't switch branches, and it defaulted to the stable branch. From G.S.

2 1

Off-axis projections -- Discrepancies between homogenized volume vs KDtree methods
by Cameron Hummels Nov. 22, 2011

Nov. 22, 2011

Hello peeps (mostly Britton, Matt and Sam), I have recently been doing some off-axis projections in my cosmological runs (using the supercool new off_axis_projection helper function Matt wrote), and I've encountered some problems. I find different results when I do the off-axis projection using a homogenized volume versus when I do not use a homogenized volume (when it uses the default behavior for camera objects -- ie a KDtree). Of course, these two results should be identical, and they are when I use a normal field like "Density". However, I'm trying to use a derived field from some code Britton wrote, part of a package called ion_balance, which creates derived fields for different atomic ions. So when I compare the CIV Number Density from these two methods, I get very different results. Even when I do this on a normal vanilla yt field, like "Density", the KDtree method takes exceptionally longer than the homogenized volume method (I think this is because I'm only doing the HV for a small subsample of the overall volume). On the other hand, they both take about the same amount of time when my sample volume is the entire box volume. I've pastebinned a demonstration script which shows this discrepancy at: http://paste.yt-project.org/show/1953. If you don't have ion_balance, you can comment that import out, and comment the line for defining the field as "CIV_Cloudy_eq_NumberDensity", and run it to see the time discrepancy between the two methods. It should work on any sort of parameter file, not just the specific one I'm using. What I do is take an off-axis projection using each method, then divide the two images against each other to form a ratio image, and then output the average and stddev for this ratio. The average of the ratio is: 2e-8. I've changed the width of the off-axis projection and it has a minimal (but nonzero) change on the overall ratio between the two. So I'm not sure what to do. It appears that the CIV field is initiated in the same way that a normal field is, with the projection_conversion set to 'cm', just as it is for "Density". Any ideas on what could be making this difference? Any ideas on which is the value to trust? Cameron

4 4

Re: [yt-dev] Memory leak issue with ParallelHOP code
by Patel, Pragneshkumar B Nov. 21, 2011

Nov. 21, 2011

________________________________ From: Patel, Pragneshkumar B Sent: Thursday, November 17, 2011 12:18 PM To: Britton Smith Cc: Geoffrey So; s(a)skory.us Subject: RE: [yt-dev] Memory leak issue with ParallelHOP code Hi Britton, I have subscribed to both yt-users and yt-dev. It may possible my requests are pending. As far as dataset is concern, Geoffrey can give you better explanation about it. Dataset contains 0-64 regions. I tried to run ParallelHOP with all regions but because of memory issue, I could not able to complete all in once. Then I ran it with only 0-3 regions to debug code "parallel_hop_interface.py". Please also find attached script. Let me know, if you have more questions. Thanks Pragnesh ________________________________ From: Britton Smith [brittonsmith(a)gmail.com] Sent: Thursday, November 17, 2011 12:00 PM To: yt-dev(a)lists.spacepope.org Cc: Patel, Pragneshkumar B Subject: Re: [yt-dev] Memory leak issue with ParallelHOP code Hi again Pragnesh, I mistakenly asked you to post this to the enzo-users mailing when I meant to say that you should post this to the yt-users mailing list. Sorry about that, not enough coffee this morning. Britton On Thu, Nov 17, 2011 at 8:56 AM, Britton Smith <brittonsmith(a)gmail.com<mailto:brittonsmith@gmail.com>> wrote: Hi Pragnesh, You need to provide us with some more information. What do you mean when you say you ran for 0-3 regions? What kind of dataset are you trying to analyze? How big is it? Can you also please post the script you are using for this? Finally, could you please sign up for the enzo-users mailing list and post this there? You've posted this to the development list, which isn't the best forum for this question. Also, if you're not on the list you won't receive any of the responses to your post. Britton On Thu, Nov 17, 2011 at 7:29 AM, Patel, Pragneshkumar B <pragnesh(a)utk.edu<mailto:pragnesh@utk.edu>> wrote: Hi Stephen, I found memory leak issue, when I ran Parallel_HOP code. It may not be in python itself but may be in packages(e.g. kdtree etc..), which we call from"parallel_hop_interface.py". Please find attach file, in which you see memory keep increase in every region (I ran program with 32 processes for 0-3 regions.) I like to get your comments/suggestions to solve this issue. I am trying to find more about it using Valgrind. I will let you know. Thanks Pragnesh _______________________________________________ yt-dev mailing list yt-dev(a)lists.spacepope.org<mailto:yt-dev@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

4 9

PR 34
by Stephen Skory Nov. 18, 2011

Nov. 18, 2011

Hi all, can I get an up/down/suggestions vote on pull request 34? It's just one line, it's been there for three days, and I think it's useful. The only thing I might see as a better alternative is for Clump to inherit from AMR3DData, but that is a bigger change, and not necessarily better. Thanks! https://bitbucket.org/yt_analysis/yt/pull-request/34/quantities-for-clumps -- Stephen Skory s(a)skory.us http://stephenskory.com/ 510.621.3687 (google voice)

3 3

Adding tracer particles
by david collins Nov. 18, 2011

Nov. 18, 2011

Hi, all-- I'm interested in adding a flock of tracer particles to an existing simulation. Has anyone done such a thing? If not I can drop what I come up with onto yt-hub, if anyone else is interested. d. -- Sent from my computer.

3 3

Memory leak issue with ParallelHOP code
by Patel, Pragneshkumar B Nov. 17, 2011

Nov. 17, 2011

Hi Stephen, I found memory leak issue, when I ran Parallel_HOP code. It may not be in python itself but may be in packages(e.g. kdtree etc..), which we call from"parallel_hop_interface.py". Please find attach file, in which you see memory keep increase in every region (I ran program with 32 processes for 0-3 regions.) I like to get your comments/suggestions to solve this issue. I am trying to find more about it using Valgrind. I will let you know. Thanks Pragnesh

2 2

confusion about tickets
by j s oishi Nov. 16, 2011

Nov. 16, 2011

Hi all, I just accepted yt-doc PR #7, https://bitbucket.org/yt_analysis/yt-doc/pull-request/7/parallel-docs-impro… which solves some of the parallel documentation, but not all of it, as is my understanding. Sam, you were going to document multilevel parallelism before we close issue #330 (https://bitbucket.org/yt_analysis/yt/issue/330/multilevel-parallelism), right? However, does this completely close issue #247 (https://bitbucket.org/yt_analysis/yt/issue/247/estimates-of-resources-for-p… If so, I'll close that ticket forthwith. thanks, j

2 2

yt devel documentation
by Geoffrey So Nov. 16, 2011

Nov. 16, 2011

Hi all, Catching up on some missed emails, I realized that to be able to implement a feature I'm still missing the documentation (and fixing my fork from stable to devel). I was looking on the site under developer's guide, and I didn't see any references as to how to edit the documentation. I found a README asking us to go to the repo http://hg.yt-project.org/yt-doc/ I was wondering how one should go about editing, then submitting changes to the doc? Do we do the same with YT: -fork yt-doc -edit yt-doc -commit to local repo -submit pull request From G.S. --be learning how to use Sphinx in the mean time

2 2

Timeline and Issues for 2.3 Release
by Sam Skillman Nov. 16, 2011

Nov. 16, 2011

Hi all, After the totally awesome G+ hangout, we've made a few decisions on the upcoming 2.3 release. First of all, we have set a deadline of December 15th. Out of the "new stuff since 2.2", shown below, I have created documentation issues https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… for each feature (except the bug fixes). In order for the feature to be included in the 2.3 release, the corresponding documentation issue must be closed. Stay tuned for Jeff's email describing what is needed for a documentation issue to be closed. If on the 15th a given feature has not been passed through this process, it will be bumped to 2.4. I have assigned many of the tasks to individual people, but there are a few that have not yet been assigned. If you see your feature on that list, assign it to yourself and start documenting! New stuff since 2.2: * Multi-level parallelism * Real, extensive answer tests * Boolean data regions * Isocontours / flux calculations * "deliberate_field" branch merge (pending PR) * PHOP memory improvements * Bug fixes for tests * Parallel data loading for RAMSES, along with other speedups and improvements there * Performance improvements for volume rendering * (beta) Rockstar interoperability * Adaptive HEALpix support (with a few bugs left) * Column density calculations * Massive speedup for 1D profiles * Lots more, bug fixes etc. (Including a bug fix for the tests!) * Re-organization of Docs/Website Your Diamond Manifold Overlord, Sam

2 2

yt release discussion
by Sam Skillman Nov. 15, 2011

Nov. 15, 2011

Hi all, We're coming up on a lot of features without another release, so a few of us were thinking we'd have a release discussion for 2.3. There are a few things we should talk about. One is to cover the outstanding issues and assign/solve/push them off: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&sort=mil… Another big area of discussion will be on documentation. Some people have been quite good about keeping up to date (e.g. Stephen), but a few of us (including but not limited to me!) have fallen behind on some of the new features like the new parallelism and testing. Additionally, Cameron and Jeff have been working quite a lot on rewriting/reorganizing the documentation. If we go back and take a look at a discussion of the roadmap: http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2011-May/001574… it looks like we have gone pretty far away from the original outline of a 2.3 release aka "Brick Trombone", so it is hereby reborn as "Diamond Manifold". "Lime Trumpet" (3.0) is slated to conquer non-cartesian datasets and possibly some new IO subsystems. Personally I think "Lime Tuba" is a more fitting name, but that can also be discussed. New stuff since 2.2: * Multi-level parallelism * Real, extensive answer tests * Boolean data regions * Isocontours / flux calculations * "deliberate_field" branch merge (pending PR) * PHOP memory improvements * Bug fixes for tests * Parallel data loading for RAMSES, along with other speedups and improvements there * Performance improvements for volume rendering * (beta) Rockstar interoperability * Adaptive HEALpix support (with a few bugs left) * Column density calculations * Massive speedup for 1D profiles * Lots more, bug fixes etc. (Including a bug fix for the tests!) In any case, if anyone wants to discuss what needs to be done and when we want to roll 2.3 out by, I suggest we have a G+ hangout at the yt page ( https://plus.google.com/107728486871834552760). I've set up a doodle -- http://www.doodle.com/95f9rccyreundr65 -- if you're interested fill it out and we'll figure out a time. Let me know if none of the times are possible. Sam

3 4

Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch
by Matthew Turk Nov. 11, 2011

Nov. 11, 2011

Hi all, Over the last couple months, Casey and I have been working -- on and off! -- on a new branch of the code called "deliberate_fields." This branch will change, in a substantial but easy-to-update way, how fields are handled in yt. I recognize this email is long. But if you use non-standard fields, a bunch of derived fields, unit modifications, any of that, it may affect you. So I *please* ask that you read it and, if you like, contribute back to the discussion. This is one of the items I really want to have done for a hypothetical 2.3 release. = Background = The way fields work currently was designed a bit haphazardly. They use FieldInfoContainers, objects which share state and which contain unions of the known derived fields and the known IO-based fields. One of the problems with this is that the only thing that separates a derived field from a known field is that function that generates the field: the IO-based fields all use a lambda which returns None, and the non-IO based fields return actual fields. This is pretty sub-optimal, and it actually lands us in trouble when (for instance) we have fields wandering around named things like "Thermal_Energy" and "ThermalEnergy"; the mechanism by which one is selected and the other not is problematic, and to get around infinite recursion, hacks have had to be applied. As it stands, to find a field, the shared-state "field info" on a parameter file is queried; this then will try to check universal fields. But because of how the fields are stored, the field info cascade can also operate in reverse. The big problem is that the field selection mechanism doesn't seem to have a bus factor >= 1.0. And, it has a number of hacks to make it work with conflicting field definitiosn and the like. Unfortunately, layering these hacks on top of each other makes it much harder for other codes to be supported; translations are not reliable, and sometimes cause too many levels of recursion to be added. Something simpler is necessary. = What this does = Essentially, this creates multi-level, explicit fallbacks. The field info container, which was a bloated, weird shared state object, is now simply a dictionary subclass with a "fallback" option. When you create them, you can either create it in isolation (with no fallback) or with a fallback. When you query it, if it does not have a field, it checks its fallback. There are, additionally, two new functions for IO: the translation function and the null function. The first is to translate, for instance, "density" to "Density" and the second is to indicate that a field is expected to be found in an output from the simulation code. There are now affiliated with each simulation code two field info objects: the "known" fields, which may appear in files, and the non-known (i.e., code-specific derived) fields. These live as the attributes _fieldinfo_fallback and _fieldinfo_known on the StaticOutput sublcass corresponding to a simulation code. When the Hierarchy (not static output) is instantiated, the first step is to create a new field_info object. This has, as a fallback, the _fieldinfo_fallback, which itself has as a fallback the universally-known derived fields. The hierarchy then queries the output file for which fields are available. This process then looks for a corresponding field in fieldinfo_known, and if it finds it, it adds it to the field_info object, *overriding* any possible derived fields. (In this manner, for instance, yt will not recalculate a "CoolingTime" field if one exists in the output.) = What it aims to do in the future = This will be utilized in three main ways: 1) Making it more clear which fields belong to which code, and which come from disk and which are derived 2) Help move IO into fields, to optimize for geometries and data containers 3) Make units more clear and specific 4) This is all designed around better supporting the GDF. = Where from here? = It would be hugely beneficial if you could test this and report back. I have created a pull request: https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-util… This is by no means a settled matter; I think we need to have testing on this, buy-in from developers and users, and to make sure that old code doesn't beak. The test cases all pass for me for Enzo. Before this can be merged, I would hope we can get some testing from: * Enzo * Nyx * FLASH * Orion and any other codes that can hear me. Thanks very much for your time; please let me know if you have any questions, concerns, jokes, comments, improvements, CDs of your band, suggestions, and so on. For this major of a change I'd like to keep discussion on list, so the record of this is a bit more prominent. Best, Matt

5 11

proposal: turning off parameter file storage by default
by Sam Skillman Nov. 10, 2011

Nov. 10, 2011

Hi all, There have been some recent issues that have popped up when parameter file storage is used when accessing different parameter files at the same time in a parallel job. While the long term solution is to move to something thread-safe like sqlite, I am of the opinion that for now we should turn off this capability by default. We would leave the capability in there, and it would be turned on with: from yt.config import ytcfg; ytcfg["yt", "storeparameterfiles"] = "True" at the very top of a yt script. Since this could disrupt some people's workflows, I think a +/-1 would be helpful, along with any strong opinions. Thanks, Sam

3 3

Written interview floss4science.com
by FLOSS FORSCIENCE Nov. 7, 2011

Nov. 7, 2011

Hi. My name is Edwood Ocasio and I am the founder of http://www.floss4science.com, a site about open source software in science and engineering. I will like to publish a written interview about the yt Project as a featured post in floss4science.com I worked for 20 years as a professor of computer science, mathematics and statistics in different universities in Puerto Rico. You can easily find traces of me on the web. I also have a group for an Spanish speaking audience interested in open source software for scientific computing ( http://groups.google.com/group/compcilibre). * The interview publishing process * 1. I will send you a private link to drafts of the interview before publishing it to allow you to make corrections. 2. Your photo and/or screenshots of your software can be included. The photo is not a requirement. 3. Interviews will be queued for weekly publishing. I will notify you the expected publishing date. * Interview license * This interview, as all content in floss4science.com, will be distributed with a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License (http://creativecommons.org/licenses/by-nc-nd/3.0/) Below are the questions. You can answer them right here and just reply to this email. You can leave questions unanswered if they do not seem to apply to you or the yt Project or cannot answer for whatever reason. Again, thanks. ====== INTERVIEW QUESTIONS ====== 1. Please, give us a brief introduction about yourself, i.e. profession, workplace, projects, etc. 2. What is the yt Project? 3. Why and when did the yt Project came to be? 4. In which [language(s)|platform(s)] is the project developed? 5. Does the yt Project have sponsors (private, academic, other)? 6. How are the sponsors supporting the project? 7. How many users you estimate the yt Project have? 8. Do you know where is the yt Project used (corporations, universities, other)? 9. How many team members does the project have? 10. In what areas of the yt Project development do you currently need help? 11. How can people get involved with the project? 12. What features are in the roadmap? 13. Which projects, blogs or sites related to open source software for science can you recommend? 14. Why do you consider free/libre open source software important for the advancement of your field? 15. Is there any other topic you would like our readers to know about? 16. Where people can contact you and learn more about the yt Project (website, blog, email, twitter, identi.ca, Facebook, etc.)? Thank you for sharing with us more about you and the yt Project. =========== -- Cordially, Edwood Ocasio Founder floss4science.com E: info(a)floss4science.com W: http://www.floss4science.com Identi.ca: http://identi.ca/floss4science Twitter: http://twitter.com/#!/floss4science

2 1

Releases, pull requests, "governance" ...
by Matthew Turk Nov. 2, 2011

Nov. 2, 2011

Hi all, There's been some discussion on the Enzo mailing list: https://groups.google.com/forum/#!topic/enzo-users/vtlCedbJILU about the stable/dev branch of yt. The takeaway is that bugs get fixed in the dev branch, but since our releases are much less frequent than that, they don't necessarily get propagated very rapidly to stable -- in large part because we also weave lots of new development in and out. Enzo deals with this by doing all development in forks, mandating pull requests, and being careful about which items to merge and when. It's not clear to me that we could implement that model, even if we wanted to. We've had recent success with developing in forks, issuing pull requests and the like, but there's no codified set of "when to develop in a fork" guidelines or anything like that. There've been some rumblings of "governance" discussions about yt, but nothing ever really formalized. So I guess what it comes down to -- do we want to start being aggressive about putting out patch (i.e., 2.2.X) releases? Do we need a set of release managers or curators, like there are in Enzo? When do we develop in a fork, when do we feel comfortable pushing to the main repo? The problem, as I see it, is that in some respects yt suffers from the same *problems* a project like Enzo does, in that some pieces of functionality are critical to a few applications, but it does not have the level of support or engagement as Enzo, so we don't have as much ability to spread workload around. Any thoughts on this? -Matt

3 2

undefined variables in amr_kdtree.py
by Britton Smith Nov. 1, 2011

Nov. 1, 2011

Hi everyone, I'm coming across instances of the variable 'nprocs' being undefined in save_kd_bricks and load_kd_bricks in amr_kdtree.py. I can't quite tell what they should be instead or where they should be defined. Does anyone have any ideas? Britton

2 1