yt-dev April 2012

yt-dev@python.org

10 participants
10 discussions

Fwd: [yt_analysis/yt] Removing the astro_objects directory, which was an ill-motivated idea. (pull request #146)

by Matthew Turk

(Sometimes I forget to forward PRs, but I have been trying to remember!) ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Mon, Apr 30, 2012 at 4:18 PM Subject: [yt_analysis/yt] Removing the astro_objects directory, which was an ill-motivated idea. (pull request #146) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/146/removing-the-astro_ob… Title: Removing the astro_objects directory, which was an ill-motivated idea. This idea wasn't well-thought out, and I'd like to see it gone. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

8 years, 5 months

Fwd: [yt_analysis/yt] Change time series objects to *not* instantiate pfs by default (pull request #143)

by Matthew Turk

Hi all, A couple people (Nathan and Britton) have mentioned memory problems with time series data because the StaticOutput objects sometimes hold on to hierarchies. I've taken a pass at preventing this from happening by keeping the static outputs from being instantiated until requested, and allowing the references to be passed rather than retained. Anyway, it works for me and I've checked that the references go to zero. If somebody else that was having memory issues could take a gander that'd be great. Here's my sample script; if you run before&after the numbers it prints out at the end should be "something" and "0". http://paste.yt-project.org/show/2302/ -Matt ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Sun, Apr 22, 2012 at 10:52 AM Subject: [yt_analysis/yt] Change time series objects to *not* instantiate pfs by default (pull request #143) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/143/change-time-series-ob… Title: Change time series objects to *not* instantiate pfs by default This change, as described in the commit message, will prevent time series data from initially instantiating any parameter file objects. By keeping the references to pfs temporary, this will prevent hierarchies from sticking around during the course of analysis, which should improve overall memory usage. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

8 years, 5 months

Vector fields

by Matthew Turk

Hi all, I got an email from someone identifying a problem in profiles. This person was dealing with vector fields, which were giving bad results. I wanted to open up discussion here, as I know Jeff and Sam and others have thought about this. I'll contribute some thoughts after giving the mic over to someone else. In an ideal world, how should a vector field -- like angular momentum, in particular -- be presented and dealt with in yt? -Matt

8 years, 5 months

2.4: Release and bug roundup

by Matthew Turk

Hi all, A couple people have mentioned to me recently (in IRC and over email) that the stable branch is missing some fixes for FLASH and Enzo data. I'd like to start a discussion on wrapping up the dev cycle and moving for a 2.4 release. The showpoint of this release I believe will be improvements to volume rendering; more on that below. Here are the open, targeted bugs for 2.4: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… These touch on a few major areas: * Minor improvements to reason and reason widgets: Adding vectors, layers (which should get postponed) a volume rendering widget (should get postponed) and a file open dialog (Nathan had been working a bit on that.) * FLASH updates to make it more user friendly: John Z and I had a brainstorming session after I spoke to some FLASH users last week, and I filled out tickets to try to address any remaining weak points. * Time Series updates: Nathan has made a few suggestions, and I'm going to implement them. I like where Time Series is going! * Volume rendering refactor (Sam, if you have a few minutes, could you give an update on what that includes and what it provides? This could be a great centerpiece of 2.4!) * Rockstar integration is incomplete and undocumented, but that may also be postponed. There are also a few bugs that need to be addressed: * MPI attributes now break pickling in some cases; I will address this * Ghost zone issues seem to be everywhere, as corner cases keep coming up. John W and I have spoken about this and we have an idea for how to fix it moving forward. * Quadproj scales poorly, but that'll be a bit of a tricky piece of work. Does anyone have any comments or ideas about these bugs, or want to tackle any of them? Are there any other concerns, or thoughts on this? How would aiming to release in a month from now sound? April 16 is a Monday, which would be a good day for a release. -Matt

8 years, 5 months

Gridding Particles

by Christopher Moody

Hi everyone, I just asked the IRC list this, but I guess no one was there. I'm trying to grid a few million particles onto the yt grids, but checking if the left and right edges of my particles are inside every cell is slow: #find particles inside cell idx = na.logical_and(na.all(le < pos,axis=1), na.all(re > pos,axis=1)) #assign them to a grid ... #remove particles from future searches pos = pos[~idx] I start with the finest grids at the highest levels, and once the particles are gridded they're removed -- so the list of particles is always decreasing, and the coarsest mesh tends to have only a few particles. I'm guessing that somewhere this has been done before - I'd rather not reinvent the wheel. Has anyone used any relevant kdtree (or cython) code in the yt utilities that can select a subvolume of particles quickly? Thanks ahead of time! chris

8 years, 5 months

Fwd: [yt_analysis/yt] Adding IPython notebook dependencies (on-by-default) to the install script. (pull request #137)

by Matthew Turk

Hi all, I thought this pull request might be of interest to the list, and we could discuss it here, rather than in the PR itself. The IPython notebook is awesome. I love Reason, and I'm super proud of the work we have done on it, but the clunkiest part of Reason is the cell-evaluation (and, it's also the part of it that we spent the least time on.) I think it would be worthwhile to discuss and consider moving forward with de-emphasizing our own notebook solution, embed IPython inside Reason, and then re-focus on providing awesome widgets. I am still reading through the API for the IPython notebook's engine interface, but I believe this could be accomplished by: 1) Spinning up an SSL-enabled, passworded IPython engine when reason is started 2) Construct actual RPC interfaces for the existing (and new) widgets, with payload delivery systems 3) Embed the IPython notebook as another widget Anyway, if you want to test it out, pull the new install script, run it, and then start the notebook with: ipython notebook There are some pretty awesome things in there. Fernando Perez and I discussed this on a G+ post of his ( https://plus.google.com/105051551851350439748/posts/gd9fDgpV8iY ) and I'd particularly draw your attention to the blog post he cited by Titus Brown ( http://ivory.idyll.org/blog/apr-12/replication-i.html ; I believe his office is just down the hall from where we held the Enzo workshop last Spring!) and the possibilities therein. Even if down the road we don't find a way to marry Reason widgets with the IPython notebook in a seamless way, I'm still issuing this PR because I would like to make it easier to get up and running with the notebook. Note that they have a detailed discussion of the security implications ( http://ipython.org/ipython-doc/rel-0.12/interactive/htmlnotebook.html#secur… ), which become very important on shared user systems, which I think make up the majority of yt installations. Thoughts on this? -Matt ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Wed, Apr 4, 2012 at 3:31 PM Subject: [yt_analysis/yt] Adding IPython notebook dependencies (on-by-default) to the install script. (pull request #137) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/137/adding-ipython-notebo… Title: Adding IPython notebook dependencies (on-by-default) to the install script. The IPython notebook requires 0MQ, Tornado and pyzmq. This change adds those dependencies. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

8 years, 5 months

Fwd: SciPy 2012 - The Eleventh Annual Conference on Scientific Computing with Python

by Matthew Turk

---------- Forwarded message ---------- From: Warren Weckesser <warren.weckesser(a)enthought.com> Date: Wed, Apr 4, 2012 at 5:30 PM Subject: [pydata] SciPy 2012 - The Eleventh Annual Conference on Scientific Computing with Python To: SciPy Users List <scipy-user(a)scipy.org>, scikit-learn-general(a)lists.sourceforge.net, SciPy Developers List <scipy-dev(a)scipy.org>, IPython Development list <ipython-dev(a)scipy.org>, enthought-dev(a)enthought.com, epd-users(a)enthought.com, Discussion of Numerical Python <numpy-discussion(a)scipy.org>, pystatsmodels(a)googlegroups.com, pydata(a)googlegroups.com SciPy 2012, the eleventh annual Conference on Scientific Computing with Python, will be held July 16–21, 2012, in Austin, Texas. At this conference, novel scientific applications and libraries related to data acquisition, analysis, dissemination and visualization using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is preceded by two days of tutorials, during which community experts provide training on several scientific Python packages. Following the main conference will be two days of coding sprints. We invite you to give a talk or present a poster at SciPy 2012. The list of topics that are appropriate for the conference includes (but is not limited to): new Python libraries for science and engineering; applications of Python in solving scientific or computational problems; high performance, parallel and GPU computing with Python; use of Python in science education. Specialized Tracks Two specialized tracks run in parallel to the main conference: High Performance Computing with Python Whether your algorithm is distributed, threaded, memory intensive or latency bound, Python is making headway into the problem. We are looking for performance driven designs and applications in Python. Candidates include the use of Python within a parallel application, new architectures, and ways of making traditional applications execute more efficiently. Visualization They say a picture is worth a thousand words--we’re interested in both! Python provides numerous visualization tools that allow scientists to show off their work, and we want to know about any new tools and techniques out there. Come show off your latest graphics, whether it’s an old library with a slick new feature, a new library out to challenge the status quo, or simply a beautiful result. Domain-specific Mini-symposia Mini-symposia on the following topics are also being organized: Computational bioinformatics Meteorology and climatology Astronomy and astrophysics Geophysics Talks, papers and posters We invite you to take part by submitting a talk or poster abstract. Instructions are on the conference website: http://conference.scipy.org/scipy2012/talks.php Selected talks are included as papers in the peer-reviewed conference proceedings, to be published online. Tutorials Tutorials will be given July 16–17. We invite instructors to submit proposals for half-day tutorials on topics relevant to scientific computing with Python. See http://conference.scipy.org/scipy2012/tutorials.php for information about submitting a tutorial proposal. To encourage tutorials of the highest quality, the instructor (or team of instructors) is given a $1,000 stipend for each half day tutorial. Student/Community Scholarships We anticipate providing funding for students and for active members of the SciPy community who otherwise might not be able to attend the conference. See http://conference.scipy.org/scipy2012/student.php for scholarship application guidelines. Be a Sponsor The SciPy conference could not run without the generous support of the institutions and corporations who share our enthusiasm for Python as a tool for science. Please consider sponsoring SciPy 2012. For more information, see http://conference.scipy.org/scipy2012/sponsor/index.php Important dates: Monday, April 30: Talk abstracts and tutorial proposals due. Monday, May 7: Accepted tutorials announced. Monday, May 13: Accepted talks announced. Monday, June 18: Early registration ends. (Price increases after this date.) Sunday, July 8: Online registration ends. Monday-Tuesday, July 16 - 17: Tutorials Wednesday-Thursday, July 18 - July 19: Conference Friday-Saturday, July 20 - July 21: Sprints We look forward to seeing you all in Austin this year! The SciPy 2012 Team http://conference.scipy.org/scipy2012/organizers.php

8 years, 6 months

Fields refactor hangout

by Casey W. Stark

Hi all. Matt will be away for the first part of the hangout, so I will host for now. Please join us if you would like to discuss the new fields design, or anything else in the yt 3.0 ideas doc. The url of the hangout is https://plus.google.com/hangouts/_/710595e7d987bde736ddd0b6024c6d8a3ba9b542. Let me know if that doesn't work. Best, Casey

8 years, 6 months

Re: [yt-dev] Interacting with data in yt 3.0 (was Field units from code to code)

by Matthew Turk

In general, I agree with the idea Nathan put out. (Also, I think this is a fine time to have a bikeshed discussion. Many of the underlying assumptions about how yt works were laid out a long time ago.) But, I'm not entirely sure I understand how different it would be -- conceptually, yes, I see what you're getting at, that we'd have a set number of attributes. In what I was thinking of for the geometry refactor so far I'm trying to get rid of the "hierarchy" as existing for every data set, and instead relying on what amounts to an object-finder and io-coordinator, which I'm calling a geometry handler. It sounds like what you would like is: 1) Get rid of accessing parameters with an implicit __getitem__ on the parameter file (i.e., pf["SomethingThatOnlyExistsInOneCode"]). I'm +10 on this. 2) Move units into the .units object (I'm mostly with Casey on this, but I think it should be a part of the field_info object) 3) Have things like current_time, domain_dimensions and so on move into basic_info and make them dict objects. I think of those, I'm in favor of one and two, but somewhat opposed to #3. Right now we have these attributes mandated for subclasses of StaticOutput: refine_by dimensionality current_time domain_dimensions domain_left_edge domain_right_edge unique_identifier current_redshift cosmological_simulation omega_matter omega_lambda hubble_constant The only ones here that I think would be okay to move out of properties would be the cosmology items, and even those I'm -0 on moving. But, in general, the idea of moving from this two-stage system of parameter file (rather than dataset) and hierarchy (rather than an implicitly-handled geometry) is something I am in support of. The geometry is something that should nearly *always* be handled by the backend, rather than by the user. So having the library require pf.h.sphere(...) is less than ideal, since it's exposing something relatively unfortunate (that building a hundred thousand grid objects can take some time). The main ways that the static output is interacted with: * Parameter information specific to a simulation code * Properties that yt needs to know about * To get at the hierarchy * Input to plot collections The main ways that the hierarchy is interacted with: * Getting data objects * Finding max * Statistics about the simulation * Inspecting individual grids (much less common use case now that it was before) All of these use cases are still valid, but I think it's clear that accessing individual grids and accessing simulation-specific parameters are not "generic" functions. What a lot of this discussion has really brought up for me is that we're talking about *generic* functionality, not code-specific functionality, and we right now do not have the best enumeration of functionality and where it lies. With the geometry_refactor, I'd like to consolidate functionality into the main "dataset" object. The geometry can still provide access to the individual grids (of course) but data objects, finding max, getting stats about the simulation, etc, should all go into the main dataset object, and the geometry handler can simply be created on the fly if necessary. This brings up two points, though -- 1) Does our method of instantiating objects still hold up? i.e., ds.sphere(...) and so on? Or does our dataset object then become overcrowded? I would also like to move *all* plotting objects into whatever we end up deciding is the location data containers come from, which for instance could look like ds.plot("slice", "x") (for instance, although we can bikeshed that later), which would return a plot window. 2) Datasets and time series should behave, if not identically, at least consistently in their APIs. Moving to a completely ds-mediated mechanism for generating, accessing and inspecting data opens up the ability to then construct very nice and simply proxy objects. As an example, while something this is currently technically possible with the current Time Series API, it's a bit tricky: ts = TimeSeriesData.from_filenames(...) plot = ts.plot("slice", "x", (100.0, 'au')) ts.seek(dt = (100, 'years')) plot.save() ts.seek(dt = (10, 'years')) plot.save() (The time-slider, as Tom likes to call it ...) In general, this idea of moving toward more thoughtful dataset-construction, rather than the hokey parameter file + hierarchy construction brings with it a mindset shift which I'd like to spread to the time series, which can continue to be a focus. What do you think? -Matt On Thu, Mar 29, 2012 at 7:08 PM, Casey W. Stark <caseywstark(a)gmail.com> wrote: > +1 on datasets, although I would like to see the unit object(s) at the field > level. > > > On Thu, Mar 29, 2012 at 4:04 PM, Cameron Hummels > <chummels(a)astro.columbia.edu> wrote: >> >> +1 on datasets. >> >> >> On 3/29/12 6:58 PM, Nathan Goldbaum wrote: >>> >>> +1. I'd also be up to help out with the sprint. Doing a virtual sprint >>> using a google hangout might help mitigate some of the distance problems. >>> >>> While we're brining up Enzo-isms that we should get rid of, I think it >>> might be a good idea to make a conceptual shift in the basic python UI. >>> Instead referring to the interface between the user and the data as a >>> parameter file, I think instead we should be talking about datasets. One >>> would instantiate a dataset just like we do now with parameter files: >>> >>> ds = load(filename) >>> >>> A dataset would also have some universal attributes which would present >>> themselves to the user as a dict, e.g. ds.units, ds.parameters, >>> ds.basic_info (like current_time, timestep, filename, and simulation code), >>> and ds.hierarchy (not sure how that would interfere with the geometry >>> refactor). >>> >>> This may be a paintibg the bike shed discussion, but I think this shift >>> will help new users understand how to access their data. Thoughts? >>> >>> On Mar 29, 2012, at 3:40 PM, Matthew Turk<matthewturk(a)gmail.com> wrote: >>> >>>> Hi Nathan and Casey, >>>> >>>> I agree with what both of you have said. The Orion/Nyx units should >>>> be made to be consistent, but more importantly I think we should >>>> continue breaking away from Enzo-isms in the code. >>>> >>>> As it stands, all of the universal fields call underlying Enzo-named >>>> aliases -- Density, ThermalEnergy, etc etc. I hope we can have a 3.0 >>>> out within a calendar year, hopefully by the end of this year. (I've >>>> been pushing on the geometry refactor, although recently other efforts >>>> have been paying off which has decreased my output there.) I am much, >>>> much less doubtful than Casey is that we cannot do this; in fact, I'm >>>> completely in favor of this and I think it would be relatively >>>> straightforward to implement. >>>> >>>> In the existing system we have a mechanism for aliasing fields. What >>>> we can do is provide an additional translation system where we >>>> enumerate the fields that are available for items in UniversalFields, >>>> and then construct aliases to those. This would mean changing what is >>>> aliased in existing non-Enzo frontends, and adding aliases in Enzo. >>>> The style of name Casey proposes is what I woudl also agree with: >>>> underscores, lower cases, and erring on the side of verbosity. The >>>> fields off hand that we would need to do this for (in their current >>>> enzo-isms): >>>> >>>> x-velocity => velocity_x (same for y, z) >>>> Density => density >>>> TotalEnergy => ? >>>> GasEnergy => thermal_energy_specific (and thermal_energy_density) >>>> Temperature => temperature >>>> >>>> and so on. >>>> >>>> Once we have these aliases in place, an overall cleanup of >>>> UniversalFields should take place. One place we should clean up is >>>> ensuring that there are no conditionals; rather than conditionals >>>> inside the functions, we should place those conditionals inside the >>>> parameter file types. So for instance, if you have a field that is >>>> calculated differently depending on the parameter HydroMethod (in Enzo >>>> for instance) you simply set a validator on the field requiring the >>>> parameter be set to a particular value, and then only the field which >>>> satisfies that validator will be called when requested. >>>> >>>> So we've gotten rid of a bunch of enzo-isms in the parameter files; >>>> after fields, what else can we address? And, I'd be up for sprinting >>>> on this (which should take just a few hours) basically any time next >>>> week or after. I'd also be up for talking more about geometry >>>> refactoring, if anyone is interested, but it's not quite to the point >>>> that I think I am satisfied enough with the architecture to request >>>> input / contributions. Sometimes (especially with big architectural >>>> things like this) I think it's a shame we do all of our work >>>> virtually, as I think a lot of this would be easier to bang out in >>>> person for a couple hours. >>>> >>>> -Matt >>>> >>>> On Wed, Mar 28, 2012 at 6:14 PM, Casey W. Stark<caseywstark(a)gmail.com> >>>> wrote: >>>>> >>>>> Hi Nathan. >>>>> >>>>> I'm also worried about this and I agree that fields with the same name >>>>> should all be consistent. I would support some sort of cleanup of >>>>> frontend >>>>> fields, and I can get the Nyx fields in line and help with Enzo. >>>>> >>>>> I doubt we can do this, but I would prefer changing the field names as >>>>> part >>>>> of the removing enzo-isms and geometry handling refactoring pushes. For >>>>> instance, the field in Orion could be thermal_energy_density and the >>>>> field >>>>> in Enzo could be specific_thermal_energy. I also noticed this issue >>>>> when I >>>>> was using "Density" in Enzo (proper density in cgs) and "density" in >>>>> Nyx >>>>> (comoving density in cgs). >>>>> >>>>> Best, >>>>> Casey >>>>> >>>>> >>>>> On Wed, Mar 28, 2012 at 1:47 PM, Nathan Goldbaum<goldbaum(a)ucolick.org> >>>>> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> On IRC today we noticed that Orion defines its ThermalEnergy field per >>>>>> unit volume but Enzo and FLASH define ThermalEnergy per unit mass. Is >>>>>> this >>>>>> a problem? Since yt defaults to the Enzo field names, should we try >>>>>> to make >>>>>> sure that all fields are defined using the same units as in Enzo? Is >>>>>> there >>>>>> a convention for how different codes should define derived fields that >>>>>> are >>>>>> aliased to Enzo fields? >>>>>> >>>>>> One problem for this particular example is that the Pressure field is >>>>>> defined in terms of ThermalEnergy in universal_fields.py so the units >>>>>> of >>>>>> ThermalEnergy become important if a user merely wants the gas pressure >>>>>> in >>>>>> the simulation. >>>>>> >>>>>> One possible solution for this issue would be the units overhaul we're >>>>>> planning. If all fields are associated with a unit object, we can >>>>>> simply >>>>>> query the units to ensure that units are taken care of correctly and >>>>>> code-to-code comparisons aren't sensitive to the units chosen for >>>>>> fields in >>>>>> the frontend. >>>>>> >>>>>> Personally, I think it would be best if we could make sure that all of >>>>>> the >>>>>> fields aliased to Enzo fields have the same units. >>>>>> >>>>>> Nathan Goldbaum >>>>>> Graduate Student >>>>>> Astronomy& Astrophysics, UCSC >>>>>> >>>>>> goldbaum(a)ucolick.org >>>>>> http://www.ucolick.org/~goldbaum >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev(a)lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev(a)lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev(a)lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> !DSPAM:10175,4f74e5073356450621218! >>>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev(a)lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> _______________________________________________ >> yt-dev mailing list >> yt-dev(a)lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev(a)lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >

8 years, 6 months

Install script issue

by Andrew Myers

Hello yt developers, I don't know if this is a "bug," exactly, but I noticed an issue when doing a clean install of yt today involving the h5py install. Namely, it looks like if you have an non-empty CFLAGS environment variable (which can happen without your knowledge if you load certain module files in a supercomputing environment, for example), the h5py build will proceed without getting the "-fno-strict-aliasing" flag and the resulting module will not work. You can get around this either by 1) clobbering CFLAGS, or 2) adding "-fno-strict-aliasing" to it and re-running the script. It seems like the install script should either detect this and work around it, or else warn you that the h5py build has gone of the rails after the installation, because the tracebacks you get from trying to use the whacked-out h5py module are not very illuminating. I would do this myself, but I'm not sure my shell scripting skills are up to the task. Thanks, Andrew

8 years, 6 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

yt-dev April 2012