Mailman 3 December 2013 - yt-dev

Units in yt and astropy
by Erik Tollerud Dec. 5, 2013

Dec. 5, 2013

Hello all, I think this may be my first post on this list - I'm Erik Tollerud, one of the members of the Astropy coordinating committee. Today I learned there are plans afoot to do some refactoring of the yt units system. I was wondering if any of you here have spent much time looking at the Astropy units framework (https://astropy.readthedocs.org/en/stable/units/index.html)? I couldn't find much in a quick glance over yt's docs about how the unit system actually works, but I'm sure there's a lot of overlap there. So I can't help but think that we should at least try to get some sort of interoperability in place. Of course, you're welcome to directly use any part of Astropy's unit or Quantity frameworks that right now suit your needs. But given that one of Astropy's main goals is to improve compatibility and interoperability in python astronomy, tools, we are happy to do some leg work on our end if it will help you. -- Erik T

3 3

Thursday: Hangout 2PM Eastern
by Matthew Turk Dec. 5, 2013

Dec. 5, 2013

Hi all, On Thursday, I'll be hosting a hangout at 2PM Eastern. This will be mostly or exclusively about the units refactor, which Nathan and I will be issuing a PR for later today or tomorrow. Agenda: * Go over the units refactor from a high level * Describe the new field plugin system * Talk about where things have changed * Get feedback as people pull, update to, and try out the units system on their data The final one, where we're all going to pull down the changes, update to them and try them out, is going to be the most important part. So, if you can attend, that'd be really great. I've created an event here: https://plus.google.com/b/107728486871834552760/events/cnsc16nn6v8trc8n0ev1… I've enabled people who are attending to invite additional people, so if you aren't already in yt's circles (but I think most people are) you can just pop in to IRC on the day of and someone will invite you. -Matt

1 1

Issue #743: Derived quantities are difficult to make unit aware (yt_analysis/yt)
by Matthew Turk Dec. 5, 2013

Dec. 5, 2013

New issue 743: Derived quantities are difficult to make unit aware https://bitbucket.org/yt_analysis/yt/issue/743/derived-quantities-are-diffi… Matthew Turk: Derived quantities at present do not know anything about the units they are to return. This presents something of a problem, as we perform several operations on them that are challenging to do in a neutral fashion; they can return scalars as well as arrays, and either or both might be unit-ful. Here is the logic for how they are defined: ``` #!python retvals = [ [] for i in range(self.n_ret)] chunks = self._data_source.chunks([], chunking_style="io") for ds in parallel_objects(chunks, -1): rv = self.func(ds, *args, **kwargs) if not iterable(rv): rv = (rv,) for i in range(self.n_ret): retvals[i].append(rv[i]) # Note that we do some fancy footwork here. # _par_combine_object and its affiliated alltoall function # assume that the *long* axis is the last one. However, # our long axis is the first one! rv = [] for my_list in retvals: data = YTArray(my_list).transpose() rv.append(self.comm.par_combine_object(data, datatype="array", op="cat").transpose()) retvals = rv return self.c_func(self._data_source, *retvals) ``` Now, there are only a relatively small number of derived quantities, and some likely have to be rewritten to take advantage of the particle type and fluid type systems. Additionally, we wanted to add them "on demand" similar to fluid and particle fields. So I see a few different options. 1. Rewrite them to be unit-aware. This could also include fixes for how reduction operations are set up and an explicit noting of how many return values are to be expected. 2. Sanitize all arrays, record the units they are transformed into (they must be in the same units when viewed as ndarrays, or else we lose unitful protections), and then re-cast after passing between processors. 3. Hack around it somehow. I think that the first option is likely the best. We can redefine the derived quantity system to behave more like the new field system, and we can also take the opportunity to make it more explicit. Many problems we've had in the past have been related to `n_ret` being unclear and whatnot. I will first attempt to do 2, then 3, then 1. I think 1 is where we will end up, however. The derived quantities currently in the code: * TotalMass * CenterOfMass * WeightedAverageQuantity * WeightedVariance * BulkVelocity * AngularMomentumVector * StarAngularMomentumVector * ParticleAngularMomentumVector * BaryonSpinParameter * ParticleSpinParameter * IsBound * Extrema * Action * MaxLocation * MinLocation * TotalQuantity * ParticleDensityCenter * HalfMass I think several of these should be removed or rewritten to merge them. For instance, we can join the various spin parameter computations, potentially the variance and averaging, and possibly simplify the TotalMass computation as opposed to the TotalQuantity. We may be able to alias WeightedAverage and CenterOfMass as well. Responsible: MatthewTurk

1 0

spread the word?
by j s oishi Dec. 5, 2013

Dec. 5, 2013

Nautilus is back up. I just logged in. I was the only person on the node. I have no idea if this means it's back or if they just stopped booting people off... j

2 2

Fwd: [AstroPy] Ureka 1.0 is now available
by Matthew Turk Dec. 4, 2013

Dec. 4, 2013

---------- Forwarded message ---------- From: Christine Slocum <cslocum(a)stsci.edu> Date: Wed, Dec 4, 2013 at 1:13 PM Subject: [AstroPy] Ureka 1.0 is now available To: "AstroPy(a)scipy.org" <AstroPy(a)scipy.org> STScI and Gemini are announcing the release of Ureka 1.0. Improvements (with respect to beta 6): - test for completeness of OS library dependencies - many installation improvements - expanded documentation - support for Ubuntu 13.10, Fedora 19, and OS X Mavericks (10.9) - menu shortcuts on Linux - optional support for MySQL and PostgreSQL - bundled Fortran run-time library is always used to avoid conflict with OS version(s) Updated packages: - Gemini IRAF 1.12 - virtualenv (1.10.1) - SciPy (0.13.0) - wheel (0.22.0) - scikit-image (0.9.3) - Pillow (2.2.1) - CFITSIO (3350) - scikit-learn (0.14.1) - Tornado (3.1.1) - setuptools (1.3) - NumPy (1.8.0) - stsci.distutils (0.3.2) - astropy (0.3) - d2to1 (0.2.11) - PyParsing (2.0.1) New packages: - PyKE (2.4.0) - Six (1.4.1) - PyTZ (2013.8) - kapteyn (2.2.1b17) - BeautifulSoup (3.2.1) - mechanize (0.2.5) - jsonschema (2.1.0) - numexpr (2.2.2) - Dateutil (2.2) - mock (1.0.1) Ureka is a binary packaging installer for common astronomical software (primarily for the UV/Optical/IR community). The goals of the Ureka installer are to: 1) minimize the number of actions needed to install all the different software components. We are seeking a "one button install" (it's not quite one button, but not far from it). 2) permit installation without requiring system privileges. 3) make installs as problem-free as possible for the great majority of users. 4) allow users to install their own software (particularly Python-based) within this framework, or update versions of software within the framework. 5) permit different Ureka installations to coexist and to easily switch between them. 6) enable installing different versions of the same software package under a particular Ureka installation. 7) support Macs and most popular Linux variants. Ureka does not use LD_LIBRARY_PATH (or its Mac equivalent), nor require PYTHONPATH, minimizing the possibility of affecting existing software after installation or use. Should conflicts nevertheless arise, Ureka can easily be disabled temporarily or enabled only in specific terminal windows. Keep in mind that no installation system is completely foolproof (that's very nearly impossible to achieve). In particular, when users update or add software to the Ureka framework, they increase the risk of breaking something, but we feel that is an option that users should have as long as they understand the possible risks. This release includes IRAF 2.16 and associated packages for IRAF, DS9, and a fairly full suite of Python scientific software packages (e.g., numpy, scipy, and matplotlib). The full listing of included software can be found at this link: http://ssb.stsci.edu/ureka/1.0/docs/components.html Ureka can be downloaded from: http://ssb.stsci.edu/ureka/ (choose "1.0"). Installation and usage instructions can be found on the same web page. Please send questions or feedback to help(a)stsci.edu. _______________________________________________ AstroPy mailing list AstroPy(a)scipy.org http://mail.scipy.org/mailman/listinfo/astropy

1 0

Issue #742: Projection pixelization is screwy in unitrefactor (yt_analysis/yt)
by Nathan Goldbaum Dec. 3, 2013

Dec. 3, 2013

New issue 742: Projection pixelization is screwy in unitrefactor https://bitbucket.org/yt_analysis/yt/issue/742/projection-pixelization-is-s… Nathan Goldbaum: The following script: ``` #!python %matplotlib inline from yt.mods import * pf = load('Enzo_64/DD0043/data0043') pp = ProjectionPlot(pf, 0, 'density') pp.save() print pp._frb['density'] ``` outputs the following: ``` #!python [[ 0.00113812 0.00113812 0.00113812 ..., 0. 0. 0. ] [ 0.00113812 0.00113812 0.00113812 ..., 0. 0. 0. ] [ 0.00113812 0.00113812 0.00113812 ..., 0. 0. 0. ] ..., [ 0. 0. 0. ..., 0. 0. 0. ] [ 0. 0. 0. ..., 0. 0. 0. ] [ 0. 0. 0. ..., 0. 0. 0. ]] g/cm**2 ``` and this image: http://i.imgur.com/TSWUCge.png The issue has to do with the `px`, `py`, `pdx`, and `pdy` fields and happens at the level of `FixedResolutionBuffer` rather than in `PlotWindow`. Responsible: MatthewTurk

1 0

filtered_type kwarg in @particle_filter
by Britton Smith Dec. 3, 2013

Dec. 3, 2013

Hi all, I'm working with particle filters for the first time and I'm unclear on the role of the filtered_type keyword. For example, in the following script: http://paste.yt-project.org/show/4099/ I define filters that should catch every single particle in an Enzo-3.0 dataset. My understanding of the filtered_type keyword is that is preselected which type of particles to pass through the filter. In this dataset, I have the following particles: Type Number DarkMatter 32768 CenOstriker 10543 Total 43311 So, my understanding is that print ad['finest_all', 'particle_mass'].size should print 43311 and print ad['finest_all', 'particle_mass'].size should print 32768. However, they both print 43311. Is this a bug or am I not understanding what filtered_type is supposed to do? Britton

2 3

Unit refactor PR
by Matthew Turk Dec. 3, 2013

Dec. 3, 2013

Hi all, Nathan and I sprinted on a few remaining things for the unit refactor, to the point that I would like to see it get tried out and tested in production. Casey and Nathan have done some really cool stuff with the units work, and I think it will be awesome. I've issued a PR against the tip of the yt-3.0 branch. Nathan and I will likely be updating this throughout the day. https://bitbucket.org/yt_analysis/yt/pull-request/662/unit-refactor What works: * All of the non-VR *unit* tests * Plotting of things * Units on arrays! Each frontend needs to be manually converted, as fields are now completely redone -- and much nicer. We have redone: * Enzo * FLASH * RAMSES * Boxlib (mostly) * Stream However, before we got too far down the road, I wanted to verify that things are heading in a direction the developers are happy with. Specifically, things like the way fields are created (check out yt/fields/*.py and the frontends/*/fields.py for converted frontends) and the new UI for dealing with things. This *needs* to land in some form or another before the 3.0 release, and I'd like to start getting testing done now so that we can land the other big item, the rebranding. Unit refactor can't be accepted until: * We get some people to use it, say what's wrong. * The other frontends get converted. * The VR interface is fixed. * Plots have a unit-ful interface, which is mostly done. So, I'd like to ask that y'all try it out by pulling the bookmark unitrefactor from my fork ( MatthewTurk/yt ). On Thursday during the dev hangout at 2PM EST, this will be the biggest part of the agenda -- actually trying it out. -Matt

2 2

2.6.1 point release
by Matthew Turk Dec. 3, 2013

Dec. 3, 2013

Hi all, Because of a regression in phase plots in yt 2.6, we've just rolled out a point release of yt 2.6.1. If you have been using phase plots, or want to use them in the future, we recommend upgrading. If you're using the "stable" branch already from source, just run "yt update". -Matt

1 0

Milliparsecs
by Matthew Turk Dec. 3, 2013

Dec. 3, 2013

Hi all, Nathan and I are working on some unit things. One that came up is that it looked like the conversion to mpc was wrong. Turns out, it's not. But what it's set up as is "mpc" for "milliparsecs" and "Mpc" for "Megaparsecs." We'd like to get rid of any usage of milliparsecs, since I think the number of people who want to use that (globally) are probably between 0 and 0, and just make Mpc and mpc the same thing. If anybody objects, write back. Silence will be taken as tacit agreement! -Matt

7 11