Mailman 3 August 2013 - yt-dev

Packaging.
by Matthew Turk 10 Sep '13

10 Sep '13

Hi all, We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.) As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types. Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable. That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple. There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages. So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front. * Conda / Anaconda: This package manager is gaining traction, and I think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers. * Canopy: This is the Enthought package manager, which Sam has had some good experience with it. I do not have a feeling for how it works on supers. * Source-only: This is the way some packages are managed, but it is essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable. * Keep trying to plug holes as they come up in the install script. What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody? Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one. -Matt PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them.

10 32

Frontend tests
by Matthew Turk 09 Sep '13

09 Sep '13

Hi all, I'd like to bring up the topic of frontend tests. As it stands, we have answer tests for a few frontends, and a large amount of sample data for other frontends. The answer tests are fine, but they are also somewhat ... difficult to write and come up with. I believe that they should still exist inside the main repo. However, what I'd like to propose is a new repository ("frontend_tests" perhaps?) that includes scripts for each frontend that load data, save images, and that we will then *version* a set of results images and data inside. This repository will be allowed to grow larger than we'd like the main yt repository to grow, and it would also mean that we could use normal pull request mechanisms for updating results, rather than AWS S3 uploads with keys and so on. My idea was that it would include something like a helper function library (for common routines like "what is the current version of the code" and "commit a new version of this image") and would also include image-generating scripts and mechanisms for viewing images. The idea is that you would run a script at the top level and it would spit out a bunch of images or data, and there would be templates of HTML that you could view old-versus-new. This could then be integrated into our CI system (I spoke with Kacper about this previously). It would serve two purposes: 1) Display results as a function of the current iteration (these results would not be quantitatively assessed; this would be the function of the answer testing we already have) 2) Tell us if loading a dataset or frontend breaks 3) Light quantitative result analysis I'm planning to create a repository similar to this regardless (for demonstrating frontend scripts) but I'm bringing it to the list to see if it is alright to host under the main yt_analysis team on Bitbucket and to integrate into testing. Does this appeal to anyone? I imagine it could be much simpler than creating new answer tests. -Matt

2 3

yt Strategy Meeting Monday the 22nd
by Matthew Turk 04 Sep '13

04 Sep '13

Hi all, I'd like to call a brief, approximately one hour, strategy meeting on Monday, April 22nd at 3PM Eastern. This will cover where yt is at, where we can go, and how things are progressing. Some of this will be dedicated to progress updates (from me and others) as well as soliciting feedback on a number of things. Many other projects -- notably Cactus -- do these on a fixed schedule, so based on the success/failure of this one we can consider having them semi-regularly. The agenda I would like to cover will include: * yt 3.0 status, including near/long term plans for development and support for non-grid codes * Rundown of all YTEPs accepted and outstanding * yt's "presence" on the web * 3.0 paper I will also be soliciting feedback in roundtable style on: * How are we doing? * How is the procedure for code contributions working? * How is the yt "infrastructure" meeting people's needs? (Mailing lists, Bitbucket, etc) * Where are energies being spent that could be better spent elsewhere? If you have either agenda items or discussion items, please reply to this email and suggest them. I think having a large presence at this meeting would be very useful, so if you are able to make it, please try to do so. It'll be held as a google hangout, and I'll send details here when it starts up. If there's a pressing need to reschedule, please write in reply to this email. Thanks, and looking forward to seeing you there, Matt

2 3

Documentation Sprint and Creating BB Issues When You See a Gap
by Cameron Hummels 03 Sep '13

03 Sep '13

Hey peeps, After talking with Matt at some length about the yt documentation, we thought it might be a good idea to have a documentation sprint in a few months when we've moved over to yt 3.0 as the focus of our development. The docs in general are good, but there are some gaps here and there with some functionality not present in the documentation. I'm not proposing an entire (or even partial) rewrite of the docs, as I think that would be counterproductive. I'm simply thinking we could fill in the holes to make sure all of the cool stuff in yt is written up so people know how to use it. Now, I know doc writing is not often fun, but I think this could be really beneficial to our user base, and actually cut down on the amount of time we have to respond to new users on the mailing list and irc (as well as making it easier for people to use yt). What I'm asking from you is if you encounter something that you don't think is well documented in the formal docs (not simply the docstrings), could you take a moment to create a bitbucket issue about it? You don't even have to fix it then, just note it, so we know where to work when we do the sprint in a few months. Anyone else have any other ideas about this? Cameron -- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org

8 17

cell padding in RegionSelector
by Douglas Harvey Rudd 01 Sep '13

01 Sep '13

Hi all, Matt and I are finishing up our changes to the selector functions and ran into a feature which may or may not be currently used. In RegionSelector, there is a parameter _dx_pad to allow for cells just outside the selected region to be included. Is anyone using that feature or can it be deprecated? Douglas Rudd Scientific Computing Consultant Research Computing Center drudd(a)uchicago.edu

2 3

Issue #643: Install script should build jinja2 (yt_analysis/yt)
by Nathan Goldbaum 29 Aug '13

29 Aug '13

New issue 643: Install script should build jinja2 https://bitbucket.org/yt_analysis/yt/issue/643/install-script-should-build-… Nathan Goldbaum: jinja2 is now an IPython dependency.

1 0

Issue #642: YT Backend on OS 10.7 (yt_analysis/yt)
by Anonymous 29 Aug '13

29 Aug '13

New issue 642: YT Backend on OS 10.7 https://bitbucket.org/yt_analysis/yt/issue/642/yt-backend-on-os-107 Anonymous: matplotlib reports a python framework install issue. This was fixed by changing the Backend in the matplotlibrc file to something other than OS X. Responsible: ngoldbaum

1 0

Issue #641: ZeroMQ 3.2.3 release tarball has a small build-breaking issue (yt_analysis/yt)
by Nathan Goldbaum 29 Aug '13

29 Aug '13

New issue 641: ZeroMQ 3.2.3 release tarball has a small build-breaking issue https://bitbucket.org/yt_analysis/yt/issue/641/zeromq-323-release-tarball-h… Nathan Goldbaum: As reported by Stephanie Tonnesen on the user's list, there is a small issue with the ZeroMQ 3.2.3 release tarball we are distributing. I think the most straightforward way to deal with this is to repackage with the newline appended to the affending file, but perhaps there's a nicer way to do it.

1 0

Issue #640: Enable changing the location of a SlicePlot (yt_analysis/yt)
by Stuart Mumford 27 Aug '13

27 Aug '13

New issue 640: Enable changing the location of a SlicePlot https://bitbucket.org/yt_analysis/yt/issue/640/enable-changing-the-location… Stuart Mumford: I would like to be able to animate changing the location of a SlicePlot, i.e. sweeping through a domain. Something like this ```python slc = yt.SlicePlot(ds_h5, 'z', 'VortXmag', origin='left-domain', center=(1e8,1e8,10*12.5e5)) slc.set_window_size(8) fig = slc.plots['VortXmag'].figure def animate(i): slc.set_center((1e8,1e8,i*12.5e5)) animation.FuncAnimation(fig, animate, frames=range(0,127,2), interval=50, blit=True) ```

1 0

Issue #639: PlotWindow save should accept directory prefix (yt_analysis/yt)
by Matthew Turk 26 Aug '13

26 Aug '13

New issue 639: PlotWindow save should accept directory prefix https://bitbucket.org/yt_analysis/yt/issue/639/plotwindow-save-should-accep… Matthew Turk: PlotWindow's save should detect if the filename supplied ends with `os.pathsep` and if so, it should simply prefix that before the filename it would otherwise generate.

1 0