Mailman 3 September 2013 - yt-dev

License switch: remove author tags
by Nathan Goldbaum 11 Sep '13

11 Sep '13

Hi all, I'd like to bring up for discussion something I brought up on the license switch PR: https://bitbucket.org/yt_analysis/yt/pull-request/596/relicensing-to-bsd-3-… Specifically, I think we should remove the author tags, which Matt left in the source code after changing the license tags. My reasons are fleshed out in a section from the book Team Geek, by Brian Fitzpatrick and Ben Collins-Sussman, the lead devs on the subversion project: http://books.google.com/books?id=Iwk_pKeBc9gC&pg=PA48&lpg=PA48&dq=%22puttin… While the poisonous patterns described in the book haven't happenned in yt in the past, there's always the chance that it will happen in the future. I think we also lose nothing by dropping the author tags since the authorship information encoded in the tags has been rendered redundant and incomplete by the authorship information encoded in the mercurial repository. I'd prefer to just have "the yt development team" as the author on all source code files. I don't want this suggestions to hold up acceptance of Matt's PR and I will happily change all of the author tags myself if we decide to remove them. Cheers, Nathan

9 16

Re: [yt-dev] License switch: remove author tags
by John Wise 11 Sep '13

11 Sep '13

+1 on removing the author names and creating a page listing all of the developers on each point release like IPython. Cheers, John Britton Smith <brittonsmith(a)gmail.com> wrote: >_______________________________________________ >yt-dev mailing list >yt-dev(a)lists.spacepope.org >http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

2 1

Packaging.
by Matthew Turk 11 Sep '13

11 Sep '13

Hi all, We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.) As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types. Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable. That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple. There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages. So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front. * Conda / Anaconda: This package manager is gaining traction, and I think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers. * Canopy: This is the Enthought package manager, which Sam has had some good experience with it. I do not have a feeling for how it works on supers. * Source-only: This is the way some packages are managed, but it is essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable. * Keep trying to plug holes as they come up in the install script. What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody? Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one. -Matt PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them.

10 32

Issue #656: HaloProfiler fails if output_dir keyword requires recursive mkdir. (yt_analysis/yt)
by Britton Smith 09 Sep '13

09 Sep '13

New issue 656: HaloProfiler fails if output_dir keyword requires recursive mkdir. https://bitbucket.org/yt_analysis/yt/issue/656/haloprofiler-fails-if-output… Britton Smith: Responsible: brittonsmith

1 0

Frontend tests
by Matthew Turk 09 Sep '13

09 Sep '13

Hi all, I'd like to bring up the topic of frontend tests. As it stands, we have answer tests for a few frontends, and a large amount of sample data for other frontends. The answer tests are fine, but they are also somewhat ... difficult to write and come up with. I believe that they should still exist inside the main repo. However, what I'd like to propose is a new repository ("frontend_tests" perhaps?) that includes scripts for each frontend that load data, save images, and that we will then *version* a set of results images and data inside. This repository will be allowed to grow larger than we'd like the main yt repository to grow, and it would also mean that we could use normal pull request mechanisms for updating results, rather than AWS S3 uploads with keys and so on. My idea was that it would include something like a helper function library (for common routines like "what is the current version of the code" and "commit a new version of this image") and would also include image-generating scripts and mechanisms for viewing images. The idea is that you would run a script at the top level and it would spit out a bunch of images or data, and there would be templates of HTML that you could view old-versus-new. This could then be integrated into our CI system (I spoke with Kacper about this previously). It would serve two purposes: 1) Display results as a function of the current iteration (these results would not be quantitatively assessed; this would be the function of the answer testing we already have) 2) Tell us if loading a dataset or frontend breaks 3) Light quantitative result analysis I'm planning to create a repository similar to this regardless (for demonstrating frontend scripts) but I'm bringing it to the list to see if it is alright to host under the main yt_analysis team on Bitbucket and to integrate into testing. Does this appeal to anyone? I imagine it could be much simpler than creating new answer tests. -Matt

2 3

Issue #655: EnzoSimulationTimeSeries fails when there are no time outputs. (yt_analysis/yt)
by Britton Smith 09 Sep '13

09 Sep '13

New issue 655: EnzoSimulationTimeSeries fails when there are no time outputs. https://bitbucket.org/yt_analysis/yt/issue/655/enzosimulationtimeseries-fai… Britton Smith: If dtDataDump is set to 0, the time_outputs list does not get defined. Responsible: brittonsmith

1 0

Pull request validation
by Matthew Turk 08 Sep '13

08 Sep '13

Hi all, Kacper just issued a PR to update YTEP-0007: https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0007.html Pull requests are now automatically validated, and a robot will leave comments to let you know if it's good to merge. As an example, yt-fido just told us: https://bitbucket.org/yt_analysis/yt/pull-request/588/move-bundled-software… This is pretty awesome. Kacper, I think I speak for everyone when I say that we're all looking forward to the next stage of development, where your robots will begin issuing pull requests. -Matt

7 6

Issue #654: deposit *_count fails in artio (yt_analysis/yt)
by Sam Leitner 06 Sep '13

06 Sep '13

New issue 654: deposit *_count fails in artio https://bitbucket.org/yt_analysis/yt/issue/654/deposit-_count-fails-in-artio Sam Leitner: For the following script applied to a DM-only ARTIO fileset ``` #!python from yt.mods import * pf = load("/home/sleitner/run1/refall2/out/agora.nb_a0.2504.art") sl = SlicePlot(pf, 'z', ('deposit', 'all_count')) ``` I get the following Traceback ``` #!python File "temp.py", line 3, in <module> sl = SlicePlot(pf, 'z', ('deposit', 'all_count')) File "/scratch/midway/sleitner/yt-3.0/yt/visualization/plot_window.py", line 1293, in __init__ slc.get_data(fields) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 533, in get_data self._generate_fields(fields_to_generate) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 549, in _generate_fields self.field_data[field] = self._generate_field(field) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 226, in _generate_field tr = self._generate_fluid_field(field) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 243, in _generate_fluid_field rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 253, in _generate_spatial_fluid for i,chunk in enumerate(self.chunks(field, "spatial", ngz = 0)): File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 453, in chunks self.get_data(fields) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 533, in get_data self._generate_fields(fields_to_generate) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 549, in _generate_fields self.field_data[field] = self._generate_field(field) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 226, in _generate_field tr = self._generate_fluid_field(field) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/data_containers.py", line 245, in _generate_fluid_field rv = finfo(gen_obj) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/field_info_container.py", line 498, in __call__ dd = self._function(self, data) File "/scratch/midway/sleitner/yt-3.0/yt/data_objects/particle_fields.py", line 74, in particle_count d = data.deposit(pos, method = "count") File "/scratch/midway/sleitner/yt-3.0/yt/frontends/artio/data_structures.py", line 178, in deposit f64 = [np.array(f, dtype="float64") for f in fields] TypeError: 'NoneType' object is not iterable ```

1 0

Issue #653: Several data objects do not have docstring entries in the API docs (yt_analysis/yt)
by Nathan Goldbaum 06 Sep '13

06 Sep '13

New issue 653: Several data objects do not have docstring entries in the API docs https://bitbucket.org/yt_analysis/yt/issue/653/several-data-objects-do-not-… Nathan Goldbaum: We rely on the docstrings somewhat to provide information beyond __init__ signatures on [this](http://yt-project.org/docs/2.5/analyzing/objects.html#available-objec… page. Several entries do not have links to the API docs because those objects have not been explicitly added. In particular, * AMRBooleanRegionBase * ExtractedRegionBase * AMRFixedResCuttingPlaneBase * AMRFixedResProjectionBase * AMRInclindedBoxBase * AMRPeriodicRegionBase * AMRQuadTreeProjBase * AMRRegionStrictBase

1 0

Issue #652: add_quantity fails silently if no combine_function is given (yt_analysis/yt)
by J.S. Oishi 06 Sep '13

06 Sep '13

New issue 652: add_quantity fails silently if no combine_function is given https://bitbucket.org/yt_analysis/yt/issue/652/add_quantity-fails-silently-… J.S. Oishi: If you add a new derived quantity but forget to give the `combine_function` keyword arg, no error is given, but the field is not added to the list of available derived quantities.

1 0