Mailman 3 May 2012 - yt-dev

What's going on with the Hub
by Matthew Turk May 12, 2012

May 12, 2012

Hi all, Last August we announced the creation of the yt Hub, at hub.yt-project.org, to replace the ailing Barn project. It was somewhat successful; it received about 30 submissions, which is 30 more than we had before it was announced. Rick Wagner emailed me Sunday morning to let me know he'd been able to find a considerable amount of Spam on the site; while we had put in restrictions keeping it from going to the front page, or being "published" and sent through the RSS Feed / email subscription, it was still possible to find them while searching the site. Britton spent some time clearing it out, and it turns out that in total we had 10,000 spam submissions on the entire site. I attribute most of the traffic to clever google searches on the part of spammers -- the strings used on the hub come from the original software, pligg, which was vulnerable to spam. Back in March, I received a grant from Amazon in Education to launch a data pastebin, which has been happily up and running -- although underused, as it hasn't been announced properly. Hooks into it have been inserted into the development branch, will undergo a tad bit of polish before 2.4, and will be played up in the 2.4 release cycle. (For instance, PlotCollection objects are already uploadable, along with projections, and slices. We have a vertex-display method, but uploads are not nice yet.) It's kind of neat, but only time will tell if this fits a niche for people. I've slowly been adding on new stuff, and the next item is going to be "named pastes" -- something that we've kind of wanted for a while, but never implemented. Anyway, the convergence of these things has led us to try to implement project listing in the data hub. This will be much less accessible to spammers, as the only upload mechanism will be via API, rather than web form. We'll also have substantially more control over the presentation, methods of interaction, and the entire software stack. This brings with it some challenges, but I believe we can overcome them. As a first step, I've imported all the projects from the old hub, as well as the users for those projects. (You can regain access by resetting your password; your passwords are all set to random values.) This first step is here: https://data.yt-project.org/ but I would like to migrate it to hub.yt-project.org and unify both offerings. Project URLs will remain functional, although through redirects. Note that the getting started instructions require a change I implemented in development tip this afternoon. I'd like to solicit feedback on this: 1) What suggestions do you have for views, style, appearance, etc? 2) What problems do you see as is? 3) What needs to be added? For what it's worth, I'm working to add editing of submitted projects, to refine the yt mechanism for submitting new projects (right now it's non-functional), and to add more documentation about how to use the hub. The repo is hosted (publicly) on BitBucket under my account as yt.hub, so if you want to help out feel free to dive in. As a word of warning, as I fiddle with things and add new things, there may be interruptions in service. Let me know off-list if you think you see one that's long-lived or broken. Thanks, Matt

1 2

Fwd: [Sunrise] Sunrise v4.0 release!
by Matthew Turk May 9, 2012

May 9, 2012

This might be of interest to people here. Congratulations to Patrik and the other developers! This is a very impressive accomplishment. ---------- Forwarded message ---------- From: Patrik Jonsson <sunrise(a)familjenjonsson.org> Date: Wed, May 9, 2012 at 1:56 PM Subject: [Sunrise] Sunrise v4.0 release! To: sunrisemcrx(a)googlegroups.com Hi Sunrise users, I'm pleased to announce the release of the long-in-the-making Sunrise v4.0! This new version includes many updates, and a paper describing them is in the works. (However, it will likely be a while until this paper is done, for reasons that will become obvious below.) Special thanks should go to Chris Moody and Greg Snyder, who have done a lot of work testing the new versions and finding bugs. You can get the version now by using the tag "v4_00" in the Mercurial repository. One of the most exciting ones is the support for distributed-memory machines using MPI. This means that you no longer will have to go hunting for machines with extremely large memory to run Sunrise on large simulations. (However, the overhead for running MPI is quite high, so it is not useful for speeding up small calculations.) See the "Using MPI" Wiki page (http://code.google.com/p/sunrise/wiki/UsingMPI) for more information. The MPI functionality works but has not been extensively tested, so you are likely to run into bugs. Please report any issues that arise. A prerequisite for getting the MPI version to work was to redo the way the adaptive mesh works. The grid is now a pure octree, meaning it always starts with one root cell that is then recursively subdivided into octants. Before, the root grid was a cartesian grid of arbitrary size, but that is no longer supported. (This was necessary to get the Hilbert ordering of the cells used for domain decomposition working efficiently.) This new grid is also more efficient than the old version. The calculation of infrared emission has also been redone. When finding the equilibrium dust temperatures/radiation fields, it now uses the "immediate reemission" method described in Fleck & Cummings (1971) and Lucy (1999). This exactly preserves the energy balance when grains absorb/reemit photons, and discretizes only the temperature field (not the radiation intensity) on the grid when iterating, which should give better accuracy at a given grid resolution. Finally, instead of MC sampling the emission distribution in the cells when making images of dust emission, the direct emission is now calculated by explicitly integrating the RTE along the line of sight. This results in nose-free images. The scattered light is still done with MC, but at the wavelengths where the dust emission dominates, the albedo is essentially zero anyway so this doesn't make much difference. The integration is adaptive in the sense that the size of the cells encountered are monitored, and if the cells are smaller than the solid angle subtended by the ray, the ray is subdivided (similar to what Abel & Wandelt 99 did, but for incoming rays as opposed to outgoing ones). Furthermore, an explicit monitoring of Monte Carlo error in the iteration, which adjusts the number of rays to achieve the specified accuracy, has been included. The previous method of testing convergence did not guarantee an accurate temperature field, only that the overall energy balance was within a certain limit. The new method monitors the error in determining the temperature of each individual cell. The flipside is that if you ask for a high accuracy, you'd better be prepared to wait a *long* time. The full force of the sqrt(N) Monte Carlo convergence rate will hit you. The polychromatic ray tracing algorithm has been upgraded with an automatic determination of reference wavelength and forced scattering positions to avoid large bias factors in situations with high optical depths. Instead of splitting rays that have been boosted to high intensities after the fact, it avoids large bias factors in the first place by forcing several scatterings along the line of propagation. This lowers the noise caused by the large bias factors and avoids the guesswork in setting the reference wavelength and forced scattering parameters. There are also many efficiency tweaks that have sped up the calculation. (However, the most important factor affecting runtime, by far, is the dust equilibrium accuracy required.) The code is capable of processing snapshots from Gadget, Gasoline, Arepo, and, using YT to export files, from Enzo and ART. If you are interested in the YT functionality, contact Matt Turk and he can point you in the right direction. I don't know anything about how the YT export works. Finally, Sunrise now handles kinematics, meaning that it can take into account the effect of velocity and Doppler shift of sources and scatterers. However, for this to work you need a SED library with logarithmic wavelength spacing (and for it to be useful, quite high resolution). A high-resolution set of Starburst99 and MAPPINGS files have been uploaded to the Downloads section of the Sunrise web site. Note that with such high spectral resolution, maintaining high spatial resolution comes at the cost of *very* large memory use and output file size. It's best to cut out a small wavelength range around the features you are interested in. (And don't even think of calculating dust emission with that wavelength resolution...) For those of you who are interested in knowing more about what's happening inside the galaxy, there are also now options to save the dust emission SEDs and the dust temperatures to the output file. All these changes naturally means that parameter keywords have changed. I'll post another message containing example configurations, but I encourage you to carefully read through the Wiki pages describing the keywords to get an idea of the changes. As usual, you have to take responsibility for verifying that the results make sense, so don't just take the example files without actually verifying that things work correctly in your particular situation. The library requirements have also changed, so review the compilation instructions and be sure to update your copy of libPJutil from the repo at https://bitbucket.org/lutorm/libpjutil. The other big update is that I will no longer be working primarily on Sunrise. Next week, I'm moving to a position as software engineer at SpaceX in LA, working on simulation software for rockets! :-) I will still have my research grant for the next few years, but I'm unlikely to have much time to work on astrophysics. Chris Hayward, who has worked a lot with Sunrise, has graciously accepted taking over as the primary maintainer of the code. However, everyone will have to help Chris by pitching in with tracking down bugs you find, and with providing improvements. You are still welcome to email me with questions, but I should probably not be the first choice as my ping time is likely to be very long. I hope that all the effort I've put into Sunrise over the years will continue to be built on so Sunrise will remain the premier tool in galaxy SED prediction! Happy Monte-Carloing, /Patrik -- You received this message because you are subscribed to the Google Groups "Sunrise" group. To post to this group, send email to sunrisemcrx(a)googlegroups.com. To unsubscribe from this group, send email to sunrisemcrx+unsubscribe(a)googlegroups.com. For more options, visit this group at http://groups.google.com/group/sunrisemcrx?hl=en.

1 0

Fwd: [yt_analysis/yt] New ART frontend and new Sunrise export, expanded Rockstar (pull request #150)
by Matthew Turk May 4, 2012

May 4, 2012

Chris submitted a pretty sizable pull request -- congrats, Chris! I am going to review this, but I'd like to invite others to do so as well. ---------- Forwarded message ---------- From: Christopher Moody <pullrequests-noreply(a)bitbucket.org> Date: Thu, May 3, 2012 at 1:01 AM Subject: [yt_analysis/yt] New ART frontend and new Sunrise export, expanded Rockstar (pull request #150) To: matthewturk(a)gmail.com A new pull request has been opened by Christopher Moody. juxtaposicion/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/150/new-art-frontend-and-… Title: New ART frontend and new Sunrise export, expanded Rockstar ART Frontend now uses Hilbert-patch coalescing a la the RAMSES frontend. Particle support included. Sunrise export is completely revamped, works with the new MPI Sunrise and it's Hilbert-ordering convention. Expanded some Rockstar options and expanded the HaloList class so that we can plot 'HOP' circles for halos found via yt. Changes to be pulled: 096be56abb5c by Christopher Moo…: "added back Msun to data_structures" 60d0eae312ac by Christopher Moo…: "reverted a lot of the last commit" 33e1017fc58a by Christopher Moo…: "something is wrong with the ART frontend units" 1503e0ae9825 by Christopher Moo…: "hilbert frameworks works; wrong ordering though" db3a9b59d37c by Christopher Moo…: "merge" 42485223eca7 by Christopher Moo…: "fixes in rockstar halo classes. allowing ART frontend to be dm only" 41bf558abeaf by Christopher Moo…: "merged" 629ca6b1128f by Christopher Moo…: "implemented rockstar halo and halo list classes" 8512e594a837 by Christopher Moo…: "updated to tip" 6cc606a417a7 by Christopher Moo…: "more rockstar options. fixed ART particle masses" fecd8f0251c2 by Christopher Moo…: "removing pdb trace" 26ad4e98e6f7 by Christopher Moo…: "fixed particle positions to unitary units" 05d7dd36866a by Christopher Moo…: "fixes to ART particles" eb9e749e7253 by Christopher Moo…: "added the ability to pass DM particle mass directly" c309f591b93f by Christopher Moo…: "added Rockstar halo class" 91a9d9d9e7d4 by Christopher Moo…: " merged with tip" a3bc652df01e by Christopher Moo…: "changed le to le-1" f42fa77f5077 by Christopher Moo…: "changed LE for ART. works!" 053ae2cf5a3a by Christopher Moo…: "fixed temp field. commented out child mask." 0834402f6cde by Christopher Moo…: "reorganized the hydro fields" 549c408579ef by Christopher Moo…: " fixed a bug with particle arrays being the wrong length. affected stars." 4433494b6bba by Christopher Moo…: "implemented child masks. Broken! causes a segfault" 761a9e79cf93 by Matthew Turk: "Making regridding more strict and closer to what we want for the actual selectio…" d03f9f1da6f3 by Matthew Turk: "Adding a progressbar and a better parentage calculator to the ART frontend." 12223d9fef86 by Christopher Moo…: "particle field definitions" 69fee55a453e by Christopher Moo…: "added a few more options to patch splitting" a97c1ad2ec24 by Christopher Moo…: "fixed so that all fields are fully loaded in on preload. previously, only up to …" 210c6d907441 by Christopher Moo…: "particle fields can be accessed" 895621adae33 by Christopher Moo…: "added time conversion to data_structure" 858542457124 by Christopher Moo…: "particle fields still not working." 9ee69e4f02cd by Christopher Moo…: "added standard le, active dim attributes to art grids" b33d607ce6ec by Christopher Moo…: "particles read in and indices gridded. fields not setup yet." e3a1f76ae1c5 by Christopher Moo…: "forced 32 bit numpy arrays for IO-cython compatibility" 0fcc360c4d58 by Christopher Moo…: "first pass at particle inclusion. change recursiv splitting to start from the mi…" d202aaa72971 by Christopher Moo…: "including particle headers" 26eb767bac7f by Christopher Moo…: "bug fixes to io. added new parameters and units to data structures." 98534589dd52 by Christopher Moo…: "new io is less memory intensive. seems to work too." a1e85e8f0816 by Christopher Moo…: " reduced memory" dd43dbad7df5 by Christopher Moo…: " working version of art reader. faster than the first pass, uses matt's hilbert …" 4c7fcdbca996 by Christopher Moo…: "bug fixes in the info messages" b1cb776fed6f by Christopher Moo…: "adding log info. starting on memory optimization; loading left edges by level." d4d82426e66c by Christopher Moo…: "fixed a bug in the hilbert grad patching a bit. added lots of comments." ca8b7105e354 by Christopher Moo…: "implemented matt's hilbert-ordered patch-splitting" 8ae85483aad1 by Christopher Moo…: "fixing root grid size. projection of Ones field works." 41c966d5bcdc by Christopher Moo…: "record arrays overflow silently. changed the numgrid calculation to 64bit." 45c690fab939 by Christopher Moo…: "special exception for the root grid" -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

3 3

Fwd: [yt-svn] commit/yt: MatthewTurk: Updating mercurial to 2.2.
by Matthew Turk May 2, 2012

May 2, 2012

Hi all, I've updated hg to 2.2 in the install script. This brings with it "hg commit --amend", moves "hg glog" into core and comes with tons of performance improvements. More details: http://mercurial.selenic.com/wiki/WhatsNew#Mercurial_2.2_.282012-05-01.29 -Matt ---------- Forwarded message ---------- From: Bitbucket <commits-noreply(a)bitbucket.org> Date: Wed, May 2, 2012 at 8:17 AM Subject: [yt-svn] commit/yt: MatthewTurk: Updating mercurial to 2.2. See: To: yt-svn(a)lists.spacepope.org 1 new commit in yt: https://bitbucket.org/yt_analysis/yt/changeset/6eed6c70ed21/ changeset: 6eed6c70ed21 branch: yt user: MatthewTurk date: 2012-05-02 14:17:30 summary: Updating mercurial to 2.2. See: http://mercurial.selenic.com/wiki/WhatsNew#Mercurial_2.2_.282012-05-01.29 for more details. affected #: 1 file diff -r 3e1f218938d83de6a169837549eee92ec0e763a5 -r 6eed6c70ed21b25fed963d934705e159d9d4e054 doc/install_script.sh --- a/doc/install_script.sh +++ b/doc/install_script.sh @@ -366,7 +366,7 @@ echo 'ffc5c9e0c8c8ea66479abd467e442419bd1c867e6dbd180be6a032869467955dc570cfdf1388452871303a440738f302d3227ab7728878c4a114cfc45d29d23c ipython-0.12.tar.gz' > ipython-0.12.tar.gz.sha512 echo 'e748b66a379ee1e7963b045c3737670acf6aeeff1ebed679f427e74b642faa77404c2d5bbddb922339f009c229d0af1ae77cc43eab290e50af6157a6406d833f libpng-1.2.43.tar.gz' > libpng-1.2.43.tar.gz.sha512 echo 'f5ab95c29ef6958096970265a6079f0eb8c43a500924346c4a6c6eb89d9110eeeb6c34a53715e71240e82ded2b76a7b8d5a9b05a07baa000b2926718264ad8ff matplotlib-1.1.0.tar.gz' > matplotlib-1.1.0.tar.gz.sha512 -echo '78715bb2bd7ed3291089948530a59d5eff146a64179eae87904a2c328716f26749abb0c5417d6001cadfeebabb4e24985d5a59ceaae4d98c4762163970f83975 mercurial-2.0.tar.gz' > mercurial-2.0.tar.gz.sha512 +echo '702f67c48e4dbe191dbe5ca0df6b5a84fa4f5c424cf1fae60b5053dfe6532531330738c7aa3012d900d49efdd743cd1ebc238bb15f354f67228e2a2c95b98a89 mercurial-2.2.tar.gz' > mercurial-2.2.tar.gz.sha512 echo 'de3dd37f753614055dcfed910e9886e03688b8078492df3da94b1ec37be796030be93291cba09e8212fffd3e0a63b086902c3c25a996cf1439e15c5b16e014d9 numpy-1.6.1.tar.gz' > numpy-1.6.1.tar.gz.sha512 echo '5ad681f99e75849a5ca6f439c7a19bb51abc73d121b50f4f8e4c0da42891950f30407f761a53f0fe51b370b1dbd4c4f5a480557cb2444c8c7c7d5412b328a474 sqlite-autoconf-3070500.tar.gz' > sqlite-autoconf-3070500.tar.gz.sha512 echo 'edae735960279d92acf58e1f4095c6392a7c2059b8f1d2c46648fc608a0fb06b392db2d073f4973f5762c034ea66596e769b95b3d26ad963a086b9b2d09825f2 zlib-1.2.3.tar.bz2' > zlib-1.2.3.tar.bz2.sha512 @@ -388,7 +388,7 @@ get_enzotools Python-2.7.2.tgz get_enzotools numpy-1.6.1.tar.gz get_enzotools matplotlib-1.1.0.tar.gz -get_enzotools mercurial-2.0.tar.gz +get_enzotools mercurial-2.2.tar.gz get_enzotools ipython-0.12.tar.gz get_enzotools h5py-2.0.1.tar.gz get_enzotools Cython-0.15.1.tar.gz @@ -531,7 +531,7 @@ if [ $INST_HG -eq 1 ] then echo "Installing Mercurial." - do_setup_py mercurial-2.0 + do_setup_py mercurial-2.2 export HG_EXEC=${DEST_DIR}/bin/hg else # We assume that hg can be found in the path. Repository URL: https://bitbucket.org/yt_analysis/yt/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email. _______________________________________________ yt-svn mailing list yt-svn(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-svn-spacepope.org

1 0

Fwd: [yt_analysis/yt] Raise an exception instead of returning None when load fails. (pull request #149)
by Matthew Turk May 1, 2012

May 1, 2012

This has been discussed a lot in the past. ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Tue, May 1, 2012 at 3:03 PM Subject: [yt_analysis/yt] Raise an exception instead of returning None when load fails. (pull request #149) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/149/raise-an-exception-in… Title: Raise an exception instead of returning None when load fails. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

1 0