Mailman 3 August 2012 - yt-dev

na => np in 3.0 and 2.x
by Matthew Turk 10 Sep '12

10 Sep '12

Hi all, On Friday, Anthony submitted a PR to the yt-3.0 branch: https://bitbucket.org/yt_analysis/yt-3.0/pull-request/4/na-is-dead-long-liv… This PR is pretty invasive, and done by regular expressions. Basically, for a long time we've been sticking with a convention I started using about ... six years ago ... when NumArray was the dominant array language. (Or when I was still too removed from Python's scientific community to see otherwise.) The array library was shortened to 'na'. Almost immediately after, NumPy took over and while we switched to NumPy we never updated the shorthand in the Python code to 'np'. (The Cython code always uses 'np') Most python tutorials use np instead of numpy, and I'd like to encourage best practices in yt as well as suggest we try to fit into the broader ecosystem of packages. Anyway, this is kind of a bandaid that needs to be ripped off at some point, and I think it's appropriate to discuss now. I see three options, which basically break down on levels of disruption and ease of the dual-lines of development we currently have. 1) Put this PR (which applies only to 3.0) on hold. This way, merging from 2.x to 3.0 can proceed easily, and the disruption is completely pushed off for a bit. 2) Accept the PR. This increases the burden on me for merging considerably, and it would fall on me. Any file where both a numpy line and another line are changed would throw a conflict that I'd have to manually resolve. But, because it would just be in the 3.0 line, disruption would be kept to a minimum. 3) Accept the PR *but* also mandate that we apply it to the main repository's development branch (2.x). This would be the most disruptive, but it would also keep merging difficulty to a minimum. As a compatibility layer, we'll keep "na" in the yt.mods namespace, which means my_plugins.py files would still work, as would existing scripts. Because this could be disruptive for any major, outstanding forks, I also think it needs to be discussed here. (I'm actually kind of -1 on big discussions happening in pull requests.) My vote is for #3. I'd rather get this over with, since we all know it probably ought to happen at some point in the future. What do people think -- of the three options, which is your favorite, and do you have strong feelings against any one option? -Matt

9 20

Coordinate handling pull request
by Matthew Turk 31 Aug '12

31 Aug '12

Hi all, I've issued a pull request to the 3.0 repository, as I think it warrants discussion. It's here: https://bitbucket.org/yt_analysis/yt-3.0/pull-request/5/initial-import-of-c… This includes a first pass at a coordinate handling system. This is distinct from a geometry handling system; the coordinates here refer to how we handle coordinates and spatial locations internally, whereas geometry refers to how data is distributed throughout a domain and throughout places on disk. For instance, coordinate handling would be cartesian, polar, spherical. The reason I'm bringing it up for discussion is that I believe we want to move as much coordinate handling and transformation into a separate, well-defined class as possible. Periodicity, distances and so on are all currently scattered throughout the code, and I'd like to try to consolidate them. Additionally, as new coordinate systems (polar, spherical) are added, we'll need clear ways to delegate responsibility for things like "How do I calculate path length as I integrate?" or "What's the way to turn this into an image?" I believe the best way to do that is to attach a coordinate system to the dataset object itself. (We now have a polar pixelizer http://i.imgur.com/a4UGg.png !) The interface is currently set such that you need to define these methods and properties in order to implement a coordinate system: coordinate_fields (this may go away, but it's for the analogs of 'x', 'y', 'z', as well as volume) pixelize convert_from_cartesian convert_to_cartesian axis_name axis_id x_axis y_axis period Some of these currently live in dictionaries in yt/utilities/definitions.py, which is pretty sub-optimal. I'd like to ask for feedback: 1) Do these methods sufficiently cover everything we need to know in yt about a coordinate system? Should any be added? 2) Do we need to directly address dimensionality as a separate subclass? 3) Should any of these be removed? This will also help address these issues: https://bitbucket.org/yt_analysis/yt/issue/418/use-a-right-handed-coordinat… https://bitbucket.org/yt_analysis/yt/issue/422/ray-casting-in-cylindrical-c… https://bitbucket.org/yt_analysis/yt/issue/421/refactor-non-cartesian-geome… https://bitbucket.org/yt_analysis/yt/issue/345/non-cartesian-geometry https://bitbucket.org/yt_analysis/yt/issue/205/periodicity -Matt

3 8

removing empty bins from radial profiles
by Wolfram Schmidt 31 Aug '12

31 Aug '12

Hi all, I am calculating radial profiles with BinnedProfile1D, where the profile values happen to be zero in a few bins. This causes interruptions in the graphs when I plot the the profiles with pylab.loglog. I don't want to make the bins broader. So I would like to get rid of the empty bins (i.e., bins with zero values) by removing them from the profile lists or somehow tell the loglog function of pylab to ignore all zero values. Unfortunately, I have found a solution either way. Can anyone help? Wolfram

3 2

Fwd: [yt_analysis/yt] Removing freetype routines and the freetype dependency. (pull request #256)
by Matthew Turk 31 Aug '12

31 Aug '12

Recently it's been brought up that the freetype2 dependency is not used or advertised in yt, and it adds a good chunk of time to the install script and dependencies too. This removes that dependency in the install script and the build-time need for it. As noted in the PR itself, I'm a bit uncertain if this will cause problems, as the freetype we installed also acted to meet the dependency in matplotlib. I am fine with reverting this change if need be, to ensure smooth sailing for installing matplotlib. Could a handful of people test this out? Thanks, Matt ---------- Forwarded message ---------- From: Matthew Turk <pullrequests-noreply(a)bitbucket.org> Date: Fri, Aug 31, 2012 at 7:49 AM Subject: [yt_analysis/yt] Removing freetype routines and the freetype dependency. (pull request #256) To: yt(a)enzotools.org A new pull request has been opened by Matthew Turk. MatthewTurk/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/256/removing-freetype-rou… Title: Removing freetype routines and the freetype dependency. In an effort to slim down the installation size, as well as remove unnecessary routines, this PR removed FreeType2 and the routines that use it directly. I have two concerns: 1) This means an additional system-level package has to be installed for Matplotlib, which may not universally be the case. 2) There may be a handful of people out there who use this on a regular basis. Changes to be pulled: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

1 0

A sad day for our community. John Hunter: 1968-2012.
by Fernando Perez 29 Aug '12

29 Aug '12

Dear friends and colleagues, [please excuse a possible double-post of this message, in-flight internet glitches] I am terribly saddened to report that yesterday, August 28 2012 at 10am, John D. Hunter died from complications arising from cancer treatment at the University of Chicago hospital, after a brief but intense battle with this terrible illness. John is survived by his wife Miriam, his three daughters Rahel, Ava and Clara, his sisters Layne and Mary, and his mother Sarah. Note: If you decide not to read any further (I know this is a long message), please go to this page for some important information about how you can thank John for everything he gave in a decade of generous contributions to the Python and scientific communities: http://numfocus.org/johnhunter. Just a few weeks ago, John delivered his keynote address at the SciPy 2012 conference in Austin centered around the evolution of matplotlib: http://www.youtube.com/watch?v=e3lTby5RI54 but tragically, shortly after his return home he was diagnosed with advanced colon cancer. This diagnosis was a terrible discovery to us all, but John took it with his usual combination of calm and resolve, and initiated treatment procedures. Unfortunately, the first round of chemotherapy treatments led to severe complications that sent him to the intensive care unit, and despite the best efforts of the University of Chicago medical center staff, he never fully recovered from these. Yesterday morning, he died peacefully at the hospital with his loved ones at his bedside. John fought with grace and courage, enduring every necessary procedure with a smile on his face and a kind word for all of his caretakers and becoming a loved patient of the many teams that ended up involved with his case. This was no surprise for those of us who knew him, but he clearly left a deep and lasting mark even amongst staff hardened by the rigors of oncology floors and intensive care units. I don't need to explain to this community the impact of John's work, but allow me to briefly recap, in case this is read by some who don't know the whole story. In 2002, John was a postdoc at the University of Chicago hospital working on the analysis of epilepsy seizure data in children. Frustrated with the state of the existing proprietary solutions for this class of problems, he started using Python for his work, back when the scientific Python ecosystem was much, much smaller than it is today and this could have been seen as a crazy risk. Furthermore, he found that there were many half-baked solutions for data visualization in Python at the time, but none that truly met his needs. Undeterred, he went on to create matplotlib (http://matplotlib.org) and thus overcome one of the key obstacles for Python to become the best solution for open source scientific and technical computing. Matplotlib is both an amazing technical achievement and a shining example of open source community building, as John not only created its backbone but also fostered the development of a very strong development team, ensuring that the talent of many others could also contribute to this project. The value and importance of this are now painfully clear: despite having lost John, matplotlib continues to thrive thanks to the leadership of Michael Droetboom, the support of Perry Greenfield at the Hubble Telescope Science Institute, and the daily work of the rest of the team. I want to thank Perry and Michael for putting their resources and talent once more behind matplotlib, securing the future of the project. It is difficult to overstate the value and importance of matplotlib, and therefore of John's contributions (which do not end in matplotlib, by the way; but a biography will have to wait for another day...). Python has become a major force in the technical and scientific computing world, leading the open source offers and challenging expensive proprietary platforms with large teams and millions of dollars of resources behind them. But this would be impossible without a solid data visualization tool that would allow both ad-hoc data exploration and the production of complex, fine-tuned figures for papers, reports or websites. John had the vision to make matplotlib easy to use, but powerful and flexible enough to work in graphical user interfaces and as a server-side library, enabling a myriad use cases beyond his personal needs. This means that now, matplotlib powers everything from plots in dissertations and journal articles to custom data analysis projects and websites. And despite having left his academic career a few years ago for a job in industry, he remained engaged enough that as of today, he is still the top committer to matplotlib; this is the git shortlog of those with more than 1000 commits to the project: 2145 John Hunter <jdh2358(a)gmail.com> 2130 Michael Droettboom <mdboom(a)gmail.com> 1060 Eric Firing <efiring(a)hawaii.edu> All of this was done by a man who had three children to raise and who still always found the time to help those on the mailing lists, solve difficult technical problems in matplotlib, teach courses and seminars about scientific Python, and more recently help create the NumFOCUS foundation project. Despite the challenges that raising three children in an expensive city like Chicago presented, he never once wavered from his commitment to open source. But unfortunately now he is not here anymore to continue providing for their well-being, and I hope that all those who have so far benefited from his generosity, will thank this wonderful man who always gave far more than he received. Thanks to the rapid action of Travis Oliphant, the NumFOCUS foundation is now acting as an escrow agent to accept donations that will go into a fund to support the education and care of his wonderful girls Rahel, Ava and Clara. If you have benefited from John's many contributions, please say thanks in the way that would matter most to him, by helping Miriam continue the task of caring for and educating Rahel, Ava and Clara. You will find all the information necessary to make a donation here: http://numfocus.org/johnhunter Remember that even a small donation helps! If all those who ever use matplotlib give just a little bit, in the long run I am sure that we can make a difference. If you are a company that benefits in a serious way from matplotlib, remember that John was a staunch advocate of keeping all scientific Python projects under the BSD license so that commercial users could benefit from them without worry. Please say thanks to John in a way commensurate with your resources (and check how much a yearly matlab license would cost you in case you have any doubts about the value you are getting...). John's family is planning a private burial in Tennessee, but (most likely in September) there will also be a memorial service in Chicago that friends and members of the community can attend. We don't have the final scheduling details at this point, but I will post them once we know. I would like to again express my gratitude to Travis Oliphant for moving quickly with the setup of the donation support, and to Eric Jones (the founder of Enthought and another one of the central figures in our community) who immediately upon learning of John's plight contributed resources to support the family with everyday logistics while John was facing treatment as well as my travel to Chicago to assist. This kind of immediate urge to come to the help of others that Eric and Travis displayed is a hallmark of our community. Before closing, I want to take a moment to publicly thank the incredible staff of the University of Chicago medical center. The last two weeks were an intense and brutal ordeal for John and his loved ones, but the hospital staff offered a sometimes hard to believe, unending supply of generosity, care and humanity in addition to their technical competence. The latter is something we expect from a first-rate hospital at a top university, where the attending physicians can be world-renowned specialists in their field. But the former is often forgotten in a world often ruled by a combination of science and concerns about regulations and liability. Instead, we found generous and tireless staff who did everything in their power to ease the pain, always putting our well being ahead of any mindless adherence to protocol, patiently tending to every need we had and working far beyond their stated responsibilities to support us. To name only one person (and many others are equally deserving), I want to thank Dr. Carla Moreira, chief surgical resident, who spent the last few hours of John's life with us despite having just completed a solid night shift of surgical work. Instead of resting she came to the ICU and worked to ensure that those last hours were as comfortable as possible for John; her generous actions helped us through a very difficult moment. It is now time to close this already too long message... John, thanks for everything you gave all of us, and for the privilege of knowing you. Fernando. ps - I have sent this with my 'mailing lists' email. If you need to contact me directly for anything regarding the above, please write to my regular address at Fernando.Perez(a)berkeley.edu, where I do my best to reply more promptly.

1 0

A sad day for our community. John Hunter: 1968-2012.
by Fernando Perez 29 Aug '12

29 Aug '12

Dear friends and colleagues, I am terribly saddened to report that yesterday, August 28 2012 at 10am, John D. Hunter died from complications arising from cancer treatment at the University of Chicago hospital, after a brief but intense battle with this terrible illness. John is survived by his wife Miriam, his three daughters Rahel, Ava and Clara, his sisters Layne and Mary, and his mother Sarah. Note: If you decide not to read any further (I know this is a long message), please go to this page for some important information about how you can thank John for everything he gave in a decade of generous contributions to the Python and scientific communities: http://numfocus.org/johnhunter. Just a few weeks ago, John delivered his keynote address at the SciPy 2012 conference in Austin centered around the evolution of matplotlib: http://www.youtube.com/watch?v=e3lTby5RI54 but tragically, shortly after his return home he was diagnosed with advanced colon cancer. This diagnosis was a terrible discovery to us all, but John took it with his usual combination of calm and resolve, and initiated treatment procedures. Unfortunately, the first round of chemotherapy treatments led to severe complications that sent him to the intensive care unit, and despite the best efforts of the University of Chicago medical center staff, he never fully recovered from these. Yesterday morning, he died peacefully at the hospital with his loved ones at his bedside. John fought with grace and courage, enduring every necessary procedure with a smile on his face and a kind word for all of his caretakers and becoming a loved patient of the many teams that ended up involved with his case. This was no surprise for those of us who knew him, but he clearly left a deep and lasting mark even amongst staff hardened by the rigors of oncology floors and intensive care units. I don't need to explain to this community the impact of John's work, but allow me to briefly recap, in case this is read by some who don't know the whole story. In 2002, John was a postdoc at the University of Chicago hospital working on the analysis of epilepsy seizure data in children. Frustrated with the state of the existing proprietary solutions for this class of problems, he started using Python for his work, back when the scientific Python ecosystem was much, much smaller than it is today and this could have been seen as a crazy risk. Furthermore, he found that there were many half-baked solutions for data visualization in Python at the time, but none that truly met his needs. Undeterred, he went on to create matplotlib (http://matplotlib.org) and thus overcome one of the key obstacles for Python to become the best solution for open source scientific and technical computing. Matplotlib is both an amazing technical achievement and a shining example of open source community building, as John not only created its backbone but also fostered the development of a very strong development team, ensuring that the talent of many others could also contribute to this project. The value and importance of this are now painfully clear: despite having lost John, matplotlib continues to thrive thanks to the leadership of Michael Droetboom, the support of Perry Greenfield at the Hubble Telescope Science Institute, and the daily work of the rest of the team. I want to thank Perry and Michael for putting their resources and talent once more behind matplotlib, securing the future of the project. It is difficult to overstate the value and importance of matplotlib, and therefore of John's contributions (which do not end in matplotlib, by the way; but a biography will have to wait for another day...). Python has become a major force in the technical and scientific computing world, leading the open source offers and challenging expensive proprietary platforms with large teams and millions of dollars of resources behind them. But this would be impossible without a solid data visualization tool that would allow both ad-hoc data exploration and the production of complex, fine-tuned figures for papers, reports or websites. John had the vision to make matplotlib easy to use, but powerful and flexible enough to work in graphical user interfaces and as a server-side library, enabling a myriad use cases beyond his personal needs. This means that now, matplotlib powers everything from plots in dissertations and journal articles to custom data analysis projects and websites. And despite having left his academic career a few years ago for a job in industry, he remained engaged enough that as of today, he is still the top committer to matplotlib; this is the git shortlog of those with more than 1000 commits to the project: 2145 John Hunter <jdh2358(a)gmail.com> 2130 Michael Droettboom <mdboom(a)gmail.com> 1060 Eric Firing <efiring(a)hawaii.edu> All of this was done by a man who had three children to raise and who still always found the time to help those on the mailing lists, solve difficult technical problems in matplotlib, teach courses and seminars about scientific Python, and more recently help create the NumFOCUS foundation project. Despite the challenges that raising three children in an expensive city like Chicago presented, he never once wavered from his commitment to open source. But unfortunately now he is not here anymore to continue providing for their well-being, and I hope that all those who have so far benefited from his generosity, will thank this wonderful man who always gave far more than he received. Thanks to the rapid action of Travis Oliphant, the NumFOCUS foundation is now acting as an escrow agent to accept donations that will go into a fund to support the education and care of his wonderful girls Rahel, Ava and Clara. If you have benefited from John's many contributions, please say thanks in the way that would matter most to him, by helping Miriam continue the task of caring for and educating Rahel, Ava and Clara. You will find all the information necessary to make a donation here: http://numfocus.org/johnhunter Remember that even a small donation helps! If all those who ever use matplotlib give just a little bit, in the long run I am sure that we can make a difference. If you are a company that benefits in a serious way from matplotlib, remember that John was a staunch advocate of keeping all scientific Python projects under the BSD license so that commercial users could benefit from them without worry. Please say thanks to John in a way commensurate with your resources (and check how much a yearly matlab license would cost you in case you have any doubts about the value you are getting...). John's family is planning a private burial in Tennessee, but (most likely in September) there will also be a memorial service in Chicago that friends and members of the community can attend. We don't have the final scheduling details at this point, but I will post them once we know. I would like to again express my gratitude to Travis Oliphant for moving quickly with the setup of the donation support, and to Eric Jones (the founder of Enthought and another one of the central figures in our community) who immediately upon learning of John's plight contributed resources to support the family with everyday logistics while John was facing treatment as well as my travel to Chicago to assist. This kind of immediate urge to come to the help of others that Eric and Travis displayed is a hallmark of our community. Before closing, I want to take a moment to publicly thank the incredible staff of the University of Chicago medical center. The last two weeks were an intense and brutal ordeal for John and his loved ones, but the hospital staff offered a sometimes hard to believe, unending supply of generosity, care and humanity in addition to their technical competence. The latter is something we expect from a first-rate hospital at a top university, where the attending physicians can be world-renowned specialists in their field. But the former is often forgotten in a world often ruled by a combination of science and concerns about regulations and liability. Instead, we found generous and tireless staff who did everything in their power to ease the pain, always putting our well being ahead of any mindless adherence to protocol, patiently tending to every need we had and working far beyond their stated responsibilities to support us. To name only one person (and many others are equally deserving), I want to thank Dr. Carla Moreira, chief surgical resident, who spent the last few hours of John's life with us despite having just completed a solid night shift of surgical work. Instead of resting she came to the ICU and worked to ensure that those last hours were as comfortable as possible for John; her generous actions helped us through a very difficult moment. It is now time to close this already too long message... John, thanks for everything you gave all of us, and for the privilege of knowing you. Fernando. ps - I have sent this with my 'mailing lists' email. If you need to contact me directly for anything regarding the above, please write to my regular address at Fernando.Perez(a)berkeley.edu, where I do my best to reply more promptly.

1 0

3.0 planning
by Matthew Turk 29 Aug '12

29 Aug '12

Hi all, Yesterday in IRC Kacper and I chatted a bit about planning, particularly as it applies to development of yt 3.0. Planning in this case serves two purposes -- to make others aware of who is doing what, and to solicit feedback on design decisions that might affect people. After going back and forth, it seems like a combination of wiki pages and issues would be good places to have planning documents and ideas stored. Nathan also brought up in IRC that while many people may be interested in what's going on, it seems as though the changes in 3.0 are probably too invasive or not ready yet for general use. That's sort of true, but things are stabilizing. I'm not going to include a status update here, but I will once it has settled down a bit. I've created two wiki pages: * https://bitbucket.org/yt_analysis/yt/wiki/GoalsFor30 (almost verbatim from the google doc we collaboratively created a few months ago) * https://bitbucket.org/yt_analysis/yt/wiki/PlanFor30 (an update from my earlier email, including a somewhat revised timeline) I'd really like to request that if you would like to update these pages, that you do so. You can use either the web interface or clone and edit. The wiki could also be a good place to record decisions made here on the mailing list about design aspects -- as an example, the other thread about coordinate systems. If somebody wants to take on fixing up the wiki as a project, that would be awesome as well. -Matt

1 0

"Cards" interface
by Matthew Turk 23 Aug '12

23 Aug '12

Hi all, The Bitbucket issue tracker is not the best, but this new "Cards" interface (mimicking Trello) is a bit nicer: http://www.bitbucketcards.com/yt_analysis/yt You can log in with OAuth. Moving cards between columns changes their state, and each card is an issue. I think it would be beneficial if we tracked more ideas and projects in an open space like the issue tracker, so I'd encourage you to experiment and explore with this. -Matt

1 0

Orientation vectors
by John ZuHone 23 Aug '12

23 Aug '12

Hi all, and specifically to those of you with experience with the Orientation vectors... Doing some testing, I set up an orienter like this: orient = Orientation([0.0,0.0,1.0]) and the output of orient.unit_vectors is: [array([ 0., -1., 0.]), array([ 1., 0., -0.]), array([ 0., 0., 1.])] which I would have expected [array([ 1., 0., 0.]), array([ 0., 1., 0.]), array([ 0., 0., 1.])] although I know I could fix this by specifying a north_vector, why would the latter not be the default? I'm asking because I was attempting to set up new coordinates like so: orient = Orientation(L) x_hat = orient.unit_vectors[0] y_hat = orient.unit_vectors[1] z_hat = orient.unit_vectors[2] xsky = x*x_hat[0] + y*x_hat[1] + z*x_hat[2] ysky = x*y_hat[0] + y*y_hat[1] + z*y_hat[2] zsky = x*z_hat[0] + y*z_hat[1] + z*z_hat[2] but this will rotate the x,y plane by 90 degrees. Best, John

2 4

Meeting about yt 3.0
by Matthew Turk 22 Aug '12

22 Aug '12

Hi all, I think the 3.0 branch is nearing readiness in most core functionality for Enzo, FLASH and RAMSES, but not yet in some of the higher profile stuff, like volume rendering or finite stencils. I am still still working to convert the Boxlib codes over, because I think I'd also like to address some IO issues that were recently mentioned to me, and to consolidate what we can in there. I'd like to propose that sometime this week, we have a hangout to talk through the current state of 3.0 and 2.x, and to figure out the best way to move forward as a team. I've set up a doodle here -- if there's strong opposition to these times, we can attempt to have it on another day or week. http://doodle.com/y5msnkhc8nycdruf Here's a rough agenda: 1) Revisiting goals for 3.0: what did we decide on at the last meeting, have there been any changes? 2) Status of milestones, what other milestones are there? 3) What would it take to move into the main development branch? 4) What would it take to move developers to 3.0? 5) Open discussion: developer meeting?, handling transition, other suggestions, gripes, etc. Hope to see you soon, Matt

1 1