Mailman 3 August 2012 - NumPy-Discussion

ANN: NumPy 1.7.0b1 release
by Ondřej Čertík 09 Sep '12

09 Sep '12

Hi, I'm pleased to announce the availability of the first beta release of NumPy 1.7.0b1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.0b1/ Please test this release and report any issues on the numpy-discussion mailing list. The following problems are known and we'll work on fixing them before the final release: http://projects.scipy.org/numpy/ticket/2187 http://projects.scipy.org/numpy/ticket/2185 http://projects.scipy.org/numpy/ticket/2066 http://projects.scipy.org/numpy/ticket/1588 http://projects.scipy.org/numpy/ticket/2076 http://projects.scipy.org/numpy/ticket/2101 http://projects.scipy.org/numpy/ticket/2108 http://projects.scipy.org/numpy/ticket/2150 http://projects.scipy.org/numpy/ticket/2189 I would like to thank Ralf for a lot of help with creating binaries and other help for this release. Cheers, Ondrej ========================= NumPy 1.7.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and refactorings. It supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * ``where=`` parameter to ufuncs (allows the use of boolean arrays to choose where a computation should be done) * ``vectorize`` improvements (added 'excluded' and 'cache' keyword, general cleanup and bug fixes) * ``numpy.random.choice`` (random sample generating function) Compatibility notes =================== In a future version of numpy, the functions np.diag, np.diagonal, and the diagonal method of ndarrays will return a view onto the original array, instead of producing a copy as they do now. This makes a difference if you write to the array returned by any of these functions. To facilitate this transition, numpy 1.7 produces a FutureWarning if it detects that you may be attempting to write to such an array. See the documentation for np.diagonal for details. Similar to np.diagonal above, in a future version of numpy, indexing a record array by a list of field names will return a view onto the original array, instead of producing a copy as they do now. As with np.diagonal, numpy 1.7 produces a FutureWarning if it detects that you may be attemping to write to such an array. See the documentation for array indexing for details. The default casting rule for UFunc out= parameters has been changed from 'unsafe' to 'same_kind'. Most usages which violate the 'same_kind' rule are likely bugs, so this change may expose previously undetected errors in projects that depend on NumPy. Full-array boolean indexing has been optimized to use a different, optimized code path. This code path should produce the same results, but any feedback about changes to your code would be appreciated. Attempting to write to a read-only array (one with ``arr.flags.writeable`` set to ``False``) used to raise either a RuntimeError, ValueError, or TypeError inconsistently, depending on which code path was taken. It now consistently raises a ValueError. The <ufunc>.reduce functions evaluate some reductions in a different order than in previous versions of NumPy, generally providing higher performance. Because of the nature of floating-point arithmetic, this may subtly change some results, just as linking NumPy to a different BLAS implementations such as MKL can. If upgrading from 1.5, then generally in 1.6 and 1.7 there have been substantial code added and some code paths altered, particularly in the areas of type resolution and buffered iteration over universal functions. This might have an impact on your code particularly if you relied on accidental behavior in the past. New features ============ Reduction UFuncs Generalize axis= Parameter ------------------------------------------- Any ufunc.reduce function call, as well as other reductions like sum, prod, any, all, max and min support the ability to choose a subset of the axes to reduce over. Previously, one could say axis=None to mean all the axes or axis=# to pick a single axis. Now, one can also say axis=(#,#) to pick a list of axes for reduction. Reduction UFuncs New keepdims= Parameter ---------------------------------------- There is a new keepdims= parameter, which if set to True, doesn't throw away the reduction axes but instead sets them to have size one. When this option is set, the reduction result will broadcast correctly to the original operand which was reduced. Datetime support ---------------- .. note:: The datetime API is *experimental* in 1.7.0, and may undergo changes in future versions of NumPy. There have been a lot of fixes and enhancements to datetime64 compared to NumPy 1.6: * the parser is quite strict about only accepting ISO 8601 dates, with a few convenience extensions * converts between units correctly * datetime arithmetic works correctly * business day functionality (allows the datetime to be used in contexts where only certain days of the week are valid) The notes in `doc/source/reference/arrays.datetime.rst <https://github.com/numpy/numpy/blob/maintenance/1.7.x/doc/source/reference/…>`_ (also available in the online docs at `arrays.datetime.html <http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html>`_) should be consulted for more details. Custom formatter for printing arrays ------------------------------------ See the new ``formatter`` parameter of the ``numpy.set_printoptions`` function. New function numpy.random.choice --------------------------------- A generic sampling function has been added which will generate samples from a given array-like. The samples can be with or without replacement, and with uniform or given non-uniform probabilities. New function isclose -------------------- Returns a boolean array where two arrays are element-wise equal within a tolerance. Both relative and absolute tolerance can be specified. The function is NA aware. Preliminary multi-dimensional support in the polynomial package --------------------------------------------------------------- Axis keywords have been added to the integration and differentiation functions and a tensor keyword was added to the evaluation functions. These additions allow multi-dimensional coefficient arrays to be used in those functions. New functions for evaluating 2-D and 3-D coefficient arrays on grids or sets of points were added together with 2-D and 3-D pseudo-Vandermonde matrices that can be used for fitting. Ability to pad rank-n arrays ---------------------------- A pad module containing functions for padding n-dimensional arrays has been added. The various private padding functions are exposed as options to a public 'pad' function. Example:: pad(a, 5, mode='mean') Current modes are ``constant``, ``edge``, ``linear_ramp``, ``maximum``, ``mean``, ``median``, ``minimum``, ``reflect``, ``symmetric``, ``wrap``, and ``<function>``. New argument to searchsorted ---------------------------- The function searchsorted now accepts a 'sorter' argument that is a permuation array that sorts the array to search. C API ----- New function ``PyArray_RequireWriteable`` provides a consistent interface for checking array writeability -- any C code which works with arrays whose WRITEABLE flag is not known to be True a priori, should make sure to call this function before writing. NumPy C Style Guide added (``doc/C_STYLE_GUIDE.rst.txt``). Changes ======= General ------- The function np.concatenate tries to match the layout of its input arrays. Previously, the layout did not follow any particular reason, and depended in an undesirable way on the particular axis chosen for concatenation. A bug was also fixed which silently allowed out of bounds axis arguments. The ufuncs logical_or, logical_and, and logical_not now follow Python's behavior with object arrays, instead of trying to call methods on the objects. For example the expression (3 and 'test') produces the string 'test', and now np.logical_and(np.array(3, 'O'), np.array('test', 'O')) produces 'test' as well. Deprecations ============ General ------- Specifying a custom string formatter with a `_format` array attribute is deprecated. The new `formatter` keyword in ``numpy.set_printoptions`` or ``numpy.array2string`` can be used instead. The deprecated imports in the polynomial package have been removed. C-API ----- Direct access to the fields of PyArrayObject* has been deprecated. Direct access has been recommended against for many releases. Expect similar deprecations for PyArray_Descr* and other core objects in the future as preparation for NumPy 2.0. The macros in old_defines.h are deprecated and will be removed in the next major release (>= 2.0). The sed script tools/replace_old_macros.sed can be used to replace these macros with the newer versions. You can test your code against the deprecated C API by #defining NPY_NO_DEPRECATED_API to the target version number, for example NPY_1_7_API_VERSION, before including any NumPy headers.

12 21

Debian/Ubuntu patch help (was: ANN: NumPy 1.6.2 release candidate 1)
by Ralf Gommers 08 Sep '12

08 Sep '12

On Sat, May 12, 2012 at 9:17 PM, Ralf Gommers <ralf.gommers(a)googlemail.com>wrote: > > > On Sat, May 12, 2012 at 6:22 PM, Sandro Tosi <matrixhasu(a)gmail.com> wrote: > >> Hello, >> >> On Sat, May 5, 2012 at 8:15 PM, Ralf Gommers >> <ralf.gommers(a)googlemail.com> wrote: >> > Hi, >> > >> > I'm pleased to announce the availability of the first release candidate >> of >> > NumPy 1.6.2. This is a maintenance release. Due to the delay of the >> NumPy >> > 1.7.0, this release contains far more fixes than a regular NumPy bugfix >> > release. It also includes a number of documentation and build >> improvements. >> > >> > Sources and binary installers can be found at >> > https://sourceforge.net/projects/numpy/files/NumPy/1.6.2rc1/ >> > >> > Please test this release and report any issues on the numpy-discussion >> > mailing list. >> ... >> > BLD: add support for the new X11 directory structure on Ubuntu & co. >> >> We've just discovered that this fix is not enough. Actually the new >> directories are due to the "multi-arch" feature of Debian systems, >> that allows to install libraries from other (foreign) architectures >> than the one the machine is (the classic example, i386 libraries on a >> amd64 host). >> >> the fix included to look up in additional directories is currently >> only for X11, while for example Debian has fftw3 that's >> multi-arch-ified and thus will fail to be detected. >> >> Could this fix be extended to include all other things that are >> checked? for reference the bug in Debian is [1]; there was also a >> patch[2] in previous versions, that was using gcc to get the >> multi-arch paths - you might use as a reference, or to implement >> something debian-systems-specific. >> >> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640940 >> [2] >> http://anonscm.debian.org/viewvc/python-modules/packages/numpy/trunk/debian… >> >> It would be awesome is such support would end up in 1.6.2 . >> > > Hardcoding some more paths to check in distutils/system_info.py should be > OK, also for 1.6.2 (will require a new RC). > > The --print-multiarch thing looks very questionable. As far as I can tell, > it's a Debian specific gcc patch, only available in gcc 4.6 and up. Ubuntu > before 11.10 release also doesn't have it. Therefore I don't think use of > --print-multiarch is appropriate for numpy for now, and certainly not a > change I'd like to make to distutils right before a release. > > If anyone with access to a Debian/Ubuntu system could come up with a patch > which adds the right paths to system_info.py, that would be great. > Hi, if there's anyone wants to have a look at the above issue this week, that would be great. If there's a patch by this weekend I can create a second RC, so we can still have the final release before the end of this month (needed for Debian freeze). Otherwise a second RC won't be needed. Ralf

3 2

How to debug reference counting errors
by Ondřej Čertík 01 Sep '12

01 Sep '12

Hi, There is segfault reported here: http://projects.scipy.org/numpy/ticket/1588 I've managed to isolate the problem and even provide a simple patch, that fixes it here: https://github.com/numpy/numpy/issues/398 however the patch simply doesn't decrease the proper reference, so it might leak. I've used bisection (took the whole evening unfortunately...) but the good news is that I've isolated commits that actually broke it. See the github issue #398 for details, diffs etc. Unfortunately, it's 12 commits from Mark and the individual commits raise exception on the segfaulting code, so I can't pin point the problem further. In general, how can I debug this sort of problem? I tried to use valgrind, with a debugging build of numpy, but it provides tons of false (?) positives: https://gist.github.com/3549063 Mark, by looking at the changes that broke it, as well as at my "fix", do you see where the problem could be? I suspect it is something with the changes in PyArray_FromAny() or PyArray_FromArray() in ctors.c. But I don't see anything so far that could cause it. Thanks for any help. This is one of the issues blocking the 1.7.0 release. Ondrej

4 5

Temporary error accessing NumPy tickets
by Ondřej Čertík 01 Sep '12

01 Sep '12

Hi, When I access tickets, for example: http://projects.scipy.org/numpy/ticket/2185 then sometimes I get: Trac detected an internal error: OperationalError: database is locked For example yesterday. A refresh in about a minute fixed the problem. Today it still lasts at the moment. Ondrej

3 4

view of recarray issue
by Ralf Gommers 31 Aug '12

31 Aug '12

Hi, Just a heads up that right now views of recarrays seem to be problematic, this doesn't work anymore: >>> import statsmodels.api as sm >>> dta = sm.datasets.macrodata.load() # returns a record array with 14 fields >>> dta.data[['infl', 'realgdp']].view((float,2)) I opened http://projects.scipy.org/numpy/ticket/2187 for this. Probably a blocker for 1.7.0. Question: is that really the recommended way to get an (N, 2) size float array from two columns of a larger record array? If so, why isn't there a better way? If you'd want to write to that (N, 2) array you have to append a copy, making it even uglier. Also, then there really should be tests for views in test_records.py. Ralf

4 7

Issues for 1.7.0
by Ondřej Čertík 31 Aug '12

31 Aug '12

Hi, I am keeping track of all issues that need to be done for the 1.7.0 release here: https://github.com/numpy/numpy/issues/396 If you have trac and github push access, here is how you can help (by closing/merging): Issues that need clarification: http://projects.scipy.org/numpy/ticket/2150 http://projects.scipy.org/numpy/ticket/2101 Issues fixed (should be closed): http://projects.scipy.org/numpy/ticket/2185 http://projects.scipy.org/numpy/ticket/2066 http://projects.scipy.org/numpy/ticket/2189 PRs that need merging: https://github.com/numpy/numpy/pull/395 https://github.com/numpy/numpy/pull/397 There are still a few more (see my github issue above), that I am working on right now. Thanks, Ondrej

2 4

Access to SPARC 64
by Ondřej Čertík 31 Aug '12

31 Aug '12

Hi, Does anyone have a SPARC 64 machine that I could have an access to, so that I can try to reproduce and fix the following issue? http://projects.scipy.org/numpy/ticket/2076 That would be greatly appreciated, as it is currently marked as a blocker for 1.7.0. Thanks, Ondrej

2 1

installing numpy in jython (in Netbeans)
by Todd Brunhoff 31 Aug '12

31 Aug '12

Being a newbie to this list, I recognize that the answer to this might be "why would you do that?". But surely it can't be worse than that. Briefly put, I want to install numpy and scipy in jython (reasons below). Running 'cd <jython distro>; jython setup.py install' runs into errors. But installing he same distro under python works fine. So the underlying question is: is there a way to install numpy under jython? Here's how I got here. The reason I would like this under jython is wrapped up in Netbeans, Octave, Matlab and Windows 7. I've been using Netbeans under linux for years for c++ and a bit of Java, and the editing environment with cross referencing and syntax-directed editing is quite good. Netbeans only presents python debugging via jython, which makes sense. Most of the work I am doing is with matrix algebra and I started with Octave, but while Octave is excellent for operating on matrices, it is not good for file format manipulations, hence for some operations I'd like to turn to python, and if I edit under Netbeans, the debugging requires that I install numpy under jython. At work I use a machine running fedora 16 .... but... I travel a bit and my travel machine is a laptop running windows 7. Therefore, Windows 7 + Netbeans + numpy + debugging ==> jython + numpy + scipy. Here's the install problems, which occur under numpy-1.7.0b1 and 1.6.2. The first install error is in numpy\distutils\exec_command.py, line 585, where it throws an exception because the java exec tests are unlimited. So I comment out those two lines, and rerun the setup.py. The next errors are notes about features not being available because other packages are also unavailable. I don't think this is really getting in the way, although I could be wrong. The libraries mentioned as missing are: * libraries mkl,vml,guide * libraries ptf77blas,ptcblas,atlas * libraries lapack_atlas So setup gets as far as this: running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build\src.java1.6.0_33-2.5 creating build\src.java1.6.0_33-2.5\numpy creating build\src.java1.6.0_33-2.5\numpy\distutils building library "npymath" sources No module named jythoncompiler in numpy.distutils; trying from distutils customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 don't know how to compile Fortran code on platform 'java' Some of these messages make it look as if the config wants to use the cygwin gnu compilers, which are ok, but under windows are not nearly as good as mingw gnu compiler or better yet, the visual studio 2010 compiler. I have both, but I don't see a way to steer the numpy setup to use them. The next error is fatal File "...\numpy-1.7.0b1\numpy\distutils\ccompiler.py", line 111, in CCompiler_object_filenames if ext not in self.src_extensions: TypeError: 'NoneType' object is not iterable This one looks as if it does not know what .o or .a or .obj files are. Fixing this one looks like hours of digging through the code. Is there a simpler solution? Thanks in advance, Todd

2 4

A sad day for our community. John Hunter: 1968-2012.
by Fernando Perez 30 Aug '12

30 Aug '12

Dear friends and colleagues, [please excuse a possible double-post of this message, in-flight internet glitches] I am terribly saddened to report that yesterday, August 28 2012 at 10am, John D. Hunter died from complications arising from cancer treatment at the University of Chicago hospital, after a brief but intense battle with this terrible illness. John is survived by his wife Miriam, his three daughters Rahel, Ava and Clara, his sisters Layne and Mary, and his mother Sarah. Note: If you decide not to read any further (I know this is a long message), please go to this page for some important information about how you can thank John for everything he gave in a decade of generous contributions to the Python and scientific communities: http://numfocus.org/johnhunter. Just a few weeks ago, John delivered his keynote address at the SciPy 2012 conference in Austin centered around the evolution of matplotlib: http://www.youtube.com/watch?v=e3lTby5RI54 but tragically, shortly after his return home he was diagnosed with advanced colon cancer. This diagnosis was a terrible discovery to us all, but John took it with his usual combination of calm and resolve, and initiated treatment procedures. Unfortunately, the first round of chemotherapy treatments led to severe complications that sent him to the intensive care unit, and despite the best efforts of the University of Chicago medical center staff, he never fully recovered from these. Yesterday morning, he died peacefully at the hospital with his loved ones at his bedside. John fought with grace and courage, enduring every necessary procedure with a smile on his face and a kind word for all of his caretakers and becoming a loved patient of the many teams that ended up involved with his case. This was no surprise for those of us who knew him, but he clearly left a deep and lasting mark even amongst staff hardened by the rigors of oncology floors and intensive care units. I don't need to explain to this community the impact of John's work, but allow me to briefly recap, in case this is read by some who don't know the whole story. In 2002, John was a postdoc at the University of Chicago hospital working on the analysis of epilepsy seizure data in children. Frustrated with the state of the existing proprietary solutions for this class of problems, he started using Python for his work, back when the scientific Python ecosystem was much, much smaller than it is today and this could have been seen as a crazy risk. Furthermore, he found that there were many half-baked solutions for data visualization in Python at the time, but none that truly met his needs. Undeterred, he went on to create matplotlib (http://matplotlib.org) and thus overcome one of the key obstacles for Python to become the best solution for open source scientific and technical computing. Matplotlib is both an amazing technical achievement and a shining example of open source community building, as John not only created its backbone but also fostered the development of a very strong development team, ensuring that the talent of many others could also contribute to this project. The value and importance of this are now painfully clear: despite having lost John, matplotlib continues to thrive thanks to the leadership of Michael Droetboom, the support of Perry Greenfield at the Hubble Telescope Science Institute, and the daily work of the rest of the team. I want to thank Perry and Michael for putting their resources and talent once more behind matplotlib, securing the future of the project. It is difficult to overstate the value and importance of matplotlib, and therefore of John's contributions (which do not end in matplotlib, by the way; but a biography will have to wait for another day...). Python has become a major force in the technical and scientific computing world, leading the open source offers and challenging expensive proprietary platforms with large teams and millions of dollars of resources behind them. But this would be impossible without a solid data visualization tool that would allow both ad-hoc data exploration and the production of complex, fine-tuned figures for papers, reports or websites. John had the vision to make matplotlib easy to use, but powerful and flexible enough to work in graphical user interfaces and as a server-side library, enabling a myriad use cases beyond his personal needs. This means that now, matplotlib powers everything from plots in dissertations and journal articles to custom data analysis projects and websites. And despite having left his academic career a few years ago for a job in industry, he remained engaged enough that as of today, he is still the top committer to matplotlib; this is the git shortlog of those with more than 1000 commits to the project: 2145 John Hunter <jdh2358(a)gmail.com> 2130 Michael Droettboom <mdboom(a)gmail.com> 1060 Eric Firing <efiring(a)hawaii.edu> All of this was done by a man who had three children to raise and who still always found the time to help those on the mailing lists, solve difficult technical problems in matplotlib, teach courses and seminars about scientific Python, and more recently help create the NumFOCUS foundation project. Despite the challenges that raising three children in an expensive city like Chicago presented, he never once wavered from his commitment to open source. But unfortunately now he is not here anymore to continue providing for their well-being, and I hope that all those who have so far benefited from his generosity, will thank this wonderful man who always gave far more than he received. Thanks to the rapid action of Travis Oliphant, the NumFOCUS foundation is now acting as an escrow agent to accept donations that will go into a fund to support the education and care of his wonderful girls Rahel, Ava and Clara. If you have benefited from John's many contributions, please say thanks in the way that would matter most to him, by helping Miriam continue the task of caring for and educating Rahel, Ava and Clara. You will find all the information necessary to make a donation here: http://numfocus.org/johnhunter Remember that even a small donation helps! If all those who ever use matplotlib give just a little bit, in the long run I am sure that we can make a difference. If you are a company that benefits in a serious way from matplotlib, remember that John was a staunch advocate of keeping all scientific Python projects under the BSD license so that commercial users could benefit from them without worry. Please say thanks to John in a way commensurate with your resources (and check how much a yearly matlab license would cost you in case you have any doubts about the value you are getting...). John's family is planning a private burial in Tennessee, but (most likely in September) there will also be a memorial service in Chicago that friends and members of the community can attend. We don't have the final scheduling details at this point, but I will post them once we know. I would like to again express my gratitude to Travis Oliphant for moving quickly with the setup of the donation support, and to Eric Jones (the founder of Enthought and another one of the central figures in our community) who immediately upon learning of John's plight contributed resources to support the family with everyday logistics while John was facing treatment as well as my travel to Chicago to assist. This kind of immediate urge to come to the help of others that Eric and Travis displayed is a hallmark of our community. Before closing, I want to take a moment to publicly thank the incredible staff of the University of Chicago medical center. The last two weeks were an intense and brutal ordeal for John and his loved ones, but the hospital staff offered a sometimes hard to believe, unending supply of generosity, care and humanity in addition to their technical competence. The latter is something we expect from a first-rate hospital at a top university, where the attending physicians can be world-renowned specialists in their field. But the former is often forgotten in a world often ruled by a combination of science and concerns about regulations and liability. Instead, we found generous and tireless staff who did everything in their power to ease the pain, always putting our well being ahead of any mindless adherence to protocol, patiently tending to every need we had and working far beyond their stated responsibilities to support us. To name only one person (and many others are equally deserving), I want to thank Dr. Carla Moreira, chief surgical resident, who spent the last few hours of John's life with us despite having just completed a solid night shift of surgical work. Instead of resting she came to the ICU and worked to ensure that those last hours were as comfortable as possible for John; her generous actions helped us through a very difficult moment. It is now time to close this already too long message... John, thanks for everything you gave all of us, and for the privilege of knowing you. Fernando. ps - I have sent this with my 'mailing lists' email. If you need to contact me directly for anything regarding the above, please write to my regular address at Fernando.Perez(a)berkeley.edu, where I do my best to reply more promptly.

2 1

broadcasting question
by Neal Becker 30 Aug '12

30 Aug '12

I think this should be simple, but I'm drawing a blank I have 2 2d matrixes Matrix A has indexes (i, symbol) Matrix B has indexes (state, symbol) I combined them into a 3d matrix: C = A[:,newaxis,:] + B[newaxis,:,:] where C has indexes (i, state, symbol) That works fine. Now suppose I want to omit B (for debug), like: C = A[:,newaxis,:] In other words, all I want is to add a dimension into A and force it to broadcast along that axis. How do I do that?

3 2