Mailman 3 February 2013 - NumPy-Discussion

ANN: NumPy 1.7.0rc2 release
by Ondřej Čertík 09 Feb '13

09 Feb '13

Hi, I'm pleased to announce the availability of the second release candidate of NumPy 1.7.0rc2. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc2/ We have fixed all issues known to us since the 1.7.0rc1 release. Please test this release and report any issues on the numpy-discussion mailing list. If there are no further problems, I plan to release the final version in a few days. I would like to thank Sandro Tosi, Sebastian Berg, Charles Harris, Marcin Juszkiewicz, Mark Wiebe, Ralf Gommers and Nathaniel J. Smith for sending patches, fixes and helping with reviews for this release since 1.7.0rc1, and Vincent Davis for providing the Mac build machine. Cheers, Ondrej ========================= NumPy 1.7.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and refactorings. It supports Python 2.4 - 2.7 and 3.1 - 3.3 and is the last release that supports Python 2.4 - 2.5. Highlights ========== * ``where=`` parameter to ufuncs (allows the use of boolean arrays to choose where a computation should be done) * ``vectorize`` improvements (added 'excluded' and 'cache' keyword, general cleanup and bug fixes) * ``numpy.random.choice`` (random sample generating function) Compatibility notes =================== In a future version of numpy, the functions np.diag, np.diagonal, and the diagonal method of ndarrays will return a view onto the original array, instead of producing a copy as they do now. This makes a difference if you write to the array returned by any of these functions. To facilitate this transition, numpy 1.7 produces a FutureWarning if it detects that you may be attempting to write to such an array. See the documentation for np.diagonal for details. Similar to np.diagonal above, in a future version of numpy, indexing a record array by a list of field names will return a view onto the original array, instead of producing a copy as they do now. As with np.diagonal, numpy 1.7 produces a FutureWarning if it detects that you may be attempting to write to such an array. See the documentation for array indexing for details. In a future version of numpy, the default casting rule for UFunc out= parameters will be changed from 'unsafe' to 'same_kind'. (This also applies to in-place operations like a += b, which is equivalent to np.add(a, b, out=a).) Most usages which violate the 'same_kind' rule are likely bugs, so this change may expose previously undetected errors in projects that depend on NumPy. In this version of numpy, such usages will continue to succeed, but will raise a DeprecationWarning. Full-array boolean indexing has been optimized to use a different, optimized code path. This code path should produce the same results, but any feedback about changes to your code would be appreciated. Attempting to write to a read-only array (one with ``arr.flags.writeable`` set to ``False``) used to raise either a RuntimeError, ValueError, or TypeError inconsistently, depending on which code path was taken. It now consistently raises a ValueError. The <ufunc>.reduce functions evaluate some reductions in a different order than in previous versions of NumPy, generally providing higher performance. Because of the nature of floating-point arithmetic, this may subtly change some results, just as linking NumPy to a different BLAS implementations such as MKL can. If upgrading from 1.5, then generally in 1.6 and 1.7 there have been substantial code added and some code paths altered, particularly in the areas of type resolution and buffered iteration over universal functions. This might have an impact on your code particularly if you relied on accidental behavior in the past. New features ============ Reduction UFuncs Generalize axis= Parameter ------------------------------------------- Any ufunc.reduce function call, as well as other reductions like sum, prod, any, all, max and min support the ability to choose a subset of the axes to reduce over. Previously, one could say axis=None to mean all the axes or axis=# to pick a single axis. Now, one can also say axis=(#,#) to pick a list of axes for reduction. Reduction UFuncs New keepdims= Parameter ---------------------------------------- There is a new keepdims= parameter, which if set to True, doesn't throw away the reduction axes but instead sets them to have size one. When this option is set, the reduction result will broadcast correctly to the original operand which was reduced. Datetime support ---------------- .. note:: The datetime API is *experimental* in 1.7.0, and may undergo changes in future versions of NumPy. There have been a lot of fixes and enhancements to datetime64 compared to NumPy 1.6: * the parser is quite strict about only accepting ISO 8601 dates, with a few convenience extensions * converts between units correctly * datetime arithmetic works correctly * business day functionality (allows the datetime to be used in contexts where only certain days of the week are valid) The notes in `doc/source/reference/arrays.datetime.rst <https://github.com/numpy/numpy/blob/maintenance/1.7.x/doc/source/reference/…>`_ (also available in the online docs at `arrays.datetime.html <http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html>`_) should be consulted for more details. Custom formatter for printing arrays ------------------------------------ See the new ``formatter`` parameter of the ``numpy.set_printoptions`` function. New function numpy.random.choice --------------------------------- A generic sampling function has been added which will generate samples from a given array-like. The samples can be with or without replacement, and with uniform or given non-uniform probabilities. New function isclose -------------------- Returns a boolean array where two arrays are element-wise equal within a tolerance. Both relative and absolute tolerance can be specified. Preliminary multi-dimensional support in the polynomial package --------------------------------------------------------------- Axis keywords have been added to the integration and differentiation functions and a tensor keyword was added to the evaluation functions. These additions allow multi-dimensional coefficient arrays to be used in those functions. New functions for evaluating 2-D and 3-D coefficient arrays on grids or sets of points were added together with 2-D and 3-D pseudo-Vandermonde matrices that can be used for fitting. Ability to pad rank-n arrays ---------------------------- A pad module containing functions for padding n-dimensional arrays has been added. The various private padding functions are exposed as options to a public 'pad' function. Example:: pad(a, 5, mode='mean') Current modes are ``constant``, ``edge``, ``linear_ramp``, ``maximum``, ``mean``, ``median``, ``minimum``, ``reflect``, ``symmetric``, ``wrap``, and ``<function>``. New argument to searchsorted ---------------------------- The function searchsorted now accepts a 'sorter' argument that is a permutation array that sorts the array to search. Build system ------------ Added experimental support for the AArch64 architecture. C API ----- New function ``PyArray_RequireWriteable`` provides a consistent interface for checking array writeability -- any C code which works with arrays whose WRITEABLE flag is not known to be True a priori, should make sure to call this function before writing. NumPy C Style Guide added (``doc/C_STYLE_GUIDE.rst.txt``). Changes ======= General ------- The function np.concatenate tries to match the layout of its input arrays. Previously, the layout did not follow any particular reason, and depended in an undesirable way on the particular axis chosen for concatenation. A bug was also fixed which silently allowed out of bounds axis arguments. The ufuncs logical_or, logical_and, and logical_not now follow Python's behavior with object arrays, instead of trying to call methods on the objects. For example the expression (3 and 'test') produces the string 'test', and now np.logical_and(np.array(3, 'O'), np.array('test', 'O')) produces 'test' as well. The ``.base`` attribute on ndarrays, which is used on views to ensure that the underlying array owning the memory is not deallocated prematurely, now collapses out references when you have a view-of-a-view. For example:: a = np.arange(10) b = a[1:] c = b[1:] In numpy 1.6, ``c.base`` is ``b``, and ``c.base.base`` is ``a``. In numpy 1.7, ``c.base`` is ``a``. To increase backwards compatibility for software which relies on the old behaviour of ``.base``, we only 'skip over' objects which have exactly the same type as the newly created view. This makes a difference if you use ``ndarray`` subclasses. For example, if we have a mix of ``ndarray`` and ``matrix`` objects which are all views on the same original ``ndarray``:: a = np.arange(10) b = np.asmatrix(a) c = b[0, 1:] d = c[0, 1:] then ``d.base`` will be ``b``. This is because ``d`` is a ``matrix`` object, and so the collapsing process only continues so long as it encounters other ``matrix`` objects. It considers ``c``, ``b``, and ``a`` in that order, and ``b`` is the last entry in that list which is a ``matrix`` object. Casting Rules ------------- Casting rules have undergone some changes in corner cases, due to the NA-related work. In particular for combinations of scalar+scalar: * the `longlong` type (`q`) now stays `longlong` for operations with any other number (`? b h i l q p B H I`), previously it was cast as `int_` (`l`). The `ulonglong` type (`Q`) now stays as `ulonglong` instead of `uint` (`L`). * the `timedelta64` type (`m`) can now be mixed with any integer type (`b h i l q p B H I L Q P`), previously it raised `TypeError`. For array + scalar, the above rules just broadcast except the case when the array and scalars are unsigned/signed integers, then the result gets converted to the array type (of possibly larger size) as illustrated by the following examples:: >>> (np.zeros((2,), dtype=np.uint8) + np.int16(257)).dtype dtype('uint16') >>> (np.zeros((2,), dtype=np.int8) + np.uint16(257)).dtype dtype('int16') >>> (np.zeros((2,), dtype=np.int16) + np.uint32(2**17)).dtype dtype('int32') Whether the size gets increased depends on the size of the scalar, for example:: >>> (np.zeros((2,), dtype=np.uint8) + np.int16(255)).dtype dtype('uint8') >>> (np.zeros((2,), dtype=np.uint8) + np.int16(256)).dtype dtype('uint16') Also a ``complex128`` scalar + ``float32`` array is cast to ``complex64``. In NumPy 1.7 the `datetime64` type (`M`) must be constructed by explicitly specifying the type as the second argument (e.g. ``np.datetime64(2000, 'Y')``). Deprecations ============ General ------- Specifying a custom string formatter with a `_format` array attribute is deprecated. The new `formatter` keyword in ``numpy.set_printoptions`` or ``numpy.array2string`` can be used instead. The deprecated imports in the polynomial package have been removed. ``concatenate`` now raises DepractionWarning for 1D arrays if ``axis != 0``. Versions of numpy < 1.7.0 ignored axis argument value for 1D arrays. We allow this for now, but in due course we will raise an error. C-API ----- Direct access to the fields of PyArrayObject* has been deprecated. Direct access has been recommended against for many releases. Expect similar deprecations for PyArray_Descr* and other core objects in the future as preparation for NumPy 2.0. The macros in old_defines.h are deprecated and will be removed in the next major release (>= 2.0). The sed script tools/replace_old_macros.sed can be used to replace these macros with the newer versions. You can test your code against the deprecated C API by #defining NPY_NO_DEPRECATED_API to the target version number, for example NPY_1_7_API_VERSION, before including any NumPy headers. The ``NPY_CHAR`` member of the ``NPY_TYPES`` enum is deprecated and will be removed in NumPy 1.8. See the discussion at `gh-2801 <https://github.com/numpy/numpy/issues/2801>`_ for more details.

4 4

Type casting problems with numpy.take
by Jaime Fernández del Río 09 Feb '13

09 Feb '13

Hi, I see there is a lot of ongoing discussion on casting rules, but I couldn't find any reference to the following issue I am facing. I am trying to 'take' from an array of uint8's, using an array of uint16's as indices. Even though the return dtype would be uint8, I want to direct the output back into the array of uint16's: >>> lut = np.random.randint(256, size=(65536,)).astype('uint8') >>> arr = np.random.randint(65536, size=(1000, 1000)).astype('uint16') >>> np.take(lut, arr, out=arr) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 103, in take return take(indices, axis, out, mode) TypeError: array cannot be safely cast to required type This is puzzling, since the only casting that should be happening is from uint8's to uint16's, which is as safe as it gets: >>> np.can_cast('uint8', 'uint16') True To make things even weirder, I can get the above code to work if the type of lut is uint16, uint32, uint64, int32 or int 64, but not if it is uint8, int8 or int16. Without looking at the source, it almost looks as if the type checking in numpy.take was reversed... Am I missing something, or is this broken? My numpy's version is: >>> np.__version__ '1.6.2' which is the one packaged in Python xy 2.7.3.1, running on a 64 bit Windows 7 system. Thanks, Jaime P.S. I have posted the same question in StackExchange: http://stackoverflow.com/questions/14782135/type-casting-error-with-numpy-t… -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

3 5

ANN: pandas 0.10.1 is released
by Wes McKinney 08 Feb '13

08 Feb '13

hi all, We've released pandas 0.10.1 which includes many bug fixes from 0.10.0 (including a number of issues with the new file parser, e.g. reading multiple files in separate threads), various performance improvements, and major new PyTables/HDF5-based functionality contributed by Jeff Reback. I strongly recommend that all users upgrade. Thanks to all who contributed to this release, especially Chang She, Jeff Reback, and Yoval P. As always source archives and Windows installers are on PyPI. What's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html Installers: http://pypi.python.org/pypi/pandas $ git log v0.10.0..v0.10.1 --pretty=format:%aN | sort | uniq -c | sort -rn 66 jreback 59 Wes McKinney 43 Chang She 12 y-p 5 Vincent Arel-Bundock 4 Damien Garaud 3 Christopher Whelan 3 Andy Hayden 2 Jay Parlar 2 Dan Allan 1 Thouis (Ray) Jones 1 svaksha 1 herrfz 1 Garrett Drapala 1 elpres 1 Dieter Vandenbussche 1 Anton I. Sipos Happy data hacking! - Wes What is it ========== pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with relational, time series, or any other kind of labeled data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Links ===== Release Notes: http://github.com/pydata/pandas/blob/master/RELEASE.rst Documentation: http://pandas.pydata.org Installers: http://pypi.python.org/pypi/pandas Code Repository: http://github.com/pydata/pandas Mailing List: http://groups.google.com/group/pydata

2 1

Will numpy 1.7.0 final be binary compatible with the rc?
by Peter Cock 07 Feb '13

07 Feb '13

Hello all, Will the numpy 1.7.0 'final' be binary compatible with the release candidate(s)? i.e. Would it be safe for me to release a Windows installer for a package using the NumPy C API compiled against the NumPy 1.7.0rc? I'm specifically interested in Python 3.3, and NumPy 1.7 will be the first release to support that. For older versions of Python I can use NumPy 1.6 instead. Thanks, Peter

3 4

Dealing with the mode argument in qr.
by Charles R Harris 06 Feb '13

06 Feb '13

Hi All, This post is to bring the discussion of PR #2965<https://github.com/numpy/numpy/pull/2965>to the attention of the list. There are at least three issues in play here. 1) The PR adds modes 'big' and 'thin' to the current modes 'full', 'r', 'economic' for qr factorization. The problem is that the current 'full' is actually 'thin' and 'big' should be 'full'. The solution here was to raise a FutureWarning on use of 'full', alias it to 'thin' for the time being, and at some distant time change 'full' to alias 'big'. 2) The 'economic' mode serves little purpose. I propose to deprecate it and add a 'qrf' mode instead, corresponding to scipy's 'raw' mode. We can't use 'raw' itself as traditionally the mode may be specified using the first letter only and that leads to a conflict with 'r'. 3) As suggested in 2, the use of single letter abbreviations can constrain the options in choosing mode names and they are not as informative as the full name. A possibility here is to deprecate the use of the abbreviations in favor of the full names. A longer term problem is the divergence between the numpy and scipy versions of qr. The divergence is enough that I don't see any easy way to come to a common interface, but that is something that would be desirable if possible. Thoughts? Chuck

4 6

Where's that function?
by josef.pktd＠gmail.com 06 Feb '13

06 Feb '13

I'm convinced that I saw a while ago a function that uses a list of interval boundaries to index into an array, either to iterate or to take. I thought that's very useful, but didn't make a note. Now, I have no idea where I saw this (I thought numpy), and I cannot find it anywhere. any clues? Thanks, Josef

3 4

Issues to fix for 1.7.0rc2.
by Ondřej Čertík 06 Feb '13

06 Feb '13

Hi, Here are the last open issues for 1.7, there are 9 of them: https://github.com/numpy/numpy/issues?milestone=3&sort=updated&state=open >From these, 3 are very simple PRs that I just posted. Let's polish these, get them in. I propose to release rc2 after that and if all is ok, do the final release. Some of the issues are not fully addressed, but I don't think we should be holding the release any longer. Let me know if that is ok with you. My apologies for a delay on my side --- my son was just born 2 weeks ago and I had to submit an important article, with the deadline yesterday. Things are settled now and I now have time to get this done. Ondrej

8 10

Building numpy for python3.3: No _numpyconfig.h
by gk230-freebsd＠yahoo.de 06 Feb '13

06 Feb '13

Hi, I'm currently trying to build numpy 1.6.2 for python python 3.3 from ports on FreeBSD. Unfortunately, the setup.py execution fails because some [1] gcc command trying to access _numpyconfig.h fails since _numpyconfig.h is not generated from _numpyconfig.h.in. How do I manually build the proper header from the .h.in, and why does that not happen automatically? -- -- Gereon [1] # gcc46 -DNDEBUG -O2 -pipe -fno-strict-aliasing -O2 -pipe -Wl,-rpath=/usr/local/lib/gcc46 -fno-strict-aliasing -fPIC -Inumpy/core/include -Ibuild/src.freebsd-9.0-RELEASE-amd64-3.3/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/local/include/python3.3m -Ibuild/src.freebsd-9.0-RELEASE-amd64-3.3/numpy/core/src/multiarray -Ibuild/src.freebsd-9.0-RELEASE-amd64-3.3/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.freebsd-9.0-RELEASE-amd64-3.3/numpy/core/src/multiarray/multiarraymodule_onefile.o Assembler messages: Fatal error: can't create build/temp.freebsd-9.0-RELEASE-amd64-3.3/numpy/core/src/multiarray/multiarraymodule_onefile.o: No such file or directory In file included from numpy/core/include/numpy/ndarraytypes.h:5:0, from numpy/core/include/numpy/ndarrayobject.h:17, from numpy/core/include/numpy/arrayobject.h:14, from numpy/core/src/multiarray/common.c:6, from numpy/core/src/multiarray/multiarraymodule_onefile.c:8: numpy/core/include/numpy/numpyconfig.h:4:26: fatal error: _numpyconfig.h: No such file or directory compilation terminated.

2 1

savez documentation flaw
by Andreas Hilboll 05 Feb '13

05 Feb '13

Hi, I noticed that on http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html there's a "see also" to a function numpy.savez_compressed, which doesn't seem to exist (neither on my system nor in the online documentation). What would be the easiest way to find out where to fix this? For someone without deeper knowledge of how numpy sources are organized it's hard to find the place where to fix things. How about adding the "source" link to the docstrings via sphinx, like in scipy? Cheers, Andreas.

3 2

faster Array.take( floatindices.astype(int) ) ?
by denis 03 Feb '13

03 Feb '13

Folks, is there a way to speed up Array.take( floatindices.astype(int) ) ? astype(int) makes a copy, floor() returns floats. (Is there a wiki of NumPy one-liners / various tricks ? would sure beat googling.) Thanks, cheers -- denis

2 2