Mailman 3 September 2012 - NumPy-Discussion

ANN: python-blosc 1.0.4 released
by Francesc Alted Sept. 14, 2012

Sept. 14, 2012

============================= Announcing python-blosc 1.0.4 ============================= What is it? =========== A Python wrapper for the Blosc compression library. Blosc (http://blosc.pytables.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc is a Python package that wraps it. What is new? ============ Optimized the amount of data copied during compression (using _PyBytes_Resize() now instead of previous PyBytes_FromStringAndSize()). This leads to improvements in compression speed ranging from 1.2x for highly compressible chunks up to 7x for mostly uncompressible data. Thanks to Valentin Haenel for this nice contribution. For more info, you can see the release notes in: https://github.com/FrancescAlted/python-blosc/wiki/Release-notes More docs and examples are available in the Quick User's Guide wiki page: https://github.com/FrancescAlted/python-blosc/wiki/Quick-User's-Guide Download sources ================ Go to: http://github.com/FrancescAlted/python-blosc and download the most recent release from there. Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for details. Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc -- Francesc Alted

1 0

Obscure code in concatenate code path?
by Matthew Brett Sept. 13, 2012

Sept. 13, 2012

Hi, While writing some tests for np.concatenate, I ran foul of this code: if (axis >= NPY_MAXDIMS) { ret = PyArray_ConcatenateFlattenedArrays(narrays, arrays, NPY_CORDER); } else { ret = PyArray_ConcatenateArrays(narrays, arrays, axis); } in multiarraymodule.c So, if the user passes an axis >= (by default) 32 the arrays to concatenate get flattened, and must all have the same number of elements (it turns out). This seems obscure. Obviously it is not likely that someone would pass in an axis no >= 32 by accident, but if they did, they would not get the result they expect. Is there some code-path that needs this? Is there another way of doing it? Best, Matthew

5 7

Contiguity of result of astype changed - intentional?
by Matthew Brett Sept. 13, 2012

Sept. 13, 2012

Hi, We hit a subtle behavior change for the ``astype`` array method between 1.6.1 and 1.7.0 beta. In 1.6.1: In [18]: a = np.arange(24).reshape((2, 3, 4)).transpose((1, 2, 0)) In [19]: a.flags Out[19]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [20]: b = a.astype(float) In [21]: b.flags Out[21]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [22]: b.strides Out[22]: (64, 16, 8) So - ``a.astype(float)`` here has made a new C-contiguous array, somewhat as implied by the 'copy' explanation in the docstring. In 1.7.0 beta, ``a`` is the same but: In [22]: b.flags Out[22]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [23]: b.strides Out[23]: (32, 8, 96) Is this intended? Is there a performance reason to keep the same strides in 1.7.0? Thanks for any pointers, Matthew

3 5

numpy sort is not working
by Jadhav, Alok Sept. 11, 2012

Sept. 11, 2012

Hi everyone, I have a numpy array of dimensions >>> allRics.shape (583760, 1) To sort the array, I set the dtype of the array as follows: allRics.dtype = [('idx', np.float), ('opened', np.float), ('time', np.float),('trdp1',np.float),('trdp0',np.float),('dt',np.float),('value' ,np.float)] >>> allRics.dtype dtype([('idx', '<f8'), ('opened', '<f8'), ('time', '<f8'), ('trdp1', '<f8'), ('trdp0', '<f8'), ('dt', '<f8'), ('value', '<f8')]) I checked and the endianness in dtype is correct. When I sort the array, the output array of sort is same as original array without any change. I want to sort the allRics numpy array on 'time'. I do the following. >>> x=np.sort(allRics,order='time') >>> x[17330:17350]['time'] array([[ 61184.4 ], [ 61188.51 ], [ 61188.979], [ 61188.979], [ 61189.989], [ 61191.66 ], [ 61194.35 ], [ 61194.35 ], [ 61198.79 ], [ 61198.145], [ 36126.217], [ 36126.217], [ 36126.218], [ 36126.218], [ 36126.219], [ 36126.271], [ 36126.271], [ 36126.271], [ 36126.293], [ 36126.293]]) Time column doesn't change its order. Could someone please advise what is missing here? Is this related to the bug http://www.mail-archive.com/numpy-discussion@scipy.org/msg23060.html (from 2010). Regards, Alok Alok Jadhav CREDIT SUISSE AG GAT IT Hong Kong, KVAG 67 International Commerce Centre | Hong Kong | Hong Kong Phone +852 2101 6274 | Mobile +852 9169 7172 alok.jadhav(a)credit-suisse.com | www.credit-suisse.com <http://www.credit-suisse.com/> =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================

3 6

status of 'present' to use optional parameters in f2py
by Mark Bakker Sept. 11, 2012

Sept. 11, 2012

Hello List, I searched the list whether it is possible to use optional arguments in fortran functions that are compiled with f2py. In 2008, this feature was not yet included and appeared to be non-trivial. Does anybody know the status of putting that feature in f2py? Thanks, Mark

1 0

sum and prod
by nicky van foreest Sept. 9, 2012

Sept. 9, 2012

Hi, I ran the following code: args = np.array([4,8]) print np.sum( (arg > 0) for arg in args) print np.sum([(arg > 0) for arg in args]) print np.prod( (arg > 0) for arg in args) print np.prod([(arg > 0) for arg in args]) with this result: 2 1 <generator object <genexpr> at 0x1c70410> 1 Is the difference between prod and sum intentional? I would expect that numpy.prod would also work on a generator, just like numpy.sum. BTW: the last line does what I need: the product over the truth values of all elements of args. Is there perhaps a nicer (conciser) way to achieve this? Thanks. Nicky

4 6

Numpy 1.7b1 API change cause big trouble
by Frédéric Bastien Sept. 9, 2012

Sept. 9, 2012

Hi, I spent up to now 2 or 3 days making change to Theano to support numpy 1.7b1. But now, I just find an interface change that will need recoding a function, not just small code change. The problem is that we can't access fields from PyArrayObject anymore, we absolutely must use the old macro/newly function. For the data field, the new function don't allow to set it. There is no function that allow to do this. After so much time spent on small syntactic change, I don't feel making more complex change today. Also, I think there should be a function PyArray_SetDataPtr as similar to PyArray_SetBaseObject. Do you plan to add one? I though that you wanted to force the removing of the old API, but I never hear you wanted to disable this. The only current work around is to create a new Array, but then we need to update the refcount and other stuff. This will make the code slower too. I'll make another post on the problem I got when making Theano work with numpy 1.7b1, but I want to make this functionality removing problem into its own thread. So, do you plan to add an PyArray_SetDataPtr? Do you agree we should have one? Fred

6 9

ANN: NumPy 1.7.0b1 release
by Ondřej Čertík Sept. 9, 2012

Sept. 9, 2012

Hi, I'm pleased to announce the availability of the first beta release of NumPy 1.7.0b1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.0b1/ Please test this release and report any issues on the numpy-discussion mailing list. The following problems are known and we'll work on fixing them before the final release: http://projects.scipy.org/numpy/ticket/2187 http://projects.scipy.org/numpy/ticket/2185 http://projects.scipy.org/numpy/ticket/2066 http://projects.scipy.org/numpy/ticket/1588 http://projects.scipy.org/numpy/ticket/2076 http://projects.scipy.org/numpy/ticket/2101 http://projects.scipy.org/numpy/ticket/2108 http://projects.scipy.org/numpy/ticket/2150 http://projects.scipy.org/numpy/ticket/2189 I would like to thank Ralf for a lot of help with creating binaries and other help for this release. Cheers, Ondrej ========================= NumPy 1.7.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and refactorings. It supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * ``where=`` parameter to ufuncs (allows the use of boolean arrays to choose where a computation should be done) * ``vectorize`` improvements (added 'excluded' and 'cache' keyword, general cleanup and bug fixes) * ``numpy.random.choice`` (random sample generating function) Compatibility notes =================== In a future version of numpy, the functions np.diag, np.diagonal, and the diagonal method of ndarrays will return a view onto the original array, instead of producing a copy as they do now. This makes a difference if you write to the array returned by any of these functions. To facilitate this transition, numpy 1.7 produces a FutureWarning if it detects that you may be attempting to write to such an array. See the documentation for np.diagonal for details. Similar to np.diagonal above, in a future version of numpy, indexing a record array by a list of field names will return a view onto the original array, instead of producing a copy as they do now. As with np.diagonal, numpy 1.7 produces a FutureWarning if it detects that you may be attemping to write to such an array. See the documentation for array indexing for details. The default casting rule for UFunc out= parameters has been changed from 'unsafe' to 'same_kind'. Most usages which violate the 'same_kind' rule are likely bugs, so this change may expose previously undetected errors in projects that depend on NumPy. Full-array boolean indexing has been optimized to use a different, optimized code path. This code path should produce the same results, but any feedback about changes to your code would be appreciated. Attempting to write to a read-only array (one with ``arr.flags.writeable`` set to ``False``) used to raise either a RuntimeError, ValueError, or TypeError inconsistently, depending on which code path was taken. It now consistently raises a ValueError. The <ufunc>.reduce functions evaluate some reductions in a different order than in previous versions of NumPy, generally providing higher performance. Because of the nature of floating-point arithmetic, this may subtly change some results, just as linking NumPy to a different BLAS implementations such as MKL can. If upgrading from 1.5, then generally in 1.6 and 1.7 there have been substantial code added and some code paths altered, particularly in the areas of type resolution and buffered iteration over universal functions. This might have an impact on your code particularly if you relied on accidental behavior in the past. New features ============ Reduction UFuncs Generalize axis= Parameter ------------------------------------------- Any ufunc.reduce function call, as well as other reductions like sum, prod, any, all, max and min support the ability to choose a subset of the axes to reduce over. Previously, one could say axis=None to mean all the axes or axis=# to pick a single axis. Now, one can also say axis=(#,#) to pick a list of axes for reduction. Reduction UFuncs New keepdims= Parameter ---------------------------------------- There is a new keepdims= parameter, which if set to True, doesn't throw away the reduction axes but instead sets them to have size one. When this option is set, the reduction result will broadcast correctly to the original operand which was reduced. Datetime support ---------------- .. note:: The datetime API is *experimental* in 1.7.0, and may undergo changes in future versions of NumPy. There have been a lot of fixes and enhancements to datetime64 compared to NumPy 1.6: * the parser is quite strict about only accepting ISO 8601 dates, with a few convenience extensions * converts between units correctly * datetime arithmetic works correctly * business day functionality (allows the datetime to be used in contexts where only certain days of the week are valid) The notes in `doc/source/reference/arrays.datetime.rst <https://github.com/numpy/numpy/blob/maintenance/1.7.x/doc/source/reference/…>`_ (also available in the online docs at `arrays.datetime.html <http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html>`_) should be consulted for more details. Custom formatter for printing arrays ------------------------------------ See the new ``formatter`` parameter of the ``numpy.set_printoptions`` function. New function numpy.random.choice --------------------------------- A generic sampling function has been added which will generate samples from a given array-like. The samples can be with or without replacement, and with uniform or given non-uniform probabilities. New function isclose -------------------- Returns a boolean array where two arrays are element-wise equal within a tolerance. Both relative and absolute tolerance can be specified. The function is NA aware. Preliminary multi-dimensional support in the polynomial package --------------------------------------------------------------- Axis keywords have been added to the integration and differentiation functions and a tensor keyword was added to the evaluation functions. These additions allow multi-dimensional coefficient arrays to be used in those functions. New functions for evaluating 2-D and 3-D coefficient arrays on grids or sets of points were added together with 2-D and 3-D pseudo-Vandermonde matrices that can be used for fitting. Ability to pad rank-n arrays ---------------------------- A pad module containing functions for padding n-dimensional arrays has been added. The various private padding functions are exposed as options to a public 'pad' function. Example:: pad(a, 5, mode='mean') Current modes are ``constant``, ``edge``, ``linear_ramp``, ``maximum``, ``mean``, ``median``, ``minimum``, ``reflect``, ``symmetric``, ``wrap``, and ``<function>``. New argument to searchsorted ---------------------------- The function searchsorted now accepts a 'sorter' argument that is a permuation array that sorts the array to search. C API ----- New function ``PyArray_RequireWriteable`` provides a consistent interface for checking array writeability -- any C code which works with arrays whose WRITEABLE flag is not known to be True a priori, should make sure to call this function before writing. NumPy C Style Guide added (``doc/C_STYLE_GUIDE.rst.txt``). Changes ======= General ------- The function np.concatenate tries to match the layout of its input arrays. Previously, the layout did not follow any particular reason, and depended in an undesirable way on the particular axis chosen for concatenation. A bug was also fixed which silently allowed out of bounds axis arguments. The ufuncs logical_or, logical_and, and logical_not now follow Python's behavior with object arrays, instead of trying to call methods on the objects. For example the expression (3 and 'test') produces the string 'test', and now np.logical_and(np.array(3, 'O'), np.array('test', 'O')) produces 'test' as well. Deprecations ============ General ------- Specifying a custom string formatter with a `_format` array attribute is deprecated. The new `formatter` keyword in ``numpy.set_printoptions`` or ``numpy.array2string`` can be used instead. The deprecated imports in the polynomial package have been removed. C-API ----- Direct access to the fields of PyArrayObject* has been deprecated. Direct access has been recommended against for many releases. Expect similar deprecations for PyArray_Descr* and other core objects in the future as preparation for NumPy 2.0. The macros in old_defines.h are deprecated and will be removed in the next major release (>= 2.0). The sed script tools/replace_old_macros.sed can be used to replace these macros with the newer versions. You can test your code against the deprecated C API by #defining NPY_NO_DEPRECATED_API to the target version number, for example NPY_1_7_API_VERSION, before including any NumPy headers.

12 21

problem with scipy's test
by 心如烛光 Sept. 8, 2012

Sept. 8, 2012

Hi,every body. I encounter the error while the scipy is testing . I wanna know why and how to fix it.(OSX lion 10.7.4) here is part of the respond: AssertionError: Not equal to tolerance rtol=4.44089e-13, atol=4.44089e-13 error for eigsh:general, typ=d, which=SA, sigma=0.5, mattype=asarray, OPpart=None, mode=buckling (mismatch 100.0%) x: array([[ 15.86892331, 0.0549568 ], [ 14.15864153, 0.31381369], [ 10.99691307, 0.37543458],... y: array([[ 3.19549052, 0.0549568 ], [ 2.79856422, 0.31381369], [ 1.67526354, 0.37543458],... ====================================================================== FAIL: test_arpack.test_symmetric_modes(True, <gen-symmetric>, 'd', 2, 'SA', None, 0.5, <function asarray at 0x10f5c2b90>, None, 'cayley') ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Python/2.7/site-packages/nose-1.1.2-py2.7.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 249, in eval_evec assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 1178, in assert_allclose verbose=verbose, header=header) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=4.44089e-13, atol=4.44089e-13 error for eigsh:general, typ=d, which=SA, sigma=0.5, mattype=asarray, OPpart=None, mode=cayley (mismatch 100.0%) x: array([[-0.36892684, -0.01935691], [-0.26850996, -0.11053158], [-0.40976156, -0.13223572],... y: array([[-0.43633077, -0.01935691], [-0.25161386, -0.11053158], [-0.36756684, -0.13223572],... ---------------------------------------------------------------------- Ran 5501 tests in 56.993s FAILED (KNOWNFAIL=13, SKIP=42, failures=76) <nose.result.TextTestResult run=5501 errors=0 failures=76>

2 1

Debian/Ubuntu patch help (was: ANN: NumPy 1.6.2 release candidate 1)
by Ralf Gommers Sept. 7, 2012

Sept. 7, 2012

On Sat, May 12, 2012 at 9:17 PM, Ralf Gommers <ralf.gommers(a)googlemail.com>wrote: > > > On Sat, May 12, 2012 at 6:22 PM, Sandro Tosi <matrixhasu(a)gmail.com> wrote: > >> Hello, >> >> On Sat, May 5, 2012 at 8:15 PM, Ralf Gommers >> <ralf.gommers(a)googlemail.com> wrote: >> > Hi, >> > >> > I'm pleased to announce the availability of the first release candidate >> of >> > NumPy 1.6.2. This is a maintenance release. Due to the delay of the >> NumPy >> > 1.7.0, this release contains far more fixes than a regular NumPy bugfix >> > release. It also includes a number of documentation and build >> improvements. >> > >> > Sources and binary installers can be found at >> > https://sourceforge.net/projects/numpy/files/NumPy/1.6.2rc1/ >> > >> > Please test this release and report any issues on the numpy-discussion >> > mailing list. >> ... >> > BLD: add support for the new X11 directory structure on Ubuntu & co. >> >> We've just discovered that this fix is not enough. Actually the new >> directories are due to the "multi-arch" feature of Debian systems, >> that allows to install libraries from other (foreign) architectures >> than the one the machine is (the classic example, i386 libraries on a >> amd64 host). >> >> the fix included to look up in additional directories is currently >> only for X11, while for example Debian has fftw3 that's >> multi-arch-ified and thus will fail to be detected. >> >> Could this fix be extended to include all other things that are >> checked? for reference the bug in Debian is [1]; there was also a >> patch[2] in previous versions, that was using gcc to get the >> multi-arch paths - you might use as a reference, or to implement >> something debian-systems-specific. >> >> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640940 >> [2] >> http://anonscm.debian.org/viewvc/python-modules/packages/numpy/trunk/debian… >> >> It would be awesome is such support would end up in 1.6.2 . >> > > Hardcoding some more paths to check in distutils/system_info.py should be > OK, also for 1.6.2 (will require a new RC). > > The --print-multiarch thing looks very questionable. As far as I can tell, > it's a Debian specific gcc patch, only available in gcc 4.6 and up. Ubuntu > before 11.10 release also doesn't have it. Therefore I don't think use of > --print-multiarch is appropriate for numpy for now, and certainly not a > change I'd like to make to distutils right before a release. > > If anyone with access to a Debian/Ubuntu system could come up with a patch > which adds the right paths to system_info.py, that would be great. > Hi, if there's anyone wants to have a look at the above issue this week, that would be great. If there's a patch by this weekend I can create a second RC, so we can still have the final release before the end of this month (needed for Debian freeze). Otherwise a second RC won't be needed. Ralf

3 2