Mailman 3 January 2013 - NumPy-Discussion

natural alignment
by Henry Gomersall Jan. 9, 2013

Jan. 9, 2013

Further to my previous emails about getting SIMD aligned arrays, I've noticed that numpy arrays aren't always naturally aligned either. For example, numpy.float96 arrays are not always aligned on 12-byte boundaries under 32-bit linux/gcc. Indeed, .alignment on the array always seems to return 4 (with 64-bit, .alignment returns 4, 8, and 16 for float32, float64 and longdouble respectively). Can I assume _anything_ in general about the alignment of a numpy array? (I mean, based on what all implementations of the underlying malloc etc will return). Should I rely on what is returned from .alignment? cheers, Henry

1 0

Bug with ufuncs made with frompyfunc
by OKB (not okblacke) Jan. 9, 2013

Jan. 9, 2013

A bug causing errors with using methods of ufuncs created with frompyfunc was mentioned on the list over a year ago: http://mail.scipy.org/pipermail/numpy-discussion/2011- September/058501.html Is there any word on the status of this bug? I wasn't able to find a ticket in the bug tracker.

2 1

Embedded NumPy LAPACK errors
by mw＠eml.cc Jan. 8, 2013

Jan. 8, 2013

Hiall, I am trying to embed numerical code in a mexFunction, as called by MATLAB, written as a Cython function. NumPy core functions and BLAS work fine, but calls to LAPACK function such as SVD seem to be made against to MATLAB's linked MKL, and this generates MKL errors. When I try this with Octave, it works fine, presumably because it is compiled against the same LAPACK as the NumPy I am embedding. Assuming I haven't made big mistakes up to here, I have the following questions: Is there a way to request numpy.linalg to use a particular LAPACK library, e.g. /usr/lib/liblapack.so ? If not, is there a reasonable way to build numpy.linalg such that it interfaces with MKL correctly ? thanks in advance for any help, Marmaduke [1] The Cython code in question : https://gist.github.com/4433635 Please see the mexFunction at the bottom.

3 3

Re: [Numpy-discussion] Which Python to use for Mac binaries
by Robert Lupton the Good Jan. 7, 2013

Jan. 7, 2013

I am sympathetic with this attitude ("Avoid using system Python for anything"), but I don't think it's the right one. For example, the project I'm working on (HSC/LSST for astrofolk) is using python/C++ for astronomical imaging, and we expect to have the code running on a significant number of end-user laptops. If the instructions start out with: 0. Install a new version of python it's a significant barrier. What if they've already involved other packages into the system python? Python is a central part of modern operating systems, and people should not have to manage two versions of python to use numpy. It's tempting to say, "First install g++ 4.7 so we can use C++11 features" it's simply not viable, and I think that saying, "first install a new python" is comparable. Yes, I know that you can have more than one python/compiler suite installed simultaneously, but that's not something for casual users to have to get involved in. R > On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: >>> Which exact Python do we need to use on Mac? Do we need to use the >>> binary installer from python.org? ... > >> Avoid using system Python for anything. The first thing to do on any new OS >> X system is install Python some other way, preferably from python.org. > > +1

2 1

ANN: HDF5 for Python (h5py) 2.1.1
by Andrew Collette Jan. 7, 2013

Jan. 7, 2013

Announcing HDF5 for Python (h5py) 2.1.1 ======================================= HDF5 for Python 2.1.1 is now available! This bugfix release also marks a number of changes for the h5py project intended to make the development process more responsive, including a move to GitHub and a switch to a rapid release model. Development has moved over to GitHub at http://github.com/h5py/h5py. We welcome bug reports and pull requests from anyone interested in contributing. Releases will now be made every 4-6 weeks, in order to get bugfixes and new features out to users quickly while still leaving time for testing. * New main website: http://www.h5py.org * Mailing list: http://groups.google.com/group/h5py What is h5py? ============= The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. H5py uses straightforward NumPy and Python metaphors, like dictionary and NumPy array syntax. For example, you can iterate over datasets in a file, or check out the .shape or .dtype attributes of datasets. You don't need to know anything special about HDF5 to get started. In addition to the easy-to-use high level interface, h5py rests on a object-oriented Cython wrapping of the HDF5 C API. Almost anything you can do from C in HDF5, you can do from h5py. Best of all, the files you create are in a widely-used standard binary format, which you can exchange with other people, including those who use programs like IDL and MATLAB. What's new in 2.1.1? ==================== This is a bugfix release. The most substantial changes were: * Fixed a memory leak related to variable-length strings (Thanks to Luke Campbell for extensive testing and bug reports) * Fixed a threading deadlock related to the use of H5Aiterate * Fixed a double INCREF memory leak affecting Unicode variable-length strings * Fixed an exception when taking the repr() of objects with non-ASCII names

1 0

high dimensional array -> python scalar/index
by Sebastian Berg Jan. 6, 2013

Jan. 6, 2013

Question for everyone, is this really reasonable: >>> import numpy as np >>> from operator import index >>> index(np.array([[5]])) 5 >>> int(np.array([[5]])) 5 >>> [0,1,2,3][np.array([[2]])] 2 To me, this does not make sense, why should we allow to use a high dimensional object like a normal scalar (its ok for 0-d arrays I guess)? Personally I would be for deprecating these usages, even if that (probably) means you cannot reshape your array with a matrix (as it is 2D) ;-): >>> np.arange(10).reshape(np.matrix([5,-1]).T) array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]])

2 2

Rank-0 arrays - reprise
by Matthew Brett Jan. 6, 2013

Jan. 6, 2013

Hi, Following on from Nathaniel's explorations of the scalar - array casting rules, some resources on rank-0 arrays. The discussion that Nathaniel tracked down on "rank-0 arrays"; it also makes reference to casting. The rank-0 arrays seem to have been one way of solving the problem of maintaining array dtypes other than bool / float / int: http://mail.scipy.org/pipermail/numpy-discussion/2002-September/001612.html Quoting from an email from Travis in that thread, replying to an email from Tim Hochberg: http://mail.scipy.org/pipermail/numpy-discussion/2002-September/001647.html <quote> > Frankly, I have no idea what the implimentation details would be, but > could we get rid of rank-0 arrays altogether? I have always simply found > them strange and confusing... What are they really neccesary for > (besides holding scalar values of different precision that standard > Pyton scalars)? With new coercion rules this becomes a possibility. Arguments against it are that special rank-0 arrays behave as more consistent numbers with the rest of Numeric than Python scalars. In other words they have a length and a shape and one can right N-dimensional code that works the same even when the result is a scalar. Another advantage of having a Numeric scalar is that we can control the behavior of floating point operations better. e.g. if only Python scalars were available and sum(a) returned 0, then 1 / sum(a) would behave as Python behaves (always raises error). while with our own scalars 1 / sum(a) could potentially behave however the user wanted. </quote> There seemed then to be some impetus to remove rank-0 arrays and replace them with Python scalar types with the various numpy precisions : http://mail.scipy.org/pipermail/numpy-discussion/2002-September/013983.html Travis' recent email hints at something that seems similar, but I don't understand what he means: http://mail.scipy.org/pipermail/numpy-discussion/2012-December/064795.html <quote> Don't create array-scalars. Instead, make the data-type object a meta-type object whose instances are the items returned from NumPy arrays. There is no need for a separate array-scalar object and in fact it's confusing to the type-system. I understand that now. I did not understand that 5 years ago. </quote> Travis - can you expand? I remember rank-0 arrays being confusing in that I sometimes get a python scalar and sometimes a numpy scalar, and I may want a python scalar, and have to special-case the rank-0 array, but I don't remember precisely why I needed the python scalar. Any other comments / records of rank-0 arrays being confusing? Best, Matthew

6 13

Re: [Numpy-discussion] Invalid value encoutered : how to, prevent numpy.where to do this?
by Eric Emsellem Jan. 5, 2013

Jan. 5, 2013

Thanks! This makes sense of course. And yes the operation I am trying to do is rather complicated so I need to rely on a prior selection. Now I would need to optimise this for large arrays and the code does go through these command line many many times. When I have to operate on the two different parts of the array, I guess just using the following is the fastest way (as you indicated) : result = np.empty_like(data) mask = (data == 0) result[mask] = 0.0 result[~mask] = 1.0/data[~mask] But if I only need to do this on one side of the selection, I guess I would just do: result = np.empty_like(data) mask = (data != 0) result[mask] += 1.0 / data[mask] I have tried using three version of "mask = " with the rest of the code being the same: 1- mask = where(data != 0) 2- mask = np.where(data != 0) 3- mask = (data != 0) and it looks like #3 is the fastest, then #2 (20% slower) then #1 (50% slower than #3). I am not sure why, but Is that making sense? Or is there even a faster way (for large data arrays, and complicated operations)? thanks Eric > If your operation doesn't factor like this though then you can always > use something more cumbersome like > result = np.empty_like(data) > mask = (data == 0) > result[mask] = 0 > result[~mask] = 1.0/data[~mask] > > Or in 1.7 this could be written > result = np.zeros_like(data) > np.divide(1.0, data, where=(data != 0), out=result) > > -n >

2 1

Re: [Numpy-discussion] Invalid value encoutered : how to, prevent numpy.where to do this?
by Eric Emsellem Jan. 5, 2013

Jan. 5, 2013

Thanks! This makes sense of course. And yes the operation I am trying to do is rather complicated so I need to rely on a prior selection. Now I would need to optimise this for large arrays and the code does go through these command line many many times. When I have to operate on the two different parts of the array, I guess just using the following is the fastest way (as you indicated) : result = np.empty_like(data) mask = (data == 0) result[mask] = 0.0 result[~mask] = 1.0/data[~mask] But if I only need to do this on one side of the selection, I guess I would just do: result = np.empty_like(data) mask = (data != 0) result[mask] += 1.0 / data[mask] I have tried using three version of "mask = " with the rest of the code being the same: 1- mask = where(data != 0) 2- mask = np.where(data != 0) 3- mask = (data != 0) and it looks like #3 is the fastest, then #2 (20% slower) then #1 (50% slower than #3). I am not sure why, but Is that making sense? Or is there even a faster way (for large data arrays, and complicated operations)? thanks Eric > If your operation doesn't factor like this though then you can always > use something more cumbersome like > result = np.empty_like(data) > mask = (data == 0) > result[mask] = 0 > result[~mask] = 1.0/data[~mask] > > Or in 1.7 this could be written > result = np.zeros_like(data) > np.divide(1.0, data, where=(data != 0), out=result) > > -n >

1 0

Re: [Numpy-discussion] Invalid value encoutered : how to, prevent numpy.where to do this?
by Eric Emsellem Jan. 5, 2013

Jan. 5, 2013

Thanks! This makes sense of course. And yes the operation I am trying to do is rather complicated so I need to rely on a prior selection. Now I would need to optimise this for large arrays and the code does go through these command line many many times. When I have to operate on the two different parts of the array, I guess just using the following is the fastest way (as you indicated) : result = np.empty_like(data) mask = (data == 0) result[mask] = 0.0 result[~mask] = 1.0/data[~mask] But if I only need to do this on one side of the selection, I guess I would just do: result = np.empty_like(data) mask = (data != 0) result[mask] += 1.0 / data[mask] I have tried using three version of "mask = " with the rest of the code being the same: 1- mask = where(data != 0) 2- mask = np.where(data != 0) 3- mask = (data != 0) and it looks like #3 is the fastest, then #2 (20% slower) then #1 (50% slower than #3). I am not sure why, but Is that making sense? Or is there even a faster way (for large data arrays, and complicated operations)? thanks Eric > If your operation doesn't factor like this though then you can always > use something more cumbersome like > result = np.empty_like(data) > mask = (data == 0) > result[mask] = 0 > result[~mask] = 1.0/data[~mask] > > Or in 1.7 this could be written > result = np.zeros_like(data) > np.divide(1.0, data, where=(data != 0), out=result) > > -n >

1 0