Mailman 3 December 2011 - NumPy-Discussion

structured array with submember assign
by Fabian Dill 26 Dec '11

26 Dec '11

2 2

Baffling TypeError using fromfunction with explicit cast
by Thomas Coffee 25 Dec '11

25 Dec '11

Somewhat new to NumPy, but I've been investigating this for over an hour and found nothing helpful: Can anyone explain why this works ... >>> import numpy >>> numpy.fromfunction(lambda i, j: i*j, (3,4)) array([[ 0., 0., 0., 0.], [ 0., 1., 2., 3.], [ 0., 2., 4., 6.]]) ... but neither of these do? >>> numpy.fromfunction(lambda i, j: float(i*j), (3,4)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/pymodules/python2.7/numpy/core/numeric.py", line 1617, in fromfunction return function(*args,**kwargs) File "<stdin>", line 1, in <lambda> TypeError: only length-1 arrays can be converted to Python scalars >>> numpy.fromfunction(lambda i, j: numpy.float(i*j), (3,4)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/pymodules/python2.7/numpy/core/numeric.py", line 1617, in fromfunction return function(*args,**kwargs) File "<stdin>", line 1, in <lambda> TypeError: only length-1 arrays can be converted to Python scalars Given that fromfunction casts the return values to float by default, why does it break when this cast is made explicit? The motivation for the question is to be able to use fromfunction with a function that can return infinity, which I only know how to create with the explicit cast float('inf'). Thanks in advance for your help! - Thomas

1 1

Change in __array_priority__ behavior breaks code.
by Charles R Harris 24 Dec '11

24 Dec '11

Hi All, The following change breaks existing code, my polynomial classes for example. File: numpy/core/src/umath/ufunc_object.c Previous Code: /* * FAIL with NotImplemented if the other object has * the __r<op>__ method and has __array_priority__ as * an attribute (signalling it can handle ndarray's) * and is not already an ndarray or a subtype of the same type. */ if ((arg_types[1] == PyArray_OBJECT) && (loop->ufunc->nin==2) && (loop->ufunc->nout == 1)) { PyObject *_obj = PyTuple_GET_ITEM(args, 1); if (!PyArray_CheckExact(_obj) /* If both are same subtype of object arrays, then proceed */ && !(Py_TYPE(_obj) == Py_TYPE(PyTuple_GET_ITEM(args, 0))) && PyObject_HasAttrString(_obj, "__array_priority__") && _has_reflected_op(_obj, loop->ufunc->name)) { loop->notimplemented = 1; return nargs; } } Current Code: /* * FAIL with NotImplemented if the other object has * the __r<op>__ method and has __array_priority__ as * an attribute (signalling it can handle ndarray's) * and is not already an ndarray or a subtype of the same type. */ if (nin == 2 && nout == 1 && dtypes[1]->type_num == NPY_OBJECT) { PyObject *_obj = PyTuple_GET_ITEM(args, 1); if (!PyArray_CheckExact(_obj)) { double self_prio, other_prio; self_prio = PyArray_GetPriority(PyTuple_GET_ITEM(args, 0), NPY_SCALAR_PRIORITY); other_prio = PyArray_GetPriority(_obj, NPY_SCALAR_PRIORITY); if (self_prio < other_prio && _has_reflected_op(_obj, ufunc_name)) { retval = -2; goto fail; } } } The difference is that the previous code ignored the value of the priority. I think that is the best thing to do, as trying to use the priority to decide the priority between objects that aren't ndarray or subclasses of ndarray is likely to lead to problems. Because there is no central repository of priorities from which to allocate them, the priorities are essentially arbitrary. I think the old approach makes more sense in this context. Even in the old code the restriction to a single output seems excessive and will fail with divmod. I thought I'd raise this issue on the list because it does reflect a design decision that should be made at some point. For the current development branch I think the code should be reverted simply because it breaks backward compatibility. Chuck

1 0

numpy int64_t incompatibility on OSX
by Adam Klein 23 Dec '11

23 Dec '11

I am cross-posting this question; I think this question is more numpy-related than cython-related. Original question is at bottom. In OSX 10.6, C system headers such as /usr/include/i386/types.h (as well as other standard headers like C99 stdint.h) define int64_t as typedef *long long* int64_t; Whereas npy_common.h defines the following incompatible type when included (I traced it to line 258 via the gcc -E switch): typedef *long* npy_int64 As a result, I get conflicts when mixing standard library and numpy int64_t variables (during normal Cythoning on OSX). A quick hackaround, I can alter the npy_common.h file to look like this: #ifdef OSX_INT64_LONG_LONG typedef long long npy_int64; typedef unsigned long long npy_uint64; #else typedef long npy_int64; typedef unsigned long npy_uint64; #endif And define the OSX_INT64_LONG_LONG macro in my build, in setup.py. Has anyone had this sort of incompatibility on OSX? Suggestions as to correct way to deal with it? Thanks, Adam On Mon, Dec 19, 2011 at 1:37 PM, mark florisson <markflorisson88(a)gmail.com>wrote: > >> On 19 December 2011 00:04, Adam Klein <adam(a)lambdafoundry.com> wrote: >> > Hi all, >> > >> > I'm getting the following error: >> > >> > source.cpp:1969: error: invalid conversion from >> ‘__pyx_t_5numpy_int64_t*’ to >> > ‘int64_t*’ >> > >> > I am on OSX (10.6). Same make compiles fine on Linux. >> > >> > The problematic compiler command: >> > >> > /usr/bin/llvm-gcc -fno-strict-aliasing -fno-common -dynamic -arch i386 >> -arch >> > x86_64 -O3 -march=core2 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall >> > -Wstrict-prototypes >> > >> -I/Volumes/HD2/adam/.virtualenvs/py27/lib/python2.7/site-packages/numpy/core/include >> > >> -I/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/include/python2.7 >> > -c source.cpp -o build/temp.macosx-10.5-intel-2.7/source.o >> > >> > I figured out that if I take out the flag "-arch x86_64", it builds. >> > >> > I don't have any ARCHFLAGS or CFLAGS defined otherwise in my >> environments. >> > >> > Any ideas? >> > >> > Thanks! >> > >> > --Adam >> >> Hm, could you paste the full compiler output and possibly the source code? >> > > I figured out, using the -E switch of gcc, that /usr/include/i386/types.h > is being included, which defines int64_t as > > #ifndef _INT64_T > > #define _INT64_T > typedef long long int64_t; > #endif > > It seems to be chosen from /usr/include/machine, > > #elif defined (__i386__) || defined(__x86_64__) > #include "i386/types.h" > > This matches the stdint.h definition in /usr/include/stdint.h. > > This all conflicts with the npy_common.h definition, > > #elif NPY_BITSOF_LONG == 64 > #define NPY_INT64 NPY_LONG > #define NPY_UINT64 NPY_ULONG > typedef long npy_int64; > > I'm still trying to find an OSX header int64_t definition that isn't > defined as "long long"... >

1 0

EPD 7.2 released
by Ilan Schnell 23 Dec '11

23 Dec '11

Hello, I am pleased to announce the release of Enthought Python Distribution, EPD version 7.2, along with its "EPD Free" counterpart. The highlights of this release are: the addition of GDAL and updates to over 30 packages, including SciPy, matplotlib and IPython. The new IPython 0.12 includes the HTML notebook, which caused the Tornado web-server also to be added to EPD. To see which libraries are included in the free vs. full version, please see: http://www.enthought.com/products/epdlibraries.php The complete list of additions, updates and fixes is in the change log: http://www.enthought.com/products/changelog.php About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python programming language, including over 90 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization tools, and many other tools. EPD is currently available as a single-click installer for Windows XP, Vista and 7, MacOS (10.5 and 10.6), RedHat 3, 4, 5 and 6, as well as Solaris 10 (x86 and x86_64/amd64 on all platforms). All versions of EPD (32 and 64-bit) are free for academic use. An annual subscription including installation support is available for individual and commercial use. Additional support options, including customization, bug fixes and training classes are also available: http://www.enthought.com/products/epd_sublevels.php - Ilan

1 0

Owndata flag
by Fabrice Silva 23 Dec '11

23 Dec '11

How can one arbitrarily assumes that an ndarray owns its data ? More explicitly, I have some temporary home-made C structure that holds a pointer to an array. I prepare (using Cython) an numpy.ndarray using the PyArray_NewFromDescr function. I can delete my temporary C structure without freeing the memory holding array, but I wish the numpy.ndarray becomes the owner of the data. How can do I do such thing ? -- Fabrice Silva

6 8

numpy errors on MacOS but works on Ubuntu
by Håkan Jonsson 22 Dec '11

22 Dec '11

Hi, I am running the same code on Ubuntu 10.04 and MacOS 10.7 with different results. On Ubuntu, it reads a dataset from a .mat Matlab file, and then tries to access specific values. On MacOS I get: Done loading matlab data. Traceback (most recent call last): File "parse_proximity.py", line 79, in <module> get_events(matlab_obj) File "parse_proximity.py", line 18, in get_events subject_mac = int(subject_object.my_mac[0][0][0], 16) AttributeError: 'numpy.void' object has no attribute 'my_mac' The code looks like this: import scipy.io from datetime import datetime, timedelta import time import sys, os start_date = 1095984000 end_date = 1096004000 def get_events(matlab_obj): subjects = matlab_obj["s"][0] for subject_object in subjects: valid = True try: subject_mac = int(subject_object.my_mac[0][0][0], 16) ... if __name__ == "__main__": matlab_filename = sys.argv[1] matlab_obj = scipy.io.loadmat(matlab_filename) print "Done loading matlab data." get_events(matlab_obj) I have tried two different versions of python 2.6.1 and 2.7.1 with same results. I don't have any other problems with using python on MacOS. Any suggestions? Regards, Håkan

1 0

Binning
by Nicola Creati 22 Dec '11

22 Dec '11

Hello, I have a cloud on sparse points that can be described by a Nx3 array (N is the number of points). Each point is defined by an x, y and z coordinate: x0 y0 z0 x1 y1 z1 . . . . . . . . . xn yn zn I need to bin the cloud to a regular 2D array according to a desired bin size assigning to each cell (bin) the minimum z of all points that fall in that cell(bin). Moreover I need indexes of points that fall in each cell(bin). Is there any way to accomplish this task in numpy? Thanks. Nicola Creati -- Nicola Creati Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS www.inogs.it Dipartimento di Geofisica della Litosfera Geophysics of Lithosphere Department CARS (Cartography and Remote Sensing) Research Group http://www.inogs.it/Cars/ Borgo Grotta Gigante 42/c 34010 Sgonico - Trieste - ITALY ncreati(a)ogs.trieste.it off. +39 040 2140 213 fax. +39 040 327307 _____________________________________________________________________ This communication, that may contain confidential and/or legally privileged information, is intended solely for the use of the intended addressees. Opinions, conclusions and other information contained in this message, that do not relate to the official business of OGS, shall be considered as not given or endorsed by it. Every opinion or advice contained in this communication is subject to the terms and conditions provided by the agreement governing the engagement with such a client. Any use, disclosure, copying or distribution of the contents of this communication by a not-intended recipient or in violation of the purposes of this communication is strictly prohibited and may be unlawful. For Italy only: Ai sensi del D.Lgs.196/2003 - "T.U. sulla Privacy" si precisa che le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario. _____________________________________________________________________

4 5

unexpected behavior of __array_wrap__ in matrix subclass
by Aronne Merrelli 22 Dec '11

22 Dec '11

Hello NumPy list, While experimenting with a subclass of numpy.matrix, I discovered cases where __array_wrap__ is not called during multiplication. I'm not sure whether this is a bug or my own misunderstanding of np.matrix & __array_wrap__; if nothing else I thought it would be helpful to describe this in case other people run into the same problem. If the matrix is created from an array of integers, then __array_wrap__ is called if the matrix is multiplied by an integer. It appears that in all other cases, __array_wrap__ is not called for multiplication (int times float scalar, float times float scalar, float matrix times float matrix, etc). For addition, __array_wrap__ is called for all cases that I checked. I did find a possible workaround. If you define a __mul__ method in the matrix subclass, and then just call np.multiply, then __array_wrap__ is called in all cases I expect it to be called. I uploaded a example script here: https://gist.github.com/1511354 Hopefully it is not too confusing. I'm basically abusing the python exception handler to tell whether or not __array_wrap__ is called for any particular case. The MatSubClass shows the problem, and the MatSubClassFixed as the __mul__ method defined. Here are the results I see in my working environment (ipython in EPD 7.1): In [1]: np.__version__ Out[1]: '1.6.0' In [2]: execfile('matrix_array_wrap_test.py') In [3]: run_test() array_wrap called for o2 = o * 2 after o=MatSubClass([1,1]) array_wrap NOT called for o2 = o * 2.0 after o=MatSubClass([1,1]) array_wrap NOT called for o2 = o * 2 after o=MatSubClass([1.0, 1.0]) array_wrap NOT called for o2 = o * 2.0 after o=MatSubClass([1.0, 1.0]) array_wrap called for o2 = o * 2 after o=MatSubClassFixed([1,1]) array_wrap called for o2 = o * 2.0 after o=MatSubClassFixed([1,1]) array_wrap called for o2 = o * 2 after o=MatSubClassFixed([1.0, 1.0]) array_wrap called for o2 = o * 2.0 after o=MatSubClassFixed([1.0, 1.0]) Thanks, Aronne

1 0

output different columns of ndarray in different formats
by Chao YUE 22 Dec '11

22 Dec '11

Dear all, Just a small question, how can I output different columns of ndarray in different formats, the manual says, "A single format (%10.5f), a sequence of formats, or a multi-format string" but I use np.savetxt('new.csv',data,fmt=['%i4','%f6.3']) or np.savetxt('new.csv',data,fmt=('%i4','%f6.3')) give strange results. In [33]: data.shape Out[33]: (6506, 2) I want the first column integer and second column float. cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 2