Mailman 3 July 2004 - NumPy-Discussion

bug in numarray.maximum.reduce ?
by Sebastian Haase 07 Jul '04

07 Jul '04

Hi, Is this a bug?: >>> # (import numarray as na ; 'd' is a 3 dimensional array) >>> d.type() Float32 >>> d[80, 136, 122] 80.3997039795 >>> na.maximum.reduce(d[:,136, 122]) 85.8426361084 >>> na.maximum.reduce(d) [136, 122] 37.3658103943 >>> >>> >>> na.maximum.reduce(d,0)[136, 122] 37.3658103943 >>> na.maximum.reduce(d,1)[136, 122] Traceback (most recent call last): File "<input>", line 1, in ? IndexError: Index out of range I was using na.maximum.reduce(d) to get a "pixelwise" maximum along Z (axis 0). But as seen above it does not get it right. I then tried to reproduce this with some simple arrays, but here it works just fine: >>> a = na.arange(4*4*4) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[48 49 50 51] [52 53 54 55] [56 57 58 59] [60 61 62 63]] >>> a = na.arange(4*4*4).astype(na.Float32) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[ 48. 49. 50. 51.] [ 52. 53. 54. 55.] [ 56. 57. 58. 59.] [ 60. 61. 62. 63.]] >>> Any hint ? Regards, Sebastian Haase

3 9

Re: Thanks!
by numpy-discussion＠lists.sourceforge.net 07 Jul '04

07 Jul '04

------------------ Virus Warning Message (on octopussy) Found virus WORM_NETSKY.D in file message_part2.pif The uncleanable file is deleted. ---------------------------------------------------------

1 0

Undeliverable: Re: Thanks!
by System Administrator 06 Jul '04

06 Jul '04

Your message To: jacques.heliot(a)framatome-anp.com Subject: Re: Thanks! Sent: Wed, 7 Jul 2004 08:56:14 +0200 did not reach the following recipient(s): jacques.heliot(a)framail.framatome-anp.com on Wed, 7 Jul 2004 08:56:06 +0200 The recipient name is not recognized The MTS-ID of the original message is: c=fr;a= ;p=fragroup;l=FPARI01MXB0407070656LWFRLMFV MSEXCH:IMS:FRAGROUP:FRAANP-FR-PARIS-PARIS:FPARI01MXB 0 (000C05A6) Unknown Recipient

1 0

RE: ANN: numarray-1.0 released
by Todd Miller 06 Jul '04

06 Jul '04

On Tue, 2004-07-06 at 02:59, jjm(a)tiscali.dk wrote: > > The PEP is now in > > numarray-1.0/Doc/header_pep.txt in docutils format. We want feedback > > and consensus before we submit to python-dev so please consider > > reading it and commenting. > > I can't find header_pep.txt! It is not in numarray-1.0.tar.gz. Oops, you're right. I attached it. Apparently I forgot to add it to CVS. Todd

1 0

Missing header_pep.txt
by Todd Miller 06 Jul '04

06 Jul '04

Somehow header_pep.txt didn't make it into the numarray-1.0 source tar-ball. It's now in CVS and also attached. Regards, Todd

1 0

inconsistencies between docs and C headers?
by Curzio Basso 06 Jul '04

06 Jul '04

Hi all, can someone explain me why in the docs functions like NA_NewArray() return a PyObject*, while in the headers they return a PyArrayObject*? Is it just the documentation which is slow to catch up with the development? Or am i missing something? thanks, curzio

2 1

ANN: numarray-1.0 released
by Todd Miller 02 Jul '04

02 Jul '04

Release Notes for numarray-1.0 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modeled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, and improved type promotions. I. ENHANCEMENTS 1. User added ufuncs There's a setup.py file in numarray-1.0/Examples/ufunc which demonstrates how a numarray user can define their own universal functions of one or two parameters. Ever wanted to write your own bessel() function for use on arrays? Now you can. Your ufunc can use exactly the same machinery as add(). 2. Ports of Numeric functions A bunch of Numeric functions were ported to numarray in the new libnumeric module. To get these import from numarray.numeric. Most notable among these are put, putmask, take, argmin, and argmax. Also added were sort, argsort, concatenate, repeat and resize. These are independent ports/implementations in C done for the purpose of best Numeric compatibility and small array performance. The numarray versions, which handle additional cases, still exist and are the default in numarray proper. 3. Faster matrix multiply The setup for numarray's matrix multiply was moved into C-code. This makes it faster for small matrices. 4. The numarray "header PEP" A PEP has been started for the inclusion of numarray (and possibly Numeric) C headers into the Python core. The PEP will demonstrate how to provide optional support for arrays (the end-user may or may not have numarray installed and the extension will still work). It may also (eventually) demonstrate how to build extensions which support both numarray and Numeric. Thus, the PEP is seeking to make it possible to distribute extensions which will still compile when numarray (or either) is not present in a user's Python installation, which will work when numarry (or either) is not installed, and which will improve performance when either is installed. The PEP is now in numarray-1.0/Doc/header_pep.txt in docutils format. We want feedback and consensus before we submit to python-dev so please consider reading it and commenting. For the PEP, the C-API has been partitioned into two parts: a relatively simple Numeric compatible part and the numarray native part. This broke source and binary compatibility with numarray-0.9. See CAUTIONS below for more information. 5. Changes to the manual There are now brief sections on numarray.mlab and numarray.objects in the manual. The discussion of the C-API has been updated. II. CAUTIONS 1. The numarray-1.0 C-API is neither completely source level nor binary compatible with numarray-0.9. First, this means that some 3rd party extensions will no longer compile without errors. Second, this means that binary packages built against numarray-0.9 will fail, probably disastrously, using numarray-1.0. Don't install numarray-1.0 until you are ready to recompile or replace your extensions with numarray-1.0 binaries because 0.9 binaries will not work. In order to support the header PEP, the numarray C-API was partitioned into two parts: Numeric compatible and numarry extensions. You can use the Numeric compatible API (the PyArray_* functions) by including arrayobject.h and calling import_array() in your module init function. You can use the extended API (the NA_* functions) by including libnumarray.h and calling import_libnumarray() in your init function. Because of the partitioning, all numarray extensions must be recompiled to work with 1.0. Extensions using *both* APIs must include both files in order to compile, and must do both imports in order to run. Both APIs share a common PyArrayObject struct. 2. numarray extension writers should note that the documented use of PyArray_INCREF and PyArray_XDECREF (in numarray) was found to be incompatible with Numeric and these functions have therefore been removed from the supported API and will now result in errors. 3. The numarray.objects.ObjectArray parameter order was changed. 4. The undocumented API function PyArray_DescrFromTypeObj was removed from the Numeric compatible API because it is not provided by Numeric. III. BUGS FIXED / CLOSED See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. 979834 convolve2d parameter order issues 979775 ObjectArray parameter order 979712 No exception for invalid axis 979702 too many slices fails silently 979123 A[n:n] = x no longer works 979028 matrixmultiply precision 976951 Unpickled numarray types unsable? 977472 CharArray concatenate 970356 bug in accumulate contiguity status 969162 object array bug/ambiguity 963921 bitwise_not over Bool type fails 963706 _reduce_out: problem with otype 942804 numarray C-API include file 932438 suggest moving mlab up a level 932436 mlab docs missing 857628 numarray allclose returns int 839401 Argmax's behavior has changed for ties 817348 a+=1j # Not converted to complex 957089 PyArray_FromObject dim check broken 923046 numarray.objects incompatibility 897854 Type conflict when embedding on OS X 793421 PyArray_INCREF / PyArray_XDECREF deprecated 735479 Build failure on Cygwin 1.3.22 (very current install). 870660 Numarray: CFLAGS build problem 874198 numarray.random_array.random() broken? 874207 not-so random numbers in numarray.random_array 829662 Downcast from Float64 to UInt8 anomaly 867073 numarray diagonal bug? 806705 a tale of two rank-0's 863155 Zero size numarray breaks for loop 922157 argmax returns integer in some cases 934514 suggest nelements -> size 953294 choose bug 955314 strings.num2char bug? 955336 searchsorted has strange behaviour 955409 MaskedArray problems 953567 Add read-write requirement to NA_InputArray 952705 records striding for > 1D arrays 944690 many numarray array methods not documented 915015 numarray/Numeric incompatabilities 949358 UsesOpPriority unexpected behavior 944678 incorrect help for "size" func/method 888430 NA_NewArray() creates array with wrong endianess 922798 The document Announce.txt is out of date 947080 numarray.image.median bugs 922796 Manual has some dated MA info 931384 What does True mean in a mask? 931379 numeric.ma called MA in manual 933842 Bool arrays don't allow bool assignment 935588 problem parsing argument "nbyte" in callStrideConvCFunc() 936162 problem parsing "nbytes" argument in copyToString() 937680 Error in Lib/numerictypes.py ? 936539 array([cmplx_array, int_array]) fails 936541 a[...,1] += 0 crashes interpreter. 940826 Ufunct operator don't work 935882 take for character arrays? 933783 numarray, _ufuncmodule.c: problem setting buffersize 930014 fromstring typecode param still broken 929841 searchsorted type coercion 924841 numarray.objects rank-0 results 925253 numarray.objects __str__ and __repr__ 913782 Minor error in chapter 12: NUM_ or not? 889591 wrong header file for C extensions 925073 API manual comments 924854 take() errors 925754 arange() with large argument crashes interpreter 926246 ufunc reduction crash 902153 can't compile under RH9/gcc 3.2.2 916876 searchsorted/histogram broken in versions 0.8 and 0.9 920470 numarray arange() problem 915736 numarray-0.9: Doc/CHANGES not up to date WHERE ----------- Numarray-1.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen K"upper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, and everyone else who has contributed with comments, bug reports, or patches. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller(a)stsci.edu

1 0

numarray and SMP
by Christopher T King 02 Jul '04

02 Jul '04

(I originally posted this in comp.lang.python and was redirected here) In a quest to speed up numarray computations, I tried writing a 'threaded array' class for use on SMP systems that would distribute its workload across the processors. I hit a snag when I found out that since the Python interpreter is not reentrant, this effectively disables parallel processing in Python. I've come up with two solutions to this problem, both involving numarray's C functions that perform the actual vector operations: 1) Surround the C vector operations with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, thus allowing the vector operations (which don't access Python structures) to run in parallel with the interpreter. Python glue code would take care of threading and locking. 2) Move the parallelization into the C vector functions themselves. This would likely get poorer performance (a chain of vector operations couldn't be combined into one threaded operation). I'd much rather do #1, but will playing around with the interpreter state like that cause any problems? Update from original posting: I've partially implemented method #1 for Float64s. Running on four 2.4GHz Xeons (possibly two with hyperthreading?), I get about a 30% speedup while dividing 10 million Float64s, but a small (<10%) slowdown doing addition or multiplication. The operation was repeated 100 times, with the threads created outside of the loop (i.e. the threads weren't recreated for each iteration). Is there really that much overhead in Python? I can post the code I'm using and the numarray patch if it's requested.

2 2

Speeding up wxPython/numarray
by Tim Hochberg 01 Jul '04

01 Jul '04

I spend some time seeing what I could do in the way of speeding up wxPoint_LIST_helper by tweaking the numarray code. My first suspect was _universalIndexing by way of _ndarray_item. However, due to some new-style machinations, _ndarray_item was never getting called. Instead, _ndarray_subscript was being called. So, I added a special case to _ndarray_subscript. This sped things up by 50% or so (I don't recall exactly). The code for that is at the end of this message; it's not gauranteed to be 100% correct; it's all experimental. After futzing around some more I figured out a way to trick python into using _ndarray_item. I added "type->tp_as_sequence->sq_item = _ndarray_item;" to _ndarray new. I then optimized _ndarray_item (code at end). This halved the execution time of my arbitrary benchmark. This trick may have horrible, unforseen consequences so use at your own risk. Finally I commented out the __del__ method numarraycore. This resulted in an additional speedup of 64% for a total speed up of 240%. Still not close to 10x, but a large improvement. However, this is obviously not viable for real use, but it's enough of a speedup that I'll try to see if there's anyway to move the shadow stuff back to tp_dealloc. In summary: Version Time Rel Speedup Abs Speedup Stock 0.398 ---- ---- _naarray_item mod 0.192 107% 107% del __del__ 0.117 64% 240% There were a couple of other things I tried that resulted in additional small speedups, but the tactics I used were too horrible to reproduce here. The main one of interest is that all of the calls to NA_updateDataPtr seem to burn some time. However, I don't have any idea what one could do about that. That's all for now. -tim static PyObject* _ndarray_subscript(PyArrayObject* self, PyObject* key) { PyObject *result; #ifdef TAH if (PyInt_CheckExact(key)) { long ikey = PyInt_AsLong(key); long offset; if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); } #endif #if _PYTHON_CALLBACKS result = PyObject_CallMethod( (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); #else result = _universalIndexing(self, key, Py_None); #endif return result; } static PyObject * _ndarray_item(PyArrayObject *self, int i) { #ifdef TAH long offset; if (NA_getByteOffset(self, 1, &i, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); #else PyObject *result; PyObject *key = PyInt_FromLong(i); if (!key) return NULL; result = _universalIndexing(self, key, Py_None); Py_DECREF(key); return result; #endif }

3 6