Mailman 3 November 2004 - NumPy-Discussion

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon May 16, 2011

May 16, 2011

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

in-situ resizing array in C
by Pearu Peterson Nov. 29, 2004

Nov. 29, 2004

Hi, I have a question to Numeric/Numarray experts: Let A be a contiguous one or two-dimensional array with shape (n,) or (m,n), respectively. Let k be a positive integer less than n. Reference count for A is 1. The objective is to execute A=A[:k] or A=A[:,:k], respctively, in C. Q: Is it safe to resize the array A by simply resetting its dimensions, that is, A->dimensions[0] = k; or A->dimensions[1] = k; if A->nd is 1 or 2, respectively, in C? This appears to work but may be there are some corner cases that I have overlooked. Would that work also when using Numarray? If not, how to resize (preferably in-situ) an array in C under the conditions given above? TIA, Pearu

2 1

(no subject)
by kevin lester Nov. 27, 2004

Nov. 27, 2004

Hi Alan, Yes, thank you. Shared elements is the out come I need. The problem I have now is how to return the shared elements back into the numarray.array([]) constructor. Numarray doesn't recognize the sets.Set([]) constructor as a python numeric type. I actually don't see any functionality/methodology to convert a sets.Set([]) back into any of the sequence types. I am also don't understand why this ability, to retrieve shared elements of two unequal arrays, is not an inherent numarry function/method. Thank you for your initial direction to the sets module, Kevin __________________________________ Do you Yahoo!? The all-new My Yahoo! - Get yours free! http://my.yahoo.com

2 1

Equality Testing
by kevin lester Nov. 26, 2004

Nov. 26, 2004

Hello, After spending a great deal of time researching how to do something as trivial as an equality test for arrrays of unequal lengths, I've given up to defer to your wisdom. a = array([19289 19291 19299 ..., 21966 21978 21990]) b = array([ 0 1 2 ..., 95956 95717 95768]) Evidently, where(equal... has no merit, as well as all the other expressions I've tried. Thank you for your help :) __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

Architecture management?
by Stephen Walton Nov. 23, 2004

Nov. 23, 2004

How are all of you, if you are at all, managing to keep ATLAS/numarray/Numeric/Scipy/etc up to date on a heterogenous network? I have to do this, really, and would like to do it with ATLAS support, but with Athlon XP, Athlon MP, P4's and a couple of older P3's in house, it is becoming both difficult to automate this and time consuming to do it manually. Not strictly a numpy question, of course, but I felt it likely others on this list have faced this problem. -- Stephen Walton, Professor, Dept. of Physics & Astronomy, Cal State Northridge stephen.walton(a)csun.edu

2 1

numarray: 3D Fourier transforms
by Christian Meesters Nov. 23, 2004

Nov. 23, 2004

Hi I only hope that you guys on this list can help me, for I didn't find a list specified on numarray. Well, may problem is that I want to write an algorithm where I combine amplitude data from one 3D-image volume with phase data from another. (Actually they are identical except for one small correction, that's why this should be legal.) Numarray only offers 1D and 2D FFTs. I could possibly implement a 3D transform, but if somebody already has written something like this, it would be very nice if one could point me on an URL for this. Thanks a lot in advance. Cheers Christian Meesters

2 2

numarray patch: use Apple's vecLib on Mac OS X
by Andrew Straw Nov. 23, 2004

Nov. 23, 2004

Hi all, Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do? Cheers! Andrew Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ---- # lapack_dirs = ['/home/jmiller/work/lib'] + lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])] ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",

5 7

ANNOUNCE: PyQwt-4.1
by Gerard Vermeulen Nov. 21, 2004

Nov. 21, 2004

I am proud to announce PyQwt-4.1. What is PyQwt? - it is a set of Python bindings for the Qwt C++ class library which extends the Qt framework with widgets for scientific, engineering and financial applications. It provides a widget to plot 2-dimensional data and various widgets to display and control bounded or unbounded floating point values. - it requires and extends PyQt, a set of Python bindings for Qt. - it really shines with either Numeric, numarray or both. Numeric and/or numarray extend the Python language with new data types which turn Python into an ideal system for numerical computing and experimentation, better than MatLab and IDL. - it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions (either Numeric, or numarray or both) and optionally SciPy in a GUI Python application or in an interactive Python session. - it runs on POSIX, MacOS/X and Windows (any operating system supported by Qt and Python). The home page of PyQwt is http://pyqwt.sourceforge.net. Main changes in PyQwt-4.1: 1. supports PyQt-3.13, -3.12, PyQt-3.11, and PyQt-3.10. 2. supports sip-4.1.1, -4.1, -4.0, sip-3.11, and sip-3.10. 3. supports Qt-3.3.3 downto -2.3.0. 4. based on Qwt-4.2.0. 5. either links with a shared (dll) Qwt-4.2.0 library or links statically with a slightly patched internal version of Qwt-4.2.0. Have fun -- Gerard Vermeulen

1 0

Error in NA_IoArray?
by Paulo J. S. Silva Nov. 19, 2004

Nov. 19, 2004

Hello, I am using swig to wrap some numerical code. I have managed to make swig understand and convert between numarrays and C arrays. However if I use NA_IoArray to make the convertion and pass an integer array when a double array is expected the changes are not copied back to the original array. Let me give a "minimal" example. When I want to wrap the following function: void inc(int len, double *v) { int i; for (i = 0; i < len; i++) v[i]++; } Swig generates the following code, that looks right to me. static PyObject *_wrap_inc(PyObject *self, PyObject *args) { PyObject *resultobj; int arg1 ; double *arg2 = (double *) 0 ; PyArrayObject *NAimg2 = NULL ; PyObject * obj1 = 0 ; if(!PyArg_ParseTuple(args,(char *)"iO:inc",&arg1,&obj1)) goto fail; { PyArrayObject *NAimg2 = NA_IoArray(obj1, tFloat64, NUM_C_ARRAY); if (!NAimg2) { printf("**** no (double) numarray *****\n"); SWIG_exception(SWIG_RuntimeError, "No double Array Found"); return 0; } arg2 = (double *) NA_OFFSETDATA(NAimg2); } inc(arg1,arg2); Py_INCREF(Py_None); resultobj = Py_None; { Py_XDECREF(NAimg2); } return resultobj; fail: { Py_XDECREF(NAimg2); } return NULL; } However if I run the following python code: >>> import test_na_ioarray >>> from numarray import * >>> v = zeros((5,)) >>> test_na_ioarray(5, v) >>> print v [0 0 0 0 0] Shouldn't the resultant array be full of ones (when casting the Float64 array of ones to int)? Thanks, Paulo Obs: The code runs fine if the original vector is already a Float64 array.

2 3

[Fwd: Re: [Numpy-discussion] Error in NA_IoArray?]
by Todd Miller Nov. 19, 2004

Nov. 19, 2004

Sorry... this is out of sequence but I forgot to reply-all.

1 0

KeyboardInterrupt shows strange behavior
by Norbert Nemec Nov. 18, 2004

Nov. 18, 2004

Hi there, I have the following dummy code, doing garbage until I interrupt it: ------------------------------------ from numarray import * A = zeros((100,100)) try: while 1: A = dot(A,A) except KeyboardInterrupt: print "Interrupted" -------------------------------------- Now, if I hit Ctrl-C to interrupt it, it behaves strangely. Most of the time, I see a message: -------------------- Exception exceptions.KeyboardInterrupt in <bound method NumArray.__del__ of array([[0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]])> ignored ------------------------- after which the program continues to run, unimpressed by the fact that I would have liked to stop it. Only one out of ten attempts of pressing Ctrl-C is successful (probably depending on the exact position in the code where I hit the keys) - but even then, the program does not quit cleanly with an exception but it crashes hard with a SegFault! I am using a self-compiled glibc including the patch by Andrew Straw (see http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2207861) Might it be that this patch causes the problem? Ciao, Nobbi

3 3

Re: Thank you!
by Numpy-discussion Nov. 17, 2004

Nov. 17, 2004

1 0

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 16, 2004

Nov. 16, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

"Head Nummie" Semi-Abdicates...
by Todd Miller Nov. 16, 2004

Nov. 16, 2004

Due to the importance of on-going and future maintenance for Numeric and the fact that my own work is limited to numarray, I think we should split the role of "Head Nummie" into "Head Numeric Nummie" and "Head numarray Nummie". This really amounts to a clarification of who is doing what. My impression is that Travis Oliphant is the one doing most or all of the current Numeric work, so he is the logical Head Numeric Nummie. I am the most active with fundamental numarray so I am the logical Head numarray Nummie. Presuming that Travis wants the title "Head Numeric Nummie", All those in favor say +1 or nothing All those opposed say -1

1 0

Re: Thanks :)
by Numpy-discussion Nov. 16, 2004

Nov. 16, 2004

1 0

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 15, 2004

Nov. 15, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

Numeric 23.6 and vecLib (Apple's optimized implementation of BLAS)
by Chris Barker Nov. 15, 2004

Nov. 15, 2004

Hi all, I'm forwarding this on from Bob Ippolito (with his permission). I'd love to see this patch applied to the next Numeric release, as well as the setup.py fixed so as not to try to default to atlas. OS-X may be the only common platform where you can count on a native BLAS being there. Let's us it! -Chris From Bob: I've patched Numeric 23.6's setup.py to link against Apple's BLAS implementation. It shouldn't cause any problems for other platforms: http://undefined.org/python/mpkg/patch/Numeric-23.6/setup.py -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Sobel question
by Bob Klimek Nov. 15, 2004

Nov. 15, 2004

Hi, I recently started playing with numarray and especially nd_image for processing grayscale and RGB images. nd_image looks very interesting and I'm looking forward to checking. My question deals with the axis parameter of the Sobel filter. What does the function return when axis=-1? I figured out that when axis=0, the function returns the vertical gradient component and when axis=1 the horizontal gradient component. Since -1 is the default I expected that it would return the gradient magnitude (sqrt(x*x + y*y)) but that's not the case. Bob

2 4

Re: Details
by numpy-discussion＠lists.sourceforge.net Nov. 12, 2004

Nov. 12, 2004

------------------ Virus Warning Message (on octopussy) Found virus WORM_NETSKY.D in file my_details.pif The uncleanable file is deleted. ---------------------------------------------------------

1 0

Use __slots__ with numarray?
by Jens Jorgen Mortensen Nov. 12, 2004

Nov. 12, 2004

A few days ago, I found this page: http://mail.python.org/pipermail/python-list/2004-May/220513.html where speedups for creating objects using __slots__ is demonstrated. I have read somewhere that creation of numarrays is much slower than creation of Numeric arrays - so maybe using __slots__ should be considered? Just a suggestion from a happy numarray user! JJ

2 1

Undeliverable: Re: Details
by System Administrator Nov. 11, 2004

Nov. 11, 2004

Your message To: jacques.heliot(a)framatome-anp.com Subject: Re: Details Sent: Fri, 12 Nov 2004 08:36:29 +0100 did not reach the following recipient(s): jacques.heliot(a)framail.framatome-anp.com on Fri, 12 Nov 2004 08:36:32 +0100 The recipient name is not recognized The MTS-ID of the original message is: c=fr;a= ;p=fragroup;l=FPARI01MXB0411120736VA6VLQB2 MSEXCH:IMS:FRAGROUP:FRAANP-FR-PARIS-PARIS:FPARI01MXB 0 (000C05A6) Unknown Recipient

1 0

RE: [Numpy-discussion] urgent problem
by Nadav Horesh Nov. 10, 2004

Nov. 10, 2004

I had also this problem undel linux. As I can recall dgesdd is a SVD routine that has no parallel in ATLAS. The simplest solution is to go back to Numeric 23.1 which uby default uses its intenal linear-algebra routine (slower, but working). The other possibility is to add lapack to "libraries_list". This worked for be, after some playing with the libraries_list order; I am running two versions of linux, each requires a bit different ordering of the libraries. Nadav. -----Original Message----- From: Hui Liu [mailto:hliu13@uic.edu] Sent: Thu 11-Nov-04 00:45 To: numpy-discussion(a)lists.sourceforge.net Cc: Subject: [Numpy-discussion] urgent problem Hi everybody, I am newbie for NumPY. I am frustrated by this problem deeply. Thanks for your help. I installed ATLAS in solaris 8 sparc station with gcc and g77. I also install Numarray and NumPY23.6 on this workstation. When I run one program. I met the following problem: " File "/usr/local/lib/python2.3/site-packages/Numeric/LinearAlgebra.py", line 8, in ? import lapack_lite ImportError: ld.so.1: /usr/local/bin/python: fatal: relocation error: file /usr/local/lib/python2.3/site-packages/Numeric/lapack_lite.so: symbol dgesdd_: referenced symbol not found". My program begins with the following: #!/usr/local/bin/python from Numeric import reshape, transpose, exp from LinearAlgebra import solve_linear_equations My Numric-23.6's setup.py is as following: undef_macros = [] # You might need to add a case here for your system if sys.platform in ['win32']: mathlibs = [] define_macros = [] undef_macros = ['HAVE_INVERSE_HYPERBOLIC'] elif sys.platform in ['mac', 'beos5']: mathlibs = [] # delete all but the first one in this list if using your own LAPACK/BLAS sourcelist = [os.path.join('Src', 'lapack_litemodule.c'), # os.path.join('Src', 'blas_lite.c'), # os.path.join('Src', 'f2c_lite.c'), # os.path.join('Src', 'zlapack_lite.c'), # os.path.join('Src', 'dlapack_lite.c') ] # set these to use your own BLAS; library_dirs_list = ['/export/home/hui/ATLAS/lib/SunOS_SunUSIII'] libraries_list = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c'] # if you also set `use_dotblas` (see below), you'll need: # ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c'] # set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct use_dotblas = 1 include_dirs = ['/export/home/hui/ATLAS/include'] # You may need to set this to find cblas.h # e.g. on UNIX using ATLAS this should be ['/usr/include/atlas']

1 0

How can I change the following code from Numeric to numarry?
by Tony H. Liu Nov. 10, 2004

Nov. 10, 2004

Hello, Thanks in advance. Because I met some problem with Numeric. #!/usr/local/bin/python from Numeric import reshape, transpose, exp from LinearAlgebra import solve_linear_equations ............... Yours, tony

2 1

Re: Thank you!
by Numpy-discussion Nov. 10, 2004

Nov. 10, 2004

1 0

urgent problem
by Hui Liu Nov. 10, 2004

Nov. 10, 2004

Hi everybody, I am newbie for NumPY. I am frustrated by this problem deeply. Thanks for your help. I installed ATLAS in solaris 8 sparc station with gcc and g77. I also install Numarray and NumPY23.6 on this workstation. When I run one program. I met the following problem: " File "/usr/local/lib/python2.3/site-packages/Numeric/LinearAlgebra.py", line 8, in ? import lapack_lite ImportError: ld.so.1: /usr/local/bin/python: fatal: relocation error: file /usr/local/lib/python2.3/site-packages/Numeric/lapack_lite.so: symbol dgesdd_: referenced symbol not found". My program begins with the following: #!/usr/local/bin/python from Numeric import reshape, transpose, exp from LinearAlgebra import solve_linear_equations My Numric-23.6's setup.py is as following: undef_macros = [] # You might need to add a case here for your system if sys.platform in ['win32']: mathlibs = [] define_macros = [] undef_macros = ['HAVE_INVERSE_HYPERBOLIC'] elif sys.platform in ['mac', 'beos5']: mathlibs = [] # delete all but the first one in this list if using your own LAPACK/BLAS sourcelist = [os.path.join('Src', 'lapack_litemodule.c'), # os.path.join('Src', 'blas_lite.c'), # os.path.join('Src', 'f2c_lite.c'), # os.path.join('Src', 'zlapack_lite.c'), # os.path.join('Src', 'dlapack_lite.c') ] # set these to use your own BLAS; library_dirs_list = ['/export/home/hui/ATLAS/lib/SunOS_SunUSIII'] libraries_list = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c'] # if you also set `use_dotblas` (see below), you'll need: # ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c'] # set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct use_dotblas = 1 include_dirs = ['/export/home/hui/ATLAS/include'] # You may need to set this to find cblas.h # e.g. on UNIX using ATLAS this should be ['/usr/include/atlas']

1 0

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 10, 2004

Nov. 10, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

Re: Hi
by Numpy-discussion Nov. 9, 2004

Nov. 9, 2004

1 0

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 8, 2004

Nov. 8, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

Re: Hello
by Numpy-discussion Nov. 8, 2004

Nov. 8, 2004

1 0

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 7, 2004

Nov. 7, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

ANN: PyTables-0.9 released
by Francesc Altet Nov. 5, 2004

Nov. 5, 2004

Announcing PyTables 0.9 ----------------------- I'm pleased to announce the availability of the latest incarnation of PyTables. On this release you will find a series of quite exciting new features, being the most important the indexing capabilities, in-kernel selections, support for complex datatypes and the possibility to modify values in both tables *and* arrays (yeah, finally :). What is ------- PyTables is a hierarchical database package designed to efficiently manage extremely large amounts of data (supports full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the peformance-critical parts of the code, makes it a very easy to use tool for high performance data saving and retrieving. It is built on top of the HDF5 library and the numarray package, and provides containers for both heterogeneous data (Tables) and homogeneous data (Array, EArray). It also sports a container for keeping lists of objects of variable length on a very efficient way (VLArray). A flexible support of filters allows you to compress your data on-the-flight by using different compressors and compression enablers. Moreover, its powerful browsing and searching capabilities allow you to do data selections over tables exceeding gigabytes of data in just tenths of second. Changes more in depth --------------------- New features: - Indexing of columns in tables. That allow to make data selections on tables up to 500 times faster than standard selections (for ex. doing a selection along an indexed column of 100 milion of rows takes less than 1 second on a modern CPU). Perhaps the most interesting thing about the indexing algorithm implemented by PyTables is that the time taken to index grows *lineraly* with the length of the data, so, making the indexation process to be *scalable* (quite differently to many relational databases). This means that it can index, in a relatively quick way, arbitrarily large table columns (for ex. indexing a column of 100 milion of rows takes just 100 seconds, i.e. at a rate of 1 Mrow/sec). See more detailed info about that in http://pytables.sourceforge.net/doc/SciPy04.pdf. - In-kernel selections. This feature allow to make data selections on tables up to 5 times faster than standard selections (i.e. pre-0.9 selections), without a need to create an index. As a hint of how fast these selections can be, they are up to 10 times faster than a traditional relational database. Again, see http://pytables.sourceforge.net/doc/SciPy04.pdf for some experiments on that matter. - Support of complex datatypes for all the data objects (i.e. Table, Array, EArray and VLArray). With that, the complete set of datatypes of Numeric and numarray packages are supported. Thanks to Tom Hedley for providing the patches for Array, EArray and VLArray objects, as well as updating the User's Manual and adding unit tests for the new functionality. - Modification of values. You can modifiy Table, Array, EArray and VLArray values. See Table.modifyRows, Table.modifyColumns() and the newly introduced __setitem__() method for Table, Array, EArray and VLArray entities in the Library Reference of User's Manual. - A new sub-package called "nodes" is there. On it, there will be included different modules to make more easy working with different entities (like images, files, ...). The first module that has been added to this sub-package is "FileNode", whose mission is to enable the creation of a database of nodes which can be used like regular opened files in Python. In other words, you can store a set of files in a PyTables database, and read and write it as you would do with any other file in Python. Thanks to Ivan Vilata i Balaguer for contributing this. Improvements: - New __len__(self) methods added in Arrays, Tables and Columns. This, in combination with __getitem__(self,key) allows to better emulate sequences. - Better capabilities to import generic HDF5 files. In particular, Table objects (in the HDF5_HL naming schema) with "holes" in their compound type definition are supported. That allows to read certain files produced by NASA (thanks to Stephen Walton for reporting this). - Much improved test units. More than 2000 different tests has been implemented which accounts for more than 13000 loc (this represents twice of the PyTables library code itself (!)). Backward-incompatible API changes: - The __call__ special method has been removed from objects File, Group, Table, Array, EArray and VLArray. Now, you should use walkNodes() in File and Group and iterrows in Table, Array, EArray and VLArray to get the same functionality. This would provide better compatibility with IPython as well. 'nctoh5', a new importing utility: - Jeff Whitaker has contributed a script to easily convert NetCDF files into HDF5 files using Scientific Python and PyTables. It has been included and documented as a new utility. Bug fixes: - A call to File.flush() now invoke a call to H5Fflush() so to effectively flushing all the file contents to disk. Thanks to Shack Toms for reporting this and providing a patch. - SF #1054683: Security hole in utils.checkNameValidity(). Reported in 2004-10-26 by ivilata - SF #1049297: Suggestion: new method File.delAttrNode(). Reported in 2004-10-18 by ivilata - SF #1049285: Leak in AttributeSet.__delattr__(). Reported in 2004-10-18 by ivilata - SF #1014298: Wrong method call in examples/tutorial1-2.py. Reported in 2004-08-23 by ivilata - SF #1013202: Cryptic error appending to EArray on RO file. Reported in 2004-08-21 by ivilata - SF #991715: Table.read(field="var1", flavor="List") fails. Reported in 2004-07-15 by falted - SF #988547: Wrong file type assumption in File.__new__. Reported in 2004-07-10 by ivilata Where PyTables can be applied? ------------------------------ PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux (Intel 32-bit) as the main development platform, but PyTables should be easy to compile/install on many other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors, with the MIPSPro compiler and running IRIX 6.5. It also runs fine on Linux 64-bit platforms, like AMD Opteron running GNU/Linux 2.4.21 Server, Intel Itanium (IA64) running GNU/Linux 2.4.21 or PowerPC G5 with Linux 2.6.x in 64bit mode. It has also been tested in MacOSX platforms (10.2 but should also work on newer versions). Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Bon profit! -- Francesc Altet

1 0

Re: [Numpy-discussion] vector cross product
by Gary Ruben Nov. 5, 2004

Nov. 5, 2004

Hi David, Sorry about the slow reply - I was studying for an exam :-) but it's over now. Thanks for letting me know about your vector class. Just a quick explanation of why it's probably not useful in my case: I'm pretty much Windows based, so at a glance pyrex doesn't seem too much help for this specific purpose - I'm thinking I'd probably be better off trying to get weave going if I want to get some extra speed out of my existing code. Also, my code was written as part of a now completed project. It may undergo further development by others, so I'd also like to be able to minimise any extra package requirements that they might need. Profiling revealed Konrad's Vector class as the biggest current bottleneck, so I just thought I'd see what others thought about the usefulness of adding Vectors to numpy. If others don't think it's appropriate for numpy, maybe the scipy is the place to suggest it instead. Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm

1 0

Re: [Numpy-discussion] Counting array elements
by Chris Barker Nov. 4, 2004

Nov. 4, 2004

Todd Miller wrote: >> What I was suggesting is that >>there should be an API for accessing the elements of an array that >>doesn't rely on the standard strides approach. I guess I'm expressing my >>disappointment that PyArrays don't follow one of the axioms of Object >>Oriented Programming: Encapsulation. I should be able to get element >>(i,j) of an array without knowing the data structures used to store the >>data. > > (I think) numarray has what you're talking about: the "element-wise > API". It's documented in the manual but AFIK is fairly slow and > probably not widely used. > Well, the "fairly slow" is the issue. Along with the not widely used. >>If we had that, then there could be a 1-d "flat" array that >>supported discontiguous arrays in a different way than the strides >>approach, while sharing the same data block as the parent N-d array. > > > The numarray "element-wise" API makes use of strides internally in order > to access array elements; it does, however, hide what it's doing. I'm no C wiz, but by being macros, it looks to me like they very much depend on the PyArrayObject that is passed in storing it's data with strides, etc. anyway, so they couldn't be used with an object with a different storage scheme. > I > don't understand the approach you're suggesting here though. Can you > elaborate? What I'm getting at is classic OO polymorphism: An Array class that has a GetElement1d(i) method that returns the element. This class could then be replaced with another class that uses a completely different internal storage mechanism, but still has a GetElement1d(i) method. I know we're working with C, rather than C++ here, but I think this kind of thing could be faked with enough typecasting. On the other hand, I don't know what the heck I'm talking about. I'm no C wiz. Your comment about performance above is key, however. If this approach has worse performance than doing pointer arithmetic by hand with Array->strides et al, then it wouldn't get used universally, and we'd be back were we started. I know even less C++ than C, but I think perhaps the only way to get this with adequate performance would be to do a lot of C++ template magic, like Blitz++. In the early days of numarray development, there was discussion about using Blitz++ (or other nifty C++ template based arrays) as the basis for numarray. I think it all really boiled down to the template magic required was not well supported by enough compilers, so it couldn't be used. I think that's a shame, as while I haven't used C++ much, templates an iterators and all look very appealing, and much better than all the hassles of pointer arithmetic an static typing of C. >>Anyway, I'm just dreaming, I suppose, we're pretty committed to the >>current approach! > > Good ideas have a way of getting adopted, so dream on... > Well, yes, but the core API of numarray is pretty well established by now. >>Very cool! I'm still using Numeric, but I think next time I need to >>write my own Ufunc extension, this may be what converts me! By the way, the two reasons I still use numeric, other than inertia, are: 1) slower small array performance: I use arrays a lot for the convenience, rather than just when I have large arrays and need the performance. 2) Much slower performance when passing arrays into wxPython, due to wxPython using the generic sequence interface, which is apparently much slower with numarray than Numeric. Has this changed? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 1

Re: Hello
by Numpy-discussion Nov. 3, 2004

Nov. 3, 2004

1 0

floating point exception weirdness
by Andrew Straw Nov. 3, 2004

Nov. 3, 2004

Hi All, I've isolated numarray.ieeespecial as the cause of a strange error which I posted on c.l.py: http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=41198B29.709… My non-expert guess as to an explanation: some statements early in numarray.ieeespecial (e.g. _na.array(1.0)/_na.array(0.0)) result in a floating point exception being set which lies dormant until I call certain functions in the Intel IPP library (e.g. ippiAddWeighted_8u32f_C1IR), at which point the latent exception bit my Python program is killed while the console says "Floating point exception". I've come to this tentative conclusion based on 3 minimal Pyrex programs: ---------------------------------------------------- Version 1: import numarray.numarrayall as _na plus_inf = inf = _na.array(1.0)/_na.array(0.0) cimport avg avg.work() # calls IPP, terminates with "Floating point exception". ---------------------------------------------------- Version 2: import numarray.numarrayall as _na # Define *ieee special values* _na.Error.pushMode(all="ignore") plus_inf = inf = _na.array(1.0)/_na.array(0.0) _na.Error.popMode() cimport avg avg.work() # calls IPP, terminates with "Floating point exception". ---------------------------------------------------- Version 3: import numarray.numarrayall as _na cimport avg avg.work() # calls IPP, terminates normally In the short term, I'll try and work around this problem by not importing numarray.ieeespecial, but in the long term, I think we should try and fix this. I'm happy to try to debug further if someone gives me advice on where to go from here. Cheers! Andrew

2 6

failure notice
by MAILER-DAEMON＠mx.teleo.net Nov. 2, 2004

Nov. 2, 2004

Hi. This is the qmail-send program at mx.teleo.net. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. <patrick.phalen(a)gmail.com>: 64.233.171.27 failed after I sent the message. Remote host said: 552 5.7.0 Illegal Attachment --- Below this line is a copy of the message.

1 0

Re: [Numpy-discussion] vector cross product
by Gary Ruben Nov. 2, 2004

Nov. 2, 2004

I guess from the lack of enthusiastic responses that there aren't many out there who think this is of interest. So, perhaps Todd could just add it to the bottom of the list of possible ideas I imagine he maintains, so that if it comes up again in the future, it is at least noted. thanks, Gary ----- Original Message ----- From: "Gary Ruben" <gazzar(a)email.com> To: numpy-discussion(a)lists.sourceforge.net Subject: [Numpy-discussion] vector cross product Date: Sun, 31 Oct 2004 22:18:56 +1000 > > Not that I have a really urgent need, but is there a reason that nice, fast C-based vector operations aren't implemented in Numeric or numarray? I notice Fernando Perez has a cross product as a useful SciPy weave example on his site. I've also seen comments elsewhere about Numpy's lack of a cross product. eg. <http://mail.python.org/pipermail/python-list/2004-March/213878.html> > I'm using Konrad Hinsen's Scientific Python for the convenience value of his Vector class, which also provides a nice angle() method but it bothers me that it's implemented in native Python. The Vector type in vpython probably does it 'properly', but I don't use it just for the convenience since it adds an extra dependency to my code. > > comments? > Gary R. -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm

2 1

Re: [Numpy-discussion] Counting array elements
by Todd Miller Nov. 2, 2004

Nov. 2, 2004

I have returned from our astronomical data systems conference and I am going to take a short cut and summarize what I saw as the key developments of this thread. I apologize for not responding sooner and individually but the web-mail system I use isn't effective for conducting any kind of discussion. You guys did a great job sorting this out this week. I marked my key points with **. The rest is probably only for people with a lot of patience. ** I've finally come to terms with the fact that functions are the right way to do numarray rather than methods. The arguments in the Numeric manual are no more persuasive now than they ever were, but Stephen Walton's remarks about method explosion finally convinced me what the "real" reason for doing functions is that using methods combines every new feature under the umbrella of a single namespace, the NumArray class. Using functions lets us partition things into modules which can be used selectively and makes a more extensible and understandable system. Thanks Stephen. A couple people remarked that using .flat might solve everything with something like a.flat.sum() or sum(ravel(a). This gets to the original motivation for the sum() method, which was the codification of a simple and storage efficient technique for reducing noncontiguous arrays. The first point is that a non-contiguous array cannot generally be reshaped without making a copy. The basic idea of the sum() method is to do *two* reductions, the first, along a single axis, results in a smaller contiguous array. In the case of astronomical images which are generally square or at least non-degenerate, the reduction result is a *much* smaller array. The second reduction handles all the remaining dimensions since .flat is guaranteed to work because the array is contiguous. The end result is a complete sum() without righting additional ufuncs or making an array copy. There was understandable confusion about why .flat is sometimes allowed to fail. Since it is an attribute, we thought it inappropriate to make it return a copy of the source array and chose instead to raise an exception. In contrast, it is reasonable for the ravel() function to return a completely different array, so it always works. (I just noticed that ravel() is not named flat()). Some of our more contemporary thinkers suggested using iterators to produce a .flat which always works. If anyone has an idea how to make this work with good performance, please let me know; I don't. ** Tim Hochberg pointed out that we can overload the reduction (and not accumulation?) axis parameter with an "all" or a tuple describing a sequence of axes to reduce along. My perception was that there was a consensus behind this and in any case I'm in agreement with Tim. Alan Isaac pointed out that None might be better here than "all" and I agree. At this point, I think sumAll() is dead, the sum() method will be deprecated, and the reductions should be expanded as Tim suggested. ** Peter Verveer made some comments about the expectations of a naive user regarding reductions, namely that "all" should be the default. My own experience bears this out, and I am torn about what to do here. Chris Barker pointed out the need for backward compatibility with Numeric, and given the current numarray goal of supporting SciPy, this need is growing stronger and more complex. SciPy uses yet another axis convention. If anyone has any ideas how to handle these multiple conventions with elegance, let me know. A number of people commented on our naming conventions, an issue which we have side stepped for the moment with sumAll(). My impression is that, for better or worse, numarray uses the lowerUpper() version of Camel case. I think this is very much a matter of personal taste and don't claim to have any. My guess is that numarray is probably inconsistent at the moment, in part because lowerUpper() often degenerates into merely lower() which degenerates into confusion. Regards, Todd

5 7

Re: floating point exception weirdness (Andrew Straw)
by Steve Chaplin Nov. 1, 2004

Nov. 1, 2004

> Just a small addendum, (which I hope will spur on bug-fixing once Todd > et al. are back from the conference -- let me know if I should file a > sourceforge bug report): I've not read all this thread so I don't know the full background. But I had a floating point / SSE problem using numarray. It turned out to be a glibc not numarray problem and was solved by upgrading glibc. http://sources.redhat.com/bugzilla/show_bug.cgi?id=10 There was also a SourceForge bug report but I can't locate it. Regards Steve

2 2

Promote Your Marketing
by Tim Oct. 31, 2004

Oct. 31, 2004

1 0