Mailman 3 December 2008 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

PyArray_ContiguousFromObject & PyDECREF
by Michiel Jan Laurens de Hoon May 16, 2011

May 16, 2011

Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

3 2

Test failures on 2.6
by T J Nov. 18, 2010

Nov. 18, 2010

Hi, I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4: nose version 0.10.4 ..........................................................................................................................................................................................................................................................................................................................................................................................................................................F................K........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(store._mask, True) /home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(test.mask, [0,1,0,0,0,0,0,0,0,0]) ...............................................E................................................................................................................................................................................ ====================================================================== ERROR: Tests the min/max functions with explicit outputs ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py", line 653, in test_minmax_funcs_with_output result = npfunc(xm,axis=0,out=nout) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py", line 1525, in amin return amin(axis, out) File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py", line 2978, in min np.putmask(out, newmask, np.nan) ValueError: cannot convert float NaN to integer ====================================================================== FAIL: test_umath.TestComplexFunctions.test_against_cmath ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py", line 268, in test_against_cmath assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j) ---------------------------------------------------------------------- Ran 1726 tests in 8.856s FAILED (KNOWNFAIL=1, errors=1, failures=1) <nose.result.TextTestResult run=1726 errors=1 failures=1>

2 1

Unpleasant behavior with poly1d and numpy scalar multiplication
by Fernando Perez Feb. 15, 2010

Feb. 15, 2010

Hi all, consider this little script: from numpy import poly1d, float, float32 p=poly1d([1.,2.]) three=float(3) three32=float32(3) print 'three*p:',three*p print 'three32*p:',three32*p print 'p*three32:',p*three32 which produces when run: In [3]: run pol1d.py three*p: 3 x + 6 three32*p: [ 3. 6.] p*three32: 3 x + 6 The fact that multiplication between poly1d objects and numbers is: - non-commutative when the numbers are numpy scalars - different for the same number if it is a python float vs a numpy scalar is rather unpleasant, and I can see this causing hard to find bugs, depending on whether your code gets a parameter that came as a python float or a numpy one. This was found today by a colleague on numpy 1.0.4.dev3937. It feels like a bug to me, do others agree? Or is it consistent with a part of the zen of numpy I've missed thus far? Thanks, f

6 17

Compact way of performing array math with specified result type?
by Russell E. Owen Feb. 8, 2010

Feb. 8, 2010

I often find myself doing simple math on sequences of numbers (which might or might not be numpy arrays) where I want the result (and thus the inputs) coerced to a particular data type. I'd like to be able to say: numpy.divide(seq1, seq2, dtype=float) but ufuncs don't allow on to specify a result type. So I do this instead: numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float) Is there a more compact solution (without having to create the result array first and supply it as an argument)? -- Russell

4 3

Is anyone knowledgeable about dll deployment on windows ?
by David Cournapeau Dec. 1, 2009

Dec. 1, 2009

Hi, My head hurt trying to understand dll management with windows. I wanted to find a sane way to use dll for numscons, but I can't see how to do this, so I was wondering if anyone on this ML had any deep knowledge on how to install dll, and reuse them with python extensions ? The problem is the following: - you have some dll (say MKL) in a path (say C:\program files\MKL). - how do you tell another dll (more exactly a python extension, a .pyd) to look for mkl dll into the MKL path ? It seems that there is no rpath-like capability on windows, and the only way to load dll is to put them into some windows directory, or to put the dll path into PATH. Is it really the only way ? cheers, David

8 25

Slicing/selection in multiple dimensions simultaneously
by Mike Ressler March 17, 2009

March 17, 2009

The following seems to be a wart: is it expected? Set up a 10x10 array and some indexing arrays: a=arange(100) a.shape=(10,10) q=array([0,2,4,6,8]) r=array([0,5]) Suppose I want to extract only the "even" numbered rows from a - then print a[q,:] <works - output deleted> Every fifth column: print a[:,r] <works - output deleted> Only the even rows of every fifth column: print a[q,r] --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /.../.../.../<ipython console> in <module>() <type 'exceptions.ValueError'>: shape mismatch: objects cannot be broadcast to a single shape But, this works: print a[q,:][:,r] [[ 0 5] [20 25] [40 45] [60 65] [80 85]] So why does the a[q,r] form have problems? Thanks for your insights. Mike -- mike.ressler(a)alum.mit.edu

8 16

Automatic differentiation (was Re: second-order gradient)
by Rob Clewley March 11, 2009

March 11, 2009

> Maybe we should focus on writing a decent 'deriv' function then. I > know Konrad Hinsen's Scientific had a derivatives package > (Scientific.Functions.Derivatives) that implemented automatic > differentiation: > > http://en.wikipedia.org/wiki/Automatic_differentiation That would be great, but wouldn't that be best suited as a utility requiring Sympy? You'll want to take advantage of all sorts of symbolic classes, especially for any source code transformation approach. IMO Hinsen's implementation isn't a very efficient or attractive solution to AD given the great existing C/C++ codes out there. Maybe we should be looking to provide a python interface to an existing open source package such as ADOL-C, but I'm all in favour of a new pure python approach too. What would be perfect is to have a single interface to a python AD package that would support a faster implementation if the user wished to install a C/C++ package, otherwise would default to a pure python equivalent. -Rob

4 4

linalg.norm missing an 'axis' kwarg?!
by Hans Meine Feb. 10, 2009

Feb. 10, 2009

Hi, I have a 2D matrix comprising a sequence of vectors, and I want to compute the norm of each vector. np.linalg.norm seems to be the best bet, but it does not support axis. Wouldn't this be a nice feature? Greetings, Hans

4 8

new incremental statistics project
by Bradford Cross Jan. 22, 2009

Jan. 22, 2009

This is a new project I just released. I know it is C#, but some of the design and idioms would be nice in numpy/scipy for working with discrete event simulators, time series, and event stream processing. http://code.google.com/p/incremental-statistics/

8 9

Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64)
by George Goussard Jan. 16, 2009

Jan. 16, 2009

Hello. I have been battling with the following error for the past week. The output from the terminal is: Traceback (most recent call last): File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\backends\backend_qt4agg.py", line 86, in paintEvent FigureCanvasAgg.draw(self) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\backends\backend_agg.py", line 261, in draw self.figure.draw(self.renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\figure.py", line 765, in draw legend.draw(renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 197, in draw self._update_positions(renderer) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 513, in _update_positions l,b,w,h = get_tbounds(self.texts[-1]) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\legend.py", line 499, in get_tbounds bboxa = bbox.inverse_transformed(self.get_transform()) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\transforms.py", line 478, in inverse_transformed return Bbox(transform.inverted().transform(self.get_points())) File "C:\development\Python\2_5_2\Lib\site-packages\matplotlib\transforms.py", line 1338, in inverted self._inverted = Affine2D(inv(mtx)) File "C:\development\Python\2_5_2\Lib\site-packages\numpy\linalg\linalg.py", line 350, in inv return wrap(solve(a, identity(a.shape[0], dtype=a.dtype))) File "C:\development\Python\2_5_2\Lib\site-packages\numpy\linalg\linalg.py", line 249, in solve raise LinAlgError, 'Singular matrix' numpy.linalg.linalg.LinAlgError: Singular matrix Initially MPL plots a graph but when you try to interact with the widget(for example resize) then the output is displayed and the MPL figure is not updated. Everything works with Windows 32-bit. Linux 32-bit and 64-bit are working correctly. Any ideas would be helpful. Thanks. George.

6 9

Re: [Numpy-discussion] Complete LAPACK needed (Frank Lagor)
by Frank Lagor Jan. 14, 2009

Jan. 14, 2009

Thanks so much for your help, David. I'm sorry I did not receive your posts previously -- I have the digest mode on and there is a bit of a delay. I'll try to change my options next time I post a request. Thanks so much again, Frank

2 1

formatting issues, locale and co
by David Cournapeau Jan. 12, 2009

Jan. 12, 2009

Hi, While looking at the last failures of numpy trunk on windows for python 2.5 and 2.6, I got into floating point number formatting issues; I got deeper and deeper, and now I am lost. We have several problems: - we are not consistent between platforms, nor are we consistent with python - str(np.float32(a)) is locale dependent, but python str method is not (locale.str is) - formatting of long double does not work on windows because of the broken long double support in mingw. 1 consistency problem: ---------------------- python -c "a = 1e20; print a" -> 1e+020 python26 -c "a = 1e20; print a" -> 1e+20 In numpy, we use PyOS_snprintf for formatting, but python itself uses PyOS_ascii_formatd - which has different behavior on different versions of python. The above behavior can be simply reproduced in C: #include <Python.h> int main() { double x = 1e20; char c[200]; PyOS_ascii_format(c, sizeof(c), "%.12g", x); printf("%s\n", c); printf("%g\n", x); return 0; } On 2.5, this will print: 1e+020 1e+020 But on 2.6, this will print: 1e+20 1e+020 2 locale dependency: -------------------- Another issue is that our own formatting is local dependent, whereas python isn't: import numpy as np import locale locale.setlocale(locale.LC_NUMERIC, 'fr_FR') a = 1.2 print "str(a)", str(a) print "locale.str(a)", locale.str(a) print "str(np.float32(a))", str(np.float32(a)) print "locale.str(np.float32(a))", locale.str(np.float32(a)) Returns: str(a) 1.2 locale.str(a) 1,2 str(np.float32(a)) 1,2 locale.str(np.float32(a)) 1,20000004768 I thought about copying the way python does the formatting in the trunk (where discrepancies between platforms have been fixed), but this is not so easy, because it uses a lot of code from different places - and the code needs to be adapted to float and long double. The other solution would be to do our own formatting, but this does not sound easy: formatting in C is hard. I am not sure about what we should do, if anyone else has any idea ? cheers, David

6 19

help with typemapping a C function to use numpy arrays
by Rich E Jan. 9, 2009

Jan. 9, 2009

Hi list, My question has to do with the Numpy/SWIG typemapping system. I recently got the typemaps in numpy.i to work on most of my C functions that are wrapped using SWIG, if they have arguments of the form (int sizeArray, float *pArray). Now I am trying to figure out how to wrap function that aren't of the form, such as the following function: /*! \brief compute magnitude spectrum of a DFT * * \param sizeMag size of output Magnitude (half of input real FFT) * \param pFReal pointer to input FFT real array (real/imag floats) * \param pFMAg pointer to float array of magnitude spectrum */ void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag) { int i, it2; float fReal, fImag; for (i=0; i<sizeMag; i++) { it2 = i << 1; fReal = pInRect[it2]; fImag = pInRect[it2+1]; pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag); } } There are two arrays, one is half the size of the other. But, SWIG doesn't know this, according to the type map it will think *pInRect is of size sizeMag and will not know anything about *pOutMag. Ideally in python, I would like to call the function as sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of nArray2, and nArray2 is of size sizeMag. I think in order to do this (although if someone has a better suggestion, I am open to it), I will have to modify the typemap in order to tell SWIG how to call the C function properly. I do not want to have to edit the wrapped C file every time it is regenerated from the interface file. Here is a start I made with the existing typemap code in numpy.i (not working): /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) */ %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY, fragment="NumPy_Macros") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) { $1 = is_array($input) && PyArray_EquivTypenums(array_type($input), DATA_TYPECODE); } %typemap(in, fragment="NumPy_Fragments") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) (PyArrayObject* array=NULL, int i=0) { array = obj_to_array_no_conversion($input, DATA_TYPECODE); if (!array || !require_dimensions(array,1) || !require_contiguous(array) || !require_native(array)) SWIG_fail; $1 = 1; for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i); $2 = (DATA_TYPE*) array_data(array); } and try to alter it to allow for a conversion of type: (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2) where ARRAY1 is size DIM1 * 2 and ARRAY2 is size DIM1. Then I can %apply this to my function that I mentioned in the last post. So here are my first two questions: 1) where is DIM1 used to declare the array size? I don't see where it is used at all, and I need to somewhere multiply it by 2 to declare the size of ARRAY1 2) I am not understanding where $input comes from, so I do not understand how to distinguish between ARRAY1 and ARRAY2. In the attempt I have already tried, I think I just use the pointer to ARRAY1 twice. If anyone has suggestions on how to solve this problem, thanks! regards, Rich

3 11

ANN: numpy.i - added managed deallocation to ARGOUTVIEW_ARRAY1 (ARGOUTVIEWM_ARRAY1)
by Egor Zindy Jan. 8, 2009

Jan. 8, 2009

Dear List, after I tried to write a simple ARGOUTVIEW_ARRAY1 example (see http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW#A_simple_ARGOUTVIEW_… ), I started wondering about memory deallocation. Turns out a few clever people already did all the thinking (see http://blog.enthought.com/?p=62 ) and a few more clever people use this in a swig file (see http://niftilib.sourceforge.net/pynifti, file nifticlib.i). All this concentrated knowledge helped me implement a single ARGOUTVIEWM_ARRAY1 fragment in numpy.i to do some testing (see attached numpy.i). How to use it? In yourfile.i, the %init function has a few lines added to it: %init %{ import_array(); /* initialize the new Python type for memory deallocation */ _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) return; %} ... and that's it! then just use ARGOUTVIEWM_ARRAY1 instead of ARGOUTVIEW_ARRAY1 and python does the deallocation for you when the python array is destroyed (see the examples attached). Everything compiles, but it would be nice to get rid of the warnings... ezalloc_wrap.c:2763: warning: initialization from incompatible pointer type ezalloc_wrap.c: In function `_wrap_alloc_managed': ezalloc_wrap.c:2844: warning: assignment from incompatible pointer type writing build\temp.win32-2.5\Release\_ezalloc.def Compilation: python setup_alloc.py build Testing: The attached test_alloc.py does 2048 allocations of 1MB each for managed and unmanaged arrays. Output on my XP laptop with 1GB RAM as follows: ARGOUTVIEWM_ARRAY1 (managed arrays) - 2048 allocations (1048576 bytes each) Done! ARGOUTVIEW_ARRAY1 (unmanaged, leaking) - 2048 allocations (1048576 bytes each) Step 482 failed TODO: Find a better name for the methods (not sure I like ARGOUTVIEWM_ARRAY), then do the missing fragments (2D arrays), clear the warnings and verify the method for a possible inclusion in the official numpy.i file. Thank you for reading! Regards, Egor

2 6

Old-style classes in tests
by Tom Wright Jan. 6, 2009

Jan. 6, 2009

I am currently working on the Ironclad project porting numpy to Ironpython. It would be quite useful for me if HermitianTestCase in test_linalg.py was a new style-class instead of an old-style class - since Ironpython has a bug where dir operations do not work for classes inheriting from both old- and new- style classes and I'd very much prefer not to patch my version of numpy. In general, it would be useful if whenever this multiple inheritence pattern is used new-style classes are used rather than old style classes. This would require the following classes to change: test_numerictypes - create_values, read_values_plain, read_values_nested test_print - create_zeros, create_values, assign_values, byteorder_values test_io - Roundtriptest test_linalg - LinalgTestCase, HermitianTestCase Would people be ameniable to these change? Thank you very much for your help, Tom Wright

3 2

Alternative to record array
by Jean-Baptiste Rudant Jan. 2, 2009

Jan. 2, 2009

Hello, I like to use record arrays to access fields by their name, and because they are esay to use with pytables. But I think it's not very effiicient for what I have to do. Maybe I'm misunderstanding something. Example : import numpy as np age = np.random.randint(0, 99, 10e6) weight = np.random.randint(0, 200, 10e6) data = np.rec.fromarrays((age, weight), names='age, weight') # the kind of operations I do is : data.age += data.age + 1 # but it's far less efficient than doing : age += 1 # because I think the record array stores [(age_0, weight_0) ...(age_n, weight_n)] # and not [age0 ... age_n] then [weight_0 ... weight_n]. So I think I don't use record arrays for the right purpose. I only need something which would make me esasy to manipulate data by accessing fields by their name. Am I wrong ? Is their something in numpy for my purpose ? Do I have to implement my own class, with something like : class FieldArray: def __init__(self, array_dict): self.array_list = array_dict def __getitem__(self, field): return self.array_list[field] def __setitem__(self, field, value): self.array_list[field] = value my_arrays = {'age': age, 'weight' : weight} data = FieldArray(my_arrays) data['age'] += 1 Thank you for the help, Jean-Baptiste Rudant

5 6

numpy.test() failures
by Nils Wagner Dec. 31, 2008

Dec. 31, 2008

>>> numpy.__version__ '1.3.0.dev6283' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 74, in check_complex_type err_msg='Failed str formatting for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed str formatting for type <type 'numpy.complex64'> ACTUAL: '(1e+10+0j)' DESIRED: '1e+10' ====================================================================== FAIL: Check formatting when using print ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 108, in check_float_type_print _test_redirected_print(float(x), tp) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 104, in _test_redirected_print err_msg='print failed for type%s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: print failed for type<type 'numpy.float32'> ACTUAL: '10000000000.0\n' DESIRED: '1e+10\n' ====================================================================== FAIL: Check formatting when using print ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 115, in check_complex_type_print _test_redirected_print(complex(x), tp) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 104, in _test_redirected_print err_msg='print failed for type%s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: print failed for type<type 'numpy.complex64'> ACTUAL: '(10000000000+0j)\n' DESIRED: '(1e+10+0j)\n' ====================================================================== FAIL: test_print.test_locale_single ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 164, in test_locale_single return _test_locale_independance(np.float32) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float32'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_double ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 169, in test_locale_double return _test_locale_independance(np.double) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float64'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_longdouble ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 174, in test_locale_longdouble return _test_locale_independance(np.longdouble) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float128'> ACTUAL: '1,2' DESIRED: '1.2' ---------------------------------------------------------------------- Ran 1797 tests in 16.303s FAILED (KNOWNFAIL=1, failures=6)

1 0

Removal of deprecated test framework stuff
by Alan McIntyre Dec. 31, 2008

Dec. 31, 2008

Hi all, Unless somebody objects, I'd like to remove from NumPy 1.3 the following numpy.testing items that were deprecated in NumPy 1.2 (since the warnings promise we'll do so ;): - ParametricTestCase (also removing the entire file numpy/testing/parametric.py) - The following arguments from numpy.testing.Tester.test() (which is used for module test functions): level, verbosity, all, sys_argv, testcase_pattern - Path manipulation functions: set_package_path, set_local_path, restore_path - NumpyTestCase, NumpyTest Thanks, Alan

2 1

"False" and "True" not singletons?
by Christopher Barker Dec. 30, 2008

Dec. 30, 2008

Hi all, I've just discovered that "False" is not a singleton: >>> import numpy as N >>> f = N.all((1,2,0)) >>> print f False >>> id(f) 17316364 >>> f is False False >>> id(False) 3294768 Should it be? This arose for me in some tests I'm using that check if a result is False: self.failUnless ( (self.B1 == self.B3) is False ) I'm doing it this way, because I want to make sure that the __eq__ method really is returning "True" or "False", rather than a value that happens to evaluate to true or false, like 1, or an empty list, or whatever. This is interesting to me because back when Python first introduced Booleans, I had thought they should be kept pure and not be subclasses of integers, and, in fact, "if" should only except boolean values. However this opinion was really a matter of my sense of purity, and this is the fist time I've run into a case that matters. I suppose I'm not being pythonic -- I should really only care if the result evaluates true or false, but I don't feel like I'm testing right if values can slip though that shouldn't. It does reinforce my opinion though -- whether zero, or an empty sequence or string should evaluate false really is a matter of specific application, not universal. Anyway, should I just give up? Or should numpy return the same "True" and "False" (like None), or is there another solution? x == False is close, but zero still slips through, even though an empty list does not: >>> 0 == False True >>> [] == False False -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

4 4

numpy.test() failures (1.2.1) on Mac OS X
by Nicholas Matsakis Dec. 30, 2008

Dec. 30, 2008

I just installed what I believe to be a completely vanilla installation of numpy on an Intel Mac OS X 10.5.6. Python 2.5 pkg from Python.org, numpy 1.2.1 pkg from scipy.org, nose installed through setup tools. Running "import numpy; numpy.test()" results in the following errors and failures: ERROR: Failure: TypeError (can't multiply sequence by non-int of type 'float') ERROR: test_definition (test_helper.TestFFTShift) ERROR: test_inverse (test_helper.TestFFTShift) ERROR: Test of inplace division FAIL: test_division_int (test_umath.TestDivision) FAIL: test_basic (test_index_tricks.TestUnravelIndex) FAIL: Test of inplace division FAIL: test_inplace_division_misc (test_core.TestMaskedArrayInPlaceArithmetics) FAIL: Test of inplace operations and rich comparisons Is this expected? Should I file a ticket? A complete dump of the test run can be found at: http://nick.matsakis.net/tmp/numpy-1.2.1-tests-12-30-2008.txt Nick Matsakis

1 1

combining recarrays
by ctw Dec. 30, 2008

Dec. 30, 2008

Hi! I'm a bit stumped by the following: suppose I have several recarrays with identical dtypes (identical field names, etc.) and would like to combine them into one rec array, what would be the best way to do that? I tried using np.rec.fromrecords, but that doesn't produce the desired result. As a minimal example consider the following code: desc = np.dtype({'names':['a','b'],'formats':[np.float,np.int]}) rec1 = np.zeros(3,desc) rec2 = np.zeros(3,desc) Now I have two recarrays of shape (3,) that both look like this: array([(0.0, 0), (0.0, 0), (0.0, 0)], dtype=[('a', '<f8'), ('b', '<i4')]) I would like to turn them into one new recarray of shape (6,) that looks like this: array([(0.0, 0), (0.0, 0), (0.0, 0), (0.0, 0), (0.0, 0), (0.0, 0)], dtype=[('a', '<f8'), ('b', '<i4')]) Any ideas?

2 1

Re: [Numpy-discussion] Thoughts on persistence/object tracking in scientific code
by Luis Pedro Coelho Dec. 29, 2008

Dec. 29, 2008

Hello, I coincidently started my own implementation of a system to manage intermediate results last week, which I called jug. I wasn't planning to make such an alpha version public just now, but it seems to be on topic. The main idea is to use hashes to map function arguments to paths on the filesystem, which store the result (nothing extraordinary here). I also added the capability of having tasks (the basic unit) take the results of other tasks and defining an implicit dependency DAG. A simple locking mechanism enables light-weight task-level parellization (this was the second of my goals: help me make my stuff parallel). A trick that helps is that I don't really use the argument values to hash (which would be unwieldy for big arrays). I use the computation path (e.g., this is the value obtained from f(g('something'),2)). Since, at least in my problems, things tend to always map back into simple file-system paths, the hash computation doesn't even need to load the intermediate results. I will make the git repository publicly available once I figure out how to do that. I append the tutorial I wrote, which explains the system. HTH, Luís Pedro Coelho PhD Student in Computational Biology Carnegie Mellon University ============ Jug Tutorial ============ What is jug? ------------ Jug is a simple way to write easily parallelisable programs in Python. It also handles intermediate results for you. Example ------- This is a simple worked-through example which illustrates what jug does. Problem ~~~~~~~ Assume that I want to do the following to a collection of images: (1) for each image, compute some features (2) cluster these features using k-means. In order to find out the number of clusters, I try several values and pick the best result. For each value of k, because of the random initialisation, I run the clustering 10 times. I could write the following simple code: :: imgs = glob('*.png') features = [computefeatures(img,parameter=2) for img in imgs] clusters = [] bics = [] for k in xrange(2,200): for repeat in xrange(10): clusters.append(kmeans(features,k=k,random_seed=repeat)) bics.append(compute_bic(clusters[-1])) Nr_clusters = argmin(bics) // 10 Very simple and solves the problem. However, if I want to take advantage of the obvious parallelisation of the problem, then I need to write much more complicated code. My traditional approach is to break this down into smaller scripts. I'd have one to compute features for some images, I'd have another to merge all the results together and do some of the clustering, and, finally, one to merge all the results of the different clusterings. These would need to be called with different parameters to explore different areas of the parameter space, so I'd have a couple of scripts just for calling the main computation scripts. Intermediate results would be saved and loaded by the different processes. This has several problems. The biggest are (1) The need to manage intermediate files. These are normally files with long names like *features_for_img_0_with_parameter_P.pp*. (2) The code gets much more complex. There are minor issues with having to issue several jobs (and having the cluster be idle in the meanwhile), or deciding on how to partition the jobs so that they take roughly the same amount of time, but the two above are the main ones. Jug solves all these problems! Tasks ~~~~~ The main unit of jug is a Task. Any function can be used to generate a Task. A Task can depend on the results of other Tasks. The original idea for jug was a Makefile-like environment for declaring Tasks. I have moved beyond that, but it might help you think about what Tasks are. You create a Task by giving it a function which performs the work and its arguments. The arguments can be either literal values or other tasks (in which case, the function will be called with the *result* of those tasks!). Jug also understands lists of tasks (all standard Python containers will be supported in a later version). For example, the following code declares the necessary tasks for our problem: :: imgs = glob('*.png') feature_tasks = [Task(computefeatures,img,parameter=2) for img in imgs] cluster_tasks = [] bic_tasks = [] for k in xrange(2,200): for repeat in xrange(10): cluster_tasks.append(Task(kmeans,feature_tasks,k=k,random_seed=repeat)) bic_tasks.append(Task(compute_bic,cluster_tasks[-1])) Nr_clusters = Task(argmin,bic_tasks) Task Generators ~~~~~~~~~~~~~~~ In the code above, there is a lot of code of the form *Task(function,args)*, so maybe it should read *function(args)*. A simple helper function aids this process: :: from jug.task import Task def TaskGenerator(function): def gen(*args,**kwargs): return Task(function,*args,**kwargs) return gen computefeatures = TaskGenerator(computefeatures) kmeans = TaskGenerator(kmeans) compute_bic = TaskGenerator(compute_bic) @TaskGenerator def Nr_Clusters(bics): return argmin(bics) // 10 imgs = glob('*.png') features = [computefeatures(img,parameter=2) for img in imgs] clusters = [] bics = [] for k in xrange(2,200): for repeat in xrange(10): clusters.append(kmeans(features,k=k,random_seed=repeat)) bics.append(compute_bic(clusters[-1])) Nr_clusters(bics) You can see that this code is almost identical to our original sequential code, except for the declarations at the top and the fact that *Nr_clusters* is now a function (actually a TaskGenerator, look at the use of a declarator). This file is called the jugfile (you should name it *jugfile.py* on the filesystem) and specifies your problem. Of course, *TaskManager* is already a part of jug and those first few lines could have read :: from jug.task import TaskGenerator Jug ~~~ So far, we have achieved seemingly little. We have turned a simple piece of sequential code into something that generates Task objects, but does not actually perform any work. The final piece is jug. Jug takes these Task objects and runs them. It's main loop is basically :: while len(tasks) > 0: for t in tasks: if can_run(t): # ensures that all dependencies have been run if need_to_run(t) and not is_running(t): t.run() tasks.remove(t) If you run jug on the script above, you will simply have reproduced the original code with the added benefit of having all the intermediate results saved. The interesting is what happens when you run several instances of jug at the same time. They will start running Tasks, but each instance will run its own tasks. This allows you to take advantage of multiple processors in a way that keeps the processors all occupied as long as there is work to be done, handles the implicit dependencies, and passes functions the right values. Note also that, unlike more traditional parallel processing frameworks (like MPI), jug has no problems with the number of participating processors varying throughout the job. Behind the scenes, jug is using the filesystem to both save intermediate results (which get passed around) and to lock running tasks so that each task is only run once (the actual main loop is thus a bit more complex than shown above). Intermediate and Final Results ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can obtain the final results of your computation by setting up a task that saves them to disk and loading them from there. If the results of your computation are simple enough, this might be the simplest way. Another way, which is also the way to access the intermediate results if you want them, is to run the jug script and then call the *load()* method on Tasks. For example, :: img = glob('*.png') features = [computefeatures(img,parameter=2) for img in imgs] ... feature_values = [feat.load() for feat in features] If the values are not accessible, this raises an exception. Advantages ---------- jug is an attempt to get something that works in the setting that I have found myself in: code that is *embarissingly parallel* with a couple of points where all the results of previous processing are merged, often in a simple way. It is also a way for me to manage either the explosion of temporary files that plagued my code and the brittleness of making sure that all results from separate processors are merged correctly in my *ad hoc* scripts. Limitations ----------- This is not an attempt to replace MPI in any way. For code that has more merge points, this won't do. It also won't do if the individual tasks are so small that the over-head of managing them swamps out the performance gains of parallelisation. In my code, most of the times, each task takes 20 seconds to a few minutes. Just enough to make the managing time irrelevant, but fast enough that the main job can be broken into thousands of tiny pieces. The system makes it too easy to save all intermediate results and run out of disk space. This is still Python, not a true parallel programming language. The abstraction will sometimes leak through, for example, if you try to pass a Task to a function which expects a real value. Recall how we had to re-write the line *Nr_clusters = argmin(bics) // 10* above. Planned Capabilities -------------------- Here are a couple of simple improvements I plan to make at some point: * jug.py cleanup: removes left-over locks, temporary files, and unsused results. * Stop & re-start. Currently, jug processes will exit if they can't make any progress for a while. In the future, I'd like them to be unblockable by other jug processes. * No result tasks. Task-like objects that don't save intermediate results. * Have tasks be passed inside *sets* and *dictionaries*. Maybe even *numpy* arrays! This will make jug even more like a real parallel programming language. * If the original arguments are files on disk, then jug should check their modification date and invalidate subsequent results.

3 4

Should object arrays have a buffer interface?
by Andreas Klöckner Dec. 29, 2008

Dec. 29, 2008

Hi all, I don't think PyObject pointers should be accessible via the buffer interface. I'd throw an error, but maybe a (silenceable) warning would do. Would have saved me some bug-hunting. >>> import numpy >>> numpy.array([55, (33,)], dtype=object) >>> x = numpy.array([55, (33,)], dtype=object) >>> x array([55, (33,)], dtype=object) >>> buffer(x) <read-only buffer for 0x8496f48, size -1, offset 0 at 0x850b060> >>> str(buffer(x)) '\xb0\x1c\x17\x08l\x89\xd7\xb7' >>> numpy.__version__ '1.1.0' Opinions? Andreas

2 6

NumPy and Python 2.6 on Windows
by Lenard Lindstrom Dec. 27, 2008

Dec. 27, 2008

Hi everyone, I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6. You are welcome to use the Pygame build tools if they will help. I also have a Python 2.6 build of NumPy 1.2.1 for Pygame testing. http://www3.telus.net/len_l/pygame/numpy-1.2.1.win32-py2.6.msi md5sum: b791f5c4b620da21f779b53252b5932e *numpy-1.2.1.win32-py2.6.msi Lenard -- Lenard Lindstrom <len-l(a)telus.net>

4 11