Mailman 3 December 2008 - NumPy-Discussion

Re: [Numpy-discussion] Complete LAPACK needed (Frank Lagor)
by Frank Lagor 14 Jan '09

14 Jan '09

Thanks so much for your help, David. I'm sorry I did not receive your posts previously -- I have the digest mode on and there is a bit of a delay. I'll try to change my options next time I post a request. Thanks so much again, Frank

2 1

formatting issues, locale and co
by David Cournapeau 12 Jan '09

12 Jan '09

Hi, While looking at the last failures of numpy trunk on windows for python 2.5 and 2.6, I got into floating point number formatting issues; I got deeper and deeper, and now I am lost. We have several problems: - we are not consistent between platforms, nor are we consistent with python - str(np.float32(a)) is locale dependent, but python str method is not (locale.str is) - formatting of long double does not work on windows because of the broken long double support in mingw. 1 consistency problem: ---------------------- python -c "a = 1e20; print a" -> 1e+020 python26 -c "a = 1e20; print a" -> 1e+20 In numpy, we use PyOS_snprintf for formatting, but python itself uses PyOS_ascii_formatd - which has different behavior on different versions of python. The above behavior can be simply reproduced in C: #include <Python.h> int main() { double x = 1e20; char c[200]; PyOS_ascii_format(c, sizeof(c), "%.12g", x); printf("%s\n", c); printf("%g\n", x); return 0; } On 2.5, this will print: 1e+020 1e+020 But on 2.6, this will print: 1e+20 1e+020 2 locale dependency: -------------------- Another issue is that our own formatting is local dependent, whereas python isn't: import numpy as np import locale locale.setlocale(locale.LC_NUMERIC, 'fr_FR') a = 1.2 print "str(a)", str(a) print "locale.str(a)", locale.str(a) print "str(np.float32(a))", str(np.float32(a)) print "locale.str(np.float32(a))", locale.str(np.float32(a)) Returns: str(a) 1.2 locale.str(a) 1,2 str(np.float32(a)) 1,2 locale.str(np.float32(a)) 1,20000004768 I thought about copying the way python does the formatting in the trunk (where discrepancies between platforms have been fixed), but this is not so easy, because it uses a lot of code from different places - and the code needs to be adapted to float and long double. The other solution would be to do our own formatting, but this does not sound easy: formatting in C is hard. I am not sure about what we should do, if anyone else has any idea ? cheers, David

6 19

help with typemapping a C function to use numpy arrays
by Rich E 09 Jan '09

09 Jan '09

Hi list, My question has to do with the Numpy/SWIG typemapping system. I recently got the typemaps in numpy.i to work on most of my C functions that are wrapped using SWIG, if they have arguments of the form (int sizeArray, float *pArray). Now I am trying to figure out how to wrap function that aren't of the form, such as the following function: /*! \brief compute magnitude spectrum of a DFT * * \param sizeMag size of output Magnitude (half of input real FFT) * \param pFReal pointer to input FFT real array (real/imag floats) * \param pFMAg pointer to float array of magnitude spectrum */ void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag) { int i, it2; float fReal, fImag; for (i=0; i<sizeMag; i++) { it2 = i << 1; fReal = pInRect[it2]; fImag = pInRect[it2+1]; pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag); } } There are two arrays, one is half the size of the other. But, SWIG doesn't know this, according to the type map it will think *pInRect is of size sizeMag and will not know anything about *pOutMag. Ideally in python, I would like to call the function as sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of nArray2, and nArray2 is of size sizeMag. I think in order to do this (although if someone has a better suggestion, I am open to it), I will have to modify the typemap in order to tell SWIG how to call the C function properly. I do not want to have to edit the wrapped C file every time it is regenerated from the interface file. Here is a start I made with the existing typemap code in numpy.i (not working): /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) */ %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY, fragment="NumPy_Macros") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) { $1 = is_array($input) && PyArray_EquivTypenums(array_type($input), DATA_TYPECODE); } %typemap(in, fragment="NumPy_Fragments") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) (PyArrayObject* array=NULL, int i=0) { array = obj_to_array_no_conversion($input, DATA_TYPECODE); if (!array || !require_dimensions(array,1) || !require_contiguous(array) || !require_native(array)) SWIG_fail; $1 = 1; for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i); $2 = (DATA_TYPE*) array_data(array); } and try to alter it to allow for a conversion of type: (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2) where ARRAY1 is size DIM1 * 2 and ARRAY2 is size DIM1. Then I can %apply this to my function that I mentioned in the last post. So here are my first two questions: 1) where is DIM1 used to declare the array size? I don't see where it is used at all, and I need to somewhere multiply it by 2 to declare the size of ARRAY1 2) I am not understanding where $input comes from, so I do not understand how to distinguish between ARRAY1 and ARRAY2. In the attempt I have already tried, I think I just use the pointer to ARRAY1 twice. If anyone has suggestions on how to solve this problem, thanks! regards, Rich

3 11

ANN: numpy.i - added managed deallocation to ARGOUTVIEW_ARRAY1 (ARGOUTVIEWM_ARRAY1)
by Egor Zindy 08 Jan '09

08 Jan '09

Dear List, after I tried to write a simple ARGOUTVIEW_ARRAY1 example (see http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW#A_simple_ARGOUTVIEW_… ), I started wondering about memory deallocation. Turns out a few clever people already did all the thinking (see http://blog.enthought.com/?p=62 ) and a few more clever people use this in a swig file (see http://niftilib.sourceforge.net/pynifti, file nifticlib.i). All this concentrated knowledge helped me implement a single ARGOUTVIEWM_ARRAY1 fragment in numpy.i to do some testing (see attached numpy.i). How to use it? In yourfile.i, the %init function has a few lines added to it: %init %{ import_array(); /* initialize the new Python type for memory deallocation */ _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) return; %} ... and that's it! then just use ARGOUTVIEWM_ARRAY1 instead of ARGOUTVIEW_ARRAY1 and python does the deallocation for you when the python array is destroyed (see the examples attached). Everything compiles, but it would be nice to get rid of the warnings... ezalloc_wrap.c:2763: warning: initialization from incompatible pointer type ezalloc_wrap.c: In function `_wrap_alloc_managed': ezalloc_wrap.c:2844: warning: assignment from incompatible pointer type writing build\temp.win32-2.5\Release\_ezalloc.def Compilation: python setup_alloc.py build Testing: The attached test_alloc.py does 2048 allocations of 1MB each for managed and unmanaged arrays. Output on my XP laptop with 1GB RAM as follows: ARGOUTVIEWM_ARRAY1 (managed arrays) - 2048 allocations (1048576 bytes each) Done! ARGOUTVIEW_ARRAY1 (unmanaged, leaking) - 2048 allocations (1048576 bytes each) Step 482 failed TODO: Find a better name for the methods (not sure I like ARGOUTVIEWM_ARRAY), then do the missing fragments (2D arrays), clear the warnings and verify the method for a possible inclusion in the official numpy.i file. Thank you for reading! Regards, Egor

2 6

Old-style classes in tests
by Tom Wright 06 Jan '09

06 Jan '09

I am currently working on the Ironclad project porting numpy to Ironpython. It would be quite useful for me if HermitianTestCase in test_linalg.py was a new style-class instead of an old-style class - since Ironpython has a bug where dir operations do not work for classes inheriting from both old- and new- style classes and I'd very much prefer not to patch my version of numpy. In general, it would be useful if whenever this multiple inheritence pattern is used new-style classes are used rather than old style classes. This would require the following classes to change: test_numerictypes - create_values, read_values_plain, read_values_nested test_print - create_zeros, create_values, assign_values, byteorder_values test_io - Roundtriptest test_linalg - LinalgTestCase, HermitianTestCase Would people be ameniable to these change? Thank you very much for your help, Tom Wright

3 2

Alternative to record array
by Jean-Baptiste Rudant 02 Jan '09

02 Jan '09

Hello, I like to use record arrays to access fields by their name, and because they are esay to use with pytables. But I think it's not very effiicient for what I have to do. Maybe I'm misunderstanding something. Example : import numpy as np age = np.random.randint(0, 99, 10e6) weight = np.random.randint(0, 200, 10e6) data = np.rec.fromarrays((age, weight), names='age, weight') # the kind of operations I do is : data.age += data.age + 1 # but it's far less efficient than doing : age += 1 # because I think the record array stores [(age_0, weight_0) ...(age_n, weight_n)] # and not [age0 ... age_n] then [weight_0 ... weight_n]. So I think I don't use record arrays for the right purpose. I only need something which would make me esasy to manipulate data by accessing fields by their name. Am I wrong ? Is their something in numpy for my purpose ? Do I have to implement my own class, with something like : class FieldArray: def __init__(self, array_dict): self.array_list = array_dict def __getitem__(self, field): return self.array_list[field] def __setitem__(self, field, value): self.array_list[field] = value my_arrays = {'age': age, 'weight' : weight} data = FieldArray(my_arrays) data['age'] += 1 Thank you for the help, Jean-Baptiste Rudant

5 6

numpy.test() failures
by Nils Wagner 31 Dec '08

31 Dec '08

>>> numpy.__version__ '1.3.0.dev6283' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 74, in check_complex_type err_msg='Failed str formatting for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed str formatting for type <type 'numpy.complex64'> ACTUAL: '(1e+10+0j)' DESIRED: '1e+10' ====================================================================== FAIL: Check formatting when using print ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 108, in check_float_type_print _test_redirected_print(float(x), tp) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 104, in _test_redirected_print err_msg='print failed for type%s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: print failed for type<type 'numpy.float32'> ACTUAL: '10000000000.0\n' DESIRED: '1e+10\n' ====================================================================== FAIL: Check formatting when using print ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 115, in check_complex_type_print _test_redirected_print(complex(x), tp) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 104, in _test_redirected_print err_msg='print failed for type%s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: print failed for type<type 'numpy.complex64'> ACTUAL: '(10000000000+0j)\n' DESIRED: '(1e+10+0j)\n' ====================================================================== FAIL: test_print.test_locale_single ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 164, in test_locale_single return _test_locale_independance(np.float32) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float32'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_double ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 169, in test_locale_double return _test_locale_independance(np.double) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float64'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_longdouble ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 174, in test_locale_longdouble return _test_locale_independance(np.longdouble) File "/usr/local/lib64/python2.5/site-packages/numpy/core/tests/test_print.py", line 157, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float128'> ACTUAL: '1,2' DESIRED: '1.2' ---------------------------------------------------------------------- Ran 1797 tests in 16.303s FAILED (KNOWNFAIL=1, failures=6)

1 0

Removal of deprecated test framework stuff
by Alan McIntyre 31 Dec '08

31 Dec '08

Hi all, Unless somebody objects, I'd like to remove from NumPy 1.3 the following numpy.testing items that were deprecated in NumPy 1.2 (since the warnings promise we'll do so ;): - ParametricTestCase (also removing the entire file numpy/testing/parametric.py) - The following arguments from numpy.testing.Tester.test() (which is used for module test functions): level, verbosity, all, sys_argv, testcase_pattern - Path manipulation functions: set_package_path, set_local_path, restore_path - NumpyTestCase, NumpyTest Thanks, Alan

2 1

"False" and "True" not singletons?
by Christopher Barker 30 Dec '08

30 Dec '08

Hi all, I've just discovered that "False" is not a singleton: >>> import numpy as N >>> f = N.all((1,2,0)) >>> print f False >>> id(f) 17316364 >>> f is False False >>> id(False) 3294768 Should it be? This arose for me in some tests I'm using that check if a result is False: self.failUnless ( (self.B1 == self.B3) is False ) I'm doing it this way, because I want to make sure that the __eq__ method really is returning "True" or "False", rather than a value that happens to evaluate to true or false, like 1, or an empty list, or whatever. This is interesting to me because back when Python first introduced Booleans, I had thought they should be kept pure and not be subclasses of integers, and, in fact, "if" should only except boolean values. However this opinion was really a matter of my sense of purity, and this is the fist time I've run into a case that matters. I suppose I'm not being pythonic -- I should really only care if the result evaluates true or false, but I don't feel like I'm testing right if values can slip though that shouldn't. It does reinforce my opinion though -- whether zero, or an empty sequence or string should evaluate false really is a matter of specific application, not universal. Anyway, should I just give up? Or should numpy return the same "True" and "False" (like None), or is there another solution? x == False is close, but zero still slips through, even though an empty list does not: >>> 0 == False True >>> [] == False False -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

4 4

numpy.test() failures (1.2.1) on Mac OS X
by Nicholas Matsakis 30 Dec '08

30 Dec '08

I just installed what I believe to be a completely vanilla installation of numpy on an Intel Mac OS X 10.5.6. Python 2.5 pkg from Python.org, numpy 1.2.1 pkg from scipy.org, nose installed through setup tools. Running "import numpy; numpy.test()" results in the following errors and failures: ERROR: Failure: TypeError (can't multiply sequence by non-int of type 'float') ERROR: test_definition (test_helper.TestFFTShift) ERROR: test_inverse (test_helper.TestFFTShift) ERROR: Test of inplace division FAIL: test_division_int (test_umath.TestDivision) FAIL: test_basic (test_index_tricks.TestUnravelIndex) FAIL: Test of inplace division FAIL: test_inplace_division_misc (test_core.TestMaskedArrayInPlaceArithmetics) FAIL: Test of inplace operations and rich comparisons Is this expected? Should I file a ticket? A complete dump of the test run can be found at: http://nick.matsakis.net/tmp/numpy-1.2.1-tests-12-30-2008.txt Nick Matsakis

1 1

NumPy-Discussion December 2008