Mailman 3 February 2008 - NumPy-Discussion

partial_sum/adj_difference?
by Neal Becker 20 Feb '08

20 Feb '08

Does numpy/scipy have a partial_sum and adj_difference function? partial_sum[i] = \sum_{j=0}^{i} x[j] adj_diff[i] = x[i] - x[i-1] : i > 1, x[i] otherwise

8 11

CFLAGS, LDFLAGS, appending and overwriting: what should be the behavior of numscons ?
by David Cournapeau 20 Feb '08

20 Feb '08

Hi, I would like to know what the UI should be for numscons wrt to compilation / link flags. This is an issue which has confused many people with distutils, and something we can fix with numscons. Several approaches are possible for numscons, but I was wondering about the preferred behavior. Right now, by default: - using CFLAGS will only change debug/warning/optimization flags (that is, you can't remove -fPIC with CFLAGS on platforms which need it). I understand than totally overwriting the flags should also be possible (e.g. the current *distutils* behavior). Is handling new compilers the only reason, in which case a better solution is available for numscons through scons tools ? - using LDFLAGS: I do not use it right now. Is anything needed on this front ? More generally, since numscons is easier to use now thanks to the merge into the numpy trunk, and can work on all major platforms since 0.4.1 release, I am interested in getting feedback from numpy developers as well. Internally, I tried to make as few choices as possible, so it should be quite flexible wrt things difficult to handle with distutils. thanks, David

1 0

[ANN] numscons 0.4.1. Building numpy with MS compiler + g77 works !
by David Cournapeau 19 Feb '08

19 Feb '08

Hi, I've just finished a new release of numscons. This version is the first one to be able to compile numpy using MS compilers and g77 compiler on windows, which was the last "big" platform not supported by numscons. This also means I can now refocus on building scipy with numscons, so that the whole numpy/scipy framework can be built with numscons on all platforms. This release is available through pypi (just use easy_install numscons, or easy_install -U numscons to upgrade a previous install), as well as on launchpad: https://launchpad.net/numpy.scons.support/0.4/0.4.1 cheers, David

4 12

Help with user-defined data type
by Neal Becker 18 Feb '08

18 Feb '08

I'm trying out user-defined data type for numpy, but I'm a bit stuck. Maybe someone can spot the problem? I'm creating a descr like this: PyArray_Descr * d = (PyArray_Descr*)(PyObject_New (PyArray_Descr, &PyArrayDescr_Type)); d->typeobj = typeptr; d->kind = 'O'; // ??? d->type = 'z'; d->byteorder = '='; d->hasobject = 0; d->elsize = sizeof (cmplx_int_t); d->alignment = __alignof__ (cmplx_int_t); d->subarray = 0; d->fields = 0; d->names = 0; PyArray_ArrFuncs * f = new PyArray_ArrFuncs; PyArray_InitArrFuncs (f); f->copyswapn = &copyswapn<cmplx_int_t>; f->copyswap = &copyswap<cmplx_int_t>; f->getitem = &getitem<cmplx_int_t>; f->setitem = &setitem<cmplx_int_t>; d->f = f; incref (typeptr); Then register: PyArray_RegisterDataType (d1); This seems OK, I get a typenum 256. But if I try to use it, bad things happen. This tests creating an array, both from typenum and from descr: PyArray_Descr* descr = PyArray_DescrFromType (typenum); npy_intp dims[] = {10}; PyObject * o1 = PyArray_NewFromDescr (&PyArray_Type, descr, 1, dims, 0, 0, 0, 0); PyObject * o2 = PyArray_New (&PyArray_Type, 1, dims, descr->type_num, 0, 0, 0, 0, 0); If I try this for typenum=1, it's fine. But if I use my typenum=256 I get >>> a,b = testit (256) >>> a python: Objects/typeobject.c:1418: extra_ivars: Assertion `t_size >= b_size' failed. Program received signal SIGABRT, Aborted. 0x0000003834c30ec5 in raise () from /lib64/libc.so.6

2 3

MaskedArray __setitem__ Performance
by Alexander Michael 16 Feb '08

16 Feb '08

In part of some code I'm rewriting from carrying around a data and mask array to using MaskedArray, I read data into an array from an input stream. By its nature this a "one at a time" process, so it is basically a loop over assigning single elements (in no predetermined order) of already allocated arrays. Unfortunately, using MaskedArray in this way is significantly slower. The sample code below demonstrates that for this particular procedure, filling the MaskedArray is 32x slower than working with the two arrays I had been carying around. It appears that I can regain the fill performance by working on _data and _mask directly. I can guarantee that the MaskedArrays I'm working with have been created with a dense mask as I've done below (there are always some masked elements, so there is no gain in shrinking to nomask). Is this safe? If not, can I make it safe for this particular performance critical section? I'm assuming that doing array operations won't incur this sort of penalty when I get further into my translation. Some overhead is acceptable for the convenience of not dragging around the mask and thinking about it all of the time, but hopefully less than 2x slower. Thanks! Alex import numpy def get_ndarrays(): return (numpy.zeros((5000,500), dtype=float), numpy.ones((5000,500), dtype=bool)) import timeit t_base = timeit.Timer( 'a[0,0] = 1.0; m[0,0] = False', 'from __main__ import get_ndarrays; a,m = get_ndarrays()' ).timeit(1000)/1000 print t_base 6.97574691756e-007 import numpy.ma def get_maskedarray(): return numpy.ma.MaskedArray( numpy.zeros((5000,500), dtype=float), numpy.ones((5000,500), dtype=bool) ) t_ma = timeit.Timer( 'a[0,0] = 1.0', 'from __main__ import get_maskedarray; a = get_maskedarray()' ).timeit(1000)/1000 print t_ma, t_ma/t_base 2.26880790715e-005 32.5242290749 t_ma_com = timeit.Timer( 'd[0,0] = 1.0; m[0,0] = False', 'from __main__ import get_maskedarray, get_setter; a = get_maskedarray(); d,m = a._data,a._mask' ).timeit(1000)/1000 print t_ma_com, t_ma_com/t_base 7.34450886914e-007 1.05286343612

2 4

best way for C code wrapping
by dmitrey 16 Feb '08

16 Feb '08

hi all, I intend to connect some C code to Python for some my purposes. What is the best software for the aim? Is it numpy.ctypes or swig or something else? IIRC ctypes are present in Python since v2.5, so it's ok to use just ctypes, not numpy.ctypes, or some difference is present? Another one question: if I'll translate a fortran code to C via f2c, which % of speed down should I expect (in average, using gfortran and gcc)? Afaik it contains operations with sparse matrices. Thank you in advance, D.

3 2

changing display options for dtype info for record arrays
by JJ 16 Feb '08

16 Feb '08

Hello: I am starting to use record arrays and would like to know how to keep numpy from displaying the dtype info. For example, I can make a record array containing a long tuple: mydescriptor = dtype([('first', 'f4'),('second', 'f4'), ('third', [(str(x),'<i2') for x in range(1000)] )]) myarr = rec.array( [(1.2, 4.5, tuple([x for x in range(1000)]) )], dtype= mydescriptor ) If in a terminal I type "myarr" and hit return, I get a printout of the data and then a very long list of dtype information. Is there some way to turn this feature off so that only the data is displayed? I realize that "print myarr" does not produce the long dtype list. Also, is there an easier way to set the format for the long tuple, such as using ('third', 'tuple') or some such expression? I know that 'tuple' is not a dtype, just thought there might be some shortcut for formatting long tuples in a rec array. Thanks, John ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs

1 0

test failures when numpy built without atlas libraries.
by Charles R Harris 15 Feb '08

15 Feb '08

When built without ATLAS, the following tests fail : ====================================================================== ERROR: check_testUfuncRegression (numpy.core.tests.test_ma.test_ufuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_ma.py", line 691, in check_testUfuncRegression self.failUnless(eq(ur.filled(0), mr.filled(0), f)) File "/usr/lib/python2.5/site-packages/numpy/ma/core.py", line 1552, in filled result = self._data.copy() File "/usr/lib/python2.5/site-packages/numpy/ma/core.py", line 1474, in _get_data return self.view(self._baseclass) TypeError: Cannot change data-type for object array. ====================================================================== FAIL: Ticket #588 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line 734, in check_dot_negative_stride assert_equal(np.dot(x,z),np.dot(x,y2)) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 143, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 225, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/lib/python2.5/site-packages/numpy/testing/utils.py", line 217, in assert_array_compare assert cond, msg AssertionError: Arrays are not equal (mismatch 100.0%) x: array([[ 55924.]]) y: array([[ 640000.]]) When built with ATLAS the second test passes, but the check_testUfuncRegression error persists. This is at revision 4807. Chuck

1 0

MaskedArray __setitem__ for Record Values
by Alexander Michael 15 Feb '08

15 Feb '08

I am attempting to work with field dtypes (i.e. record arrays) as MaskedArrays. I don't want to use MaskedRecords because I want the mask to apply to the whole record since the first field is primary and the rest are "tag along" values associated with the primary data value. The numpy.ndarray supports the ability to set a record with a (heterogenous) sequence, but MaskedArray does not: >>> import numpy >>> from numpy import ma >>> d = numpy.empty((5,4), dtype=[('value', float),('source', int)]) >>> a = ma.MaskedArray(d, mask=True, fill_value=(0.0,0)) >>> a[0,0] = (10.0, 1) numpy\ma\core.py in __setitem__(self, indx, value) 1359 return 1360 #.... -> 1361 dval = getdata(value).astype(self.dtype) 1362 valmask = getmask(value) 1363 if self._mask is nomask: TypeError: expected a readable buffer object I suggest replacing " dval = getdata(value).astype(self.dtype)" with something along the lines of: dval = None if self.dtype.fields is None: dval = getdata(value).astype(self.dtype) else: dval = value but I do not understand all of the activity in the __setitem__ method, so this may be too naive a fix (but it *appears* to work). Thanks, Alex

2 3

List Array References?
by Alexander Michael 15 Feb '08

15 Feb '08

Is there a way to list all of the arrays that are referencing a given array? Similarly, is there a way to get a list of all arrays that are currently in memory? Thanks, Alex

4 6