Mailman 3 July 2008 - NumPy-Discussion

NumPy, buffers (PEP 3118), complex floats, and Cython
by Dag Sverre Seljebotn 06 Jul '08

06 Jul '08

I'd like some advice for what way people feel would be the best for supporting complex datatypes in NumPy in Cython; as well as ask in what way it is likely that NumPy will make use of PEP 3118. It seems like NumPy defines its complex data to be a struct of two doubles, for instance: typedef struct { double real, imag; } npy_cdouble; According to PEP 3118 [1], it would be natural to export this as "dd" (record of two doubles) rather than "Zd" (complex double), when exporting ndarrays through a buffer interface. Right? I'm implementing native Cython support/syntax candy for PEP 3118 [2] and at the same time a backwards-compatability wrapper so that Cython can access NumPy through PEP 3118 for Python <= 2.5 (essentially I implement PyObject_GetBuffer etc. on behalf of NumPy for Python <= 2.5). Then the question arise about how to deal with complex datatypes. For Cython, it seems most user-friendly to use the C99 standard for complex numbers (not currently supported, but it wouldn't be much work) and have code like this: cdef ndarray[complex double, 2] arr = ... cdef complex double item item = arr[34, 23] So the natural questions then are: - Is it ok to assume that (complex double*) can be safely casted to (npy_cdouble*) on all platforms which both a) NumPy compiles and runs on, and b) supports complex double? (It seems to be ok with GCC but I wouldn't know about other compilers.) - Will NumPy ever be rewritten to use the C99 complex datatype if available? (I.e. #ifdef how npy_cdouble is defined, and define corresponding #ifdef-ed macros for all complex operations). This would remove the need for such a cast. - Does NumPy plan to support PEP3118, and if so, will complex numbers be exported as "dd" or "Zd"? - [1]: http://www.python.org/dev/peps/pep-3118/ [2]: http://wiki.cython.org/enhancements/buffer -- Dag Sverre

2 2

calling numpy from java
by wilson 06 Jul '08

06 Jul '08

hi all, is it possible to use numpy functions (like eigh()..etc)from java code? isthere a java wrapper for numpy? thanks wilson

4 3

Numpy on AIX 5.3
by Marek Wojciechowski 06 Jul '08

06 Jul '08

Hi! I'm trying to install numpy-1.1 on AIX 5.3 but i'm getting an error: running build running scons customize UnixCCompiler Found executable /usr/bin/cc_r customize IBMFCompiler Found executable /usr/bin/xlf90 Found executable /usr/bin/xlf Found executable /usr/bin/xlf95 Creating /tmp/tmp5j_OiW/qV0MJ4_xlf.cfg customize IBMFCompiler Creating /tmp/tmp5j_OiW/-LWcxB_xlf.cfg customize UnixCCompiler customize UnixCCompiler using scons Traceback (most recent call last): File "setup.py", line 96, in <module> setup_package() File "setup.py", line 89, in setup_package configuration=configuration ) File "/home/marek/tmp/numpy-1.1.0/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/home/marek/apython/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/home/marek/apython/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/home/marek/apython/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/home/marek/tmp/numpy-1.1.0/numpy/distutils/command/build.py", line 38, in run self.run_command('scons') File "/home/marek/apython/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/home/marek/apython/lib/python2.5/distutils/dist.py", line 993, in run_command cmd_obj.ensure_finalized() File "/home/marek/apython/lib/python2.5/distutils/cmd.py", line 117, in ensure_finalized self.finalize_options() File "/home/marek/tmp/numpy-1.1.0/numpy/distutils/command/scons.py", line 289, in finalize_options self.cxxcompiler = cxxcompiler.cxx_compiler() File "/home/marek/tmp/numpy-1.1.0/numpy/distutils/ccompiler.py", line 303, in CCompiler_cxx_compiler + cxx.linker_so[2:] TypeError: can only concatenate list (not "str") to list Setting CXX=xlc++_r (which is proper C++ compiler) does not work. How to fix this? Greetings -- Marek Wojciechowski

2 1

failure with numpy.inner
by Gregor Thalhammer 06 Jul '08

06 Jul '08

After upgrading to NumPy 1.1.0 (I installed numpy-1.1.0-win32-superpack-pyhon2.5) I observed a fatal failure with the following code which uses numpy.inner import numpy F = numpy.zeros(shape = (1,79), dtype = numpy.float64) #this suceeds FtF = numpy.inner(F,F.copy()) #this fails FtF = numpy.inner(F,F) The failure (Exception code 0xc0000005) happens in _dotblas.pyd. I use Windows XP on a Intel Core2Duo system. Further, I could observe that it depends on the size of the array wether this failure appears or not. I submitted this report also as ticket #844. I hope somebody can track down the reason for this behaviour (and fix it). Gregor Thalhammer

2 1

ANN: PyTables 2.0.4 released
by Francesc Alted 05 Jul '08

05 Jul '08

=========================== Announcing PyTables 2.0.4 =========================== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. After some months without new versions (I have been busy for a while doing things not related with PyTables, unfortunately), I'm happy to announce the availability of PyTables 2.0.4. It fixes some important issues, and now it is possible to use table selections in threaded environments. Also, ``EArray.truncate(0)`` can be used so that you can completely void existing EArrays (only enabled if you have a recent version, i.e. >= 1.8.0, of the HDF5 library installed). Finally, the usage of recent versions of NumPy (1.1) and HDF5 (1.8.1) has been tested and, fortunately, they work just fine. In case you want to know more in detail what has changed in this version, have a look at ``RELEASE_NOTES.txt``. Find the HTML version for this document at: http://www.pytables.org/moin/ReleaseNotes/Release_2.0.4 You can download a source package of the version 2.0.4 with generated PDF and HTML docs and binaries for Windows from http://www.pytables.org/download/stable/ For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.0.4 *Important note for PyTables Pro users*: due to lack of resources, I'll not be delivering a MacOSX binary version of Pro for the time being (this is pretty easy to compile, though). However, I'll continue offering the all-in-one binary for Windows (32-bit). Migration Notes for PyTables 1.x users ====================================== If you are a user of PyTables 1.x, probably it is worth for you to look at ``MIGRATING_TO_2.x.txt`` file where you will find directions on how to migrate your existing PyTables 1.x apps to the 2.x versions. You can find an HTML version of this document at http://www.pytables.org/moin/ReleaseNotes/Migrating_To_2.x Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last, but not least thanks a lot to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy your data! -- Francesc Alted

1 0

[numscons] 0.8.1 released
by David Cournapeau 05 Jul '08

05 Jul '08

Hi, I am pleased to announce numscons 0.8.1. numscons is a tentative for a new build system for numpy/scipy and other packages depending on numpy.distutils for extension code. You can get it through the usual channels: http://projects.scipy.org/scipy/numpy/wiki/NumScons I did not announce releases of numscons for quite some time, but this did not mean I did not continue working on numscons. The goal is to have a beta available for scipy conference in August, where I will present numscons. Most changes since 0.8.0 are internal, with code simplification (compiler customization, etc...). The goal is to put most of the code upstream into scons (numscons is now around 3700 lines of code, and I hope to be able to reduce this to ~ 2000 lines of code). In particular: - no custom code for fortran anymore, everything is now integrated upstream. - python extensions are now built using a scons tool. This means more reliability, and hopefully at some point upstream integration. There are still some rough edges, but hopefully, nothing fundamental. - VS 2003/2005/2008 are supported as well (numpy can build with python 2.6 alpha and Visual Studio express), using a method similar to distutils in python 2.6 to get informations from the var*.bat files (much simpler and more robust than relying on the brain-dead registry). Since 0.8, the changes are not backward compatible with < 0.8. In particular the names of python builders have changed, and you should now use 2 scons scripts instead of one. I am sorry I have not yet updated the documentation, but I really want to get a beta ready for scipy conference, and I don't have so much time to keep the doc in sync. The examples are up to date, though. cheers, David

1 0

Ctypes required? Fails to build.
by James Turner 05 Jul '08

05 Jul '08

Hello, I'm trying to build Python 2.5.1 on Solaris 9 with the Sun WorkShop 6 compiler, but it is failing to build the ctypes extension. Can anyone tell me whether NumPy 1.1 (or 1.04) can work without ctypes, please? What about SciPy 0.6? Maybe that's a silly question, but I can't see how to make it compile so I'm a bit stuck! I have been through one previous iteration where I tried building NumPy 1.0.4 after ignoring the ctypes problem and it produces undefined symbol errors for various math functions like "exp" and "sqrt". So perhaps that answers my question ... but I'm not 100% clear about it being due to the ctypes problem so I'd be grateful if someone can confirm that. In fact, ctypes is not listed under NumPy "prerequisites" (which, incidentally, is mistyped) in the FAQ and I believe it wasn't part of Python either until 2.5. Thanks, James.

3 4

Python ref count leak in numpy
by Michael Abbott 05 Jul '08

05 Jul '08

I hope this is going to the right place. I've tried to submit a Trac ticket at http://projects.scipy.org/scipy/numpy/ but unfortunately it won't let me log in, even though I've registered and successfully logged in as a SciPy user! Grr. The bug itself is very easy to see in Python debug mode: adding arrays of differing shapes causes the reference count to increase on each operation. For example: array(1) + 1 # Leaks one ref count per call 1 + array(1) # Leaks 12 ref counts per call(!) It turns out that PyArray_CanCoerceScalar is being called rather a lot of times (some of the lower level numpy code is really not nice, which is a pity as there's some really clean code there too) ... and it has a ref count leak. The patch below fixes this problem. This patch was made against version 5331 from subversion. The leak was introduced in revision 2575 in June '06, over two years ago -- I'm a little surprised this hasn't been discovered already. commit ea66a7ee65e1ae855bbc432a48eb1f48176a85d9 Author: Michael Abbott <michael.abbott(a)diamond.ac.uk> Date: Fri Jul 4 12:08:24 2008 +0100 Fix memory leak in PyArray_CanCoerceScalar This routine (in multiarraymodule.c) was calling PyArray_DescrFromType without properly decrementing the reference count afterwards. As this is use widely, this results in memory leaks in the most basic of operations. diff --git a/numpy/core/src/multiarraymodule.c b/numpy/core/src/multiarraymodule.c index 1b752d6..e44ebac 100644 --- a/numpy/core/src/multiarraymodule.c +++ b/numpy/core/src/multiarraymodule.c @@ -2143,8 +2143,13 @@ PyArray_CanCoerceScalar(int thistype, int neededtype, if (from->f->cancastscalarkindto && (castlist = from->f->cancastscalarkindto[scalar])) { while (*castlist != PyArray_NOTYPE) - if (*castlist++ == neededtype) return 1; + if (*castlist++ == neededtype) { + Py_DECREF(from); + return 1; + } } + Py_DECREF(from); + switch(scalar) { case PyArray_BOOL_SCALAR: case PyArray_OBJECT_SCALAR:

5 8

Should the numpy complex print the same way as python complex numbers?
by Charles R Harris 05 Jul '08

05 Jul '08

Currently they print differently: In [6]: str(np.complex128(0)) Out[6]: '(0.0+0.0j)' In [7]: str(complex(0)) Out[7]: '0j' It looks pretty easy to make numpy complex print the same as python complex. Shall we make that change? Chuck

1 0

Combination of element-wise and matrix multiplication
by Jonno 04 Jul '08

04 Jul '08

I have two 2d arrays a & b for example: a=array([c,d],[e,f]) b=array([g,h],[i,j]) Each of the elements of a & b are actually 1d arrays of length N so I guess technically a & b have shape (2,2,N). However I want to matrix multiply a & b to create a 2d array x, where the elements of x are created with element-wise math as so: x[0,0] = c*g+d*i x[0,1] = c*h+d*j x[1,0] = e*g+f*i x[1,1] = e*h+f*j What is the simplest way to do this? I ended up doing the matrix multiplication of a & b manually as above but this doesn't scale very nicely if a & b become larger in size. Cheers, Jonno. -- "If a theory can't produce hypotheses, can't be tested, can't be disproven, and can't make predictions, then it's not a theory and certainly not science." by spisska on Slashdot, Monday April 21, 2008

2 2