Mailman 3 December 2006 - NumPy-Discussion

Fixes to Numexpr under 64 bit platforms
by Ivan Vilata i Balaguer 26 Dec '06

26 Dec '06

Hi all, here you have a patch that fixes some type declaration bugs which cause Numexpr to crash under 64 bit platforms. All of them are confusions between the ``int`` and ``intp`` types, which happen to be the same under 32 bit platforms but not under 64 bit ones, which caused garbage values to be used as shapes and strides. The errors where easy to spot by looking at the warnings yielded by the compiler. Changes have been tested under a Dual Core AMD Opteron 270 running SuSE 10.0 X86-64 with Python 2.4 and 2.5. Have nice holidays, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ Cárabos Coop. V. V V Enjoy Data ""

2 1

Bug in numarray<->numpy interaction
by Norbert Nemec 25 Dec '06

25 Dec '06

The following snippet demonstrates a problem in the interaction of numarray 1.5.2 with numpy 1.0.1 (and older versions): ------------------- #!/usr/bin/env python import numarray, numpy na = numarray.array(0.) np = numpy.array(0.) na[...] = np ------------------- the last linec causes the error "TypeError: NA_setFromPythonScalar: bad value type." The problem occured in a perfectly normal piece of code combining PyTables (internally based on numarray) with calculations done in NumPy. AFAICS, it should work but simply is a bug somewhere in numarray or numpy. Thanks, Norbert

3 3

nested recarrays
by Matthew Koichi Grimes 24 Dec '06

24 Dec '06

(Newbie alert.) I'm having trouble making a nested record array. I'm trying to work from the following example on the scipy.org examples page: >>> mydescriptor = dtype([('x', 'f4'),('y', 'f4'), # nested recarray ... ('nested', [('i', 'i2'),('j','i2')])]) >>> myarr = array([(1.0, 2.0, (1,2))], dtype=mydescriptor) ... but this isn't really a nested recarray, since you can't refer to fields 'x', 'y', or 'nested' as attributes: >>> myarr.x AttributeError: 'numpy.ndarray' object has no attribute 'x' You have to use the more cumbersome bracket notation: >>> myarr['x'] array([ 1.], dtype=float32) When I try modifying the above example by simply replacing the 'array' constructor with 'recarray', I get the following error, which I haven't really grokked yet: >>> myrecarr = N.recarray([(1.0, 2.0, (1,2))], dtype=dt) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/mkg/<ipython console> /usr/lib/python2.4/site-packages/numpy/core/records.py in __new__(subtype, shape, dtype, buf, offset, strides, formats, names, titles, byteorder, aligned) 176 177 if buf is None: --> 178 self = sb.ndarray.__new__(subtype, shape, (record, descr)) 179 else: 180 self = sb.ndarray.__new__(subtype, shape, (record, descr), TypeError: an integer is required I'm trying to make a record array that stores two record arrays called "states" and "controls", each of which store two float arrays "x" and "dx". The dtype would be something like: dtype([('states', [('x', 'f4'), ('dx, 'f4')]), ('controls', [('x', 'f4'), ('dx', 'f4')])]) It'd be great if I could address elements as: myarr.states.x[0] as opposed to myarr['states']['x'][0] Any tips would be greatly appreciated. Once I figure this out, I'd be glad to post the solution as an example under the "recarray()" entry in the examples list. -- Matt

2 1

repr of bool matrix
by Keith Goodman 23 Dec '06

23 Dec '06

Bool matrices and arrays don't line up when you display them because False has 5 letters and True has 4. After several columns it can become impossible to tell which column an element belongs to. Could the repr of bool matrices print 'True ' instead of 'True'? Some truths are more true than others.

2 1

Type of 1st argument in Numexpr where()
by Ivan Vilata i Balaguer 22 Dec '06

22 Dec '06

Hi all, I noticed that the set of ``where()`` functions defined by Numexpr all have a signature like ``xfxx``, i.e. the first argument is a float and the return, second and third arguments are of the same type (whatever it is). Since the first argument effectively represents a condition, wouldn't it make more sense for it to be a boolean? Booleans are already supported by Numexpr, maybe the old signatures are just a legacy from the time when Numexpr didn't support them. I have attached a patch to the latest version of Numexpr which implements this. Cheers, PS: It seems that http://numpy.scipy.org/ still points to the old SourceForge list address. :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ Cárabos Coop. V. V V Enjoy Data ""

2 4

Automatic matrices
by Paulo Jose da Silva e Silva 21 Dec '06

21 Dec '06

Hello, If a numpy user is specially concerned with numerical linear algebra (or more generally with Math), it may find unconvenient the use of the dot function instead of the * operator. This behavior may be specially unpleasant for someone migrating from Matlab. I believe that this is the may reason for the existence of the matrix class within numpy. However, after trying to use the matrix class I have came across a major roadblock: many numpy/scipy functions return an array by default and not matrices. Then, we need then to add many conversion calls to the 'mat' function in our code. This is also unconvenient. I have had the idea of trying to write some code to automatically call the mat function for me. I have a very simple and inefficient prototype now that can be downloaded at: http://www.ime.usp.br/~pjssilva/matrix_import.py The use is simple. Instead of importing a numerical module with import, use the special class MatrixfiedModule from the above file. Let me give an example: --- ipython session --- In [2]:import matrix_import as mi In [3]:num = mi.MatrixfiedModule('numpy') Importing numpy In [4]:la = mi.MatrixfiedModule('scipy.linalg') Importing scipy.linalg In [5]:A = num.random.rand(3,4) Importing numpy.random In [6]:Q, R = la.qr(A) In [7]:la.norm(Q*R - A) Out[7]:6.0555516793379748e-16 ----- End session ----- For now the solution is very inefficient: every function call to a MatrixfiedModule function is wrapped on the fly to search for array return values ad convert them to matrix. This can certainly be improved the wrapping all the functions in the original module first. I plan to add this possibility soon. It is also incomplete: The automatic conversion only happens for return values of module function. It doesn't try to deal with special objects like finfo(float).eps or mgrid[0:9.,0:6.]. I am not sure how to deal with this. I can donate the code to scipy if there is any interest. Any comments? Best, Paulo -- Paulo José da Silva e Silva Professor Assistente, Dep. de Ciência da Computação (Assistant Professor, Computer Science Dept.) Universidade de São Paulo - Brazil e-mail: pjssilva(a)ime.usp.br Web: http://www.ime.usp.br/~pjssilva Teoria é o que não entendemos o (Theory is something we don't) suficiente para chamar de prática. (understand well enough to call) (practice)

5 11

Assignment when there is nothing to assign
by Keith Goodman 21 Dec '06

21 Dec '06

I have the following two lines in a for loop: if len(idx): x[idx,i] = x[idx,i-1] where idx is the output of a where statement. I need the 'if len(idx)' line to prevent an error when idx is empty. Would it make sense to allow x[idx,i] = x[idx,i-1] when idx is empty instead of raising an error? The error I get is ValueError: array is not broadcastable to correct shape

1 1

ANN: PyTables 1.4 released
by Francesc Altet 21 Dec '06

21 Dec '06

=========================== Announcing PyTables 1.4 =========================== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. It is based on the HDF5 library for doing the I/O and leverages the numarray/NumPy/Numeric packages so as to deliver the data to the user in convenient in-memory containers. This is a new major release of PyTables, and probably the last major one of the 1.x series (i.e. with numarray at the core). On it, we have implemented better code to deal with table buffers, enhanced the capability for reading native HDF5 files, enhanced support for 64-bit platforms (but not with Python 2.5: see ``Special Warning`` section below), better support for AIX, optional automatic parent creation and the traditional amount of bug fixes. Go to the PyTables web site for downloading the beast: http://www.pytables.org/ or keep reading for more info about the new features and bugs fixed. Changes more in depth ===================== Improvements: - Table buffers code refactored: now each Row read iterator has its own buffers, completely independent of their table (although write iterators still share a single buffer in the same table). This separation makes the logic of buffering much more clear and less prone to errors (in fact, some of them have been solved). Performance and memory consumption are more or less equal than before. - When flushing the complete file (i.e. when calling File.flush()), only the buffers of those nodes that are alive (i.e. referenced from user code) are actually flushed. This brings much better efficiency (and also stability) to situations where one has to flush (and hence, close) files with many nodes on it. - Better support for AIX by renaming the internal LONLONG_MAX C constant (it was used internally by the xlc compiler). Thanks to Brian Granger for the report. - Added optional automatic parent creation support during node creation, copying and moving operations. See the release notes for more information. - Improved support for Python2.4 and 64-bit platforms (but beware, there are still known issues when using Python2.5 in combination with 64-bit platforms). Thanks to Gerard Vermeulen for his patches for Win64 platforms. - Implemented a workaround for a leak present in numarray --> Numeric conversions when using the array protocol, as can be seen in: http://comments.gmane.org/gmane.comp.python.numeric.general/12563 The workaround can potentially be far slower than the array protocol (because a copy of the arrays is always made), but at least the new code doesn't leak anymore. Bug fixes: - Previously, when the size for memory compounds type was less than the size of the type on disk (for example, when one have padding or aligned fields), PyTables was unable to read info on them. This has been fixed. This allows reading general compound types in HDF5 files written with other tools than PyTables. - When many tables with indexed columns were created simultaneously, a bug make PyTables to crash. This has been fixed (for more info, see bug #26). - Fixed a typo in the code that prevented recognizing complex data in non-PyTables files. - Table.createIndex() now refuses to index complex columns. - Now, it is possible to index several nested columns that hangs from the same column parent. Fixes bug #24. - Fixed a typo in nctoh5 utility that prevented using filters properly. Thanks to Lou Wicker for reporting this. - When setting/appending an array in-memory to an Array (or descendant) object and they have mismatched byteorders, the array was set/appended without being byteswapped first. This has been fixed. Thanks to Elias Collas for the report. Deprecated features: - None Backward-incompatible changes: - Please, see ``RELEASE-NOTES.txt`` file. Special Warning for Python 2.5 and 64-bit platforms users ========================================================= Unfortunately, and due to problems with the combination numarray 1.5.2, Python2.5 and 64-bit platforms, PyTables cannot be safely used yet in such scenario. This will be solved either when numarray can address this issue (hopefully with numarray 1.5.3), or when PyTables 2.x series (with NumPy at its core) will be out. Important note for Windows users ================================ If you are willing to use PyTables with Python 2.4 or 2.5 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win.ZIP Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC (and PowerPC64) and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray About NumPy: http://numpy.scipy.org/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to Acusim (http://www.acusim.com/) for sponsoring many of the job done for releasing this version of PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

1 0

PyArray_DIMS problem
by Gennan Chen 20 Dec '06

20 Dec '06

Hi! I have problem with this function call under FC6 X86_64 for my own numpy extension printf("\n %d %d %d", PyArray_DIM(imgi,0),PyArray_DIM(imgi, 1),PyArray_DIM(imgi,2)) it gave me 166 256 256 if I tried: int *dim; dim = PyArray_DIMS(imgi) printf("\n %d %d %d", dim[0], dim[1], dim[2]); it gave me 166 0 256 Numpy version: In [2]: numpy.__version__ Out[2]: '1.0.2.dev3487' I did test it under OS X 10.4.8 on MacPro. Those two methods gave me the exact results. So, what happens here ?? Gen-Nan Chen

4 4

A question about argmax and argsort
by asaf david 20 Dec '06

20 Dec '06

Hello Let's say i have an N sized array, and i want to get the positions of the K largest items. for K = 1 this is simply argmax. is there any way to generalize it for k !=1? currently I use argsort and take only K items from it, but I'm paying an additional ~lg(N)... Thanks in advance, asaf

4 3