Mailman 3 December 2005 - NumPy-Discussion

Example of power of new data-type descriptors.
by Travis Oliphant 28 Dec '05

28 Dec '05

I'd like more people to know about the new power that is in scipy core due to the general data-type descriptors that can now be used to define numeric arrays. Towards that effort here is a simple example (be sure to use latest SVN -- there were a coupld of minor changes that improve usability made recently). Notice this example does not use a special "record" array subclass. This is just a regular array. I'm kind of intrigued (though not motivated to pursue) the possibility of accessing (or defining) databases directly into scipy_core arrays using the record functionality. # Define a new data-type descriptor >>> import scipy >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], 'formats': ['S30', 'i2', 'f4']}) >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) # the argument to dtypedescr could have also been placed here as the argument to dtype >>> print a['name'] [Bill Fred] >>> print a['age'] [31 15] >>> print a['weight'] [ 260. 135.] >>> print a[0] ('Bill', 31, 260.0) >>> print a[1] ('Fred', 15, 135.0) It seems to me there are some very interesting possibilities with this new ability. The record array subclass adds an improved scalar type (the record) and attribute access to get at the fields: (e.g. a.name, a.age, and a.weight). But, if you don't need attribute access you can use regular arrays to do a lot of what you might need a record array to accomplish for you. I'd love to see what people come up with using this new facility. The new array PEP for Python basically proposes adding a very simple array object (just the basic PyArrayObject * of Numeric with a bare-bones type-object table) plus this new data-type descriptor object to Python and a very few builtin data-type descriptors (perhaps just object initially). This would basically add the array interface to Python directly and allow people to start using it generally. The PEP is slow going because it is not on my priority list right now because it is not essential to making scipy_core work well. But, I would love to have more people ruminating on the basic ideas which I think are crystallizing. Best wishes for a new year, -Travis Oliphant

2 2

Building numarray with atlas on FC4
by Christopher Barker 27 Dec '05

27 Dec '05

Hi all, I just built numarray with atlas on Fedora core 4 Linux. I used the copy of atlas in extras (yum install atlas-sse2). To get it to build, I edited cfg_packages.py this way: elif os.path.exists('/usr/local/lib/atlas'): # uses atlas, installed in /usr/local/lib lapack_dirs = ['/usr/local/lib/atlas'] lapack_include_dirs += ["/usr/local/include/atlas"] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] elif os.path.exists('/usr/lib/sse2/'): # uses atlas, installed in /usr/lib/sse2 ( Fedora Core4 extras ) lapack_dirs = ['/usr/lib/sse2'] lapack_include_dirs += ["/usr/include/atlas"] lapack_libs = ['lapack_atlas','lapack', 'blas'] Did I do that right? would you like to add that stanza to the distribution? I do think it's a good idea to check for the existence of the atlas dir before assuming that it's there. Maybe an exception could be raised if nothing is found? I did find it odd that I couldn't find instructions for how to do this. did I miss something? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

WE WISH numpy-discussion A MERRY CHRISTMAS
by Fernando Allen 25 Dec '05

25 Dec '05

Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold

1 0

Re: scipy.stats.itemfreq: overflow with add.reduce
by Tim Churches 22 Dec '05

22 Dec '05

Hans Georg Krauthaeuser wrote: > Hans Georg Krauthaeuser schrieb: > >>Hi All, >> >>I was playing with scipy.stats.itemfreq when I observed the following >>overflow: >> >>In [119]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, stats.itemfreq(l), l.count(0) >> .....: >>254 [ [ 0 254]] 254 >>255 [ [ 0 255]] 255 >>256 [ [0 0]] 256 >>257 [ [0 1]] 257 >>258 [ [0 2]] 258 >> >>itemfreq is pretty small (in stats.py): >> >>---------------------------------------------------------------------- >>def itemfreq(a): >> """ >>Returns a 2D array of item frequencies. Column 1 contains item values, >>column 2 contains their respective counts. Assumes a 1D array is passed. >> >>Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>""" >> scores = _support.unique(a) >> scores = sort(scores) >> freq = zeros(len(scores)) >> for i in range(len(scores)): >> freq[i] = add.reduce(equal(a,scores[i])) >> return array(_support.abut(scores, freq)) >>---------------------------------------------------------------------- >> >>It seems that add.reduce is the source for the overflow: >> >>In [116]:from scipy import * >> >>In [117]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, add.reduce(equal(l,0)) >> .....: >>254 254 >>255 255 >>256 0 >>257 1 >>258 2 >> >>Is there any possibility to avoid the overflow? Apropos the preceding, herewith a thread on the Numpy list from a more than a few months ago. The take-home message is that for integer arrays, add.reduce is very fast at producing results which fall into two categories: a) correct or b) incorrect due to overflow. Unfortunately there is no equally quick method of determining into which of these two categories any specific result returned by add.reduce falls. Tim C

3 2

ANN: PyTables 1.2.1
by Francesc Altet 21 Dec '05

21 Dec '05

========================= Announcing PyTables 1.2.1 ========================= I'm happy to announce a new version of PyTables. This is a maintenance version and only bugs has been fixed in it. In particular, the documentation has not change at all, so it is not necessary that you look at it for changes. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ or keep reading for more info about the bugs fixed in this version. I take the opportunity to announce as well that PyTables is adopting the bazaar-ng (http://www.bazaar-ng.org/) distributed version control, in order to easy the contributions from developers throughout the world. See more info about this new facility (currently in beta) in: http://sourceforge.net/mailarchive/forum.php?thread_id=9253854&forum_id=137… Changes more in depth ===================== Bug fixes: - Table.flush() is called automatically before disposing a table object from the user space. This avoids a problem that appears when the user does not explicitely do the flush and the table is unbounded and rebounded after on (using h5file.getNode() for example). - A small typo has been fixed in the ptrepack utility. This prevented ptrepack from working correctly when asking for statistics on operations done (-v flag). Known issues: - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of an HDF5 limitation. See SF bug #1234709 for more info. Backward-incompatible changes: - None. Important note for MacOSX users =============================== From PyTables 1.2 on, the UCL compressor seems to work well again on MacOSX platforms. We don't know exactly why, but the fact is that all the test suite passes (using UCL) executes flawlessly. So, from now on, support for UCL in MacOSX is enabled again by default (i.e. you don't need to use the flag ``--force-ucl``, which has disappeared). Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and numarray (Numeric is also supported) package for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://pytables.sourceforge.net/ About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (necessarily incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

1 0

A patch to fix a Numeric 24.2 dotblas innerproduct problem
by Janne Sinkkonen 14 Dec '05

14 Dec '05

With Numeric 24.2 one gets the following error: >>> innerproduct(ones((10,3),Float), ones((10,3),Float)) ldc must be >= MAX(N,1): ldc=3 N=10Parameter 14 to routine cblas_dgemm was incorrect It seems to be fixed by the following patch to Packages/dotblas/dotblas/_dotblas.c : --- _dotblas.c~ 2005-04-07 08:15:47.000000000 +0300 +++ _dotblas.c 2005-12-15 11:31:46.148482242 +0200 @@ -530,28 +530,28 @@ ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (double *)ap1->data, lda, (double *)ap2->data, ldb, - 0.0, (double *)ret->data, ldb); + 0.0, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_FLOAT) { cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (float *)ap1->data, lda, (float *)ap2->data, ldb, - 0.0, (float *)ret->data, ldb); + 0.0, (float *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CDOUBLE) { cblas_zgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneD, (double *)ap1->data, lda, (double *)ap2->data, ldb, - zeroD, (double *)ret->data, ldb); + zeroD, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CFLOAT) { cblas_cgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneF, (float *)ap1->data, lda, (float *)ap2->data, ldb, - zeroF, (float *)ret->data, ldb); + zeroF, (float *)ret->data, ap1->dimensions[0]); } } else { -- Janne Sinkkonen, PhD Xtract Ltd

2 1

Re: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core
by Peter Verveer 14 Dec '05

14 Dec '05

On 13 Dec, 2005, at 23:39, Travis Oliphant wrote: > >> However, even though I am the author, I am most likely not going >> to support it outside numarray. I do realize that at some point >> numarray may be retired, but I have not decided yet if I will be >> using/supporting nd_image beyond that point. It would be nice to >> have a common code-base with the numarray version so that all can >> profit from any bug-fixes that I am still doing in the numarray >> version. > > > So, do I take this to mean that you are not enthusiastic about a > merger between the Numeric and numarray communities? I would love > to hear why you feel this way. > From my perspective, it just makes unification that much harder to > have important third-party authors make these kinds of statements, > so I'm curious as to why you don't want to convert to a common code- > base. Because others may have the same feelings, it would help all > of us if we could address them. Don't worry about offending me or > others in the scipy world. We have pretty thick skins by this point. I think I should clarify what I intent to do with nd_image in the future. It affects current nd_image users and the numarray package also, so I am also sending this to the numpy list. First of all, I should stress that I am quite happy about a possible merger. It is quite likely that I will be using python for numerical work in the future, and a merger of numarray and Numeric into a package of equal or better quality will only make my life easier. Currently I am the only software developer within a scientific laboratory, but software development is not my main focus. I used python with numarray for virtual all of my numerical work, and nd_image was sort of a side effect of that. Right now I do not have the time to make a switch to a new package, nor can I really further expand nd_image. It would be nice to try moving to scipy_core now, which would help the community too I guess, but I just cannot affort it at the moment, so I am sticking with numarray for the time being. My situation will most likely change next year, since our group will move and expand a lot, and I will hopefully be able to have other people working on software development. I guess around that time we will see scipy_core mature, so that would be a good time to switch. I understand the numarray people take a wait-and-see approach, but intent to switch when scipy_core matures and fulfills their requirements. I guess I am following their example. So next year I will might be in the position of having more resources and I then will start thinking about developing some open-source scientific software. An imaging package like nd_image would be one of them. However, in contrast to the current design, I would like to move to a package that is fully written in C, with a thin interface layer to numarray/scipy_core for use with python. That would allow me to use the routines in other environments than python,. Sofar I have been very comfortable to make my package available under the license that is used by numarray. The same would be true for a scipy_core/scipy version. However, for a general C package that could have a much broader audience in the same way, I feel different. I would most likely release such a package using the GPL. In addition, that would allow me to use other GPL licensed software such as GSL with it. However, that would mean that a python package based on it would also be released with that license. I am not complete sure what the implications for distribution with a package like scipy would be, this may not be possible. But it would be freely available as a separate module which we would maintain ourself. If it pans out this way, it means that nd_image will not be further developed at some point. I am currently doing bug-fixes in the numarray version, and I certainly would not mind if there would be a common code-base allowing it to work with scipy_core. But if I am going to developer package as described above, I will to focus on that and recommend nd_image users to switch at some point. > > Even if your statement is made from the perspective of someone who > is moving away from Python altogether, I would still very-much > appreciate hearing your reasons. Now, perhaps, your just going to > experience a career-change altogether in which case, best wishes, > and thanks for all you've contributed. I will certainly stick with Python :-) > > Regardless of whether or not you have time to respond, I do > appreciate all your efforts. The nd_image package is very nice as > I've told you before. Thank you! Thanks! nd_image seems to find some users, and I would like to continue to provide something like nd_image. It might be a while though until nd_image or a replacement will start moving again and further developing. Meanwhile I am committed to keep nd_image working with numarray and doing minor bug-fixes. If we can at some point find a solution for a single code-base which would work with scipy_core, I am happy to expand this commitment to scipy_core. I cannot promise indefinite support, but as long as I am in the field and as long as I do not release a 'successor' package, I will try... Cheers, Peter

4 3

numarray bug: random_array/dtest.py fails on 64 bit machine
by Edward C. Jones 13 Dec '05

13 Dec '05

I have an Athlon 64 chip. I use Debian unstable but I compile all Python related programs myself. They go into /usr/local. I have Python 2.4.2 and numarray 1.5.0. I got the following error from testall.test(): File "/usr/local/lib/python2.4/site-packages/numarray/random_array/dtest.py", line 57, in numarray.random_array.dtest Failed example: x = multivariate_normal(num.array([10,20]), num.array(([1,2],[2,4]))); x Expected: array([ 9.95170432, 19.90340867]) Got: array([ 9.95170432, 19.90340866])

1 0

Re: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core
by Travis Oliphant 12 Dec '05

12 Dec '05

>That certainly would be nice. We are starting to look migrating some >Python code. It may be a little while >(a couple months?) before we can start tackling migrating some of the C >extension code so we won't be exercising that right away (but maybe >someone else can). > > I think I will have something basic in-place by then. We can add needed API calls as extensions get ported. On a related note, where should the Packages (like nd_image) from numarray go? Should they all go into scipy core, full scipy, or be separately downloadable as scipy_core addons? -Travis

2 1

Thoughts on making it easier for numarray users to transition to scipy_core
by Travis Oliphant 12 Dec '05

12 Dec '05

A few days ago I played with nd_image and was able to make it compile for scipy_core. In the process, I had some ideas for making the transition to scipy_core easier for numarray users: 1) It would be nice if there was someway to document on the Python level any needed changes. In the process, we might find things that need to be added to scipy_core. I would like to have some kind of program for automatically making most of those needed changes before a 1.0 release of scipy_core. 2) On the C-API side, my experience with nd_image showed me that quite a few of the numarray C-API calls can be written as macros while some will need to be additional functions. I think we could easily write a numcompatmodule.c and associated numcompat.h file so that users needing the numarray C-API could include the numcompat.h file and then the import_libnumarray() command would load the numcompatmodule.c with it's compatibility functions. In this way, the transition for C-API users could be as easy as changing the include file to numcompat.h? -Travis

2 1