Mailman 3 September 2005 - NumPy-Discussion

A bug in numarray?
by Nadav Horesh 22 Sep '05

22 Sep '05

It seems that the tostring method fails on rank 0 arrays: a = N.array(-4) >>> a array(-4) >>> a.tostring() Traceback (most recent call last): File "<pyshell#18>", line 1, in -toplevel- a.tostring() File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", line 746, in tostring self._strides, self._itemsize) MemoryError >>> N.__version__ '1.4.0' Nadav.

2 1

numarray speed problem
by Humufr 20 Sep '05

20 Sep '05

Hello, I have a problem with numarray and especially the function numarray.all. I want to compare two files to do this I read the files with a function readcol2 who can put them in a list or numarray format (string or numerical). I'm doing a comparaison on each line of the file. If I'm using the array format and the numarray.all function, that take forever to do the comparaison for 2 big files. If I'm using python list object, it's very fast. I think there are some problem or at least some improvement to do. If I understand correctly the goal of numarray, it has been write to speed up some part of python but here it slow down a lot. An very simple sample to see the effect is at the bottom of this mail. Thanks for numarray, I hope to not bother you. My comments are more to improve numarray than other things. I have been able to find the problem so no I can avoied it. H. def readcol(fname,comments='%',columns=None,delimiter=None,dep=0,arraytype='list'): """ Load ASCII data from fname into an array and return the array. The data must be regular, same number of values in every row fname can be a filename or a file handle. Input: - Fname : the name of the file to read Optionnal input: - comments : a string to indicate the charactor to delimit the domments. the default is the matlab character '%'. - columns : list or tuple ho contains the columns to use. - delimiter : a string to delimit the columns - dep : an integer to indicate from which line you want to begin to use the file (useful to avoid the descriptions lines) - arraytype : a string to indicate which kind of array you want ot have: numeric array (numeric) or character array (numstring) or list (list). By default it's the list mode used matfile data is not currently supported, but see Nigel Wade's matfile ftp://ion.le.ac.uk/matfile/matfile.tar.gz Example usage: x,y = transpose(readcol('test.dat')) # data in two columns X = readcol('test.dat') # a matrix of data x = readcol('test.dat') # a single column of data x = readcol('test.dat,'#') # the character use like a comment delimiter is '#' initial function from pylab (J.Hunter). Change by myself for my specific need """ from numarray import array,transpose fh = file(fname) X = [] numCols = None nline = 0 if columns is None: for line in fh: nline += 1 if dep is not None and nline <= dep: continue line = line[:line.find(comments)].strip() if not len(line): continue if arraytype=='numeric': row = [float(val) for val in line.split(delimiter)] else: row = [val.strip() for val in line.split(delimiter)] thisLen = len(row) if numCols is not None and thisLen != numCols: raise ValueError('All rows must have the same number of columns') X.append(row) else: for line in fh: nline +=1 if dep is not None and nline <= dep: continue line = line[:line.find(comments)].strip() if not len(line): continue row = line.split(delimiter) if arraytype=='numeric': row = [float(row[i-1]) for i in columns] elif arraytype=='numstring': row = [row[i-1].strip() for i in columns] else: row = [row[i-1].strip() for i in columns] thisLen = len(row) if numCols is not None and thisLen != numCols: raise ValueError('All rows must have the same number of columns') X.append(row) if arraytype=='numeric': X = array(X) r,c = X.shape if r==1 or c==1: X.shape = max([r,c]), elif arraytype == 'numstring': import numarray.strings # pb if numeric+pylab X = numarray.strings.array(X) r,c = X.shape if r==1 or c==1: X.shape = max([r,c]), return X ------------------------------------------- files_test_creation.py ------------------------------------------- f1 = file('test1.dat','w') for i in range(10000): f1.write(str(i)+' '+str(i+1)+' '+str(i+2)+'\n') f1.close() f2 = file('test2.dat','w') for i in range(10000): f2.write(str(i)+' '+str(i+1)+' '+str(i+2)+'\n') f2.close() ------------------------------------------- numarray_pb_sample.py ------------------------------------------- import numarray data1 = readcol2.readcol('test1.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='numstring') data2 = readcol2.readcol('test2.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='numstring') #or in non string array form (same result) ## data1 = readcol2.readcol('test1.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='numeric') ## data2 = readcol2.readcol('test2.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='numeric') for a_i in range(data1.shape[0]): for b_i in range(data2.shape[0]): if numarray.all(data1[a_i,:] == data2[b_i,:]): print a_i,b_i ------------------------------------------- python_list_sample.py ------------------------------------------- data1 = readcol2.readcol('test1.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='list') data2 = readcol2.readcol('test2.dat',columns=[1,2,3],comments='#',delimiter=' ',dep=1,arraytype='list') for a_i in range(len(data1)): for b_i in range(len(data2)): if data1[a_i] == data2[b_i]: print a_i,b_i

2 2

How to install Numeric on Zope?
by shashank karnik 18 Sep '05

18 Sep '05

Hello everyone Can anyone help me to install Numeric extension or package for my Zope server 2.4 version running on Windows Xp? You see i am a beginner at python and zope both and need to use a product in Zope(GNOWSYS) which requires Python -Numeric and Python-XMLBase packages.. I downloaded the Numeric py package for Windows from this link http://prdownloads.sourceforge.net/numpy/Numeric-23.8.win32-py2.4.exe?downl… However...i cant figure out how to install it so that my Zope server recognises it. The installer currently just installs the package so that it is recognised by the Python interpreter...but the Zope server is stored in Program Files on my machine..how do i make it understand that the Numeric package has been installed.. I tried putting the numeric folder in this path C:\Program Files\Zope\lib\python\Products But it doesnt work Please help me out If think that this question should be asked on a Zope forum...please let me know! Thank you

2 1

scipy core now builds from svn
by Travis Oliphant 17 Sep '05

17 Sep '05

This is to officially announce that the new replacement for Numeric (scipy_core) is available at SVN. Read permission is open to everyone so a simple checkout: svn co http://svn.scipy.org/svn/scipy_core/branches/newcore newcore should get you the distribution that should install with cd newcore python setup.py install I'm in the process of adding the linear algebra routines, fft, random, and dotblas from Numeric. This should be done by the conference. I will make a windows binary release for the SciPy conference, but not before then. There is a script in newcore/scipy/base/convertcode.py that will take code written for Numeric (or numerix) and convert it to code for the new scipy base object. This code is not foolproof, but it takes care of the minor incompatibilities (a few search and replaces are done). The compatibility issues are documented (mostly in the typecode characters and a few method name changes). The one bigger incompatibility is that a.flat does something a little different (a 1-d iterator object). The convert code script changes uses of a.flat that are not indexing or set attribute related to a.ravel() C-code should build for the new system with a change of #include Numeric/arrayobject.h to #include scipy/arrayobject.h --- though you may want to enhance your code to take advantage of the new features (and more extensive C-API). I also still need to add the following ufuncs: isnan, isfinite, signbit, isinf, frexp, and ldexp. This should not take too long. -Travis O.

3 5

scipy.base (Numeric3) now on scipy.org svn server
by Travis Oliphant 14 Sep '05

14 Sep '05

Now that the new scipy.base (that can replace Numeric) is pretty much complete, I'm working on bringing the other parts of Numeric along so that scipy_core can replace Numeric (and numarray in functionality) for all users. I'm now using a branch of scipy_core to do this work. The old Numeric3 CVS directory on sourceforge will start to wither... The branch is at http://svn.scipy.org/svn/scipy_core/branches/newcore I'm thinking about how to structure the new scipy_core. Right now under the new scipy_core we have Hierarchy Imports as ==================== base/ --> scipy.base (namespace also available under scipy itself) distutils/ --> scipy.distutils test/ --> scipy.test weave/ --> weave We need to bring over basic linear algebra, statistics, and fft's from Numeric. So where do we put them and how do they import? Items to consider: * the basic functionality will be expanded / replaced by anybody who installs the entire scipy library. * are we going to get f2py to live in scipy_core (I say yes)... * I think scipy_core should install a working basic scipy (i.e. import scipy as Numeric) should work and be an effective replacement for import Numeric). Of course the functionality will be quite a bit less than if full scipy was installed, but basic functions should still work. With that in mind I propose the additions Hiearchy Imports as ========================== corelib/lapack_lite/ --> scipy.lapack_lite corelib/fftpack_lite/ --> scipy.fftpack_lite corelib/random_lite/ --> scipy.random_lite linalg/ --> scipy.linalg fftpack/ --> scipy.fftpack stats/ --> scipy.stats Users would typically use only the functions in scipy.linalg, scipy.fftpack, and scipy.stats. Notice that scipy also has modules names linalg, fftpack, and stats. These would add / replace functionality available in the basic core system. Comments, -Travis O.

2 1

ANN: PyTables 1.1.1 released
by Francesc Altet 14 Sep '05

14 Sep '05

========================== Announcing PyTables 1.1.1 ========================== This is a maintenance release of PyTables. In it, several optimizations and bug fixes have been made. As some of the fixed bugs were quite important, it's strongly recommended for users to upgrade. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ or keep reading for more info about the improvements and bugs fixed. Changes more in depth ===================== Improvements: - Optimized the opening of files with a large number of objects. Now, files with table objects open a 50% faster, and files with arrays open more than twice as fast (up to 2000 objects/s on a Pentium 4@2GHz). Hence, a file with a combination of both kinds of objects opens between a 50% and 100% faster than in 1.1. - Optimized the creation of ``NestedRecArray`` objects using ``NumArray`` objects as columns, so that filling a table with the ``Table.append()`` method achieves a performance similar to PyTables pre-1.1 releases. Bug fixes: - ``Table.readCoordinates()`` now converts the coords parameter into ``Int64`` indices automatically. - Fixed a bug that prevented appending to tables (though ``Table.append()``) using a list of ``NumArray`` objects. - ``Int32`` attributes are handled correctly in 64-bit platforms now. - Correction for accepting lists of numarrays as input for ``NestedRecArrays``. - Fixed a problem when creating rank 1 multi-dimensional string columns in ``Table`` objects. Closes SF bug #1269023. - Avoid errors when unpickling objects stored in attributes. See the section ``AttributeSet`` in the reference chapter of the User's Manual for more information. Closes SF bug #1254636. - Assignment for ``*Array`` slices has been improved in order to solve some issues with shapes. Closes SF bug #1288792. - The indexation properties were lost in case the table was closed before an index was created. Now, these properties are saved even in this case. Known bugs: - Classes inheriting from ``IsDescription`` subclasses do not inherit columns defined in the super-class. See SF bug #1207732 for more info. - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of a HDF5 limitation. See SF bug #1234709 for more info. Backward-incompatible changes: - None (that we are aware of). Important note for MacOSX users =============================== UCL compressor works badly on MacOSX platforms. Recent investigation seems to point to a bug in the development tools in MacOSX. Until the problem is isolated and eventually solved, UCL support will not be compiled by default on MacOSX platforms, even if the installer finds it in the system. However, if you still want to get UCL support on MacOSX, you can use the ``--force-ucl`` flag in ``setup.py``. Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and numarray (Numeric is also supported) package for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further problems. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://pytables.sourceforge.net/ About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package!. And last but not least, a big thanks to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

1 0

[Fwd: compression in storage of Numeric/numarray objects]
by Joost van Evert 12 Sep '05

12 Sep '05

6 6

Linux to Windows porting question
by Daniel Sheltraw 09 Sep '05

09 Sep '05

Hello NumPy Listees I am trying to port some code to Windows that works fine under Linux. The offending line is: blk = fromstring(f_fid.read(BLOCK_LEN), num_type).byteswapped().astype(Float32).tostring() The error I get is: ValueError: string size must be a multiple of element size Does anyone have an idea where the problem might be? BLOCK_LEN is specified in bytes and num_type is Int32. Thanks, Daniel

4 4

Security Measures - Are You Traveling?
by PayPal Security Departament 08 Sep '05

08 Sep '05

1 0

scipy core (Numeric3) win32 binaries to play with
by Travis Oliphant 05 Sep '05

05 Sep '05

<http://www.scipy.org/download/misc/folder_contents> If anybody has just been waiting for a windows binary to try out the new Numeric (scipy.base) you can download this. from scipy.base import * (replaces from Numeric import *) The installer is here: http://numeric.scipy.org/files/scipy_core-0.4.0.win32-py2.4.exe <http://www.scipy.org/download/misc/folder_contents>

3 2