Mailman 3 July 2003 - SciPy-User

ANN: PyTables 0.7
by Francesc Alted July 31, 2003

July 31, 2003

Announcing PyTables 0.7 ----------------------- PyTables is a hierarchical database package designed to efficently manage very large amounts of data. PyTables is built on top of the HDF5 library and the numarray package and features an object-oriented interface that, combined with C-code generated from Pyrex sources, makes it a fast, yet extremely easy to use tool for interactively save and retrieve large amounts of data. Release 0.7 is the third public beta release. The version 0.6 was internal and will never be released. On this release you will find: - new AttributeSet class - 25% I/O speed improvement - fully multidimensional table cells support - new column descriptors - row deletion in tables is finally here - much more! More in detail: What's new ----------- - A new AttributeSet class has been added. This will allow the addition and deletion of generic attributes (any scalar type plus any Python object supported by Pickle) as easy as this: table.attrs.date = "2003/07/28 10:32" # Attach a string to table group._v_attrs.tempShift = 1.2 # Attach a float to group array.attrs.detectorList = [1,2,3,4] # Attach a list to array del array.attrs.detectorList # Detach detectorList attr from array - PyTables now has support for fully multidimensional table cells. This has been made possible in part by implementation of multidimensional cells in numarray.records.RecArray object. Thanks to numarray crew, and especially to Jin-chung Hsu, for willingly accepting to do that, and also for including some cache improvements in RecArray. - New column descriptors added: IntCol, Int8Col, UInt8Col, Int16Col, UInt16Col, Int32Col, UInt32Col, Int64Col, UInt64Col, FloatCol, Float32Col, Float64Col and StringCol. I think they are more explicit and easy-to-use than the now deprecated (but still supported) Col() descriptor. All the examples and user's manual has been accordingly updated. - The new Table.removeRows(start, stop) function allows you to remove rows from tables. This feature was requested a long time ago. There are still limitations, however: you cannot delete rows in extremely large Tables (as the remaining rows after the stop parameter are stored in memory). Nor is the performance optimized. These issues will hopefully be addressed in future releases. - Added iterators to File, Group and Table (they now support the special __iter__() method). They make the object much more user-friendly, especially in interactive mode. See documentation for usage examples. - Added a __getitem__() method to Table that works more or less like read(), but with extended slices support. - As a consequence of rewriting table iterators in C (with the help of Pyrex, of course) the table read performance has been improved between 20% and 30%. Data selections in PyTables are now starting to beat powerful relational databases like SQLite, even compared to in-core selects (!). I think there is still room for another 20% or 30% speed improvement, so stay tuned. - A checksum is now added automatically when using LZO (not with UCL where I'm having some difficulties implementing that capability). The Adler32 algorithm has been chosen because of its speed. With that, the compressing/decompressing speed has dropped 1% or 2%, which is hardly noticeable. I think this addition will allow the cautious user to be a bit more confident about this excellent compressor. Code has been added to be able to read files created without this checksum (so you can be confident that you will be able to read your existing files compressed with LZO and UCL). - Recursion has been removed from PyTables. Before, this made the maximum depth tree to be less than the Python recursion limit (which depends on implementation, but is around 900, at least in Linux). Now, the limit has been set (somewhat arbitrarily) at 2048. Thanks to John Nielsen for implementing the new iterative method!. - A new rootUEP parameter to openFile() has been added. You can now define the root from which you want to start to build the object tree. Thanks to John Nielsen for the suggestion and a first implementation. - A small bug fixed when dealing with non-native PyTables files that prevented the use of the "classname" filter during a listNodes() call. Thanks to Jeff Robbins for reporting that. - Some (non-serious) bugs were discovered and fixed. - Updated documentation to explain all these new bells and whistles. It is also available on the web: http://pytables.sourceforge.net/html-doc/usersguide-html.html - Added more unit tests (more than 350 now!) - PyTables 0.7 *needs* numarray 0.6 or higher and HDF-1.6.0 or higher to compile and work. It has been tested with Python 2.2 and 2.3 and should work fine on both versions. What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seems to be quite a strange requirement for an language like Python, that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors and running IRIX 6.5. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP, but it should also work with other flavors. An example? ----------- For online code examples, have a look at http://pytables.sourceforge.net/tut/tutorial1-1.html and http://pytables.sourceforge.net/tut/tutorial1-2.html Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted falted(a)openlc.org

1 0

Please help : wxplt line-plot to box-plot?
by Sajec, Mike TQO July 30, 2003

July 30, 2003

Hello, Can anyone give an example (or fill in below) how to change a value's plot-type from line to bar without manually editing through chaco? I'd like to create a histogram generating function based on wxplt.plot . Thanks in advance, Mike ################################### from scipy import * from chaco import wxplt as plt import RandomArray as ra #data for testing... data = ra.normal(0,3,100) mu = mean(data) sigma = std(data) mhist = stats.histogram(data) xhist = mhist[1] yhist = mhist[0] fig1 = plt.plot(xhist,yhist) ## CHANGE FROM LINE PLOT TO BAR PLOT????? ## # fig1.canvas.???

1 0

gui_thread
by daishi＠egcrc.net July 30, 2003

July 30, 2003

I'm having some problems following the tutorial found here: http://www.scipy.org/site_content/tutorials/plot_tutorial In particular, when in the Python interpreter I do >>> import gui_thread nothing seems to happen - I simply get an interpreter prompt. I took a look at gui_thread/__init__.py, and it appears that there are only import statements - is there something that I'm missing? I noticed that in a previous version of __init__.py there was a call to main.start(), so I tried re-inserting that into the file, but this simply hangs the interpreter. (The interpreter must be killed from another terminal). What is somewhat odd is that if I leave the code as is (i.e., without the explicit call to main.start()), and do: >>> import gui_thread >>> gui_thread.main.start() Things seem to work as advertised. I.e., <Importing wxPython... done.> >>> from scipy import plt >>> plt.plot(range(3)) <scipy.plt.wxplt.plot_frame instance; proxy of C++ wxFrame instance at _8664858_wxFrame_p> >>> shows an appropriate window. Is there a reason why calling main.start() from the interpreter behaves differently then when called from within the __init__.py? (I realize the contexts are different, but having looked at what main.start() does, it's not clear to me why that would matter). On a separate note, trying to use Chaco results in the following: >>> from chaco import wxplt Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.2/site-packages/chaco/wxplt.py", line 19, in ? from wxplot import * File "/usr/local/lib/python2.2/site-packages/chaco/wxplot.py", line 17, in ? from wxplot_window import PlotWindow, PlotTraitSheet File "/usr/local/lib/python2.2/site-packages/chaco/wxplot_window.py", line 25, in ? from wxplot_trait_sheet import TraitPanel, TraitSheet, trait_sheet_handler File "/usr/local/lib/python2.2/site-packages/chaco/wxplot_trait_sheet.py", line 18, in ? from traits.wxtrait_sheet import * File "/usr/local/lib/python2.2/site-packages/traits/__init__.py", line 20, in ? from traits import HasTraits, HasDynamicTraits, Trait, Disallow, \ File "/usr/local/lib/python2.2/site-packages/traits/traits.py", line 100, in ? from trait_delegates import TraitDelegate File "/usr/local/lib/python2.2/site-packages/traits/trait_delegates.py", line 38, in ? class TraitDelegate: File "/usr/local/lib/python2.2/site-packages/traits/trait_delegates.py", line 40, in TraitDelegate def __init__ ( self, delegate = None, mutate_or_prefix = False ): NameError: name 'False' is not defined >>> Does one need Python 2.3 to use Chaco? (I know that wxPython uses 'True' and 'False' without requiring 2.3 - I believe it's dynamically bound when one does 'import wx'?) Thanks in advance for any insight, d Some system info follows: ------ os.name='posix' ------ sys.platform='linux2' ------ sys.version: 2.2 (#1, Feb 26 2002, 15:49:15) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-81)] ------ sys.prefix: /usr/local ------ Found Numeric version '21.3' in /usr/local/lib/python2.2/site-packages/Numeric/Numeric.pyc ------ Found f2py2e version '2.35.229-1492' in /usr/local/lib/python2.2/site-packages/f2py2e/f2py2e.pyc ------ Found scipy_distutils version '0.2.0_alpha_3.288' in '/usr/local/lib/python2.2/site-packages/scipy_distutils/__init__.pyc' Importing scipy_distutils.command.build_flib ... ok ------ atlas_info: FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/local/lib'] blas_info: FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] blas_src_info: NOT AVAILABLE dfftw_info: NOT AVAILABLE dfftw_threads_info: NOT AVAILABLE djbfft_info: NOT AVAILABLE fftw_info: FOUND: libraries = ['rfftw', 'fftw'] library_dirs = ['/usr/local/lib'] define_macros = [('SCIPY_FFTW_H', None)] include_dirs = ['/usr/local/include'] fftw_threads_info: NOT AVAILABLE lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/usr/local/lib'] lapack_src_info: NOT AVAILABLE sfftw_info: NOT AVAILABLE sfftw_threads_info: NOT AVAILABLE x11_info: FOUND: libraries = ['X11'] library_dirs = ['/usr/X11R6/lib'] include_dirs = ['/usr/X11R6/include'] ------ Gnu version='0.5.26' F77='g77' F77FLAGS=' -Wall -fno-second-underscore -fPIC ' F77OPT=' -O3 -funroll-loops -march=i686 -malign-double -fomit-frame-pointer ' LIBS='-lg2c'

2 1

RE: [SciPy-user] RE: [Numpy-discussion] Removing rows and columns of a matrix
by Perry Greenfield July 29, 2003

July 29, 2003

> from scipy import * > from RandomArray import * > from numarray import * > I believe the problem is in this part. You are getting some functions (like rand) which return a numeric array which some numarray functions don't recognize (we will be fixing this soon, they should accept any Python sequence that fits the basic requirements). For the moment you will need to generate a numarray array for that function to work. In the latest release you would do the imports as follows (for version 0.6 and later) from numarray.numeric import * from numarray.random_array import * a = random((5,5)) and so forth Perry

1 0

RE: [Numpy-discussion] Removing rows and columns of a matrix
by Perry Greenfield July 29, 2003

July 29, 2003

Nils Wagner writes: > > In any case I am interested in a reliable workaround for this feature. > > A small example would be appreciated. > Thanks in advance. > Numarray (and Numeric to a certain extent) currently is centered around what rows or columns you want rather than those you don't. Writing a reasonably efficient function to do this in a more convenient way is not hard. Here is an example for numarray (not tested, nor does it have any error checking to ensure the input array has sufficient dimensions or that the index is within the array bounds): def remove(arr, index, dim=0): """Remove subarray at index and dimension specified""" # make selected dimension the first one swparr = swapaxes(arr, 0, dim) indices = range(swparr.shape[0]) del indices[index] newarray = swparr[indices] # reorder axes as they originally appeared. return swapaxes(newarray, 0, dim) Then you can say x = remove(x, 5) to remove the 6th row or x = remove(x, 4, dim=1) to remove the 5th column. (depending on what you call rows or columns; in the examples I'm taking an image-oriented view rather than a matrix-oriented view) Perry

2 2

generalized eigenvalue problem with singular matrices a,b
by Nils Wagner July 29, 2003

July 29, 2003

Hi all, Attention should be paid to the output of linalg.eig when both matrices are singular. Please find attached a small example. Any comment ? Nils from scipy import * a = diag(arange(0,10)) b = identity(10) b[0,0] = b[0,0]-1 # # Uncomment #w,vr = linalg.eig(a,b) #print w w = linalg.eigvals(a,b) print w Traceback (most recent call last): File "singular.py", line 7, in ? w,vr = linalg.eig(a,b) File "/usr/local/lib/python2.1/site-packages/scipy/linalg/decomp.py", line 108, in eig return _geneig(a1,b,left,right,overwrite_a,overwrite_b) File "/usr/local/lib/python2.1/site-packages/scipy/linalg/decomp.py", line 68, in _geneig vr = _make_complex_eigvecs(w, vr, t) File "/usr/local/lib/python2.1/site-packages/scipy/linalg/decomp.py", line 29, in _make_complex_eigvecs vnew.real = scipy_base.take(vin,ind[::2],1) ValueError: matrices are not aligned for copy

1 0

xplt.hold
by Nils Wagner July 29, 2003

July 29, 2003

Hi all, Is it possible to use xplt.hold('on') for window(1) and xplot.hold('off') for window(2) ? A small example would be appreciated. Nils

1 0

Re: [Numpy-discussion] Removing rows and columns of a matrix
by Nils Wagner July 28, 2003

July 28, 2003

Konrad Hinsen schrieb: > > On Monday 28 July 2003 16:26, Nils Wagner wrote: > > Konrad Hinsen schrieb: > > > > How can I delete a column/row from a matrix. > > > > > > As an in-place operation, not at all. To get a copy of an array with some > > > columns/rows removed, use Numeric.take. > > > > The in-place operation of matlab is a nice feature. Is it thinkable to > > have this > > in scipy or numarray at a later date ? > > I'd say that the obvious way to implement it is as > > del a[2:3,:] > > That can certainly be done, but someone has to do it. > > Personally, I am not convinced that it is very useful. The consequences of an > in-place change can be disastrous when many references point to that array. > But then, we have other in-place operations which sometimes are handy, so my > usefulness estimation may be wrong. > > Konrad. In my opinion, this operation is quite useful. For example a finite element model where we have to impose several homogeneous boundary conditions. This can be done by deleting columns and associated rows. Nils In any case I am interested in a reliable workaround for this feature. A small example would be appreciated. Thanks in advance. > -- > ------------------------------------------------------------------------------- > Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr > Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 > Rue Charles Sadron | Fax: +33-2.38.63.15.17 > 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ > France | Nederlands/Francais > ------------------------------------------------------------------------------- > > ------------------------------------------------------- > This SF.Net email sponsored by: Free pre-built ASP.NET sites including > Data Reports, E-commerce, Portals, and Forums are available now. > Download today and enter to win an XBOX or Visual Studio .NET. > http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion

1 0

Re: [Numpy-discussion] Removing rows and columns of a matrix
by Nils Wagner July 28, 2003

July 28, 2003

Konrad Hinsen schrieb: > > > How can I delete a column/row from a matrix. > > As an in-place operation, not at all. To get a copy of an array with some > columns/rows removed, use Numeric.take. > The in-place operation of matlab is a nice feature. Is it thinkable to have this in scipy or numarray at a later date ? Nils > Konrad. > -- > ------------------------------------------------------------------------------- > Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr > Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 > Rue Charles Sadron | Fax: +33-2.38.63.15.17 > 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ > France | Nederlands/Francais > ------------------------------------------------------------------------------- > > ------------------------------------------------------- > This SF.Net email sponsored by: Free pre-built ASP.NET sites including > Data Reports, E-commerce, Portals, and Forums are available now. > Download today and enter to win an XBOX or Visual Studio .NET. > http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion

2 1

Removing rows and columns of a matrix
by Nils Wagner July 28, 2003

July 28, 2003

Hi all, How can I delete a column/row from a matrix. In Matlab: >> a=rand(4,4) a = 0.9501 0.8913 0.8214 0.9218 0.2311 0.7621 0.4447 0.7382 0.6068 0.4565 0.6154 0.1763 0.4860 0.0185 0.7919 0.4057 use >> a(:,2)=[] a = 0.9501 0.8214 0.9218 0.2311 0.4447 0.7382 0.6068 0.6154 0.1763 0.4860 0.7919 0.4057 >> a(2,:)=[] a = 0.9501 0.8214 0.9218 0.6068 0.6154 0.1763 0.4860 0.7919 0.4057 >> Is there something similar in scipy, numpy or numarray ? Any suggestion ? Nils

1 0