Mailman 3 July 2003 - NumPy-Discussion

ANN: PyTables 0.7
by Francesc Alted 31 Jul '03

31 Jul '03

Announcing PyTables 0.7 ----------------------- PyTables is a hierarchical database package designed to efficently manage very large amounts of data. PyTables is built on top of the HDF5 library and the numarray package and features an object-oriented interface that, combined with C-code generated from Pyrex sources, makes it a fast, yet extremely easy to use tool for interactively save and retrieve large amounts of data. Release 0.7 is the third public beta release. The version 0.6 was internal and will never be released. On this release you will find: - new AttributeSet class - 25% I/O speed improvement - fully multidimensional table cells support - new column descriptors - row deletion in tables is finally here - much more! More in detail: What's new ----------- - A new AttributeSet class has been added. This will allow the addition and deletion of generic attributes (any scalar type plus any Python object supported by Pickle) as easy as this: table.attrs.date = "2003/07/28 10:32" # Attach a string to table group._v_attrs.tempShift = 1.2 # Attach a float to group array.attrs.detectorList = [1,2,3,4] # Attach a list to array del array.attrs.detectorList # Detach detectorList attr from array - PyTables now has support for fully multidimensional table cells. This has been made possible in part by implementation of multidimensional cells in numarray.records.RecArray object. Thanks to numarray crew, and especially to Jin-chung Hsu, for willingly accepting to do that, and also for including some cache improvements in RecArray. - New column descriptors added: IntCol, Int8Col, UInt8Col, Int16Col, UInt16Col, Int32Col, UInt32Col, Int64Col, UInt64Col, FloatCol, Float32Col, Float64Col and StringCol. I think they are more explicit and easy-to-use than the now deprecated (but still supported) Col() descriptor. All the examples and user's manual has been accordingly updated. - The new Table.removeRows(start, stop) function allows you to remove rows from tables. This feature was requested a long time ago. There are still limitations, however: you cannot delete rows in extremely large Tables (as the remaining rows after the stop parameter are stored in memory). Nor is the performance optimized. These issues will hopefully be addressed in future releases. - Added iterators to File, Group and Table (they now support the special __iter__() method). They make the object much more user-friendly, especially in interactive mode. See documentation for usage examples. - Added a __getitem__() method to Table that works more or less like read(), but with extended slices support. - As a consequence of rewriting table iterators in C (with the help of Pyrex, of course) the table read performance has been improved between 20% and 30%. Data selections in PyTables are now starting to beat powerful relational databases like SQLite, even compared to in-core selects (!). I think there is still room for another 20% or 30% speed improvement, so stay tuned. - A checksum is now added automatically when using LZO (not with UCL where I'm having some difficulties implementing that capability). The Adler32 algorithm has been chosen because of its speed. With that, the compressing/decompressing speed has dropped 1% or 2%, which is hardly noticeable. I think this addition will allow the cautious user to be a bit more confident about this excellent compressor. Code has been added to be able to read files created without this checksum (so you can be confident that you will be able to read your existing files compressed with LZO and UCL). - Recursion has been removed from PyTables. Before, this made the maximum depth tree to be less than the Python recursion limit (which depends on implementation, but is around 900, at least in Linux). Now, the limit has been set (somewhat arbitrarily) at 2048. Thanks to John Nielsen for implementing the new iterative method!. - A new rootUEP parameter to openFile() has been added. You can now define the root from which you want to start to build the object tree. Thanks to John Nielsen for the suggestion and a first implementation. - A small bug fixed when dealing with non-native PyTables files that prevented the use of the "classname" filter during a listNodes() call. Thanks to Jeff Robbins for reporting that. - Some (non-serious) bugs were discovered and fixed. - Updated documentation to explain all these new bells and whistles. It is also available on the web: http://pytables.sourceforge.net/html-doc/usersguide-html.html - Added more unit tests (more than 350 now!) - PyTables 0.7 *needs* numarray 0.6 or higher and HDF-1.6.0 or higher to compile and work. It has been tested with Python 2.2 and 2.3 and should work fine on both versions. What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seems to be quite a strange requirement for an language like Python, that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors and running IRIX 6.5. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP, but it should also work with other flavors. An example? ----------- For online code examples, have a look at http://pytables.sourceforge.net/tut/tutorial1-1.html and http://pytables.sourceforge.net/tut/tutorial1-2.html Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted falted(a)openlc.org

1 0

buglet in interaction between pygame, Numeric & copy in Python 2.3
by Alex Martelli 31 Jul '03

31 Jul '03

I'm sending this buglet-report to the discussion lists for each of pygame, Numeric, and Python-Dev, because it ends with a recommendation for changes (at least to docs) in each of the two extension packages and Python 2.3.1. I'm not a regular user of pygame [I am one of Numeric]. I helped some pygame users find out why their program was breaking (in just one point) when migrating Python 2.2 -> 2.3 (Numeric 23.0, pygame 1.5.6 from cvs snapshot). The problem can be narrowed down the following little buglet.py script: import copy, pygame.display, pygame.surfarray screen = pygame.display.set_mode((640,480),0,24) surf = pygame.surfarray.pixels3d(screen) xx = copy.copy(surf[10:20,30]) With python 2.2, this runs without errors. With 2.3: Traceback (most recent call last): File "buglet.py", line 6, in ? xx = copy.copy(surf[10:20,30]) File "/usr/local/lib/python2.3/copy.py", line 101, in copy raise Error("un(shallow)copyable object of type %s" % cls) copy.Error: un(shallow)copyable object of type <type 'array'> Workaround for the problem: add an "import Numeric" at the top of buglet.py. Reason for the problem: copy.copy uses different mechanisms in 2.3 vs 2.2 for shallow copies of Numeric.array instances. In 2.2, it looked for a method __copy__ on the instance, found it, and happily used it. In 2.3, it relies on getting a 'reductor' function from the copy_reg.dispatch_table. Module Numeric installs such a reductor *WHEN IMPORTED* -- but pygame never imports it even though functions such as the above-used pixels3d do return instances of Numeric.array. So, Numeric.py doesn't run, and therefore cannot install its pickle_array function with the copy_reg module. Suggested fix in pygame: I would suggest making sure Numeric has been imported when Numeric.array instances are to be returned. That's up to the pygame maintainers, of course; at the very least, I think this issue should be mentioned in pygame's docs (asking the user to make sure "import Numeric" happens in any pygame-using program that manipulates such instances). Suggested fix in Numeric: Numeric cannot DO anything much about this issue, I think, but it should probably *mention* (in the docs regarding interfacing to Numeric from other C-coded extensions) that other extensions should make sure Numeric is _imported_ (from a Python POV), if said extensions return instances of Numeric.array, to ensure such instances have full functionality (and in particular shallow copyability). Suggested fix in Python: a mention in the "what's new" document, section "porting to Python 2.3", is probably warranted -- not so much about Numeric and/or pygame extensions per se, though these are both popular 3rd party extensions, but rather the excellent changes in the shallow copy mechanics, which may indeed require code changes on the part of extension authors. Thanks, Alex

1 0

assigning to submatrix
by John Hunter 29 Jul '03

29 Jul '03

If I have a large matrix, say a = zeros( (20,20) ) and a small matrix b = ones( 3,4 ) and a list of row indices and a list of col indices rind = [2,4,9,15] cind = [1,4,12] what is the best way to assign the submatrix to a[rind,cind]. I understand this is possible in numarray, but is there a good way to do it in Numeric? In matlab, you could do a(rind, cind)=b; Is there some reshape, put magic I can do to make this efficient in Numeric? Thanks, John Hunter

2 5

RE: [Numpy-discussion] Removing rows and columns of a matrix
by Perry Greenfield 29 Jul '03

29 Jul '03

Nils Wagner writes: > > In any case I am interested in a reliable workaround for this feature. > > A small example would be appreciated. > Thanks in advance. > Numarray (and Numeric to a certain extent) currently is centered around what rows or columns you want rather than those you don't. Writing a reasonably efficient function to do this in a more convenient way is not hard. Here is an example for numarray (not tested, nor does it have any error checking to ensure the input array has sufficient dimensions or that the index is within the array bounds): def remove(arr, index, dim=0): """Remove subarray at index and dimension specified""" # make selected dimension the first one swparr = swapaxes(arr, 0, dim) indices = range(swparr.shape[0]) del indices[index] newarray = swparr[indices] # reorder axes as they originally appeared. return swapaxes(newarray, 0, dim) Then you can say x = remove(x, 5) to remove the 6th row or x = remove(x, 4, dim=1) to remove the 5th column. (depending on what you call rows or columns; in the examples I'm taking an image-oriented view rather than a matrix-oriented view) Perry

2 2

Mac problem with numarray-0.6; use numarray-0.6.1 source
by Todd Miller 29 Jul '03

29 Jul '03

I discovered today (some bug reports have hinted at this but didn't spell it out completely) that I forgot Mac OS 10 testing for numarray-0.6. As it turns out, the numarray examples and add-ons don't compile correctly due to a small untested change. I corrected the problem and uploaded a new tarball to Source Forge: http://osdn.dl.sourceforge.net/sourceforge/numpy/numarray-0.6.1.tar.gz Non-Mac users of numarray-0.6 should just ignore numarray-0.6.1. This is a source update only. Todd -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

Grow Your Customers
by Laura 28 Jul '03

28 Jul '03

E-mail Marketing is one of the most effective and inexpensive ways to promote your products and services. We offer a complete E-mail Marketing solution with quality service and the lowest prices. The result is that you will enjoy more success. 1. Targeted E-mail Addresses We can supply targeted e-mail addresses according to your requirements, which are compiled only on your order, such as region / country / field / occupation / Domain Name etc. We will customize your customer e-mail addresses. * We have millions of e-mail addresses in a wide variety of categories. 2. Targeted Mailing If you are worried about any complications or consequences with sending out targeted e-mails, or want to avoid the work of sending out targeted e-mails. We will do it for you! We can send your e-mail message to your targeted customers. * We can Bullet-Proof your Web Site. We also offer a wide variety of e-mail marketing software. For more details, you can refer to our web site: http://www.aid-biz.com Our services will help you get more business opportunities. Regards! Ms Laura www.aid-biz.com Customer Support E-mail Marketing, at Great Fees. ************************************************************************ Receiving this email because you registered to receive special offers from one of our partners. If you would prefer not to receive future e-mail, Click here: Booster(a)ywzc.net?subject=REMOVE ************************************************************************

1 0

ANNOUNCE: numarray-0.6
by Todd Miller 28 Jul '03

28 Jul '03

I have just uploaded numarray-0.6 to Source Forge. http://sourceforge.net/project/showfiles.php?group_id=1369 This is the first release with numarray repackaged as a Python package. I. ENHANCEMENTS 1. Repackaging numarray has been modified from a collection of Python modules into a Python package. See Doc/Repackaging in the source distribution for more details. 2. Median Filter Function See numarray.image.combine.median for a function which computes the median pixels in a stack of images. 3. Thread Safety Work Modifications to ufunc caching and IEEE error handling enable numarray to work in multi-threaded programs. II. BUGS FIXED 733534 Broadcasts of non-array sequences fails 733535 NA_typeNoToTypeObject sometimes fails 736674 PyArray_ContiguousFromObject() fails for some scalars 737041 Build failure on Cygwin 1.3.22 (very current install). 750320 CharArray.raw() is broken 750326 building RPMS fails 754836 Missing Src/_convmodule.c during setup 756050 chararray multi-d copy fails for recarray 772519 memory.new_memory() 64-bit scanning bug III. CAUTIONS 1. Due to the distribution repackaging, numarray-0.6 will not install correctly on top of an existing numarray installation. Before installing numarray-0.6 remove your old version of numarray. -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

4 7

Removing rows and columns of a matrix
by Nils Wagner 28 Jul '03

28 Jul '03

Hi all, How can I delete a column/row from a matrix. In Matlab: >> a=rand(4,4) a = 0.9501 0.8913 0.8214 0.9218 0.2311 0.7621 0.4447 0.7382 0.6068 0.4565 0.6154 0.1763 0.4860 0.0185 0.7919 0.4057 use >> a(:,2)=[] a = 0.9501 0.8214 0.9218 0.2311 0.4447 0.7382 0.6068 0.6154 0.1763 0.4860 0.7919 0.4057 >> a(2,:)=[] a = 0.9501 0.8214 0.9218 0.6068 0.6154 0.1763 0.4860 0.7919 0.4057 >> Is there something similar in scipy, numpy or numarray ? Any suggestion ? Nils

4 6

new mail UwEuOebt
by rmgolbeck＠transilvania.eu.org 25 Jul '03

25 Jul '03

Мы рады предложить вам новый бесплатный почтовый сервис http://www.mail15.com. Его отличительные особенности: 1) размер ящика 15 мб; 2) защищенность и надежность; 3) возможность использования любых почтовых программ(POP,IMAP,SMTP); 4) доступ из любого места в любое время; 5) простой и доступный вебинтерфейс с ПОЛНЫМ ОТСУТСТВИЕМ РЕКЛАМЫ; 6) антивирусный и антиспамовый контроль; 7) мгновенная пересылка почты. Если вы не хотите получать более данную рассылку, пишите mailto:unsubscribe@mail15.com?subject=unsubscribe ************* We are glad to invite you at new free mail service http://www.mail15.com. The advantages of this service are: 1) mailbox, up to 15 Mb; 2) absolute privacy and high reliability; 3) ability to use mail clients (POP3, IMAP4, SMTP); 4) access from anywhere, anytime; 5) flexible light-weight web interface without advertising banners; 6) antivirus and antispam control; 7) fast mail transfer; 8) high speed network channel; 9) flexible light-weight web interface; 10) wide spread ability of mail filtering and forwarding mail; 11) clock around support; If you wish to be removed: mailto:unsubscribe@mail15.com?subject=unsubscrib R2t0vLtgLZ

1 0

[ANN] SciPy 03 Workshop - Scientific Computing with Python
by Travis N. Vaught 25 Jul '03

25 Jul '03

------------------ SciPy '03 Workshop ------------------ The SciPy '03 Workshop on Scientific Computing with Python is being held at Cal Tech again this year. Online registration is now available. After last year's success, we expect another great turnout with compelling presentations/tutorials. It is a two day workshop held September 11-12 in Pasadena, CA. More information may be found at: http://www.scipy.org/site_content/scipy03 Online registration is available at: https://www.enthought.com/scipy03 Please register early. Last minute registrants will have to suffer the indignity of a handwritten name tag. It currently looks like breakfast (bagels, etc.), lunch and supper will be provided Thursday, and breakfast and lunch will be provided Friday. Discussion about the workshop may be directed to the SciPy-user mailing list (http://www.scipy.org/site_content/MailList) to which you may post messages at scipy-user(a)scipy.org. Presenters: If you would like to present at the workshop, there are two formats: - Lightning Talks: Short (5-15 minute) presentations of a particular scientific problem solved with Python. - Presentations: Traditional presentation format ~45 minute talks. If you would like to present, contact Eric Jones at eric(a)enthought.com. For any other questions, please feel free to call or email Travis Vaught: travis(a)enthought.com, (512)536-1057. You can call the same number to register via phone (Business Hours CST). Links: SciPy.org community site: http://www.scipy.org Workshop Home: http://www.scipy.org/site_content/scipy03 Summary of last year's workshop: http://www.scipy.org/site_content/scipy02/scipy02summary.htm

1 0

NumPy-Discussion July 2003