NumPy-Discussion March 2003

numpy-discussion@python.org

19 participants
20 discussions

by Paul Miller

The Numeric-23 source drop doesn't contain Windows Visual Studio project files, and I don't know how to build the debug version of the DLLs with the setup file. I'm trying to use Numeric in a debug session where I also have debug versions of the python dlls. But it won't import if I'm linked with the debug python. I'm assuming having the debug dlls for Numeric will help. Could someone just send me their copies if they have them, or send a VStudio project with a debug target?

18 years, 8 months

PyObject typecodes are gone in numarray?

by Berthold Höllmann

Hello, I am trying to convert a larger project of ours from Numeric to numarray under some time pressure. One stumbling point is that numarray does not support the Numeric "PyObject" typecode for arrays. I am mapping some kind or database structure to Numeric arrays and am using PyObject arrays for storing String columns for consistency with the numeric columns. How would this be solved with numarray? Example: >python Python 2.2.1 (#1, Dec 13 2002, 10:37:32) [GCC 3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric as N >>> a=N.zeros(3, N.PyObject) >>> a[:] = ['abc', 'def', 'ghi'] >>> a array([abc , def , ghi ],'O') Thanks Berthold -- Dipl.-Ing. Berthold Höllmann __ Address: hoel(a)GL-Group.com G / \ L Germanischer Lloyd phone: +49-40-36149-7374 -+----+- Vorsetzen 32/35 P.O.Box 111606 fax : +49-40-36149-7320 \__/ D-20459 Hamburg D-20416 Hamburg **************************************************** Beachten Sie: Wir moechten Sie informieren, dass die E-Mail-Adresse des Germanischen Lloyd sowie unsere Web-Adresse mit Wirkung vom 1. Maerz 2003 auf den Namen gl-group.com umgestellt wurde. Dies bedeutet, dass die bisherige Adresse Kurzzeichen(a)germanlloyd.org durch die neue Adresse Kurzzeichen(a)gl-group.com ersetzt wird. Die Homepage des GL ist kuenftig ueber die Adresse 'http://www.gl-group.com' aufrufbar. Die bisher verwendeten Adressen bleiben für eine Uebergangsfrist erreichbar. **************************************************** Please notice: We would like to inform you that the e-mail address of Germanischer Lloyd as well as our internet address had been changed to gl-group.com with effect from 1st March 2003. This means that the previous address shortmark(a)germanlloyd.org will be replaced by shortmark(a)gl-group.com. From now on the GL homepage can be accessed at the address 'http://www.gl-group.com'. The old addresses remain valid for a transitional period. **************************************************** This e-mail contains confidential information for the exclusive attention of the intended addressee. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc.

18 years, 8 months

Message de Elodie

by sophie.l＠deux.fr

18 years, 8 months

ANNOUNCE: Pytables 0.4

by Francesc Alted

Announcing PyTables 0.4 ----------------------- I'm happy to announce the first beta release of PyTables. It is labelled beta because it has been thoroughly tested, even in production environments, and it is getting fairly mature. Besides, from now on, the API will remain mostly stable, so you can start using PyTables now with the guarantee that your code will also work (well, mostly ;-) in the future. The large amount of unit tests included in PyTables will also ensure the backward compatibility as well as the quality of future releases will remain at least as good as it is now (although hopefully it should increase!). What's new ----------- - numarray objects (NumArray, CharArray and RecArray) supported - As a consequence of a large internal code redesign (numarray is at the core of PyTables now), performance has been improved by a factor of 10 (see "How Fast Is It?" section) - It consumes far less memory than previous version - Support for reading generic HDF5 files added (!) - Some bugs and memory leaks existing in 0.2 solved - Updated documentation - Added more unit tests (more than 200 now!) What it is ---------- In short, PyTables provides a powerful and very Pythonic interface to process table and array data. Its goal is to enable the end user to manipulate easily scientific data tables and Numerical and numarray Python objects in a persistent hierarchical structure. The foundation of the underlying hierarchical data organization is the excellent HDF5 library (http://hdf.ncsa.uiuc.edu/HDF5). A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and strict "data types" seems to be quite a strange requirement for an interpreted language like Python, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. Quite a bit effort has been invested to make browsing the hierarchical data structure a pleasant experience. PyTables implements just two (orthogonal) easy-to-use methods for browsing. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. How fast is it? --------------- PyTables can write table records between 20 and 30 times faster than cPickle and between 3 and 10 times faster than struct (it is a module present in the Standard Library); and retrieves information around 100 times faster than cPickle and between 8 and 10 times faster than struct. When compared with SQLite (http://www.sqlite.org/), one of the fastest (free) relational databases available, PyTables achieves between a 60% and 80% the speed of SQLite during selects of dataset sizes that fit in the O.S. filesystem memory cache. However, when those sizes does not fit in the cache (i.e. when dealing with large amounts of data), PyTables beats SQLite by a factor of 2 or even more (depending on the kind of record selected), and its performance in this case is only limited by the I/O speed of the disk subsystem. Go to http://pytables.sourceforge.net/doc/PyCon.html#section4 for a detailed description on the conducted benchmarks. Platforms --------- I'm using Linux as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors and running IRIX 6.5. If you are using Windows and you get the library to work, please let me know. An example? ----------- At the bottom of this message there is some code that shows basic capabilities of PyTables. You may also look at http://pytables.sourceforge.net/tut/tutorial1-1.html and http://pytables.sourceforge.net/tut/tutorial1-2.html for online code. Web site -------- Go to the PyTables web site for downloading and more details: http://pytables.sf.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted falted(a)openlc.org *-*-*-**-*-*-**-*-*-**-*-*- Small code example *-*-*-**-*-*-**-*-*-**-*-*-* from tables import * class Particle(IsDescription): identity = Col("CharType", 22, " ", pos = 0) # character String idnumber = Col("Int16", 1, pos = 1) # short integer speed = Col("Float32", 1, pos = 1) # single-precision # Open a file in "w"rite mode fileh = openFile("objecttree.h5", mode = "w") # Get the HDF5 root group root = fileh.root # Create the groups: group1 = fileh.createGroup(root, "group1") group2 = fileh.createGroup(root, "group2") # Now, create a table in "group0" group array1 = fileh.createArray(root, "array1", ["string", "array"], "String array") # Create 2 new tables in group1 table1 = fileh.createTable(group1, "table1", Particle) table2 = fileh.createTable("/group2", "table2", Particle) # Create the last table in group2 array2 = fileh.createArray("/group1", "array2", [1,2,3,4]) # Now, fill the tables: for table in (table1, table2): # Get the record object associated with the table: row = table.row # Fill the table with 10 records for i in xrange(10): # First, assign the values to the Particle record row['identity'] = 'This is particle: %2d' % (i) row['idnumber'] = i row['speed'] = i * 2. # This injects the Record values row.append() # Flush the table buffers table.flush() # Select actual data from table. # on entries where TDCcount field is greater than 3 and pressure less than 50 out = [ x['identity'] for x in table.iterrows() if x['idnumber'] > 3 and 4 < x['speed'] < 10 ] print out # Finally, close the file (this also will flush all the remaining buffers!) fileh.close()

18 years, 8 months

Re: [Numpy-discussion] UInt64 is not supported on 64-bit architecture?

by Francesc Alted

A Dimarts 18 Març 2003 16:47, Todd Miller va escriure: > > It may be a fairly simple matter of an unsupported platform; I don't > have access to the SGI machine you're dealing with. The default block > in numarray's setup.py compiles for 32-bit without support for UInt64. > Add a case for > SGI and you may be done. If so, please send a patch. I see. I'm attaching the patch for IRIX 6.5 (IRIX64 nut 6.5 07121149 IP27): --- setup.py.orig Tue Mar 18 17:24:40 2003 +++ setup.py.irix Tue Mar 18 17:42:37 2003 @@ -32,6 +32,8 @@ if sys.platform == "osf1V5": EXTRA_COMPILE_ARGS.extend(["-ieee"]) LP64, HAS_UINT64, HAS_FLOAT128 = 1, 1, 1 +elif sys.platform == "irix646": + LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 elif sys.platform == "linux2": LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 elif sys.platform == "sunos5": I've only set the UInt64 support because I don't know if this platform supports floats of 128 bits and neither what LP64 exactly means. With that, numarray supports UInt64 arrays on IRIX perfectly: nut 435$ python Python 2.2.2 (#2, Nov 19 2002, 18:46:18) [GCC 2.95.2 19991024 (release)] on irix646 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray >>> numarray.ones((3,), numarray.UInt64) array([1, 1, 1], type=UInt64) >>> ^D Thanks for the pointer, -- Francesc Alted

18 years, 8 months

Re: [Numpy-discussion] typo in arrayobject.h?

by christiaan.kok＠philips.com

Dear Konrad, >There should be a semicolon there according to my understanding of C syntax! >Check the C extension which you are compiling. It very probably defines >PY_ARRAY_UNIQUE_SYMBOL to something before including arrayobject.h. I suspect >that there is something wrong with that definition. >Konrad. According to my understanding of C there should indeed be a semicolon as well, but the C compiler gives a syntax errro warning. The first lines of my C code (which runs perfectly under Linux with gcc BTW) are as follows: #include "c:\python22\include\Python.h" #include "c:\python22\include\numeric\arrayobject.h" #define PY_ARRAY_UNIQUE_SYMBOL #include "libnumpy-access.h" static PyObject* find_minimum(PyObject *self, PyObject *args){ ........ Ok, I found the problem. Within libnumpy-access.h yet again arrayobject.h is called which causes the problem. So changing this to #include "c:\python22\include\Python.h" #include "c:\python22\include\numeric\arrayobject.h" #include "libnumpy-access.h" #define PY_ARRAY_UNIQUE_SYMBOL static PyObject* find_minimum(PyObject *self, PyObject *args){ ........ Now it works perfectly. Probably my code runs under linux anyway due to silly CR LF things which always bug multiplatform development. Kudos to Fernando Perez because his libnumpy-access files make my code a lot more readable. with best regards, Christiaan Kok

18 years, 8 months

Masked arrays in C extensions

by Kuzminski, Stefan R

18 years, 8 months

UInt64 is not supported on 64-bit architecture?

by Francesc Alted

Hi, numarray seems to not support UInt64 types in 64-bit architecture. For example, on SGI Origin 2000 with R12000 processors, I'm getting: Python 2.2.2 (#2, Nov 19 2002, 18:46:18) [GCC 2.95.2 19991024 (release)] on irix646 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray >>> numarray.ones((3,), numarray.UInt64) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/scratch1/fernande/libs/lib/python2.2/site-packages/numarray/numarray.py", line 959, in ones retarr = _fillarray(ndarray.product(shape), 1, 0, type) File "/scratch1/fernande/libs/lib/python2.2/site-packages/numarray/numarray.py", line 160, in _fillarray if outtype > ptype: # Hack for Int64/UInt64 on 32-bit platforms. File "/scratch1/fernande/libs/lib/python2.2/site-packages/numarray/numerictypes.py", line 112, in __cmp__ return (genericTypeRank.index(self.name) - ValueError: list.index(x): x not in list >>> while in my laptop (pentium IV, i.e. 32-bit), I've no problem with that: Python 2.2.2 (#3, Dec 26 2002, 16:40:05) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray >>> numarray.ones((3,), numarray.UInt64) array([1, 1, 1], type=UInt64) >>> This is bizarre that a 64-bit machine does not support the UInt64 type while a 32-bit one do. I've been looking at the numarray code and that seems to be intentional, but I can't understand why. Somebody can enlighten my mind? Thanks, -- Francesc Alted

18 years, 8 months

Mixing numarray and Numeric under McMillan Installer

by Berthold Höllmann

Hello, I am trying to freeze a program using the McMillan Installer version 5, alpha 1. The program has to use Numeric and numarray in parallell. But the generated executable just gives a segmentation fault. Here' what I get: >cat aa.py import Numeric as N print "AA" import numarray as n print "BB" print N.arange(19) print n.arange(19) >python -OO ~/work/Installer/Makespec.py --name aaa aa.py ... >python -OO ~/work/Installer/Build.py aaa.spec ... >./distaaa/aaa Adding parser accelerators ... Done. AA Segmentation fault (core dumped) >python aa.py Adding parser accelerators ... Done. AA BB [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18] [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18] [23388 refs] gdb gives: >gdb distaaa/aaa core GNU gdb 5.2.1 Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... Core was generated by `./distaaa/aaa'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libpthread.so.0...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libutil.so.1...done. Loaded symbols for /lib/libutil.so.1 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libz.so.1...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from ./distaaa/struct.so...done. Loaded symbols for ./distaaa/struct.so Reading symbols from ./distaaa/zlib.so...done. Loaded symbols for ./distaaa/zlib.so Reading symbols from ./distaaa/_codecs.so...done. Loaded symbols for ./distaaa/_codecs.so Reading symbols from ./distaaa/multiarray.so...done. Loaded symbols for ./distaaa/multiarray.so Reading symbols from ./distaaa/_numpy.so...done. Loaded symbols for ./distaaa/_numpy.so Reading symbols from ./distaaa/umath.so...done. Loaded symbols for ./distaaa/umath.so Reading symbols from ./distaaa/strop.so...done. Loaded symbols for ./distaaa/strop.so Reading symbols from ./distaaa/math.so...done. Loaded symbols for ./distaaa/math.so Reading symbols from ./distaaa/cStringIO.so...done. Loaded symbols for ./distaaa/cStringIO.so Reading symbols from ./distaaa/memory.so...done. Loaded symbols for ./distaaa/memory.so Reading symbols from ./distaaa/operator.so...done. Loaded symbols for ./distaaa/operator.so Reading symbols from ./distaaa/_conv.so...done. Loaded symbols for ./distaaa/_conv.so #0 0x402656ea in init_funcDict () at Src/_convmodule.c:2337 ---Type <return> to continue, or q <return> to quit--- 2337 NA_add_cfunc(dict, "('Bool', 'asPyValue')", (void *) &BoolasPyValue_descr); I get this with Python 2.2.1, Numeric 21.0, numarray 0.4 as well as with Python 2.2.2, Numeric 22.0, numarray 0.4, both under Linux. Is there any fix or workarround? Thanks Berthold -- Dipl.-Ing. Berthold Höllmann __ Address: hoel(a)GL-Group.com G / \ L Germanischer Lloyd phone: +49-40-36149-7374 -+----+- Vorsetzen 32/35 P.O.Box 111606 fax : +49-40-36149-7320 \__/ D-20459 Hamburg D-20416 Hamburg **************************************************** Beachten Sie: Wir moechten Sie informieren, dass die E-Mail-Adresse des Germanischen Lloyd sowie unsere Web-Adresse mit Wirkung vom 1. Maerz 2003 auf den Namen gl-group.com umgestellt wurde. Dies bedeutet, dass die bisherige Adresse Kurzzeichen(a)germanlloyd.org durch die neue Adresse Kurzzeichen(a)gl-group.com ersetzt wird. Die Homepage des GL ist kuenftig ueber die Adresse 'http://www.gl-group.com' aufrufbar. Die bisher verwendeten Adressen bleiben für eine Uebergangsfrist erreichbar. **************************************************** Please notice: We would like to inform you that the e-mail address of Germanischer Lloyd as well as our internet address had been changed to gl-group.com with effect from 1st March 2003. This means that the previous address shortmark(a)germanlloyd.org will be replaced by shortmark(a)gl-group.com. From now on the GL homepage can be accessed at the address 'http://www.gl-group.com'. The old addresses remain valid for a transitional period. **************************************************** This e-mail contains confidential information for the exclusive attention of the intended addressee. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc.

18 years, 8 months

typo in arrayobject.h?

by christiaan.kok＠philips.com

Dear people, When trying to compile a C extension to Numeric 23.0 the compiler complains (with a syntax error) about a semicolon in line 284 of arrayobject.h. When I remove this semicolon my application seems to run. System: Windows 2000 Visual C++ 6.0 Numeric 23.0 Python 2.2.1 Is this a typo or do I do something strange? with best regards, Christiaan Kok /* C API address pointer */ #if defined(NO_IMPORT) || defined(NO_IMPORT_ARRAY) extern void **PyArray_API; #else #if defined(PY_ARRAY_UNIQUE_SYMBOL) void **PyArray_API /* <--------------------------- semicolon here removed! */ #else static void **PyArray_API; #endif #endif

18 years, 8 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion March 2003