NumPy-Discussion January 2003

numpy-discussion@python.org

26 participants
35 discussions

numarray bug?
by verveer＠embl.de 19 Jan '03

19 Jan '03

Hi, The following gives an error: >>> print numarray.Int8 == numarray.Any Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.2/site-packages/numarray/numerictypes.py", line 102, in __cmp__ return genericTypeRank.index(self.name) - genericTypeRank.index(other.name) ValueError: list.index(x): x not in list A bug? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: verveer(a)embl-heidelberg.de

1 0

Accessing a Numeric 'c' array from numarray
by Francesc Alted 18 Jan '03

18 Jan '03

Hi, I'm trying to make a C array from a Numeric "c" (Character) typecode array using the high level call: NA_InputArray(PyObject *numarray, NumarrayType t, int requires) with no success. As I have been able to access all the other types (i.e. '1','b','s','i','l','f','d') successfully, perhaps character type is not supported? In the NumarrayType enum, there is no tChar, but I've tried tUInt8 and tAny as the value for NumarrayType parameter, but both choices issues the same error: Traceback (most recent call last): File "table-tree2.py", line 77, in ? h5file.createArray('/columns', 'name', array(names), "Name column") File "/home/falted/PyTables/pytables-0.3/tables/File.py", line 400, in createArray setattr(group, name, object) File "/home/falted/PyTables/pytables-0.3/tables/Group.py", line 355, in __setattr__ value._f_putObjectInTree(name, self) File "/home/falted/PyTables/pytables-0.3/tables/Leaf.py", line 71, in _f_putObjectInTree self.create() File "/home/falted/PyTables/pytables-0.3/tables/Array.py", line 83, in create self.createArray(self.object, self.title) File "/home/falted/PyTables/pytables-0.3/src/hdf5Extension.pyx", line 913, in createArray array = NA_InputArray(arr, numfmt2[arr.typecode()], C_ARRAY) libnumarray.error: getShape: sequence object nested more than MAXDIM deep. although I was passing only a Numeric 'c' with a rather small shape (10,16). I just want to access the buffer data, and the shape of this object from C (well, I'm actually using Pyrex, but I think this is not important). Is that possible by only using numarray C calls? Thanks, -- Francesc Alted

1 0

make C array accessible to python without copy
by Sebastian Haase 17 Jan '03

17 Jan '03

Hi, What is the C API to make an array that got allocated, let's say, by a = new short[512*512], accessible to python as numarray. I tried NA_New - but that seems to make a copy. I would need it to use the original memory space so that I can "observe" the array from Python WHILE the underlying C array changes (it's actually a camera image) Thanks, Sebastian Haase

2 1

Bin selection
by Magnus Lie Hetland 16 Jan '03

16 Jan '03

I have a set of limits, e.g. array([0, 35, 45, 55, 75]) and I want to use these to "classify" a set of numbers (another one-dimensional array). The "class" is the number of the first limit that is lower than or equal to the number I want to classify. E.g. I'd classify 17 as 0 and 42 as 1. My current approach is: sum(nums[:,NewAxis] >= lims, dim=-1) It seems a bit unnecessary to compare each number with all the limits when O(log(n)) would suffice (the limits are ordered); or even with O(n) running time, a smarter implementation could get an average of n/2 comparisons... Suggestions? -- Magnus Lie Hetland http://hetland.org

7 15

Re: [Numpy-discussion] Optionally using Numeric in another compiled
by gvermeul＠grenoble.cnrs.fr 15 Jan '03

15 Jan '03

> Gerard Vermeulen wrote: > > I just want to point out that PyQwt plots NumPy arrays. I have played > > a little bit with the Scipy-wxWindows interface, but it is no match > > for PyQwt (I display x-y data with 16000 points). > > Thanks for the tip, I'll check it out. I think what you have there is > that the plotting is all done at the C++ level, expecting some kind of > sequence of data points. That's exactly what I want to adress with > wxPython: being able to pass in a whole sequence and have the looping > done at the C++ level. > Yes, I am using PyArray_ContiguousFromObject() to convert any sequence into a NumPy array before copying the data into Qwt's double arrays. > > Have you ever tested whether it's fster or slower to plot data passed in > as a list vs. a NumPy array? > I did not test it, but there is certainly more overhead if you pass a list or a tuple into PyArray_ContiguousFromObject() than a NumPy array > > How do you access the data in the passed in sequence? Do you use: > PySequence_GetItem ? > No, see above. The code looks like (in "sip" language, sip is a sort of swig, but more specialized to C++ and Qt): void setData(double *, double *, int); %MemberCode PyObject *xSeq, *ySeq; $C *ptr; if (sipParseArgs(&sipArgsParsed, sipArgs, "mOO", sipThisObj, sipClass_$C, &ptr, &xSeq, &ySeq)) { PyArrayObject *x = (PyArrayObject *) PyArray_ContiguousFromObject(xSeq, PyArray_DOUBLE, 1, 0); if (!(x)) return 0; PyArrayObject *y = (PyArrayObject *) PyArray_ContiguousFromObject(ySeq, PyArray_DOUBLE, 1, 0); if (!(y)) return 0; int size; Py_BEGIN_ALLOW_THREADS size = (x->dimensions[0] < y->dimensions[0]) ? x->dimensions[0] : y->dimensions[0]; ptr->setData((double*)(x->data), (double*)(y->data), size); Py_END_ALLOW_THREADS Py_DECREF(x); Py_DECREF(y); Py_INCREF(Py_None); return Py_None; } %End The setData calls copy the data. > > thanks for the tip. Qwt (and PyQwt) look very nice, I may have to > reconsider using PyQT! > Gerard > > -Chris > > > > > > Take a look at http://gerard.vermeulen.free.fr > > > > PyQwt is an addon for PyQt (a Python wrapper for Qt) that knows nothing > > about NumPy > > > > Maybe it is possible to make a NumPy plot add-on for wxWindows, too. > > > > Gerard > > > > On Wed, Jan 15, 2003 at 09:22:20AM -0800, Chris Barker wrote: > > > Hi folks, > > > > > > I use Numeric an wxPython together a lot (of course I do, I use Numeric > > > for everything!). > > > > > > Unfortunately, since wxPython is not Numeric aware, you lose some real > > > potential performance advantages. For example, I'm now working on > > > expanding the extensions to graphics device contexts (DCs) so that you > > > can draw a whole bunch of objects with a single Python call. The idea is > > > that the looping can be done in C++, rather than Python, saving a lot of > > > overhead of the loop itself, as well as the Python-wxWindows translation > > > step. > > > > > > For drawing thousands of points, the speed-up is substantial. It's less > > > substantial on more complex objects (rectangles give a factor of two > > > improvement for ~1000 objects), due to the longer time it takes to draw > > > the object itself, rather than make the call. > > > > > > Anyway, at the moment, Robin Dunn has the wrappers set up so that you > > > can pass in a NumPy array (or, indeed, and sequence) rather than a list > > > or tuple of coordinates, but it is faster to use a list than a NumPy > > > array, because for arrays, it uses the generic PySequence_GetItem call. > > > If we used the NumPy API directly, it should be faster than using a > > > list, not slower! THis is how a representative section of the code looks > > > now: > > > > > > > > > bool isFastSeq = PyList_Check(pyPoints) || > > > PyTuple_Check(pyPoints); > > > . > > > . > > > . > > > // Get the point coordinants > > > if (isFastSeq) { > > > obj = PySequence_Fast_GET_ITEM(pyPoints, i); > > > } > > > else { > > > obj = PySequence_GetItem(pyPoints, i); > > > } > > > > > > . > > > . > > > . > > > > > > So you can see that if a NumPy array is passed in, PySequence_GetItem > > > will be used. > > > > > > What I would like to do is have an isNumPyArray check, and then access > > > the NumPy array directly in that case. > > > > > > The tricky part is that Robin does not want to have wxPython require > > > Numeric. (Oh how I dream of the day that NumArray becomes part of the > > > standard library!) > > > How can I check if an Object is a NumPy array (and then use it as such), > > > without including Numeric during compilation? > > > > > > I know one option is to have condition compilation, with a NumPy and > > > non-Numpy version, but Robin is managing a whole lot of different > > > version as it is, and I don't think he wants to deal with twice as many! > > > > > > Anyone have any ideas? > > > > > > By the way, you can substitute NumArray for NumPy in this, as it is the > > > wave of the future, and particularly if it would be easier. > > > > > > -Chris > > > > > > > > > -- > > > Christopher Barker, Ph.D. > > > Oceanographer > > > > > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > > > 7600 Sand Point Way NE (206) 526-6329 fax > > > Seattle, WA 98115 (206) 526-6317 main reception > > > > > > Chris.Barker(a)noaa.gov > > > > > > > > > ------------------------------------------------------- > > > This SF.NET email is sponsored by: A Thawte Code Signing Certificate > > > is essential in establishing user confidence by providing assurance of > > > authenticity and code integrity. Download our Free Code Signing guide: > > > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker(a)noaa.gov > ------------------------------------------------------------- This message was sent using HTTPS service from CNRS Grenoble. ---> https://grenoble.cnrs.fr <---

1 0

Sparse Arrays in NumPy?
by Costas Malamas 13 Jan '03

13 Jan '03

Hello all, I have been trying to find a package/addon that will provide a sparse array class to NumPy, or will at least trick NumPy to use a sparse array as a regular array, to no avail. By sparse array here, I donot mean a sparse matrix equation solver, but an array class that accepts a "default value". In other words, I would like to instantiate a 1000x1000x1000 (1e9) array that will have at most 5-10% populated (i.e. non-zero) elements. The current NumPy will instantiate the entire 1e9 array, which is a non-starter if you would like to calculate an expression with say 4-5 arrays. Instead, I'd like a class that will only store the populated cells, and return the default value for the others (ideally, but doing some smart disk I/O to preserve memory). I've tried SciPy, Scientific Python, and a few other modules floating around; none seem to do the trick, yet I can't help but wonder that this is not un uncommon setup for a lot of problem domains. Is there a package out there? If there isn't, where should I start looking to create one? From their description I think SparseLib++ at least would be a good starting point as a base library. As a secondary issue, is anyone aware of a package that can handle storage of such arrays? netCDF and HDF do not seem to fit the bill; a B-Tree library seems a more natural fit... Thanks in advance --any and all input appreciated, Costas

2 1

questions about array types
by eric hagemann 12 Jan '03

12 Jan '03

Rereading the numeric docs I see the reference to types Float, Float32, Float64 -- which make sense, however I am curious to understand the usefulness of types Float0, Float8 and Float16 which all seem synonyms for Float32. Was there some thinking that there would be a converter written for 8bit floats? >>> from Numeric import * >>> a = array([1,2,3,4],Float32) >>> fromstring(a.tostring(),Float32) array([ 1., 2., 3., 4.],'f') >>> fromstring(a.tostring(),Float) array([ 2.00000047, 512.00012255]) # corrupt, as would be expected >>> fromstring(a.tostring(),Float0) #seems to convert back as if Float0 == Float32 array([ 1., 2., 3., 4.],'f') >>> fromstring(a.tostring(),Float8) array([ 1., 2., 3., 4.],'f') >>> fromstring(a.tostring(),Float16) array([ 1., 2., 3., 4.],'f') >>>

1 0

Some datatypes missing in numarray recarray?
by Francesc Alted 10 Jan '03

10 Jan '03

Hi, I think there are some data types missing in the recarray module. I can create recarrays using the fromarrays function with no problems except if I use UInt16, UInt32 and UInt64. As these types are well supported by numarray, is there any reason why they don't appear on numfmt and revfmt mappings in recarray module?. Is it safe to add them by hand in the source? Thanks, -- Francesc Alted

2 1

some recarray rework
by Francesc Alted 08 Jan '03

08 Jan '03

Hi, In the context of optimizing the PyTables support for numarray and recarray objects I have been playing with recarray module, and ended with a somewhat improved version of it. Roughly, the modifications done are: - Addition of a cache to quickly access the columns (numarrays) in recarrays. This object is a map (dictionary) where keys are the name fields and values are the pointers to columns regarded as numarrays entities. This dictionary is accessible through the new attribute "_fields". - Addition of an attribute for recarray objects named "_record" which points to a special object ("Record2" class) and that it is aware of the "_fields" cache. It that can be used to access the different rows in recarray objects in an efficient way. - The "_record" object is callable (it defines the "__call__" method) so as to select the recarray row that is active during access to the different fields. Advantages - Access to rows and columns (fields) in recarray objects are one order of magnitude faster (!). - The new "_fields" and "_record" attributes provides convenient and intuitive ways to access the information in recarrays. - The "_record" attribute suports the "__getattr__" and "__setattr__" methods that are very convenient to access fields in a row. Drawbacks - "_record" attribute points always to the same object and you must pass it the row over which you want to operate. So, if you want to have two different objects pointing to different rows, you can't use the "_record" attribute to get them (but you can still use the existing Record class through by calling the "__getitem__" method of a recarray object). - Two new attributes are added to the already large number of recarray variables. However, this new variables has no special space requirements as "_record" object has only three scalar variables and "_fields" is a dictionary with many entries as fields in recarray, which should be not a large amount. I'm attaching this modified version as well as a testbed program in order to test their new access methods and improved performance. The output of this program ran in a pentium4@2GHz machine is also included. Feel free to play with it and/or take/adapt the parts you consider better suited to recarray module. -- Francesc Alted PGP KeyID: 0x61C8C11F

1 0

package vs module
by Perry Greenfield 07 Jan '03

07 Jan '03

Back in December the issue of whether numarray should be a package or set of modules came up. When I asked about the possibility of making numarray a package (on the scipy mailing list but I can't seem to find the thread where it was discussed), I got only positive comments. The issue needs to be raised here also. Is there any objection to making numarray package based? The implications are that 3rd party modules (e.g. FFT) will be imported as part of the package structure, i.e., import numarray.FFT or from numarray.FFT import * instead of import FFT As usual there are advantages and disadvantages. The advantages are that we will not have name collisions with existing Numeric modules (currently we name FFT as FFT2 for this reason). It also potentially reduces name collision issues in general. Most feel it is a cleaner way to organize the software (at least based on the feedback so far). The main disadvantages I see so far are: 1) One will either have to change import statements in old code to match the new style (a pain, but generally changing imports is not terribly difficult since they are easy to identify) or explicitly add the path to each 3rd party module to Python Path (or some equivalent). 2) If numarray were accepted into the Python Standard Library, it would be the first case (as far as I can tell) of a standard library package where we would expect to add sub modules to it (e.g., FFT)). Normally these would not be distributed with the standard library, so some general mechanism will be needed to allow numarray to find 3rd party packages outside of the Python directory structure. For example, I don't think we can require having people install FFT in the Standard Library directory structure after Python is installed. Rather, we would probably have numarray look for extension modules in a standard named site-packages directory (or site-numarray?) or otherwise check a numarraypath environmental variable so that import numarray.FFT works properly. Perhaps others have ideas about how to best handle this. Any other issues being overlooked? Feedback? Thanks, Perry

7 6

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion January 2003