Mailman 3 December 2005 - NumPy-Discussion

Re: [SciPy-dev] New release on Monday
by Travis Oliphant 11 Dec '05

11 Dec '05

Ed Schofield wrote: >On 11/12/2005, at 8:53 AM, Travis Oliphant wrote: > > >Is there any chance of including type checking for this release for >unsafe in-place operations? Several people supported the idea in the >thread [In-place operators and casting] a few weeks ago. In summary, >the existing behaviour would still be achievable with an explicit cast: > >>> my_int_array += my_float_array.astype(int) > > Actually, this would not do the same thing because it would force the entire float array into memory as an integer array and then add it. The current behavior creates only bufsize memories for this kind of copying. So, for large-array performance this approach would be worse, which is a big reason I'm not really supportive of a switch. I'm also hesitant to change this because it is the default behavior of numarray, so I'd like to receive more feedback from members of that community who are coming over to scipy_core before doing something different. I think, however, Numeric raises an error in this circumstance. So, I would advise changing this behavior in the current release, but I don't see this issue as closed. While I would never support changing the data-type of the array when using an inplace operator, I could see the logic in raising an error when the type cannot be cast safely. -Travis

1 0

Re: [SciPy-user] transition problems
by Travis Oliphant 11 Dec '05

11 Dec '05

Christian Kristukat wrote: >Hi, >I noticed that using the new core together with older packages that are based on >numeric, like the plotting module from wxPython, matplotlib, grace_np.py, etc. >is problematic since numeric seems to be unable to recognize new core arrays as >equivalent to the numeric ones. > If you get a recent version (>=24.0) of Numeric then it shouldn't be a problem. One of the purposes of the array interface is to ease the transition, but only recent versions of Numeric use it properly. If there are problems with recent versions of Numeric and the array interface then please post them. Older versions of Numeric would need to use tostring and fromstring (better than tolist...) >In some cases it is sufficient to add a >.tolist() to array parameters but that is not really satisfactory. >Will a final version of new core have solved these incompatibilities? > > Unfortunately, I don't know how to solve incompatibilities with older versions of Numeric. All third-party libraries should either start using only the array interface or switch to scipy_core. I've noticed there is some confusion still circulating on the net that the new arrayobject in scipy_core is going to get into Python at some point, and some people are holding off making any changes until that happens. I am probably the source of that confusion since at one time I did have that goal. However, early on (in March) after discussions with Paul, Perry, and Guido we decided that trying to force an arrayobject into Python would place us on a release cycle that would be constraining. So, don't wait to try out scipy_core because there is nothing going into Python any time soon that is even as capable as Numeric. There are plans for getting a very simple arrayobject into Python (largely to enshrine the array interface), but we really need help moving that forward. With the recent changes to scipy_core, I think we have a very good design for a generic array object with associated data-descriptor that could go into Python. For python itself, I would only say that descriptors should only be written for Object, integer, float, and complex-float arrays. A PEP has been started and sits at http://svn.scipy.org/svn/PEP waiting for more people to help push it forward. Best, -Travis

1 0

《[朗咸平] 中国企业战略突围研讨会》
by ghkkuk 11 Dec '05

11 Dec '05

1 0

Re: Scipy Core (Was Numeric 3.0)
by Colin J. Williams 11 Dec '05

11 Dec '05

Travis, This is intended to restore the context of your response. TO is Travis Oliphant, CW is Colin Williams TO: The dtype parameter is still used and it is still called the same thing. It's just that what constitutes a data-type has changed significantly. For example now tuples and dictionaries can be used to describe a data-type. These definitions are recursive so that whenever data-type is used it means anything that can be interpreted as a data-type. And I really mean data-descriptor, but data-type is in such common usage that I still use it. CW: This would appear to be a good step forward but with all of the immutable types (int8, FloatType, TupleType, etc.) the data is stored in the ArrayType instance (array_data?) whereas, with a dictionary, it would appear to be necessary to store the items outside the array. Is that desirable? TO: I don't even understand what you are questioning here. Perhaps you misunderstood me. Items are always stored in the array. Even records "items" are stored in the array. The descriptor just allows you to describe what kind of data and how it is stored in the array. Response: Yes, I had assumed that int8 and FloatType are equally data-descriptors, i.e. objects which describe the elements of a numeric array. Wrongly, I assumed that TupleType or DictType are also a data descriptors. Mea culpa. On another subject, it would help, for those of us who are not afficionados of C, if you provided the equivalent Python def statement for those function implemented in C, especially the ArrayType's __new__ method. perhaps in the docstrings. Colin W. Travis Oliphant wrote: > Colin J. Williams wrote: > >> >> This would appear to be a good step forward but with all of the >> immutable types (int8, FloatType, TupleType, etc.) the data is stored >> in the ArrayType instance (array_data?) whereas, with a dictionary, >> it would appear to be necessary to store the items outside the >> array. Is that desirable? > > > I don't even understand what you are questioning here. Perhaps you > misunderstood me. Items are always stored in the array. Even records > "items" are stored in the array. The descriptor just allows you to > describe what kind of data and how it is stored in the array. > >> >> Even the tuple can have its content modified, as the example below >> shows: >> > I don't understand what you mean to show by the > tuple-content-modifying example. > >>> dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description >>> of this item. >>> dtype=(str, 10) --- a length-10 string >> >> >> >> So dtype now contains both the data type of each element and the >> shape of the array? This seems a significant change from numarray or >> Numeric. > > > No, no. Standard usage is the same. In normal usage you would not > create an array this way. You could, of course, but it's not the > documented procedure. The reason for this descriptor is to allow > you to have a field-element that itself is an array of items. > >> >> >>> dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor >>> that acts >>> >>> like an int16 array mathematically >>> >>> (in ufuncs) but has real and imag >>> >>> >>> fields. >>> >>> >> This adds complexity, is there a compensating benefit? Do all of the >> complex operations apply? > > > I'm only showing what is possible and that the notion of data-type > descriptor is complete. > >> >> Why not clean things up by dropping typechar? These seemed to be one >> of the warts in numarray, only carried forward for >> compatibility reasons. Could the compatibility objectives of the >> project not be achieved, outside the ArrayType object, with a wrapper >> of some sort? > > > Too hard to do at this point. Too much code uses the characters. I > also don't mind the characters so much (the struct module and the > Python array module use them). > >> >> Couldn't the use of records avoid the cumbersome use of keys? > > > Yes. But, this is how you specify fields generally. > >> >>> format2 (and how it's stored internally) >>> >>> {key1 : (data-type1, offset1 [, title1]), >>> key2 : (data-type2, offset2 [, title2]), >>> ... >>> keyn : (data-typen, offsetn [, titlen]) >>> } >>> >> This is cleaner, but couldn't this inormation be contained within the >> Record instance? > > > > Yes, but I am describing a general concept of data-type description > not just something applicable to records. > >>> Thus, you can do something like this: >>> >>> >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, >>> 1)})) >>> >>> a['imag'] = 2 >>> >>> a['real'] = 1 >>> >>> a.tostring() >>> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >>> >>> >> Or, one could have something like: >> class SmallComplex(Record): >> ..''' This class typically has no instances in user code. ''' >> ..real= (int8, ) >> ..imag= (int8) >> ..def __init__(self): >> .... >> ..def __new__(self): >> .... >> >> >>> a = ones((4,3), dtype= SmallComplex) >> >>> a.imag = 2 >> >>> a.real = 1 >> >>> a.tostring() >> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >> > > > Yes, something like this should be possible, though I have not fleshed > out the user-interface at all. I've just been working on the basic > structure that would support such things. Do: > > class small_complex(record): > dtypedescr = {'r':(int8,0),'i':(int8,1)} > > a = ndrecarray((4,3), formats=small_complex) > a.r = 1 > a.i = 2 > a.tostring() > > and your example would work (with current SVN...) > > The ndrecarray subclass allows the attribute-to-field conversion, > regular arrays do not. > > -Travis > > > > > > > > > > -Travis > >

1 0

Data type change completed
by Travis Oliphant 10 Dec '05

10 Dec '05

I've committed the data-type change discussed at the end of last week to the SVN repository. Now the concept of a data type for an array has been replaced with a "data-descriptor". This data-descriptor is flexible enough to handle an arbitrary record specification with fields that include records and arrays or arrays of records. While nesting may not be the best data-layout for a new design, when memory-mapping an arbitrary fixed-record-length file, this capability allows you to handle even the most obsure record file. While the basic core tests pass for me, there may be lurking problems and so testing of the SVN trunk of scipy core will be appreciated. I've bumped up the version number because the C-API has changed (a few new functions and some functions becoming macros). I'd like to make a release of the new version by the end of the week (as soon as Chris Hanley at STSCI and I get records.py working better), so please test. Recently some intel c-compiler tests were failing on a 64-bit platform. It would be nice to figure out why that is happening as well, but I will probably not have time for that this week. Thanks, -Travis

6 9

Important eBay! Account Information.
by eBay 09 Dec '05

09 Dec '05

1 0

ACML and dotblas
by Arnd Baecker 07 Dec '05

07 Dec '05

Hi, has anyone figured out how to use dotblas with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)? Darren Dale asked this a month ago here http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285 but I could not find any answer. Many thanx, Arnd

2 1

ANN: numarray-1.5.0
by Todd Miller 06 Dec '05

06 Dec '05

Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.5.0 I. ENHANCEMENTS 1. Implementation of scipy newcore array interface and __array_struct__ support (supplying and consuming) for numarray. This should facilitate better interoperability with scipy newcore and Numeric. This protocol is documented here: http://numeric.scipy.org/array_interface.html but also in numarray's source code in Include/numarray/arrayif.h. Thanks to everyone who gave feedback on 1.4.1 and particularly to Francesc Altet for his diligent testing (and doctest) of Numeric<->numarray data interchange. II. BUGS FIXED / CLOSED 1365121 Definition of PyObject* operator conflicts with C++ 1350954 nummacro.h:27: error: expected type-specifier before ';' tok 1346470 Please document more of what byteswap and byteswapped do 1364215 Cannot combine array and masked array (e.g. via divide) 1363723 my_array = +my_other_array uncovers a bug 1364811 Infinite loop converting empty CharArrays to lists 1364815 Strange behaviour when creating empty Int32 arrays 1340983 Fix __get_array_data__ 1346480 byteswapped doesn't match doc string 1346425 Please document if copy "fixes" byte order 1346426 Please document what array does with numarray arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be binary compatible with numarray-1.3.x. New extensions compiled against 1.5.0 may not work with prior versions of numarray. WHERE ----------- Numarray-1.5.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://www.stsci.edu/resources/software_hardware/numarray Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.5.0 requires Python 2.2.3 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller(a)stsci.edu

1 0

Re: [Numpy-discussion] performance of nd_image affine_transform
by Sebastian Haase 06 Dec '05

06 Dec '05

(Peter, I hope you meant to reply to the mailing list...) Hi Peter, Does that mean that the performance hit happens "at the lowest level", i.e. there is lot's of extra stuff in the in the innermost c loop that iterates over the output array ? Or is there just so much setup and cleanup code that makes the overall function slow ? How could I profile C extensions anyway - pointers/hints are appreciated ! Thanks for your nd_image (In case that wasn't clear - I like it a lot !) Sebastian Haase On Tuesday 06 December 2005 01:32, you wrote: > Hi Sebastian, > > The interpolation routines are not really optimized very well. The > underlying code is quite generic for spline order, data type, and > arbritatry array dimension and data layout. Your fortran code is > probably much more optimized. It would be nice to have more optimized > code in nd_image, but I do not really have the resources to do that. > > > Hi, > > Thanks Peter for checking on the problem I reported in my last > > posting... > > > > Now I'm looking into using nd_image.affine_transform inplace of a > > fortran > > routine that I have been using to do this. > > a) I need to run this on Windows - where I don't have Fortran > > b) My Fortran routine does only do linear interpolation and I like > > the idea of > > experimenting with splines. > > > > A and B would of course be good reasons to use nd_image, > > BUT c) > > for a 512x512 float32 image my fortran takes about 14ms > > nd.affine_transform with given output array, prefilter=0 and order=1 > > takes about 132ms ! > > With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it > > takes > > 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 > > 222ms) > > All these are averaged over 10 runs on Linux (P4 2.8GHz) > > > > Why is nd_image 10x slower ? (spline order 1 does the same as linear > > (non-spline) interpolation, right ?) I would call this many (100, > > 1000 ?) > > times inside a simplex algorithm which takes already many seconds > > to complete > > using the Fortran routine... > > > > Thanks, > > Sebastian Haase > > > > On Monday 05 December 2005 14:31, Peter Verveer wrote: > >> Works for me with the latest CVS version. > >> > >> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > >>> Hi, > >>> When I call nd_image.rotate with reshape=False I always get > >>> "output shape not correct" > >>> > >>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], > >>>>>> order=1, > >>> > >>> mode="constant", cval=0.0, prefilter=0) > >>> Traceback (most recent call last): > >>> File "<input>", line 1, in ? > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 351, in > >>> rotate > >>> output, order, mode, cval, prefilter) > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 205, in > >>> affine_transform > >>> output_type) > >>> File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > >>> 73, in > >>> _get_output > >>> raise RuntimeError, "output shape not correct" > >>> RuntimeError: output shape not correct > >>> > >>> I tracked the problem down to "inputShape != outputShape" one > >>> being a tuple > >>> the output shape being a list. > >>> (Pdb) p shape > >>> [128, 528] > >>> (Pdb) p output.shape > >>> (128, 528) > >>> (Pdb) p shape != output.shape > >>> 1 > >>> (Pdb) p shape , output.shape > >>> ([128, 528], (128, 528)) > >>> (Pdb) > >>> > >>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > >>> Apr 22 > >>> 20:35:27 2005//THEAD) > >>> but I took a look at the current CVS and it seems to still be a > >>> problem > >>> > >>> Looks like I'm the only one who doesn't want the reshape ;-) > >>> > >>> Thanks, > >>> Sebastian Haase > >>> > >>> > >>> ------------------------------------------------------- > >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >>> log files > >>> for problems? Stop! Download the new AJAX search engine that makes > >>> searching your log files as easy as surfing the web. DOWNLOAD > >>> SPLUNK! > >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >>> _______________________________________________ > >>> Numpy-discussion mailing list > >>> Numpy-discussion(a)lists.sourceforge.net > >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >> > >> ------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >> log > >> files for problems? Stop! Download the new AJAX search engine > >> that makes > >> searching your log files as easy as surfing the web. DOWNLOAD > >> SPLUNK! > >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion(a)lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD > > SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion

2 1

BUG in nd_image
by Sebastian Haase 05 Dec '05

05 Dec '05

Hi, When I call nd_image.rotate with reshape=False I always get "output shape not correct" >>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], order=1, mode="constant", cval=0.0, prefilter=0) Traceback (most recent call last): File "<input>", line 1, in ? File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 351, in rotate output, order, mode, cval, prefilter) File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 205, in affine_transform output_type) File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line 73, in _get_output raise RuntimeError, "output shape not correct" RuntimeError: output shape not correct I tracked the problem down to "inputShape != outputShape" one being a tuple the output shape being a list. (Pdb) p shape [128, 528] (Pdb) p output.shape (128, 528) (Pdb) p shape != output.shape 1 (Pdb) p shape , output.shape ([128, 528], (128, 528)) (Pdb) I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri Apr 22 20:35:27 2005//THEAD) but I took a look at the current CVS and it seems to still be a problem Looks like I'm the only one who doesn't want the reshape ;-) Thanks, Sebastian Haase

2 2