Mailman 3 September 2003 - NumPy-Discussion

Migrating Char arrays
by Marc Poinot 30 Sep '03

30 Sep '03

Hi all, I'm migrating my pyCGNS package to numarray. I think that numarray is better suited for the in-memory share of array I'd like to achieve. Now, with Numeric, I use to create arrays with integers, floats, but also char type. In CGNS (http://www.cgns.org) we have types I4, I8, R4 and R8, but also C1 which is an array of chars. It looks like, in numarray, we should consider chars as unsigned ints. This is not false from a system point of view, but from an application point of view I have to make a difference between *real* char arrays, and unsigned int arrays. In other words, when I have a C1 array, I have to translate it as a String, when I have an I4 or even I8, I have to translate it as Integer array. Now I only have integers arrays, and I have to find out a way to know that my array of int is a string. The CGNS->numarray translation is OK, but now I'm loosing information when I'm going back numarray->CGNS. You can say: It's up to your application to understand that this list of ints is a string. I'll reply: So why don't you consider a list of bytes, it's up to the application to know that you're 32 or 64 bits... Is there something in numarray that could replace the Numeric interface for this string type? Is it difficult to add a 'Character' type? Is the (undocumented) CharArrayType the solution for this? Marcvs [alias Not using unicode...] ----------------------------------------------------------------------- Marc POINOT Alias: marcvs Email: poinot(a)onera.fr ONERA -MFE/DSNA/ELSA Tel: 01.46.73.42.84 Info: elsa-info(a)onera.fr 29, Div. Leclerc Fax: 01.46.73.41.66 Site: 92322 Chatillon FRANCE Project: elsA Web: http://www.onera.fr

2 3

numarray unfriendly to user defined types
by Tim Hochberg 30 Sep '03

30 Sep '03

I've run into another issue in my attempt to transition over to numarray: it's less friendly to user defined types than Numeric. I think this is mainly accidental friednliness on Numeric's part, but it's handy nonetheless. The attached file illustrates the issue. Given some object, in this case *zero*, that a numarray array does not now how to handle, it ends up raising an exception instead of giving the object a chance to try. Thus ``zero + numarray.arange(5)`` works while ``numarray.arange(5) + zero`` fails, since in the first case Zero.__add__ is called first, while in the second NumArray.__add__ is called first. This should probably be fixed, but it's not clear what the best way is. My first thought was to always give the other object a chance to use __rop__ first. This is simple and easy to explain, but fails miserably in the common case of multiplying a list and an array. Just to be concrete, __mul__ would be replaced by:: def __mul__(self, operand): try: return operand.__rmul__(self) except: return ufunc.multiply(self, operand) Next thought is to catch exceptions when they occur in numarray and then give the other operand a chance:: def __mul__(self, operand): try: return ufunc.multiply(self, operand) except: return operand.__rmul__(self) This appears like it would fix my particular problem, but still is not ideal. Since numarray is the base of libraries that it will know nothing about, it should defer to classes it doesn't know about whenever possible. Otherewise it's not possible (or maybe just hard) to create new classes that try to be clever, but still interact with numarray in a reasonable way. In my case I'm thinking of proxy objects that don't do some computations till they are actually required. So my current thinking is that __mul__ and friends would be best implemented as:: def __mul__(self, operand): if not isinstance(operand, knownTypes): try: return ufunc.multiply(self, operand) except: pass return operand.__rmul__(self) Where knownTypes is something like (int, long, float, complex, tuple, list). Anyway, that's my excessively long two cents on this. -tim import numarray as na import Numeric as np class Zero: def __add__(self, other): return other __radd__ = __add__ zero = Zero() #~ print zero + np.arange(5) #~ print np.arange(5) + zero #~ print zero + na.arange(5) #~ print na.arange(5) + zero a = na.arange(5) import copy copy.deepcopy(a)

4 18

Re: Cygwin compilation
by Jochen Küpper 30 Sep '03

30 Sep '03

On 29 Sep 2003 17:05:09 -0400 Todd Miller wrote: >> _ufunc.dll needs to be linked against /usr/lib/libmingwex.a. Doing >> that manually seems to work fine. Todd> In the file generate.py, a clause for sys.platform == "cygwin?" Todd> probably needs to be added. Then, define EXTRA_LINK_ARGS for Todd> cygwin. Ok, did that. ,----[cvs diff -u generate.py] | Warning: No xauth data; using fake authentication data for X11 forwarding. | Index: generate.py | =================================================================== | RCS file: /cvsroot/numpy/numarray/generate.py,v | retrieving revision 1.14 | diff -u -u -r1.14 generate.py | --- generate.py 16 Sep 2003 20:02:41 -0000 1.14 | +++ generate.py 30 Sep 2003 08:33:38 -0000 | @@ -51,6 +51,9 @@ | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 | elif sys.platform == "win32": | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 0, 0 | +elif sys.platform == "cygwin": | + LP64, HAS_UINT64, HAS_FLOAT128 = 0, 0, 0 | + EXTRA_LINK_ARGS += ["-L/lib/mingw -lmingwex"] | elif sys.platform == "darwin": | LP64, HAS_UINT64, HAS_FLOAT128 = 0, 1, 0 | EXTRA_COMPILE_ARGS.extend(["-Ddarwin"]) `---- Should work, but I get the following strange behavior: ,---- | > python setup.py build | Using EXTRA_COMPILE_ARGS = [] | running build | running build_py | copying Lib/numinclude.py -> build/lib.cygwin-1.5.5-i686-2.3/numarray | running build_ext | building 'numarray._ufunc' extension | gcc -shared -Wl,--enable-auto-image-base -march=pentium3 -mcpu=pentium4 -O2 -Wall -pipe build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o -L/usr/lib/python2.3/config -lpython2.3 -o build/lib.cygwin-1.5.5-i686-2.3/numarray/_ufunc.dll -L/lib/mingw -lmingwex | build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o(.text+0x14f): In function `_CheckFPErrors': | /home/software/programming/numeric/numarray/Src/_ufuncmodule.c:136: undefined reference to `_fetestexcept' | build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o(.text+0x17b):/home/software/programming/numeric/numarray/Src/_ufuncmodule.c:142: undefined reference to `_feclearexcept' | collect2: ld returned 1 exit status | error: command 'gcc' failed with exit status 1 | | > gcc -shared -Wl,--enable-auto-image-base -march=pentium3 -mcpu=pentium4 -O2 -Wall -pipe build/temp.cygwin-1.5.5-i686-2.3/Src/_ufuncmodule.o -L/usr/lib/python2.3/config -lpython2.3 -o build/lib.cygwin-1.5.5-i686-2.3/numarray/_ufunc.dll -L/lib/mingw -lmingwex `---- That is, it does not work when called from setup.py. When I just copy 'n' paste the link statement to the command-line, it just works.(#$%) Any ideas?? >> Is it save to use this mingw library in "normal" Cygwin programs? Todd> I don't know. I didn't get any farther than this, perhaps not as Todd> far. I got the impression that mingw was a minimal library Todd> intended for porting gcc (but it has IEEE stuff), while the Todd> cygwin libc is intended for real programs (but has no IEEE stuff Todd> because it is a lightweight knock off of glibc used to bypass Todd> the GPL). Well, as I see it Cygwin is a Unix emulation layer for Windows, whereas MINGW is a port of gcc and some other GNU tools to Windows (not using Cygwin at all). But looking around using it this way should be ok. Maybe someone who knows cygwin better can speak up here? I would carbon-copy this email to the cygwin list, but that is subscribers only nowadays:( >> Moreover I have the follwoing test error: >> ,---- >> | Failure in example: log10([1+1j]) >> | from line #2267 of numarray.numtest >> | Expected: array([ 0.150515+0.34109409j]) >> | Got: array([ 0.150515+5.21807245j]) >> `---- Todd> One of our guys found a bug in the complex log10 function; both Todd> the expected result and the implementation were changed. Seems to be resolved. Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Liberté, Égalité, Fraternité GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.)

2 3

Crash with numarray testall.py
by Colin J. Williams 29 Sep '03

29 Sep '03

C:\Program Files\Python23\Lib\site-packages\numarray>python Python 2.3.1 (#47, Sep 23 2003, 23:47:32) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import testall The 21K file created by Windows to contain information about the crash is not readable. Windows XP and numarray 0.7 used. Colin W.

2 1

Blitz++, NumArray...
by Fernando Perez 26 Sep '03

26 Sep '03

Hi all, as some of you know, I'm a big fan of weave.inline() for the ease of development it offers. Essentially C++ becomes a scripting part of python, with only the occasional recompile overhead (but this is transparent to the users). inline() makes heavy use of the blitz++ library, to represent numpy arrays as blitz arrays and therefore index them as Arr(i,j,k...). This seems to me like a very sound decision, as the blitz library was specifically written to provide simultaneously fortran-like performance with high-level object orientation. With this in mind, I have a few questions: - Is there anything in Numarray's design which prevents this same trick from being used? The core of getting a Numpy array into a blitz one is simply a call to: // Convert a Numpy array to a blitz one, using the original's data (no copy) template<class T, int N> static blitz::Array<T,N> py_to_blitz(PyArrayObject *arr_obj) { blitz::TinyVector<int,N> shape(0); blitz::TinyVector<int,N> strides(0); for (int i=0;i<N;i++) { shape[i] = arr_obj->dimensions[i]; strides[i] = arr_obj->strides[i]/sizeof(T); } return blitz::Array<T,N>((T*) arr_obj->data,shape,strides, blitz::neverDeleteData); } I would not be surprised if something much like this would work for Numarrays, but I'd like to know for sure. - If the above is indeed possible for Numarray, do you guys feel that blitz should be considered an integral part of python's low-level arsenal, or not? I view its advantages especially in the field of an extremely clean, simple syntax (compared to the low-level Numpy API, I'm not familiar with Numarray yet). I've seen examples where a numpy algorithm can be recoded using blitz in a _far_ smaller amount of code, and with performance gains to boot (in some cases, not all). Please note that I am NOT second-guessing any of the design decisions made by the Numarray crowd. I just want to clarify if it's possible to have a C++ API which gives access to numarray with the simplest, highest-level syntax and behavior which is reasonable. This is important for teams where not everyone is an expert on C pointer manipulation details, but they may be perfectly ok with writing loops over things which can be indexed as A(i,j,k). - If any of this seems worthy of more detailed discussion, I told Eric Jones that I'd be happy to coordinate something along these lines at Scipy'03. If there's time in the schedule I'd be glad to do it officially, otherwise we can just try to keep it in mind for a lunch/dinner meeting. Documenting a 'Best Practices for low-level numerical work in python' would probably be a beneficial thing to do for the community. Right now there are so many options that I'm sure I'm not the only one to feel a bit confused. - If Blitz++ is indeed deemed a worthy piece of this puzzle, I wonder if we could contact some of its developers to bring them into this discussion. I worry a bit that activity on its mailing lists seems rather slow, but I noticed that an apparently central guy, Julian C. Cummings, is at CACR/Caltech. Perhaps he could stop by and let us know what the future for blitz looks like. I can fire off an email to him if there is interest in the blitz thing. Anyway, I'm just trying to 'pre-soak' a few ideas for discussion at scipy'03 on these topics. Best regards, Fernando.

3 3

numarray incompatibility: searchsorted
by Tim Hochberg 26 Sep '03

26 Sep '03

There are some compatibility and doc issues and perhaps a bug in numarray.searchsorted. The compatibility issue is that Numeric.searchsorted(a, v) accepts either a sequence or scalar value for v. Numarray.searchsorted accepts only sequence values. Second, the doc issue. The docstring for numarray.searchsorted states:: searchsorted(bins, values) searchsort(bins, values) returns the array of greatest indices 'i' such that each values[i] <= bins[i]. I assume that should really read something more like:: searchsorted(bins, values) searchsort(bins, values) returns the array A[j] of greatest indices 'i' such that each values[j] <= bins[i]. Third, the possible bug: # na = numarray, np = NumPy >>> na.searchsorted([1,2,3,4], [2.5, 3.5]) array([1, 2]) >>> np.searchsorted([1,2,3,4], [2.5, 3.5]) array([2, 3]) Hmmm. It appears that numarray result does match the numarray docs, (at least as I read them), but I like the Numeric behaviour better. The Numeric behaviour also matches the behaviour of the bisect module, which is described as:: bisect = bisect_right(a, x, lo=0, hi=None) Return the index where to insert item x in list a, assuming a is sorted. The return value i is such that all e in a[:i] have e <= x, and all e in a[i:] have e > x. So if x already appears in the list, i points just beyond the rightmost x already there. Optional args lo (default 0) and hi (default len(a)) bound the slice of a to be searched. I'd recomend matching the behaviour of the two existing modules (bisect and Numeric). -tim

2 1

Re: [Numpy-discussion] Subclassing numarray arrays
by Nadav Horesh 25 Sep '03

25 Sep '03

On Wed, 2003-09-24 at 14:31, Colin J. Williams wrote: > Nadav Horesh wrote: > > >Recently I had an idea to subclass numarray's array in order to > >implement a different scheme of array indexing via the __call__ method > >(ugly indeed, by my experience shows that it can be useful). It occurred > >to me that the array class is not in the spirit of Python 2.2+ > >classes/types unification: > > > I agree. > > > * The class name is not "trivial": numarraycore.NumArray > > * The class constructor is not built to be used by the commons. > > > I'm not sure what you have in mind here by "trivial" and "commons". commons == End users whose most scripts life-time is few days. trivial == A name one would expect, like "Array" or similar. > > >Since arrays are normally generated by functions (zeros, array, > >fromfunction etc.) and not by the constructor, subclassing > >numarraycore.NumArray is not very useful. If, for instance, "array" (or > >"Array") would be the class name, where the constructor would optionally > >have the same parameters as the array function (with the option to act > >as a copy constructor as the array function does), then it would be easy > >to subclass it: > > > >class MyArray(array): > > ... > > > > > >new_array = MyArray(zeros((10,5), type=Int64)) > > > > > > > This is desirable, with the present design, one has to go through a two > step process: > > new_array = MyArray(type=Int64, shape= (10, 5)) > load_some_data_into_array(new_array, zeros(10, 5)) > > Colin W. As I recall, one of the targets of the numarray project is to be included as a standard python library --- shouldn't it eventually comply with the new-classes standard? > > > Any comments? > > > > Nadav. > > > > > > > >------------------------------------------------------- > >This sf.net email is sponsored by:ThinkGeek > >Welcome to geek heaven. > >http://thinkgeek.com/sf > >_______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion(a)lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > >

3 4

Re: [Numpy-discussion] Subclassing numarray arrays
by Todd Miller 25 Sep '03

25 Sep '03

On Thu, 2003-09-25 at 12:28, Paul Dubois wrote: > I have not followed this thread in any detail but I continue to be > puzzled by the idea of subclassing numarray arrays. Any subclass worth > its salt has an unpredictable constructor argument list, so that > virtually every ufunc and every array method have to be overridden in > order to return the subclass object rather than a numarray. I think there is a need to effectively subclass numarray where the parameter lists don't change. Doing a Matrix subclass sounds reasonable to me and we should explore the OOP necessary to make it convenient. STSCI hasn't asked for it, but a couple different numarray contributors have. > I gather > that the defers-to-me stuff allows the subclass to get control of a > mixed operation but that was possible with the radd, etc. operators > already as in the old MA. radd does not work for numarray + MA. Instead, the expression is resolved as numarray.__add__(MA). Were MA a numarray subclass, it would work, given the facilities provided by Python alone. The defers-to-me-stuff is a simple work around to avoid having to reverse engineer this one Numeric behavior. > This reasoning led me to believe that subclassing had virtually no value > and to the design of MA that has-a numeric array rather than is-a. Since we want to port MA as-is to numarray, we're not changing that. > Is there some big breakthrough here I missed? No, but there are two issues: (1) effectively supporting subclasses (2) effectively supporting interoperability with other classes. (1) needs more discussion. (2) is already solved. -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

Subclassing numarray arrays.
by Nadav Horesh 24 Sep '03

24 Sep '03

Recently I had an idea to subclass numarray's array in order to implement a different scheme of array indexing via the __call__ method (ugly indeed, by my experience shows that it can be useful). It occurred to me that the array class is not in the spirit of Python 2.2+ classes/types unification: * The class name is not "trivial": numarraycore.NumArray * The class constructor is not built to be used by the commons. Since arrays are normally generated by functions (zeros, array, fromfunction etc.) and not by the constructor, subclassing numarraycore.NumArray is not very useful. If, for instance, "array" (or "Array") would be the class name, where the constructor would optionally have the same parameters as the array function (with the option to act as a copy constructor as the array function does), then it would be easy to subclass it: class MyArray(array): ... new_array = MyArray(zeros((10,5), type=Int64)) Any comments? Nadav.

2 1

ANN: Job posting -- developer position available
by eric jones 23 Sep '03

23 Sep '03

Hey group, We have a developer position open here at Enthought, and I thought I'd see if anyone is interested. The posting is below. There is also a tech writer and HCI opening listed on the website: http://www.enthought.com/careers.html thanks! eric Position: Python Developer Type: Full-Time Enthought, Inc. is a technology startup in the Austin, TX area that is looking for a motivated Python Developer to join our development team. Responsibilities include developing and deploying scientific applications within the Fortune 500. Our size and prospects provide significant opportunity for growth and advancement for Enthought employees. Candidates should have the following qualifications: - BS or MS in Computer Science or Engineering or other scientific field - 3+ years experience in the software industry - Experience developing solutions in the following languages: - Python - C/C++ - User interface design experience - Developing and deploying custom solutions for clients - Excellent verbal and written communication skills Ideal candidates will also demonstrate the following skills: - Use of the wxPython user interface library - Experience with Geophysics applications - Experience in the fields of statistics, linear algebra and calculus Please send resumes to jobs(a)enthought.com.

3 3

numarray boolean array formatting poll results
by Todd Miller 23 Sep '03

23 Sep '03

The thread on what to do with boolean array formatting in numarray has died out, so I thought I should explain (what I saw) as the results of the poll: (1) There was a clear majority of numarray users who actually prefer the status quo of 0,1 for representing arrays of boolean values. (2) There was a sizable minority that would like to see consistency with Python-2.3 by using False, True for representing arrays of boolean values. (3) There was a (good) suggestion of using F, T for representing boolean values. However, on reflection and reviewing the poll comments, F, T is not really as good as 0, 1 visually, nor is it likely to satisfy those people interested in consistency with Python, so it is unlikely to make most numarray users any happier. (4) Also suggested was supporting configurable boolean formats. This approach is reasonable to implement, but I have a reservation that adding support for multiple formats may cause as much confusion as it gives satisfaction. Is it *really* worth it? (more later) Given 1-4, I think the best thing to do for now is keep 0,1 as the only format for boolean array values. If someone wants False, True bad enough to submit a complete patch for supporting multiple formats (and defaulting to 0,1), we will still accept it. If anyone is still interested in doing this, the patch should include modified TeX documentation and pass all the self tests in setupall.py (all of core numarray and all current add-ons). The patch should also run cleanly on both Python-2.2 and Python-2.3. Thanks everyone for responding, Todd -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

[ANN] PyTables 0.7.2 released
by Francesc Alted 22 Sep '03

22 Sep '03

Announcing PyTables 0.7.2 ------------------------- PyTables is a hierarchical database package designed to efficently manage very large amounts of data. PyTables is built on top of the HDF5 library and the numarray package. It features an object-oriented interface that, combined with natural naming and C-code generated from Pyrex sources, makes it a fast, yet extremely easy to use tool for interactively save and retrieve large amounts of data. Besides, it provides flexible indexed access on disk to anywhere in the data you want to go. On this release you will not find any exciting new features. It is mainly a maintenance release where the next issues has been addressed: - a memory leak was fixed - memory consumption is being addressed and lowered - much faster opening of files - Some important index patterns cases in table reads has been optimized More in detail: What's new ----------- - Fixed a nasty memory leak located on the C libraries (it was happening during HDF5 attribute writes). After that, the memory consumption when using large object trees has dropped quite a bit. However, there remains some small leaks that has been tracked down to the underlying numarray library. These leaks has been reported, and hopefully they should be fixed more sooner than later. - Table buffers are built dinamically now, so if Tables are not accessed for reading or writing this memory will not be booked. This will help to reduce the memory consumption. - The opening of files with lots of nodes has been accelerated between a factor 2 or 3. For example, a file with 10 groups and 3000 tables that takes 9.3 seconds to open in 0.7.1, now takes only 2.8 seconds. - The Table.read() method has been refactored and optimized and some parts of its code has been moved to Pyrex. In particular, in the special case of step=1, up to a factor 5 of speedup (reaching 160 MB/s on a Pentium4 @ 2 GHz) when reading table contents can be achieved now. - Done some cosmetic changes in the user manual, but, as no new features has been added, you won't need to read the manual again :-) What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seems to be quite a strange requirement for an language like Python, that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors and running IRIX 6.5. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP, but it should also work with other flavors. An example? ----------- For online code examples, have a look at http://pytables.sourceforge.net/tut/tutorial1-1.html and http://pytables.sourceforge.net/tut/tutorial1-2.html Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted

1 0

Promoting Your Business
by Betty 20 Sep '03

20 Sep '03

Email Marketing is one of the most effective and inexpensive ways to promote your products and services. We offer a complete Email Marketing solution with quality service and the lowest prices. The result is that you will enjoy more success. 1. Targeted Email Addresses We can supply targeted email addresses according to your requirements, which are compiled only on your order. We will customize your customer email addresses. * We have millions of email addresses in a wide variety of categories. 2. Targeted Mailing If you are worried about any complications or consequences with sending out targeted emails, or want to avoid the work of sending out targeted emails. We will do it for you! We can send your email message to your targeted customers. * We can Bullet-Proof your Web Site. We also offer a wide variety of marketing software. For more details, you can refer to: www.marketingboosting.com Our services will help you get more business opportunities. Regards! Betty Jones Customer Support www.marketingboosting.com Sales(a)marketingboosting.com ************************************************************************ Receiving this email because you registered to receive special offers from one of our partners. If you would prefer not to receive future email, click: Http://optin.garyshawkey.com/optin/r.php3 ************************************************************************

1 0

R E S U M E
by ELECTRICIAN 19 Sep '03

19 Sep '03

Resume from: Rich for Job or Service " no job too small " E L E C T R I C I A N Tel. (408) 482-2102 rysio3(a)yahoo.com WIRING & INSTALLATION Hands on electrical installations perform fitting, mounting, laying cables on Commercial, Industrial, residential new & existing buildings. Electrical Power Supply for Lights, Plugs, Receptacles, Panels, & Fuse boxes, Emergency Generators wiring and testing, Transformers, Power Lines & conduit layout, bending and mounting, parking lighting, lamps, switches, SOLAR PROJECTS, posts and underground installations. Shopping Centers; grocery stories, hardware stories, restaurants & residential - housing areas, computer business & fast food units installation & buildings; Solar Panels, Sun Tracking, Flywheel Storage & electric cars systems modify, Natural Energy in Remote areas install. LOW VOLTAGE Office Home Yard Patio Parking 12 / 24 Volt audio & video equipment, Computer & data network wiring, data backup and UPS; Monitoring Video Control & backup tapes set up and mounting, electro-optical assemblies & subsystems. DC Power Supply, Switch & Motion sensors Alarm. Fire & safety systems install. Fiber Optics systems, PLC setup, Master Control Center, cable modems & cable TV install. Network, UPS Battery Backup mounting and charging systems; Power supply testing, troubleshooting, and analyzing to a components level. Electric Vehicles Design, Assembly & Installations. CC TV & Cameras, Security Systems & Sensors for Safety, Fire sprinklers and traffic Monitoring & Door Control. Telephones / Net move & install. TECHNICIAN Use lab & shop equipment, mechanical, electrical & electronic tools, measurement & testing equipment, video cameras & microscopes. Support scientists & electronic engineers. Mechanical & Electro-Mech. Design. OFFICE, ELECTRICAL AND MECHANICAL PROJECTS Electrical & Network Sketches, one line diagrams, and "as is" drawings update. Customizing Electronic and Electrical Components & Parts, Layouts electronic and electrical schematic, connectors and mechanical detailing. Quotes, supply, bids and job estimating. Customers contact, inspection, project mgmt & supervision of electricians & material handling; Use CAD, Windows and applications; ELECTRICAL & MAINTENANCE SERVICE US Citizen; open for travel .

1 0

Re: [Numpy-discussion] array sort questions
by CL WU 18 Sep '03

18 Sep '03

Great! It works much more efficiently. Thank you so much. Best, Chunlei Tim Hochberg wrote: > CL WU wrote: > >> Thank you, Tim. argsort() and take() does provide a easy way to sort >> an array based on any col or row. But for the second question, it >> doesn't return the result I want. >> As below, softrank or softrank1 are functions I am currently using >> for get the rank of a vector(first is more efficient). It returns the >> index of each value from original array/list in sorted array/list. > > > Hmmm. It seems that argsort and sortrank are inverses of a sort, so it > should be possible to do what you want efficiently, but I'm not sure how. > > <think> > > Ah, it appears to be quite simple. I believe: > > argsort(argsort(a)) > > is equivalent to your sortrank and should be much faster. > > regards, > > -tim > > >> I hope there is an efficient function in array level to do the same >> work. >> >>> from Numeric import * >> >>> a=array([5,2,3]) >> >>> argsort(a) >> array([1, 2, 0]) >> >>> def sortrank(list): >> ... n=len(list) >> ... li_a=[(i,list[i]) for i in range(n)] >> ... li_a.sort(lambda a,b:cmp(a[1],b[1])) >> ... li_b=[(i,li_a[i]) for i in range(n)] >> ... li_b.sort(lambda a,b:cmp(a[1][0],b[1][0])) >> ... return [x[0] for x in li_b] >> ... >>> sortrank(a) >> [2, 0, 1] >> >>> def sortrank2(li): >> ... li_sorted=li[:] >> ... li_sorted.sort() >> ... return [li_sorted.index(x) for x in li] >> >>> sortrank1(list(a)) >> [2, 0, 1] >> >> >> Thanks again. >> >> Chunlei >> >> Tim Hochberg wrote: >> >>> CL WU wrote: >>> >>>> Hi, group, >>>> I am new to numpy. I have 2 questions for array sort. >>>> >>>> 1. How to sort an array by its one column or one row? >>>> I know python build-in sort() can do it for list by passing own >>>> cmp function. but array function sort() will sort each column or >>>> row seperately,as I know. I don't want to convert array to list to >>>> sort and then convert back to array. >>> >>> >>> >>> >>> I think you want argsort plus take. For example, the following sorts >>> on the second column of a: >>> >>> a = array([[4,5,6], [1,2,3], [7,8,9]]) >>> arg = argsort(a[:,1]) >>> take(a, arg, 0) >>> >>>> 2. How to get the rank of a rank-0 array? The first "rank" means >>>> the order of each element after sorting, instead of the "dimension" >>>> meaning in numpy. Just like "rank()" function in splus. >>> >>> >>> >>> >>> If I understand you correctly, you want argsort as mentioned above. >>> >>> Regards, >>> >>> -tim >>> >>> >>>> >>>> Thank you >>>> >>>> Chunlei >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------- >>>> This sf.net email is sponsored by:ThinkGeek >>>> Welcome to geek heaven. >>>> http://thinkgeek.com/sf >>>> _______________________________________________ >>>> Numpy-discussion mailing list >>>> Numpy-discussion(a)lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>>> >>> >>> >>> >>> >>> >>> ------------------------------------------------------- >>> This sf.net email is sponsored by:ThinkGeek >>> Welcome to geek heaven. >>> http://thinkgeek.com/sf >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion(a)lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>> >> >> >> >> >> ------------------------------------------------------- >> This sf.net email is sponsored by:ThinkGeek >> Welcome to geek heaven. >> http://thinkgeek.com/sf >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > > > >

3 4

array sort questions
by CL WU 17 Sep '03

17 Sep '03

Hi, group, I am new to numpy. I have 2 questions for array sort. 1. How to sort an array by its one column or one row? I know python build-in sort() can do it for list by passing own cmp function. but array function sort() will sort each column or row seperately,as I know. I don't want to convert array to list to sort and then convert back to array. 2. How to get the rank of a rank-0 array? The first "rank" means the order of each element after sorting, instead of the "dimension" meaning in numpy. Just like "rank()" function in splus. Thank you Chunlei

2 2

numarray latest version
by Nadav Horesh 17 Sep '03

17 Sep '03

How comes that numarray 0.6.2 succeeds 0.7? Are there 2 active branches? Nadav.

2 1

Re: MacPython and Numeric
by Jack Jansen 14 Sep '03

14 Sep '03

Folks, I got this report on Numeric as included with MacPython (through the package manager) and it looks promising. But: I'm building Numeric simply by running setup.py, so in order to get the added performance Numeric's setup.py would need to be taught about veclib. Is anyone willing to look into this? On Sunday, September 14, 2003, at 09:35 PM, Alun Carr wrote: > Jack, > > I would like to thank you for all your hard work in producing > MacPython 2.3 for OS X. It's superb. > > One point to note: I think that you may have compiled the Numeric > stuff from scratch. Apple supply hand-tuned BLAS, LAPACK, and FFT code > in their vecLib framework. Despite the name, according to Apple's > documentation this framework is suitable for both G3 and G4 > processors, and it will automatically use vectorized code if it > detects a G4. > > From <http://developer.apple.com/hardware/ve/quickstart.html>: > >> If you are a developer who would like to make more use of AltiVec in >> your application, you should first explore the functionality that is >> available in Apple's vecLib.framework . This framework can be found >> in /System/Library/Frameworks on any MacOS X installation. There you >> will find many highly optimized routines for a diversity of tasks. >> There are single and double precision FFT's. Some of these are up to >> five times faster than FFT-W . There are also convolutions, 1D and 2D >> matrix operations and much, much more. MacOS X.2 (Jaguar) rounds this >> out with a full BLAS and LAPACK implementation in both single and >> double precision. These functions also work on G3 and earlier >> processors, meaning that you can call them safely regardless of what >> machine you are running on. > > For future releases of Numeric for MacPython, it might be worth using > vecLib, to (i) reduce the effort on your part, (ii) reduce the size of > the distribution, and (iii) to use code which has been optimized by > Apple for maximum performance. > > Anyway, thanks again for your efforts, and please keep up the good > work. > > Best regards, > > Alun > > == > > Dr Alun J. Carr Phone: +353-(0)1-716 1989 (Office) > Mechanical Engineering Dept +353-(0)1-716 1752 (BRC) > University College Dublin Fax: +353-(0)1-283 0534 > Belfield Email: alun.j.carr(a)ucd.ie (UCD) > Dublin 4 carraj(a)attglobal.net (Home) > Ireland > -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

2 1

Boolean array formatting poll
by Todd Miller 11 Sep '03

11 Sep '03

Now that Python has a bool type, I have received a patch for numarray's arrayprint module which alters the formatting of Boolean arrays from: >>>> a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a array([0, 1, 0, 1, 0, 1], type=Bool) to: >>>> a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a array([False, True, False, True, False, True], type=Bool) Which format do you prefer? Please vote for or against changing the format: +1 (for) or -1 (against). -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

10 13

Why stripping spaces in strings?
by Francesc Alted 10 Sep '03

10 Sep '03

Hi, A user of pytables recently asked me why it truncates the trailing spaces on strings. I have to explain him that the information is there, but the strings module of the numarray package deliberately strips off the trailing spaces when printing: >>> from numarray import strings >>> a=strings.array("hello world ") >>> print a ['hello world'] >>> print "<%s>" % a <['hello world']> >>> len(a[0]) 11 however: >>> print "<%s>" % a._data <hello world > >>> a._byteView() array([[104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 32, 32]], type=UInt8) >>> a._itemsize 13 In the beginning I thought that this was a nice feature, but when people is using numarray.strings objects as containers (for example for Pickled objects), that can be rather confusing, and I would prefer to see all the trailing blanks. But as I am not completely sure, I just would like to discuss the pro's and con's of the current printing apprach. Cheers, -- Francesc Alted

2 2

NA_ReturnOutput question
by Peter Verveer 09 Sep '03

09 Sep '03

The documentation states that NA_ReturnOutput(numarray, shadow) should be used with the value returned by NA_OptionalOutputArray() for the shadow parameter. Is it also possible to use a shadow array allocated otherwise, e.g. by NA_NewArray()? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: Peter.Verveer(a)embl-heidelberg.de

2 1

Re: [Numpy-discussion] Attempting to sub-class NumArray
by Colin J. Williams 07 Sep '03

07 Sep '03

Todd Miller wrote: >I think the problem you are having is caused by the assumption that >instance initialization is equivalent to an unordered assignment of all >instance attributes. That's not true because the private attributes are >interrelated; assigning "shape" affects "_shape" and "_strides", >_itemsize must be known before _strides can be computed from shape, and >so on. > It would be helpful if you could let us know what assignment ordering would be appropriate, or, if there were information about the interactions, perhaps we could work these things out for ourselves. As we've discussed before, there is little information available on private objects and their roles. Even to have the source code of some of the C code might be useful. I wasn't able to spot the relevant C code among the CVS directories. However, the assumption would appear to apply to the crude initialization of 'special'. The problem which is being reported seems to be connected with the function setshape in generic.py. This is far from the unordered assignment, although I recognize that ther might be some connection. Colin W. > >On Fri, 2003-09-05 at 20:32, Colin J. Williams wrote: > > >>Todd, >> >>Thanks again for your prompt response. >>Todd Miller wrote: >> >> >>>Hi Colin, >>> >>>I haven't looked at this in detail, but special.__init__() is making me >>>queasy. >>> >>> >>We don't call __init__ directly, but only when we construct the >>instance, a, of the class special. >> >> >>>Why doesn't it call NumArray.__init__()? >>> >>> >>Because we have data to install in the newly created a. We go off to >>the factory function, numarray.array, to create the array, which we >>then specialize. >> >>I don't see how NumArray.__init__ would help. This cumbersome >>procedure seems to work OK in most respects, but fails to display >>properly. >> >>I don't follow the code sufficiently well to be sure, but my hunch is >>that the problem lies with setShape. It fails to update _strides. >> >>Colin W. >> >> >>>Todd >>> >>>On Fri, 2003-09-05 at 15:03, Colin J. Williams wrote: >>> >>> >>> >>>>When ravel() is used on an instance of NumArray, the _strides attribute >>>>is recalculated correctly. >>>>When ravel() is used on an instance of special, a sub-class of NumArray, >>>>the _strides attribute keeps its old value. This appears to cause the >>>>display problem which is reported. >>>> >>>>The essentials of the code used are below. >>>> >>>>To investigate this problem, two print statements were added to the >>>>method setshape in generic.py: >>>> >>>> if newnelements == nelements: >>>> self._shape = tuple(shape) >>>> print 'In setShape self._shape:', self._shape >>>> self._strides = self._stridesFromShape() >>>> print 'In setShape self._strides:', self._strides >>>> else: >>>> raise ValueError("New shape is not consistent with the >>>> old shape") >>>> >>>>It seems that only the first of these print statements is executed with >>>>special, but both are executed with NumArray. In the second case, the >>>>output is not in the expected sequence. >>>> >>>>I would appreciate any suggestion as to a workaround. >>>> >>>>Colin W, >>>> >>>>Grief when attempting to sub-class NumArray >>>> >>>>Code used: >>>> >>>># Try to sub-class >>>>class special(N.NumArray): >>>> def __init__(self, data= None, shape= None, eType= None): >>>>## eType= _nt.Any or AnyType not acceptable %% >>>> arr= N.array(sequence= data, shape= shape, type= eType) >>>> for attr in dir(arr): >>>>## This is a longwinded way of setting up special >>>>## There must be a better way %% >>>>## Perhaps we should use generic._view >>>>## It's not documented TRY IT LATER >>>>## There is no need to transfer methods! >>>> if attr[0] == '_' and attr[1] != '_': >>>> print attr >>>> exec 'self.' + attr + '= ' 'arr.' + attr >>>> exec 'print self.' + attr >>>> pass >>>> >>>>a= special(data= [1, 2, 3, 4], shape= (2, 2)) >>>>a= N.array([1, 2, 3, 4], shape= (2, 2)) >>>>print 'In ts a._strides:', a._strides >>>>b= _gen.ravel(a) >>>>print 'In ts b._strides:', b._strides # <<< Unchanged with >>>>special, OK with NumArray >>>>print 'b:', b >>>> >>>> >>>> >>>>USING THE SUBCLASS special >>>> >C:\Progra~1\Python23\pythonw -u ts.py >>>>In ts a._strides: (8, 4) >>>>In setShape self._shape: (4,) >>>>In ts b._strides: (8, 4) >>>>b: In setShape self._shape: (4,) >>>>Traceback (most recent call last): >>>> File "ts.py", line 37, in ? >>>> print 'b:', b >>>> File >>>>"C:\PROGRA~1\Python23\lib\site-packages\numarray\numarraycore.py", line >>>>618, in __str__ >>>> MAX_LINE_WIDTH, PRECISION, SUPPRESS_SMALL, ' ', "") >>>> File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", >>>>line 176, in array2string >>>> separator, prefix) >>>> File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", >>>>line 138, in _array2string >>>> max_str_len = max(len(str(max_reduce(data))), >>>>libnumarray.error: maximum_Int32_reduce: access beyond buffer. offset=27 >>>>buffersize=16 >>>> >Exit code: 1 >>>>__________________________________________________________ >>>> >>>># USING NumArray >>>> >>>>In setShape self._shape: (2, 2) >>>>In ts a._strides: (8, 4) >>>>In setShape self._shape: (4,) >>>>In ts b._strides: (4,) >>>>b: In setShape self._shape: (4,) >>>>[1 2 3 4] >>>> >Exit code: 0 >>>> >>>> >>>> >>>> >>>>------------------------------------------------------- >>>>This sf.net email is sponsored by:ThinkGeek >>>>Welcome to geek heaven. >>>>http://thinkgeek.com/sf >>>>_______________________________________________ >>>>Numpy-discussion mailing list >>>>Numpy-discussion(a)lists.sourceforge.net >>>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>>> >>>> >>>>

2 2

Matrix square root
by Andrew Nesbit 06 Sep '03

06 Sep '03

I need a function equivalent to Matlab's sqrtm, i.e., a square root for matrices. I constructed the following definition: def sqrtm(M): (U,S,VT) = LinearAlgebra.singular_value_decomposition(M) D = MLab.diag(sqrt(S)) return matrixmultiply(matrixmultiply(U,D),VT) but this technique only works for Hermitian, positive definite matrices. The matrices I operate on don't necessarily satisfy that condition. Does anybody know of a more general technique, preferably with an accompanying Python implementation? Thanks. Andrew.

4 4

Attempting to sub-class NumArray
by Colin J. Williams 05 Sep '03

05 Sep '03

When ravel() is used on an instance of NumArray, the _strides attribute is recalculated correctly. When ravel() is used on an instance of special, a sub-class of NumArray, the _strides attribute keeps its old value. This appears to cause the display problem which is reported. The essentials of the code used are below. To investigate this problem, two print statements were added to the method setshape in generic.py: if newnelements == nelements: self._shape = tuple(shape) print 'In setShape self._shape:', self._shape self._strides = self._stridesFromShape() print 'In setShape self._strides:', self._strides else: raise ValueError("New shape is not consistent with the old shape") It seems that only the first of these print statements is executed with special, but both are executed with NumArray. In the second case, the output is not in the expected sequence. I would appreciate any suggestion as to a workaround. Colin W, Grief when attempting to sub-class NumArray Code used: # Try to sub-class class special(N.NumArray): def __init__(self, data= None, shape= None, eType= None): ## eType= _nt.Any or AnyType not acceptable %% arr= N.array(sequence= data, shape= shape, type= eType) for attr in dir(arr): ## This is a longwinded way of setting up special ## There must be a better way %% ## Perhaps we should use generic._view ## It's not documented TRY IT LATER ## There is no need to transfer methods! if attr[0] == '_' and attr[1] != '_': print attr exec 'self.' + attr + '= ' 'arr.' + attr exec 'print self.' + attr pass a= special(data= [1, 2, 3, 4], shape= (2, 2)) a= N.array([1, 2, 3, 4], shape= (2, 2)) print 'In ts a._strides:', a._strides b= _gen.ravel(a) print 'In ts b._strides:', b._strides # <<< Unchanged with special, OK with NumArray print 'b:', b USING THE SUBCLASS special >C:\Progra~1\Python23\pythonw -u ts.py In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (8, 4) b: In setShape self._shape: (4,) Traceback (most recent call last): File "ts.py", line 37, in ? print 'b:', b File "C:\PROGRA~1\Python23\lib\site-packages\numarray\numarraycore.py", line 618, in __str__ MAX_LINE_WIDTH, PRECISION, SUPPRESS_SMALL, ' ', "") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 176, in array2string separator, prefix) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\arrayprint.py", line 138, in _array2string max_str_len = max(len(str(max_reduce(data))), libnumarray.error: maximum_Int32_reduce: access beyond buffer. offset=27 buffersize=16 >Exit code: 1 __________________________________________________________ # USING NumArray In setShape self._shape: (2, 2) In ts a._strides: (8, 4) In setShape self._shape: (4,) In ts b._strides: (4,) b: In setShape self._shape: (4,) [1 2 3 4] >Exit code: 0

2 1

bug?
by verveer＠embl.de 05 Sep '03

05 Sep '03

possible bug? >>> a = array([[[1, 1], [0, 1]]]) >>> print logical_or.reduce(a, 0) [[1 0] [1 1]] should be: >>> print a[0, ...] [[1 1] [0 1]] Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242

1 0

Re: sum and mean methods behaviour
by Peter Verveer 04 Sep '03

04 Sep '03

Hi all, I was thinking a bit more about the changes to reduce() that Todd proposed, and have some questions: The problem that the output may not be able to hold the result of an operation is not unique to the reduce() method. For instance adding two arrays of type UInt can also give you the wrong answer: >>> array(255, UInt8) + array(255, UInt8) 254 So, if this is a general problem, why should only the reduce method be enhanced to avoid this? If you implement this, should this capability not be supported more broadly than only by reduce(), for instance by universal functons such as 'add'? Would it not be unexpected for users that only reduce() provides such added functionality? However, as Paul Dubois pointed out earlier, the original design philosphy of Numeric/numarray was to let the user deal with such problems himself and keep the package small and fast. This seems actually a sound decision, so would it not be better to avoid complicating numarray with these type of changes and also leave reduce as it is? Personally I don't have a need for the proposed changes to the reduce function. My original complaint that started the whole discussion was that the mean() and sum() array methods did not give the correct result in some cases. I still think they should return a correct double precision value, even if the universal functions may not. That could be achieved by a separate implementation that does not use the universal functions. I would be prepared to provide that implementation either to replace the mean and sum methods, or as a separate add-on. Cheers, Peter > 1. Add a type parameter to sum which defaults to widest type. > > 2. Add a type parameter to reductions (and fix output type handling). > Default is same-type as it is now. No major changes to C-code. > > 3. Add a WidestType(array) function: > > Bool --> Bool > Int8,Int16,Int32,Int64 --> Int64 > UInt8, UInt16,UInt32,UInt64 --> UInt64 (Int64 on win32) > Float32, Float64 --> Float64 > Complex32, Complex64 --> Complex64 > > The only thing this really leaves out, is a higher performance > implementation of sum/mean which Peter referred to a few times. > Peter, if you want to write a specialized module, we'd be happy > to put it in the add-ons package.

2 2

Extracting column vectors from matrix.
by Andrew Nesbit 04 Sep '03

04 Sep '03

Say that I have a rank-2 array (matrix) of shape (m,n). Call it A. When extracting column vectors from A, I often want to keep the result as a rank-2 column vector (for subsequent matrix multiplications, etc), so I usually end up writing something ugly like this: column_vector = reshape(A[:,col],(m,1)) I've got a function to wrap this, but is there a builtin (or cleaner) way to do this sort of operation? TIA. Andrew.

2 2

Re: Numpy-discussion digest, Vol 1 #693 - 3 msgs
by Peter Verveer 04 Sep '03

04 Sep '03

> 1. Add a type parameter to sum which defaults to widest type. > > 2. Add a type parameter to reductions (and fix output type handling). > Default is same-type as it is now. No major changes to C-code. > > 3. Add a WidestType(array) function: > > Bool --> Bool > Int8,Int16,Int32,Int64 --> Int64 > UInt8, UInt16,UInt32,UInt64 --> UInt64 (Int64 on win32) > Float32, Float64 --> Float64 > Complex32, Complex64 --> Complex64 This sounds like a good solution. > The only thing this really leaves out, is a higher performance > implementation of sum/mean which Peter referred to a few times. Is this really the case? It depends on how you plan to implement the output conversion. If you will do this by allocating a temporary converted copy of the complete input before the calculations then this potentially requires a lot of temporary storage. But it is certainly possible to come up with an implementation that avoids this. Have you given this some thought? > Peter, > if you want to write a specialized module, we'd be happy to put it in > the add-ons package. I hope that the reduce methods can be made sufficiently efficient so that this will not be necessary. Cheers, Peter

2 1

Re: [Numpy-discussion] sum and mean methods behaviour
by Todd Miller 03 Sep '03

03 Sep '03

I want to thank everyone who participated in this discussion: Peter, Fernando, Robert, Paul, Perry, and Tim. Tim's post has IMO a completely synthesized solution: 1. Add a type parameter to sum which defaults to widest type. 2. Add a type parameter to reductions (and fix output type handling). Default is same-type as it is now. No major changes to C-code. 3. Add a WidestType(array) function: Bool --> Bool Int8,Int16,Int32,Int64 --> Int64 UInt8, UInt16,UInt32,UInt64 --> UInt64 (Int64 on win32) Float32, Float64 --> Float64 Complex32, Complex64 --> Complex64 The only thing this really leaves out, is a higher performance implementation of sum/mean which Peter referred to a few times. Peter, if you want to write a specialized module, we'd be happy to put it in the add-ons package. Thanks again everybody, Todd -- Todd Miller <jmiller(a)stsci.edu>

1 0

Re: [Numpy-discussion] sum and mean methods behaviour
by Peter Verveer 03 Sep '03

03 Sep '03

I also believe that the current behavior for numarray/Numeric reduce method (not to cast) is the right one. It is fine to leave the user with the responsibility to be careful in the case of the reduce operation. But to correctly calculate a mean or a sum by the array methods that are provided you have to convert the array first to a more precise type, and then do the calculation. That wastes space and is slow, and seems not very elegant considering that these are very common statistical operations. A separate implementation for the mean() and sum() methods that uses double precision in the calculation without first converting the array would be straightforward. Since calculating a mean or a sum of a complete array is such a common case I think this would be useful. That leaves the same problem for the reduce method which in some cases would require first a conversion, but this is much less of a problem (at least for me). Having to convert before the operation can be wasteful though. I do like the idea that was also proposed on the list to supply an optional argument to specify the output type. Then the user has full control of the output type (nice if you want high precision in the result without converting the input), and the code can easily be used to implement the mean() and sum() methods. The default behavior of the reduce method can then remain unchanged, so this would not be an obtrusive change. But, I imagine that this may complicate the implementation. Cheers, Peter On Wednesday 03 September 2003 17:13, Paul Dubois wrote: > So after you get the result in a higher precision, then what? > a. Cast it down blindly? > b. Test every element and throw an exception if casting would lose > precision? > c. Test every element and return the smallest kind that "holds" the answer? > d. Always return the highest precision? > > a. is close to equivalent to the present behavior > b. and c. are expensive. > c. makes the type of the result unpredictable, which has its own problems. > d. uses space > > It was the originally design of Numeric to be fast rather than careful, > user beware. There is now a another considerable portion of the > community that is for very careful, and another that is for keeping it > small. You can't satisfy all those goals at once. > > If you make it slow or big in order to be careful, it will always be > slow or big, while the opposite is not true. If you make it fast, the > user can be careful. > > Todd Miller wrote: > > On Mon, 2003-09-01 at 05:34, Peter Verveer wrote: > >>Hi All, > >> > >>I noticed that the sum() and mean() methods of numarrays use the > >> precision of > >> > >>the given array in their calculations. That leads to resuls like this: > >>>>>array([255, 255], Int8).sum() > >> > >>-2 > >> > >>>>>array([255, 255], Int8).mean() > >> > >>-1.0 > >> > >>Would it not be better to use double precision internally and return the > >>correct result? > >> > >>Cheers, Peter > > > > Hi Peter, > > > > I thought about this a lot yesterday and today talked it over with > > Perry. There are several ways to fix the problem with mean() and > > sum(), and I'm hoping that you and the rest of the community will help > > sort them out. > > > > (1) The first "solution" is to require users to do their own up-casting > > prior to calling mean() or sum(). This gives the end user fine control > > over storage cost but leaves the C-like pitfall/bug you discovered. I > > mention this because this is how the numarray/Numeric reductions are > > designed. Is there a reason why the numarray/Numeric reductions don't > > implicitly up-cast? > > > > (2) The second way is what you proposed: use double precision within > > mean and sum. This has great simplicity but gives no control over > > storage usage, and as implemented, the storage would be much higher than > > one might think, potentially 8x. > > > > (3) Lastly, Perry suggested a more radical approach: rather than > > changing the mean and sum methods themselves, we could alter the > > universal function accumulate and reduce methods to implicitly use > > additional precision. Perry's idea was to make all accumulations and > > reductions up-cast their results to the largest type of the current > > family, either Bool, Int64, Float64, or Complex64. By doing this, we > > can improve the utility of the reductions and accumulations as well as > > fixing the problem with sum and mean.

1 0

sum and mean methods behaviour
by Peter Verveer 03 Sep '03

03 Sep '03

Hi All, I noticed that the sum() and mean() methods of numarrays use the precision of the given array in their calculations. That leads to resuls like this: >>> array([255, 255], Int8).sum() -2 >>> array([255, 255], Int8).mean() -1.0 Would it not be better to use double precision internally and return the correct result? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: Peter.Verveer(a)embl-heidelberg.de

5 5

bug?
by Peter Verveer 02 Sep '03

02 Sep '03

Consider conversion of a slice using the astype() method: >>> a = array([[1,2],[3,4]]) >>> b = a[1, ...] >>> print b [3 4] Conversion using the astype() method goes wrong: >>> print b.astype(Float32) [ 1. 2.] This fixes it: >>> print b.copy().astype(Float32) [ 3. 4.] A bug? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: Peter.Verveer(a)embl-heidelberg.de

3 3

type info
by Peter Verveer 01 Sep '03

01 Sep '03

Hi all, Is there some way in numarray to find out what the maximum and minimum values are that a type can hold? For instance, is there a convenient way to find out what the maximal possible value of the Float32 type is? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: Peter.Verveer(a)embl-heidelberg.de

3 3