Mailman 3 April 2006 - NumPy-Discussion

Re: Vote: complex64 vs complex128
by Joe Harrington 05 Apr '06

05 Apr '06

When I first heard of Complex128, my first response was, "Cool! I didn't even know there was a Double128!" Folks seem to agree that precision-based naming would be most intuitive to new users, but that length-based naming would be most intuitive to low-level programmers. This is a high-level package, whose purpose is to hide the numerical details and programming drudgery from the user as much as possible, while still offering high performance and not limiting capability too much. For this type of package, a good metric is "when it doesn't restrict capability, do what makes sense for new/naiive users". So, I favor Complex32 and Complex64. When you say "complex", everyone knows you mean 2 numbers. When you say 32 or 64 or 128, in the context of bits for floating values, almost everyone assumes you are talking that many bits of precision to represent one number. Consider future conversations about precision and data size. In precision discussions, you'd always have to clarify that complex128 had 64 bits of precision, just to make sure everyone was on the same key (particularly when 128-bit machines arrive). In data-size discussions, everyone would know to double the size for the two components. No extra clarification would be needed. IDL's behavior is irrelevant to us, since they just say "complex", and "dcomplex" for 32-bit and 64-bit precision. --jh--

8 11

kron with matrices
by Sven Schreiber 04 Apr '06

04 Apr '06

Hi, first of all thanks for including kron in numpy, it's very useful. Now I have just built numpy from svn for the first time in order to spot matrix-related bugs before a new release as promised. That worked well, thanks to the great wiki instructions. The old bugs (in linalg) are gone, but I wonder whether the following behavior is another one: >>> import numpy as n >>> n.kron(n.asmatrix(n.ones((1,2))), n.asmatrix(n.zeros((2,2)))) array([[0, 0, 0, 0], [0, 0, 0, 0]]) I would prefer if kron returned a matrix at least if both inputs are matrices, as in the given example. Thanks, Sven

1 0

Fw: numpy-discussion
by ijcvyash 04 Apr '06

04 Apr '06

----- Original Message ----- From: Armstrong Nicholas To: mgucfjwruye(a)bondavalli.com Sent: Saturday, April 01, 2006 10:21 PM Subject: numpy-discussion

1 0

string matrices
by Ryan Krauss 04 Apr '06

04 Apr '06

I am trying to use NumPy to generate some matrix inputs to Maxima for symbolic analysis. I am using a fair number of matrix.astype('S%d'%maxlen) statements. This seems to work very well. It also doesn't seem to pad the elements in anyway if maxlen is bigger than I need, which is great. This may seem like a dumb computer science question, but what is the memory/performance cost of making maxlen bigger than I want (but making sure that it is way bigger than I need so that the elements don't get truncated)? If my biggest matrices will be 13x13, how long can the strings be before I consume more than a few megs (or a few dozen megs) of memory? Thanks, Ryan

4 4

Re: [Numpy-discussion] first impressions with numpy
by Tim Hochberg 04 Apr '06

04 Apr '06

Sebastian Haase wrote: > Thanks Tim, > that's OK - I got the idea... > BTW, is there a (policy) reason that you sent the first email just to > me and not the mailing list !? No. Just clumsy fingers. Probably the same reason the functions got all garbled! > > I would really be more interested in comments to my first point ;-) > I think it's important that numpy will not be to cryptic and only for > "hackers", but nice to look at ... (hope you get what I mean ;-) Well, I think it's probably a good idea and it sounds like Travis like the idea " for some of the builtin types". I suspect that's code for "not types for which it doesn't make sense, like recarrays". > I'm have developed an "image analysis algorithm development" platform > (based on wxPython + PyShell) that more and more people in our lab are > using (instead of Matlab !) and I changed the default sys.displayhook > to print str(...) instead of repr(...) > mainly to get .3 instead of .2999999999998 > but seing int32 instead of <i4 would also be much "nicer". I agree that str should display something nicer. Repr should probably stay the same though. > Thanks for your great work .. Oh, I'm not doing much in the way of great work. I'm mostly just causing Travis headaches. -tim > Sebastian > > > Tim Hochberg wrote: > >> Tim Hochberg wrote: >> >>> Sebastian Haase wrote: >>> >>>> Hi, >>>> I'm a long time user of numarray. Now I downloaded numpy for the >>>> first time - and am quite excited to maybe soon being able to use >>>> things like weave ! >>>> Thanks for all the good work ! >>>> >>> [SNIP] >>> >>>> >>>> >>>> 2) >>>> This is probably more numarray related: >>>> Why does numarray.asarray( numpy.array([1]) ) return a numpy array, >>>> not a numarray ??? This is even true for numarray.array( >>>> numpy.array([1]) ) !! >>>> >>> I expect that numarray is grabbing the output of __array__() and >>> using that for asarray. What's happening in the second case is hard >>> to know, but I bet it's some snafu with assuming that anything that >>> isn't a numarray, but implements __array__ must be returning a new >>> numarray. That's just a guess. FWIW, I'm using the following two >>> functions to go back and forth between numarray and numpy. So far >>> they seem to work and they're faster (or actually work in the case >>> of asnumarray). You may want to rename them to suit your personal >>> preferences. If you use them, please let me know if you find any >>> problems. >>> >>> Regards, >>> >> I see this got thoroughly mangled somehow. If you want a copy, let me >> know and I'll just send you it as a file. >> >> -tim >> > > >

9 14

[Fwd: Re: [Numpy-discussion] first impressions with numpy]
by Sebastian Haase 02 Apr '06

02 Apr '06

supposedly meant for the whole list ... From: Tim Hochberg <tim.hochberg(a)cox.net> Sebastian Haase wrote: > Tim Hochberg wrote: > <snip> > >> This would work fine if repr were instead: >> >> dtype([('x', float64), ('z', complex128)]) >> >> Anyway, this all seems reasonable to me at first glance. That said, I >> don't plan to work on this, I've got other fish to fry at the moment. > > > A new point: Please remind me (and probably others): when did it get > decided to introduce 'complex128' to mean numarray's complex64 > and the 'complex64' to mean numarray's complex32 ? I haven't the faintest idea -- it happened when I was off in Numarray land I assume. Or it was always that way? No idea. Hopefully Travis will answer this. -tim > > I do understand the logic that 128 is really the bit-size of one > (complex) element - but I also liked the old way, because: > 1. e.g. in fft transforms, float32 would "go with" complex32 > and float64 with complex64 > 2. complex128 is one character extra (longer) and also > (alphabetically) now sorts before(!) complex64 > 3 Mostly of course: this new naming will confuse all my code and > introduce hard to find bugs - when I see complex64 I will "think" the > old way for quite some time ... > > > These might just be my personal (idiotic ;-) comments - but I would > appreciate some feedback/comments. > Also: Is it now to late to (re-)start a discussion on this !? > > Thanks > - Sebastian Haase > >

2 1

extension to xrange for numpy
by Arnd Baecker 02 Apr '06

02 Apr '06

Dear numpy enthusiasts, one python command which is extremely useful in 1D situations is `xrange`. However, for higher dimensional settings we strongly lack the commands `yrange` and `zrange`. These could be shorthands for the corresponding constructs with `:,NewAxis` added. Any comments, suggestion and even implementations are very welcome, Arnd P.S.: What I am not sure about is the right command for the 4-dimensional case - which letter should be used after the "z"? (it seems that "a" would be a very natural choice...)

6 9

observations
by Tim Hochberg 02 Apr '06

02 Apr '06

I've been doing a *lot* of playing with numpy over the last several days, so expect various observations to trickle from my abode over the next week or so. Here's the first installment. * tostring probably needs the order flag. I think you want the string generated from a multidimensional array in Fortran and C order to differ. * With the evolution of the order flag, ascontiguousarray is probably redundant, scarcely after it was added. b = asarray(a, order="C") Is actually clearer in intent than: b = ascontiguousarray(a) Does the latter leave a contiguous, Fortran order array alone? That's probably almost never what one wants. Unless your working with Fortran arrays, in which case the opposite ambiguity applies. Regards, -tim

1 0

numpy error handling
by Tim Hochberg 02 Apr '06

02 Apr '06

I've just been looking at how numpy handles changing the behaviour that is triggered when there are numeric error conditions (overflow, underflow, etc.). If I understand it correctly, and that's a big if, I don't think I like it nearly as much as the what numarray has in place. It appears that numpy uses the two functions, seterr and geterr, to set and query the error handling. These set/read a secret variable stored in the local scope. I assume that the various ufuncs then examine that value to determine how to handle errors. The secret variable approach is a little clunky, but that's not what concerns me. What concerns me is that this approach is *only* useful for built in numpy functions and falls down if we call any user defined functions. Suppose we want to be warned on underflow. Setting this is as simple as: def func(*args): numpy.seterr(under='warn') # do stuff with args return result Since seterr is local to the function, we don't have to reset the error handling at the end, which is convenient. And, this works fine if all we are doing is calling numpy functions and methods. However, if we are calling a function of our own devising we're out of luck since the called function will not inherit the error settings that we have set. Thus we have no way to influence the error settings of functions downstream from us. Compare this to the numarray approach, where you write code like the following: def func(*args) numarray.Error.pushMode(underflow='warn') try: # do stuff with args return result finally: numarray.Error.popMode() At first glance this looks a lot more clunky. However, it's also a lot more useful. In this case, pushMode pushes a new error mode value onto a global[1] stack of mode values. Thus, any functions that func calls, will inherit it's error settings, unless they explicitly override them. As for the clunkiness factor, that's partly just the choice of camelCase versus smashwords, but also the try-finally. The second, however, will be addressed in Python 2.5 with the 'with' statement. At that point, we could have a numpy equivalent to the above numarray code that looked something like: def func(*args) with numpy.errors(underflow='warn'): # do stuff with args return result I believe we should rework the error handling to follow the numarray model before things get set in stone at 1.0. The 'with' statement support can be worked in a bit later if need be, although I suspect that will be easy. I also would prefer more verbose keys ala numarray (underflow, overflow, dicidebyzero and invalid) than those currently used by numpy (under, over, divide and invalid). And (will he never stop) I like numarrays defaults better here too: overflow='warn', underflow='ignore', dividebyzero='warn', invalid='warn'. Currently, numpy defaults to ignore for all cases. These last points are relatively minor though. Regards, -tim [1] I believe it's actually thread local, and if it's not it should be, but that detail is not important for our discussion here.

4 7

ANN: PyTables 1.3 released
by Francesc Altet 01 Apr '06

01 Apr '06

========================= Announcing PyTables 1.3 ========================= This is a new major release of PyTables. The most remarkable feature added in this version is a complete support (well, almost, because unicode arrays are not there yet) for NumPy objects. Improved support for native HDF5 is there as well. As an aside, I'm happy to inform you that the PyTables web site (http://www.pytables.org) has been converted into a wiki so that users can contribute to the project with recipes or any other document. Try it out! Go to the (new) PyTables web site for downloading the beast: http://www.pytables.org/ or keep reading for more info about the new features and bugs fixed. Changes more in depth ===================== Improvements: - Support for NumPy objects in all the objects of PyTables, namely: Array, CArray, EArray, VLArray and Table. All the numerical and character (except unicode arrays) flavors are supported as well as plain and nested heterogeneous NumPy arrays. PyTables leverages the adoption of the array interface (http://numeric.scipy.org/array_interface.html) for a very efficient conversion between all the numarray (which continues to be the native flavor for PyTables) object to/from NumPy/Numeric. - The FLAVOR schema in PyTables has been refined and simplified. Now, the only 'flavors' allowed for data objects are: "numarray", "numpy", "numeric" and "python". The changes has been made so that they are fully backward compatible with existing PyTables files. However, when users would try to use old flavors (like "Numeric" or "Tuple") in existing code, a ``DeprecationWarning`` will be issued in order to encourage them to migrate to the new flavors as soon as possible. - Nested fields can be specified in the "field" parameter of Table.read by using a '/' as a separator between fields (e.g. 'Info/value'). - The Table.Cols accessor has received a new ``__setitem__()`` method that allows doing things like: table.cols[4] = record table.cols.x[4:1000:2] = array # homogeneous column table.cols.Info[4:1000:2] = recarray # nested column - A clean-up function (using ``atexit``) has been registered so that remaining opened files are closed when a user hits a ^C, for example. That would help to avoid ending with corrupted files. - Native HDF5 compound datasets that are contiguous are supported now. Before, only chunked datasets were supported. - Updated (and much improved) sections about compression issues in the User's Guide. It includes new benchmarks made with PyTables 1.3 and a exhaustive comparison between Zlib, LZO and bzip2. - The HTML version of manual is made now from the docbook2html package for an improved look (IMO). Bug fixes: - Solved a problem when trying to save CharArrays with itemsize = 0 as attributes of nodes. Now, these objects are pickled in order to prevent HDF5 from crashing. - Fixed some alignment issues with nested record arrays under certain architectures (e.g. PowerPC). - Fixed automatic conversions when a VLArray is read in a platform with a byte ordering different from the file. Deprecated features: - Due to recurrent problems with the UCL compression library, it has been declared deprecated from this version on. You can still compile PyTables with UCL support (using the --force-ucl), but you are urged to not use it anymore and convert any existing datafiles with UCL to other supported library (zlib, lzo or bzip2) with the ``ptrepack`` utility. Backward-incompatible changes: - Please, see ``RELEASE-NOTES.txt`` file. Important note for Windows users ================================ If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and numarray (but NumPy and Numeric are also supported) package for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC (and PowerPC64) and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"

1 0