Mailman 3 October 2009 - NumPy-Discussion

rc1 of the chebyshev module.
by Charles R Harris 04 Oct '09

04 Oct '09

Attached is the first rc of the chebyshev module. The module documentation is not yet complete and no doubt the rest of the documentation needs to be reviewed. The tests cover basic functionality at this point but need to be extended to cover the Chebyshev object. Nevertheless, the module should be usable. Note that the most convenient way to do the least squared fits is with the static method Chebyshev.fit, which will return a Chebyshev object that contains both the resulting Chebyshev series and its domain. Some naming questions remain. ISTM that "lstsq" or "leastsq" might be a better name than fit. Likewise, I have kept the poly1d names "deriv" and "integ", but "der" and "int" might be more appropriate. Operators behave as expected for +, -, and * but there is no truedivision unless both operands can be interpreted as scalars. When division hasn't been imported from __future__, the / and // operators are both floordivision and % returns the remainder. Divmod behaves as expected. Any feedback is welcome. Chuck

1 1

A numpy accumulator...
by Christopher Barker 03 Oct '09

03 Oct '09

Hasi all, This idea was inspired by a discussion at SciPY, in which we spent a LOT of time during the numpy tutorial talking about how to accumulate values in an array when you don't know how big the array needs to be when you start. The "standard practice" is to accumulate in a python list, then convert the final result into an array. This is a good idea because Python lists are standard, well tested, efficient, etc. However, as was pointed out in that lengthy discussion, if what you are doing is accumulating is a whole bunch of numbers (ints, floats, whatever), or particularly if you need to accumulate a data type that plain python doesn't support, there is a lot of overhead involved: a python float type is pretty heavyweight. If performance or memory use is important, it might create issues. You can use and array.array, but it doesn't support all numpy types, particularly custom dtypes. I talked about this on the cython list (as someone asked how to do accumulate in cython), and a few folks thought it would be useful, so I put together a prototype. What I have in mind is very simple. It would be: - Only 1-d - Support append() and extend() methods - support indexing and slicing - Support any valid numpy dtype - which could even get you pseudo n-d arrays... - maybe it would act like an array in other ways, I'm not so sure. - ufuncs, etc. It could take the place of using python lists/arrays when you really want a numpy array, but don't know how big it will be until you've filled it. The implementation I have now uses a regular numpy array as the "buffer". The buffer is re-sized as needed with ndarray.resize(). I've enclosed the class, a bunch of tests (This is the first time I've ever really done test-driven development, though I wouldn't say that this is a complete test suite). A few notes about this implementation: * the name of the class could be better, and so could some of the method names. * on further thought, I think it could handle n-d arrays, as long as you only accumulated along the first index. * It could use a bunch more methods - deleting part of eh array - math - probably anything supported by array.array would be good. * Robert pointed me to the array.array implimentation to see how it expands the buffer as you append. It did tricks to get it to grow fast when the array is very small, then eventually to add about 1/16 of the used array size to the buffer. I imagine that this would gets used because you were likely to have a big array, so I didn't bother and start with a buffer at 128 elements, then add 1/4 each time you need to expand -- these are both tweakable attributes. * I did a little simple profiling, and discovered that it's slower than a python list by a factor of more than 2 (for accumulating python ints, anyway). With a bit of experimentation, I think that's because of a couple factors: - an extra function call -- the append() method needs to then do an assignemt to the buffer - Object conversion -- python lists store python objects, so the python int can jsut go right in there. with numpy, it needs to be converted to a C int first -- a bit if extra overhead. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

3 3

ufunc and errors
by Dag Sverre Seljebotn 03 Oct '09

03 Oct '09

I looked and looked in the docs, but couldn't find an answer to this: When writing a ufunc, is it possible somehow to raise a Python exception (by acquiring the GIL first to raise it, set a flag and a callback which will be called with the GIL, or otherwise?). Or should one always use NaN even if the input does not make any sense (like, herhm, passing anything but integers or half-integers to a Wigner 3j symbol). I know how I'd to it manually in a wrapper w/ passed in context if not, but wanted to see. Also, will the arguments always be named x1, x2, x3, ..., or can I somehow give them custom names? Dag Sverre

3 3

A numpy accumulator...
by Christopher Barker 03 Oct '09

03 Oct '09

(I clicked send too early the last time -- sorry about that!) Hi all, This idea was inspired by a discussion at the SciPy conference, in which we spent a LOT of time during the numpy tutorial talking about how to accumulate values in an array when you don't know how big the array needs to be when you start. The "standard practice" is to accumulate in a python list, then convert the final result into an array. This is a good idea because Python lists are standard, well tested, efficient, etc. However, as was pointed out in that lengthy discussion, if what you are doing is accumulating is a whole bunch of numbers (ints, floats, whatever), or particularly if you need to accumulate a data type that plain python doesn't support, there is a lot of overhead involved: a python float type is pretty heavyweight. If performance or memory use is important, it might create issues. You can use and array.array, but it doesn't support all numpy types, particularly custom dtypes. I talked about this on the cython list (as someone asked how to do accumulate in cython), and a few folks thought it would be useful, so I put together a prototype. What I have in mind is very simple. It would be: - Only 1-d - Support append() and extend() methods - support indexing and slicing - Support any valid numpy dtype - which could even get you pseudo n-d arrays... - maybe it would act like an array in other ways, I'm not so sure. - ufuncs, etc. It could take the place of using python lists/arrays when you really want a numpy array, but don't know how big it will be until you've filled it. The implementation I have now uses a regular numpy array as the "buffer". The buffer is re-sized as needed with ndarray.resize(). I've enclosed the class, a bunch of tests (This is the first time I've ever really done test-driven development, though I wouldn't say that this is a complete test suite). A few notes about this implementation: * the name of the class could be better, and so could some of the method names. * on further thought, I think it could handle n-d arrays, as long as you only accumulated along the first index. * It could use a bunch more methods - deleting part of the array - math - probably anything supported by array.array would be good. * Robert pointed me to the array.array implimentation to see how it expands the buffer as you append. It did tricks to get it to grow fast when the array is very small, then eventually to add about 1/16 of the used array size to the buffer. I imagine that this would gets used because you were likely to have a big array, so I didn't bother and start with a buffer at 128 elements, then add 1/4 each time you need to expand -- these are both tweakable attributes. * I'm keeping the buffer a hidden variable, and slicing and __array__ return copies - this is so that it won't get multiple references, and then not be expandable. * I did a little simple profiling, and discovered that it's slower than a python list by a factor of more than 2 (for accumulating python ints, anyway). With a bit of experimentation, I think that's because of a couple factors: - an extra function call -- the append() method needs to then do an assignment to the buffer - Object conversion -- python lists store python objects, so the python int can just go right in there. with numpy, it needs to be converted to a C int first -- a bit if extra overhead. Though a straight assignment into a pre-allocated array i faster than a list. I think it's still an improvement for memory use. Maybe it would be worth writing in C or Cython to avoid some of this. In particular, it would be nice if you could use it in Cython, and put C types directly it... * This could be pretty useful for things like genfromtxt. What do folks think? is this useful? What would you change, etc? -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Database with Nulls to Numpy Structure
by Mark Janikas 03 Oct '09

03 Oct '09

Hello All, I was hoping you could help me out with a simple little problem I am having: I am reading data from a database that contains NULL values. There is more than one field being read in with equal length, but if any of them are NULL in a row, then I do NOT want to include it in my numpy structure (I.e. no records for that row across fields). As the values from each field are of the same type, I can pre-allocate the space for the entire dataset (if all were not NULL), but there may be less observations after accounting for the NULLS. So, do I use lists and append then create the arrays... Or do I fill up the pre-allocated "empty" arrays and slice off the ends? Thoughts? Thanks much... MJ Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas(a)esri.com<mailto:mjanikas@esri.com>

2 3

merging docs from wiki
by Ralf Gommers 02 Oct '09

02 Oct '09

Hi, I'm done reviewing all the improved docstrings for NumPy, they can be merged now from the doc editor Patch page. Maybe I'll get around to doing the SciPy ones as well this week, but I can't promise that. There are a few docstrings on the Patch page I did not mark "Ok to apply": 1. the generic docstrings. Some are marked Ready for review, but they refer mostly to "self" and to "generic" which I don't think is very helpful. It would be great if someone could do just one of those docstrings and make it somewhat informative. There are about 50 that can then be done in the same way. 2. get_numpy_include: the docstring is deleted because the function is deprecated. I don't think that is helpful but I'm not sure. Should this be reverted or applied? Cheers, Ralf

4 9

Question about improving genfromtxt errors
by Skipper Seabold 02 Oct '09

02 Oct '09

There have been some recent attempts to improve the error reporting in genfromtxt <http://projects.scipy.org/numpy/ticket/1212>, which is great, because hunting down the problems reading in big and messy files is not fun. I am working on a patch that keeps up with the line number and column number of where you are in parsing the file, so that this can be reported in the error. Is there a way to catch a raised error and add to it? For instance, I have a problem in my file which leads to this error being raised from np.lib._iotools.StringCoverter.upgrade ValueError: Converter is locked and cannot be upgraded I added this into np.lib.io.genfromtxt around line 995. linenum = 0 [...] if dtype is None: try: colnum = 0 for (converter, item) in zip(converters, values): converter.upgrade(item) colnum += 1 except: raise ValueError, "I don't report the error from _iotools.StringConverter.upgrade, but I do know that there is a problem trying to convert a value at line %s and column %s" % (linenum,colnum) [...] linenum += 1 I'd like to add line and column number information to original error from _iotools. Any suggestions? Cheers, Skipper

5 36

Windows 64-bit
by Klaus Noekel 02 Oct '09

02 Oct '09

Hi all, at the end of July David answered my question about future 64-bit Windows support as follows: "There were some discussion about pushing 1.4.0 'early', but instead, I think we let it slipped - one consequence is that there will be enough time for 1.4.0 to be released with proper AMD64 support on windows. The real issue is not numpy per-se, but making scipy work on top of numpy in 64 bits mode. It is hard to give an exact date as to when those issues will be fixed, but it is being worked on." As our project needs 64-bit numpy under Windows quite soon, I am curious about the state of the project: - Is a stable 64-bit Windows installer (with or without numpy 1.4.0) going to be released anytime SOON? - We need only numpy, not scipy. Does that imply that we have a good chance of producing an install ourselves with the current sources? I am a bit concerned, because earlier posts indicated that the issue is not trivial. Or have all the hard aspects to do with scipy? Thanks for an update! Cheers, Klaus Nökel

2 1

More guestions on Chebyshev class.
by Charles R Harris 01 Oct '09

01 Oct '09

The Chebyshev class is now working pretty well, but I would like to settle some things up front. 1) Order in which coefficients are stored/passed/accessed. The current poly1d class ctor is called with the coefficients in high to low order, yet the __getitem__ and __setitem__ methods access them in reverse order. This seems confusing and I think both should go in the same order and my preference would be from low to high. The low to high order also works a bit better for implementation. 2) poly1d allows the size of the coefficient array to be dynamically extended. I have mixed feeling about that and would prefer not, but there are arguments for that: students might find it easier to fool with. 3) The poly1d class prunes leading (high power) zeros. Because the Cheb class has a fit static method that returns a Cheb object, and because when fitting with Chebyshev polynomials the user often wants to see *all* of the coefficients, even if some of the leading ones are zero, the Cheb class does not automatically prune the zeros, instead there are methods for that. 4) All the attributes of the Cheb class are read/write. The poly1d class attempts to hide some, but the method used breaks the copy module. Python really doesn't have private attributes, so I left all the attributes exposed with the usual Python proviso: if you don't know what it does, don't fool with it. 5) Is Cheb the proper name for the class? Thoughts? Chuck

2 1

difficulty with numpy.where
by Dr. Phillip M. Feldman 01 Oct '09

01 Oct '09

I've defined the following one-line function that uses numpy.where: def sin_half_period(x): return where(0.0 <= x <= pi, sin(x), 0.0) When I try to use this function, I get an error message: In [4]: z=linspace(0,2*pi,9) In [5]: sin_half_period(z) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) The truth value of an array with more than one element is ambiguous. Use a.any () or a.all() Any suggestions will be appreciated. -- View this message in context: http://www.nabble.com/difficulty-with-numpy.where-tp25702676p25702676.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 3