NumPy-Discussion September 2011

numpy-discussion@python.org

88 participants
79 discussions

iterate over multiple arrays
by David Froger 01 Oct '11

01 Oct '11

Hy everybody, I'm wondering what is the (best) way to apply the same function to multiple arrays. For example, in the following code: from numpy import * def f(arr): return arr*2 a = array( [1,1,1] ) b = array( [2,2,2] ) c = array( [3,3,3] ) d = array( [4,4,4] ) a = f(a) b = f(b) c = f(c) d = f(d) I would like to replace : a = f(a) b = f(b) c = f(c) d = f(d) with something like that, but which really modify a,b,c and d: for x in [a,b,c,d]: x = f(x) So having something like a pointer on the arrays. Thanks for your help ! David --

6 10

Weird upcast behavior with 1.6.x, working as intended?
by Olivier Delalleau 30 Sep '11

30 Sep '11

Hi, This is with numpy 1.6.1 under Linux x86_64, testing the upcast mechanism of "scalar + array": >>> import numpy; print (numpy.array(3, dtype=numpy.complex128) + numpy.ones(3, dtype=numpy.float32)).dtype complex64 Since it has to upcast my array (float32 is not "compatible enough" with complex128), why does it upcast it to complex64 instead of complex128? As far as I can tell 1.4.x and 1.5.x versions of numpy are indeed upcasting to complex128. Thanks, -=- Olivier

3 5

datetime64 y2k38 bug
by Benjamin Root 30 Sep '11

30 Sep '11

I was working on adding some test cases in numpy for the argmin/max functions with some datetime64s. I found that on my 32-bit machine, it fails to parse a date past the Y2.038k date. I find this odd because the datetime is supposed to be 64-bits, but I guess there is some arch-dependent code somewhere? Ben Root

4 5

Comping Numpy in Cython
by Sean Poust 30 Sep '11

30 Sep '11

I keep getting this error: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API" and a bunch of undefined variables when comping cython code with numpy I am new to cython and numpy and I am having trouble getting a *.pyx extension to compile with this in the header: from __future__ import division import numpy as np cimport numpy as np DTYPE = np.int ctypedef np.int_t DTYPE_t I complie to C code using: cython -a montcarl.pyx And it seems to work fine. I then attempt to compile using: gcc -I /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks/ Python.framework/Versions/2.6/include/python2.6 -I /usr/local/lib/ python2.7/site-packages/numpy/core/include -shared -pthread -fPIC - fwrapv -O2 -fno-strict-aliasing -o montcar.so montcarl.c And I get a bunch of undefined variables: spoust-m:old spoust$ gcc $CFLAGS -o montcar.so montcarl.c In file included from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/ndarraytypes.h:1965, from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/ndarrayobject.h:17, from /usr/local/lib/python2.7/site-packages/numpy/ core/include/numpy/arrayobject.h:14, from montcarl.c:225: /usr/local/lib/python2.7/site-packages/numpy/core/include/numpy/ npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API" Undefined symbols: "_PyLong_AsLong", referenced from: ___Pyx_PyInt_AsLong in ccvkIDLQ.o ___Pyx_PyInt_AsLong in ccvkIDLQ.o ... Any ideas on how to get this to compile and how to get rid of these numpy undefined variables? The cython guys directed me here.

2 1

datetimes with date vs time units, local time, and time zones
by Mark Wiebe 30 Sep '11

30 Sep '11

Towards a reasonable behavior with regard to local times, I've made the default repr for datetimes use the C standard library to print them in a local ISO format. Combined with the ISO8601-prescribed behavior of interpreting datetime strings with no timezone specifier to be in local times, this allows the following cases to behave reasonably: >>> np.datetime64('now') numpy.datetime64('2011-06-15T15:16:51-0500','s') >>> np.datetime64('2011-06-15T18:00') numpy.datetime64('2011-06-15T18:00-0500','m') As noted in another thread, there can be some extremely surprising behavior as a consequence: >>> np.array(['now', '2011-06-15'], dtype='M') array(['2011-06-15T15:18:26-0500', '2011-06-14T19:00:00-0500'], dtype='datetime64[s]') Having the 15th of June print out as 7pm on the 14th of June is probably not what one would generally expect, so I've come up with an approach which hopefully deals with this in a good way. One firm principal of the datetime in NumPy is that it is always stored as a POSIX time (referencing UTC), or a TAI time. There are two categories of units that can be used, which I will call *date units* and *time units*. The date units are 'Y', 'M', 'W', and 'D', while the time units are 'h', 'm', 's', ..., 'as'. Time zones are only applied to datetimes stored in time units, so there's a qualitative difference between date and time units with respect to string conversions and calendar operations. I would like to place an 'unsafe' casting barrier between the date units and the time units, so that the above conversion from a date into a datetime will raise an error instead of producing a confusing result. This only applies to datetimes and not timedeltas, because for timedeltas the day <-> hour case is fine, it is just the year/month <-> other units which has issues, and that is already treated with an 'unsafe' casting barrier. Two new functions will facilitate the conversions between datetimes with date units and time units: date_as_datetime(datearray, hour, minute, second, microsecond, timezone='local', unit=None, out=None), which converts the provided dates into datetimes at the specified time, according to the specified timezone. If 'unit' is specified, it controls the output unit, otherwise it is the units in 'out' or the amount of precision specified in the function. datetime_as_date(datetimearray, timezone='local', out=None), which converts the provided datetimes into dates according to the specified timezone. In both functions, timezone can be any of 'UTC', 'TAI', 'local', '+/-####', or a datetime.tzinfo object. The latter will allow NumPy datetimes to work with the pytz library for flexible time zone support. I would also like to extend the 'today' input string parsing to accept strings like 'today 12:30' to allow a convenient way to express different local times occurring today, mostly useful for interactive usage. I welcome any comments on this design, particularly if you can find a case where this doesn't produce a reasonable behavior. Cheers, Mark

3 4

Re: [Numpy-discussion] Functions for finding the relative extrema of numeric data
by Jacob Silterra 30 Sep '11

30 Sep '11

Hello all, I just opened a pull request to add scipy.signal._peak_finding.find_peaks, which finds the relative maxima in a 1d ndarray. The algorithm behind this function has been discussed on the scipy list, but here's a quick recap: The idea is basically convolve the function with a wavelet of varying widths (default is ricker, or "mexican hat" wavelet) and identify as peaks those locations where there are relative maxima (meaning they are larger than their nearest neighbor on each side) at each of several widths, and have a sufficiently high SNR. More details are provided in the docstrings of the functions. Pull request at https://github.com/scipy/scipy/pull/85. I believe the code quality/docstrings/test coverage are pretty good, however it would be of great benefit for someone with experience with similar algorithms to review the code. -Jacob PS I've sent this to both the scipy and numpy lists, apologies to those who get it twice.

1 0

Comparing NumPy/IDL Performance
by Keith Hughitt 30 Sep '11

30 Sep '11

Hi all, Myself and several colleagues have recently started work on a Python library for solar physics <http://www.sunpy.org/>, in order to provide an alternative to the current mainstay for solar physics<http://www.lmsal.com/solarsoft/>, which is written in IDL. One of the first steps we have taken is to create a Python port<https://github.com/sunpy/sunpy/blob/master/benchmarks/time_test3.py>of a popular benchmark for IDL (time_test3) which measures performance for a variety of (primarily matrix) operations. In our initial attempt, however, Python performs significantly poorer than IDL for several of the tests. I have attached a graph which shows the results for one machine: the x-axis is the test # being compared, and the y-axis is the time it took to complete the test, in milliseconds. While it is possible that this is simply due to limitations in Python/Numpy, I suspect that this is due at least in part to our lack in familiarity with NumPy and SciPy. So my question is, does anyone see any places where we are doing things very inefficiently in Python? In order to try and ensure a fair comparison between IDL and Python there are some things (e.g. the style of timing and output) which we have deliberately chosen to do a certain way. In other cases, however, it is likely that we just didn't know a better method. Any feedback or suggestions people have would be greatly appreciated. Unfortunately, due to the proprietary nature of IDL, we cannot share the original version of time_test3, but hopefully the comments in time_test3.py will be clear enough. Thanks! Keith

8 10

Problem with importing csv into datetime64
by Grové 29 Sep '11

29 Sep '11

Hi, I am trying out the latest version of numpy 2.0 dev: np.__version__ Out[44]: '2.0.0.dev-aded70c' I am trying to import CSV data that looks like this: date,system,pumping,rgt,agt,sps,eskom_import,temperature,wind,pressure,weather 2007-01-01 00:30,481.9,,,,,481.9,15,SW,1040,Fine 2007-01-01 01:00,471.9,,,,,471.9,15,SW,1040,Fine 2007-01-01 01:30,455.9,,,,,455.9,,,, etc. by using the following code: convertdict = {0: lambda s: np.datetime64(s, 'm'), 1: lambda s: float(s or 0), 2: lambda s: float(s or 0), 3: lambda s: float(s or 0), 4: lambda s: float(s or 0), 5: lambda s: float(s or 0), 6: lambda s: float(s or 0), 7: lambda s: float(s or 0), 8: str, 9: str, 10: str} dt = [('date', np.datetime64),('system', float), ('pumping', float),('rgt', float), ('agt', float), ('sps', float) ,('eskom_import', float), ('temperature', float), ('wind', str), ('pressure', float), ('weather', str)] a = np.recfromcsv(fp, dtype=dt, converters=convertdict, usecols=range(0-11), names=True) The dtype it generates for a.date is 'object': array([2007-01-01T00:30+0200, 2007-01-01T01:00+0200, 2007-01-01T01:30+0200, ..., 2007-12-31T23:00+0200, 2007-12-31T23:30+0200, 2008-01-01T00:00+0200], dtype=object) But I need it to be datetime64, like in this example (but including hrs and minutes): array(['2011-07-11', '2011-07-12', '2011-07-13', '2011-07-14', '2011-07-15', '2011-07-16', '2011-07-17'], dtype='datetime64[D]') It seems that the CSV import creates an embedded object datetype for 'date' rather than a datetime64 data type. Any ideas on how to fix this? Grové

2 1

Fancy indexing with masks
by Stéfan van der Walt 27 Sep '11

27 Sep '11

Hi all, Matthew Brett showed me an interesting code snippet this evening: # Construct input data In [15]: x Out[15]: array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]]) # Fancy indexing with 1D boolean array In [16]: x[np.array([True, False, True])] Out[16]: array([[0, 1, 2], [6, 7, 8]]) # Fancy indexing with 2D boolean array In [17]: x[np.array([[True, False, True]])] Out[18]: array([0, 2]) I guess it's been a long day, but why does this work at all? I expected the first example to break, because the 1D mask does not match the number of rows in x. In the second example, I expected an error because the 2D mask was not of the same shape as x. But, oddly, both work. There's also no attempt at broadcasting indexes. Any hints would be appreciated! Thanks Stéfan

6 7

array.tofile() refuses to write into a StringIO
by OC 27 Sep '11

27 Sep '11

Hello, as said in the subject, the following code produces an error. Is it normal ? ********************************************************** A = r_[1] file_buffer = StringIO() A.tofile(file_buffer) IOError: first argument must be a string or open file **********************************************************

4 5

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2011