Mailman 3 June 2011 - NumPy-Discussion

code review for datetime arange
by Mark Wiebe 12 Jun '11

12 Jun '11

I've replaced the previous two pull requests with a single pull request rolling up all the changes so far. The newest changes include finishing the generic unit and np.arange function support. https://github.com/numpy/numpy/pull/87 Because of the nature of datetime and timedelta, arange has to be slightly different than with all the other types. In particular, for datetime the primary signature is np.arange(datetime, datetime, timedelta). I've implemented a simple extension which allows for another way to specify a date range, as np.arange(datetime, timedelta, timedelta). Here (start, delta) represents the datetime range [start, start+delta). Some examples: >>> np.arange('2011', '2020', dtype='M8[Y]') array(['2011', '2012', '2013', '2014', '2015', '2016', '2017', '2018', '2019'], dtype='datetime64[Y]') >>> np.arange('today', 10, 3, dtype='M8') array(['2011-06-09', '2011-06-12', '2011-06-15', '2011-06-18'], dtype='datetime64[D]') >>> a = np.datetime64('today', 'M') >>> np.arange(a, a+1, dtype='M8[D]') array(['2011-06-01', '2011-06-02', '2011-06-03', '2011-06-04', '2011-06-05', '2011-06-06', '2011-06-07', '2011-06-08', '2011-06-09', '2011-06-10', '2011-06-11', '2011-06-12', '2011-06-13', '2011-06-14', '2011-06-15', '2011-06-16', '2011-06-17', '2011-06-18', '2011-06-19', '2011-06-20', '2011-06-21', '2011-06-22', '2011-06-23', '2011-06-24', '2011-06-25', '2011-06-26', '2011-06-27', '2011-06-28', '2011-06-29', '2011-06-30'], dtype='datetime64[D]') Please build the branch available here, https://github.com/m-paradox/numpy try it out, and then discuss! -Mark

7 16

datetime business day API
by Mark Wiebe 10 Jun '11

10 Jun '11

Here's a possible design for a business day API for numpy datetimes: The 'B' business day unit will be removed. All business day-related calculations will be done using the 'D' day unit. A class *BusinessDayDef* to encapsulate the definition of the business week and holidays. The business day functions will either take one of these objects, or separate weekmask and holidays parameters, to specify the business day definition. This class serves as both a performance optimization and a way to encapsulate the weekmask and holidays together, for example if you want to make a dictionary mapping exchange names to their trading days definition. The weekmask can be specified in a number of ways, and internally becomes a boolean array with 7 elements with True for the days Monday through Sunday which are valid business days. Some different notations are for the 5-day week include [1,1,1,1,1,0,0], "1111100" "MonTueWedThuFri". The holidays are always specified as a one-dimensional array of dtype 'M8[D]', and are internally used in sorted form. A function *is_busday*(datearray, weekmask=, holidays=, busdaydef=) returns a boolean array matching the input datearray, with True for the valid business days. A function *busday_offset*(datearray, offsetarray, roll='raise', weekmask=, holidays=, busdaydef=) which first applies the 'roll' policy to start at a valid business date, then offsets the date by the number of business days specified in offsetarray. The arrays datearray and offsetarray are broadcast together. The 'roll' parameter can be 'forward'/'following', 'backward'/'preceding', 'modifiedfollowing', 'modifiedpreceding', or 'raise' (the default). A function *busday_count*(datearray1, datearray2, weekmask=, holidays=, busdaydef=) which calculates the number of business days between datearray1 and datearray2, not including the day of datearray2. For example, to find the first Monday in Feb 2011, >>>np.busday_offset('2011-02', 0, roll='forward', weekmask='Mon') or to find the number of weekdays in Feb 2011, >>>np.busday_count('2011-02', '2011-03') This set of three functions appears to be powerful enough to express the business-day computations that I've been shown thus far. Cheers, Mark

1 1

Returning the same dtype in Cython as np.argmax
by Keith Goodman 10 Jun '11

10 Jun '11

What is the rule to determine the dtype returned by numpy functions that return indices such as np.argmax? I assumed that np.argmax() returned the same dtype as np.int_. That works on linux32/64 and win32 but on win-amd64 np.int_ is int32 and np.argmax() returns int64. Someone suggested using intp. But I can't figure out how to do that in Cython since I am unable to cimport NPY_INTP. So this doesn't work: from numpy cimport NPY_INTP cdef np.ndarray[np.intp_t, ndim=1] output = PyArray_EMPTY(1, dims, NPY_INTP, 0) Is there another way? All I can think of is checking whether np.intp is np.int32 or int64 when I template the code.

2 2

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication
by Brandt Belson 10 Jun '11

10 Jun '11

Hi, Thanks for getting back to me. I'm doing element wise multiplication, basically innerProduct = numpy.sum(array1*array2) where array1 and array2 are, in general, multidimensional. I need to do many of these operations, and I'd like to split up the tasks between the different cores. I'm not using numpy.dot, if I'm not mistaken I don't think that would do what I need. Thanks again, Brandt Message: 1 > Date: Thu, 09 Jun 2011 13:11:40 -0700 > From: Christopher Barker <Chris.Barker(a)noaa.gov> > Subject: Re: [Numpy-discussion] Using multiprocessing (shared memory) > with numpy array multiplication > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: <4DF128FC.8000807(a)noaa.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Not much time, here, but since you got no replies earlier: > > > > > I'm parallelizing some code I've written using the built in > > multiprocessing > > > module. In my application, I need to multiply many large arrays > > together > > is the matrix multiplication, or element-wise? If matrix, then numpy > should be using LAPACK, which, depending on how its built, could be > using all your cores already. This is heavily dependent on your your > numpy (really the LAPACK it uses0 is built. > > > > and > > > sum the resulting product arrays (inner products). > > are you using numpy.dot() for that? If so, then the above applies to > that as well. > > I know I could look at your code to answer these questions, but I > thought this might help. > > -Chris > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker(a)noaa.gov > > >

2 1

numpy function error
by jonasr 10 Jun '11

10 Jun '11

Hello, i have the following problem, the following code doesnt work def f1(x): return self.lgdata[2*i][0]*float(x)+self.lgdata[2*i][1] def f2(x): return self.lgdata[2*i+1][0]*float(x)+self.lgdata[2*i+1][1] f1=f1(self.ptdata[i]) 2=f2(self.ptdata[i]) t=abs(f1-f2) deltat.append(t) temptot.append(f1) rezipr.append(1/t) xul , xur = self.ptdata[i]-0.001, self.ptdata[i]+0.001 print xul,xur,f1,f2,"test \n" verts =[(xul,f1(xul)),(xul,f2(xul)),(xur,f2(xur)),(xur,f1(xur))] it gives me the following error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/matplotlib/backends/backend_gtk.py", line 265, in key_press_event FigureCanvasBase.key_press_event(self, key, guiEvent=event) File "/usr/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 1523, in key_press_event self.callbacks.process(s, event) File "/usr/lib/python2.7/site-packages/matplotlib/cbook.py", line 265, in process proxy(*args, **kwargs) File "/usr/lib/python2.7/site-packages/matplotlib/cbook.py", line 191, in __call__ return mtd(*args, **kwargs) File "auswertung.py", line 103, in __call__ verts =[(xul,f1(xul)),(xul,f2(xul)),(xur,f2(xur)),(xur,f1(xur))] TypeError: 'numpy.float64' object is not callable have no idea where the problem is, since ptdata is an 1dim array there should be no problem to pass it to a function ? -- View this message in context: http://old.nabble.com/numpy-function-error-tp31817481p31817481.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 2

numpy function error
by jonasr 10 Jun '11

10 Jun '11

Hello, i have the following problem, the following code doesnt work def f1(x): return self.lgdata[2*i][0]*float(x)+self.lgdata[2*i][1] def f2(x): return self.lgdata[2*i+1][0]*float(x)+self.lgdata[2*i+1][1] f1=f1(self.ptdata[i]) f2=f2(self.ptdata[i]) t=abs(f1-f2) deltat.append(t) temptot.append(f1) rezipr.append(1/t) xul , xur = self.ptdata[i]-0.001, self.ptdata[i]+0.001 print xul,xur,f1,f2,"test \n" verts =[(xul,f1(xul)),(xul,f2(xul)),(xur,f2(xur)),(xur,f1(xur))] it gives me the following error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/matplotlib/backends/backend_gtk.py", line 265, in key_press_event FigureCanvasBase.key_press_event(self, key, guiEvent=event) File "/usr/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 1523, in key_press_event self.callbacks.process(s, event) File "/usr/lib/python2.7/site-packages/matplotlib/cbook.py", line 265, in process proxy(*args, **kwargs) File "/usr/lib/python2.7/site-packages/matplotlib/cbook.py", line 191, in __call__ return mtd(*args, **kwargs) File "auswertung.py", line 103, in __call__ verts =[(xul,f1(xul)),(xul,f2(xul)),(xur,f2(xur)),(xur,f1(xur))] TypeError: 'numpy.float64' object is not callable have no idea where the problem is, since ptdata is an 1dim array there should be no problem to pass it to a function ? -- View this message in context: http://old.nabble.com/numpy-function-error-tp31817562p31817562.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

1 0

silly user wish
by Paul Anton Letnes 10 Jun '11

10 Jun '11

Dear numpy developers, users, and fans: I wish for a numpy that will magically detect all my errors. Barring the invention of a numpy version which knows what I need, not what I ask of it, I have the following, simpler wish: if one takes myarray.real or myarray.imag of a float (or possibly integer), give a warning saying "well that didn't make much sense, did it?". (Maybe there is some good reason why this is a bad idea, but the idea has been presented to you, nonetheless.) Best regards Paul.

2 1

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication
by Brandt Belson 09 Jun '11

09 Jun '11

Hello all, Since I haven't heard back, I decided to simplify my problem even more. I'm attaching two files which demonstrate the problem very clearly. My original message is below. The script can be run with "python shared_mem.py". Thanks, Brandt > > Hello, > > I'm parallelizing some code I've written using the built in > multiprocessing > > module. In my application, I need to multiply many large arrays together > > and > > sum the resulting product arrays (inner products). I noticed that when I > > parallelized this with myPool.map(...) with 8 processes (on an 8-core > > machine), the code was actually about an order of magnitude slower. I > > realized that this only happens when I use the numpy array > multiplication. > > I > > defined my own array multiplication and summation. As expected, this is > > much > > slower than the numpy versions, but when I parallelized it I saw the > > roughly > > 8x speedup I expected. So something about numpy array multiplication > > prevented me from speeding it up with multiprocessing. > > > > Is there a way to speed up numpy array multiplication with > multiprocessing? > > I noticed that numpy's SVD uses all of the available cores on its own, > does > > numpy array multiplication do something similar already? > > > > I'm copying the code which can reproduce and summarize these results. > > Simply > > run: "python shared_mem.py" with myutil.py in the same directory. > > > > Thanks, > > Brandt >

2 1

Default unit for datetime/timedelta
by Mark Wiebe 09 Jun '11

09 Jun '11

The NEP and current implementation of the datetime specifies microseconds as the default unit when constructing and converting to datetimes and timedeltas. Having the np.arange function work in a general, intuitive way for a wide variety of datetime/timedelta/integer inputs is turning out to be very complicated, and has made me realized that adding a "generic time unit" which gets assigned to the data type when unspecified would simplify this code and also make it more consistent with the behavior of +/- operations. Here are some current behaviors that are inconsistent with the microsecond default, but consistent with the "generic time unit" idea: >>> np.timedelta64(10, 's') + 10 numpy.timedelta64(20,'s') >>> np.datetime64('2011-03-12') + 3 numpy.datetime64('2011-03-15','D') I'd like to make 'M8' and 'm8' be datetime data types with generic time units instead of microseconds as they are currently. This would also allow the possibility of extending the behavior of detecting the unit from the input string as: >>> np.datetime64('2011-03-12T13') numpy.datetime64('2011-03-12T13-0600','h') to also work with arrays, which currently work like this: >>> np.array(['2011-03-12T13', '2012'], dtype='M8') array(['2011-03-12T13:00:00.000000-0600', '2011-12-31T18:00:00.000000-0600'], dtype='datetime64[us]') This is similar to some cases of the string data type, where the size of the string is detected based on the input. -Mark

3 8

numpy c api general array handling
by Yoshi Rokuko 09 Jun '11

09 Jun '11

hey, i'm writing my first python module in c using numpy/arrayobject.h and i have some problems with different platforms (both linux but really different setup) so i suspect my array handling is not cor- rect. i'm used to c arrays and want to access large numpy arrays from within my c module with no strange array-iter-methods. So what are your remarks to the following: PyArg_ParseTuple(args, "OO|ii", &arg1, &arg2, &start, &stop); index = (PyArrayObject *) PyArray_ContiguousFromObject(arg1, PyArray_INT, 1, 1); ix = (int *)index->data; then using like: for(k = ix[i]; k < ix[i+1]; k++) l = ix[k]; this seems to work pretty well, but real problems come with: PyArg_ParseTuple(args, "O", &arg); adjc = (PyArrayObject *) PyArray_ContiguousFromObject(arg, PyArray_INT, 2, 2); a = (int *)adjc->data; and then using like: aik = a[i + n * k]; it seems like on one system it does not accept 2d numpy arrays just 1d ones or i must hand him a list of 1d numpy arrays like that: >>> A = np.array([[1,0],[0,1]]) >>> B = my.method(list(A)) i would prefer to avoid the list() call in python. what are your remarks on performance would it be faster to do: PyArg_ParseTuple(args, "OO|ii", &arg1, &arg2, &start, &stop); index = (PyArrayObject *) PyArray_ContiguousFromObject(arg1, PyArray_INT, 1, 1); ix = malloc(n * sizeof(int)); for(i = 0; i < n; i++) ix[i] = (int *)index->data[i]; and then use ix[] (i call a lot on ix). thank you and best regards, yoshi

3 4