Mailman 3 October 2017 - NumPy-Discussion

Vectorization of variant of piecewise or interpolation function
by Seth Ghandi 16 Oct '17

16 Oct '17

Hi everybody, I am new to newpy and am trying to define a variant of piecewise or zero holder interpolation function, say ZeroOrderInterpolation(t,a), where t is an 1D array of size, say p, consisting of real numbers, and a is a 2D array of size, say nxm, with first column consisting of increasing real numbers. This function should return an array, say y, of size px(m-1) such that y[i,:] is equal to a[n,1:] if a[n,0] <= t[i], and a[k,1:] if k < n and a[k,0] <= t[i] < a[k+1,0]. Note that t[0] is assumed to be at least equal to a[0,0]. I have the following script made of "for loops" and I am trying to vectorize it so as to make it faster for large arrays. def ZeroOrderInterpolation(t,a): import numpy as np p = t.shape[0] n, m = a.shape if n == 1: return a[0,1:] y = np.zeros((p,m-1)) for i in range(p): if a[n-1,0] <= t[i]: y[i] = a[n-1,1:] else: for j in range(n-1): if (a[j,0] <= t[i]) and (t[i] <= a[j+1,0]): y[i] = a[j,1:] return y import numpy as np t = np.array([0.5,1,1.5,2.5,3,10]) table = np.array([[0,3],[1,0],[2,5],[3,-1]]) ZeroOrderInterpolation(t,table) [Out]: array([[ 3.], [ 0.], [ 0.], [ 5.], [-1.], [-1.]]) Any help for a vectorization "à la numpy" of this fucntion will be apprecaited. Best regards,

2 2

List comprehension and loops performances with NumPy arrays
by Andrea Gavana 11 Oct '17

11 Oct '17

Hi All, I have this little snippet of code: import timeit import numpy class Item(object): def __init__(self, name): self.name = name self.values = numpy.random.rand(8, 1) def do_something(self): sv = self.values.sum(axis=0) array = numpy.empty((8, )) f = numpy.dot(0.5*numpy.ones((8, )), self.values)[0] array.fill(f) return array In my real application, the method do_something does a bit more than that, but I believe the snippet is enough to start playing with it. What I have is a list of (on average) 500-1,000 classes Item, and I am trying to retrieve the output of do_something for each of them in a single, big 2D numpy array. My current approach is to use list comprehension like this: output = numpy.asarray([item.do_something() for item in items]).T (Note: I need the transposed of that 2D array, always). But then I though: why not preallocating the output array and make a simple loop: output = numpy.empty((500, 8)) for i, item in enumerate(items): output[i, :] = item.do_something() I was expecting this version to be marginally faster - as the previous one has to call asarray and then transpose the matrix, but I was in for a surprise: if __name__ == '__main__': repeat = 1000 items = [Item('item_%d'%(i+1)) for i in xrange(500)] statements = [''' output = numpy.asarray([item.do_something() for item in items]).T ''', ''' output = numpy.empty((500, 8)) for i, item in enumerate(items): output[i, :] = item.do_something() '''] methods = ['List Comprehension', 'Empty plus Loop '] setup = 'from __main__ import numpy, items' for stmnt, method in zip(statements, methods): elapsed = timeit.repeat(stmnt, setup=setup, number=1, repeat=repeat) minv, maxv, meanv = min(elapsed), max(elapsed), numpy.mean(elapsed) elapsed.sort() best_of_3 = numpy.mean(elapsed[0:3]) result = numpy.asarray((minv, maxv, meanv, best_of_3))*repeat print method, ': MIN: %0.2f ms , MAX: %0.2f ms , MEAN: %0.2f ms , BEST OF 3: %0.2f ms'%tuple(result.tolist()) I get this: List Comprehension : MIN: 7.32 ms , MAX: 9.13 ms , MEAN: 7.85 ms , BEST OF 3: 7.33 ms Empty plus Loop : MIN: 7.99 ms , MAX: 9.57 ms , MEAN: 8.31 ms , BEST OF 3: 8.01 ms Now, I know that list comprehensions are renowned for being insanely fast, but I though that doing asarray plus transpose would by far defeat their advantage, especially since the list comprehension is used to call a method, not to do some simple arithmetic inside it... I guess I am missing something obvious here... oh, and if anyone has suggestions about how to improve my crappy code (performance wise), please feel free to add your thoughts. Thank you. Andrea.

3 4

Re: [Numpy-discussion] converting list of int16 values to bitmask and back to bitmask and back to list of int32\float values
by Nissim Derdiger 08 Oct '17

08 Oct '17

Hi again, I realize that my question was not clear enough, so I've refined it into one runnable function (attached below) My question is basically - is there a way to perform the same operation, but faster using NumPy (or even just by using Python better..) Thanks again and sorry for the unclearness.. Nissim. import struct def Convert(): Endian = '<I' # Big endian ParameterFormat = 'f' # float32 RawDataList = [17252, 26334, 16141, 58057,17252, 15478, 16144, 43257] # list of int32 registers NumOfParametersInRawData = int(len(RawDataList)/2) Result = [] for i in range(NumOfParametersInRawData): # pack every 2 registers, take only the first 2 bytes from each one, change their endianess than unpack them back to the Parameter format Result.append((struct.unpack(ParameterFormat,(struct.pack(Endian,RawDataList[(i*2)+1])[0:2] + struct.pack('<I',RawDataList[i*2])[0:2])))[0])

2 2

[ANN] Matplotlib 2.1 released
by Thomas Caswell 07 Oct '17

07 Oct '17

We are happy to announce the release of Matplotlib 2.1. This is the second minor release in the Matplotlib 2.x series and the first release with major new features since 1.5. This release contains approximately 2 years worth of work by 275 contributors across over 950 pull requests. Highlights from this release include: - support for string categorical values - export of animations to interactive javascript widgets - major overhaul of polar plots - reproducible output for ps/eps, pdf, and svg backends - performance improvements in drawing lines and images - GUIs show a busy cursor while rendering the plot along with many other enhancements and bug fixes. The gallery, examples, and tutorials have been overhauled and consolidated: Examples: http://matplotlib.org/gallery/index.html Tutorials: http://matplotlib.org/tutorials/index.html A big thank you to everyone who contributed to this release! Wheels are available on pypi for win/mac/manylinux and for conda via conda-forge. Full whats new: http://matplotlib.org/users/whats_new.html#new-in-matplotlib-2-1 Full API changes: http://matplotlib.org/api/api_changes.html#api-changes-in-2-1-0 github stats: http://matplotlib.org/users/github_stats.html Tom

1 0

Deprecate zipf distribution?
by Charles R Harris 07 Oct '17

07 Oct '17

Hi All, The current NumPy implementation of the truncated zipf distribution has several drawbacks. - Extremely poor performance when the parameter `a` is near 1. For instance, when `a = 1.000001` a simple change in the implementation speeds things up by a factor of 1,657. When the parameter is closer to 1, the algorithm effectively hangs. - Because the distribution is truncated, say to integers in the range of int64, the parameter could be allowed to take all values > 0, even though the untruncated series diverges. There is some indication that such values of `a` can be useful in modeling because of the heavy distribution in the tail. Because fixing these problems will change the output stream, I suggest implementing a truncated zeta distribution, which is an alternative name for the same distribution, and deprecating the the zipf distribution. Furthermore, rather than truncate at the value of C long, which varies, truncate at max(int64), or some possibly smaller value, say 2**44, which allows all integers up to that value to be realized with approximately correct probabilities when using double precision for the intermediate computations. Thoughts? Chubk

2 1

Sustainability
by Charles R Harris 05 Oct '17

05 Oct '17

Hi All, I and a number of others representing various open source projects under the NumFocus umbrella will be attending as meeting next Tuesday do discuss the problem of sustainability. In preparation for that meeting I would be interested in any ideas that the folks who follow this list may have on the subject. Chuck

7 7