Mailman 3 October 2011 - NumPy-Discussion

simulate AR
by Alan G Isaac 14 Oct '11

14 Oct '11

As a simple example, if I have y0 and a white noise series e, what is the best way to produces a series y such that y[t] = 0.9*y[t-1] + e[t] for t=1,2,...? 1. How can I best simulate an autoregressive process using NumPy? 2. With SciPy, it looks like I could do this as e[0] = y0 signal.lfilter((1,),(1,-0.9),e) Am I overlooking similar (or substitute) functionality in NumPy? Thanks, Alan Isaac

4 13

statistics in python
by Radim 14 Oct '11

14 Oct '11

Hi Rense (cross-posting to the numpy mailing list because these guys are awesome), On Oct 13, 10:01 pm, Rense Lange <rense.la...(a)gmail.com> wrote: > I have potentially millions of tuples <v1,v2,v3 ..., observation> and > I want to create frequency distributions conditional on the values of > discrete variables v1, v2, ... (e.g. the sums for boys vs. girls), or > combinations thereof (poor boys, poor girls, rich boys, rich girls). > Very few of the v1 x v2 x ... combinations might actually occur. Also, > it is sometimes necessary to combine different data sets. > > Should I just use some DB system (and if so, which one is best within > Python), or are there sparse matrix methods that are to be preferred? yes, that sounds like a job for a database. Sqlite is built-in (=part of standard Python library: `import sqlite3`). Numpy supports structured arrays (records) as well: >>> import numpy >>> dt = numpy.dtype([('name', 'a10'), ('wealth', numpy.int32), ('sex', 'a1')]) >>> x = numpy.array([('Mary', 10000, 'F'), ('John', 1000, 'M'), ('unknown', -1, '?')], dtype=dt) >>> print x[(x['wealth'] > 5000) & (x['sex'] == 'F')] # print records for all rich girls [('Mary', 10000, 'F')] so perhaps that could also fit your bill. There are no indexes but it's more pleasant to work with, imo. Note that "potentially millions of records" is not particularly big, so as long as you don't have too many variables, some in-memory db should be ok and will save you from headaches of dealing with complex db setups. I have also heard very good things about pytables, though i've never used it myself (gensim uses plain float matrices), you can have a look there: http://pytables.org HTH, Radim > > On Oct 12, 10:01 pm, Rense Lange <rense.la...(a)gmail.com> wrote: > > > > Gensim must be storing data very efficiently, and I need to do something > > > similar for another application. Can you tell me what Python programming > > > approach was used in Gensim, and are there perhaps particular sections in > > > the Gensim code that I shoGuld be looking at for inspiration and examples? > > > that question is too broad. What kind of data and application do you > > have? What sort of efficiency are you after? (fast access/little disk > > space/fast load time/...?) > > > Radim

1 0

the difference of np.nan, np.NaN, np.NAN?
by Chao YUE 14 Oct '11

14 Oct '11

Dear all, is there any difference between np.nan, np.NaN and np.NAN? they really confuse me.... they are all Not a Number? In [75]: np.nan==np.NaN Out[75]: False In [77]: np.NaN==np.NAN Out[77]: False Thanks a lot, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 4

the axis parameter in the np.ma.concatenate is not working?
by Chao YUE 14 Oct '11

14 Oct '11

Dear all, I use numpy version 1.5.1 which is installed by default when I do sudo apt-get install numpy on ubuntu 11.04. but it seems that for np.ma.concatenate(arrays, axis), the axis parameter is not working? In [460]: a=np.arange(10) In [461]: a=np.ma.masked_array(a,a<3) In [462]: a Out[462]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 999999) In [463]: b=np.arange(10) In [464]: b=np.ma.masked_array(a,b>7) In [465]: b Out[465]: masked_array(data = [-- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False True True], fill_value = 999999) In [466]: c=np.ma.concatenate((a,b),axis=0) In [467]: c Out[467]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9 -- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False False False True True True False False False False False True True], fill_value = 999999) In [468]: c.shape Out[468]: (20,) In [469]: c=np.ma.concatenate((a,b),axis=1) In [470]: c.shape Out[470]: (20,) cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 2

wanted: decent matplotlib alternative
by Christoph Groth 14 Oct '11

14 Oct '11

Hello, Is it just me who thinks that matplotlib is ugly and a pain to use? So far I haven't found a decent alternative usable from within python. (I haven't tried all the packages out there.) I'm mostly interested in 2d plots. Who is happy enough with a numpy-compatible plotting package to recommend it? Thanks, Christoph A few things I can't stand about matplotlib: * It works as a state machine. (There is an OO-API, too, but it's ugly and cumbersome to use, and most examples use the state machine mode or, even worse, a mixture of OO and global state.) * It uses inches by default. (I propose to switch to nails: 1 nail = 3 digits = 2 1/4 inches = 1/16 yard.) * subplot(211) (ugh!) * Concepts are named in a confusing way. ("ax = subplot(112)" anyone?)

11 14

NumPy foundations
by Linus Jundén 13 Oct '11

13 Oct '11

Hello everyone! I am about to make a NumPy presentation for my colleges in about a week. I want to tell them something about the history of the library and what kind of code it relies on. While researching and preparing for this presentation I found it very hard to find information about the origins of the numeric code. I might of course dive into the source code itself but thought that someone else might have the same questions. That's what the list is for, right? Is NumPy based on some external code like e.g. BLAS, LAPACK etc or is it coded from scratch? Anyone out there that can settle the question? Regards Linus Jundén Umeå University, Sweden

5 4

how to list all the values in a ndarray without repeat (like filter in excel)
by Chao YUE 13 Oct '11

13 Oct '11

Dear all, if I have a ndarray like array([1,2,3,2,3,1,1,1,2,2,....,2,2,3]) containing some values that are flag for data quality. how can list all the values in this array, like doing a descriptive statistics. I guess I should use Scipy statistics ? Thanks for any ideas. Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 3

np.finfo().maxexp confusing
by Matthew Brett 11 Oct '11

11 Oct '11

Hi, I realize it is probably too late to do anything about this, but: In [72]: info = np.finfo(np.float32) In [73]: info.minexp Out[73]: -126 In [74]: info.maxexp Out[74]: 128 minexp is correct, in that 2**(-126) is the minimum value for the exponent part of float32. But maxexp is not correct, because 2**(127) is the maximum value for the float32 exponent part: http://en.wikipedia.org/wiki/Single_precision_floating-point_format There is the same maxexp+1 feature for the other float types. Is this a sufficiently quiet corner of the API that it might be changed in the future with suitable warnings? Best, Matthew

1 1

Rounding to next lowest float
by Matthew Brett 11 Oct '11

11 Oct '11

Hi, Can anyone think of a clever way to round an integer to the next lowest integer represented in a particular floating point format? For example: In [247]: a = 2**25+3 This is out of range of the continuous integers representable by float32, hence: In [248]: print a, int(np.float32(a)) 33554435 33554436 But I want to round down (floor) the integer in float32. That is, in this case I want: >>> floor_exact(a, np.float32) 33554432 I can break the float into its parts to do it: https://github.com/matthew-brett/nibabel/blob/f687bfc88d1676a09fc76c968a346… but that's obviously rather ugly... Is there a simpler way? I'm sure there is and I haven't thought of it... Best, Matthew

2 2

speeding up operations on small vectors
by Christoph Groth 11 Oct '11

11 Oct '11

Dear numpy experts, I could not find a satisfying solution to the following problem, so I thought I would ask: In one part of a large program I have to deal a lot with small (2d or 3d) vectors and matrices, performing simple linear algebra operations with them (dot products and matrix multiplications). For several reasons I want the code to be in python. The most natural choice seemed to be to use numpy. However, the constant time cost when dealing with numpy arrays seems to be immense, as demonstrated by this toy program: **************************************************************** import numpy as np from time import time def points_tuple(radius): rr = radius**2 def inside(point): return point[0]**2 + point[1]**2 < rr M = ((1, 0), (0, 1)) for x in xrange(-radius, radius + 1): for y in xrange(-radius, radius + 1): point = (M[0][0] * x + M[0][1] * y, M[1][0] * x + M[1][1] * y) if inside(point): yield point def points_numpy(radius): rr = radius**2 def inside(point): return np.dot(point, point) < rr M = np.identity(2, dtype=int) for x in xrange(-radius, radius + 1): for y in xrange(-radius, radius + 1): point = np.dot(M, (x, y)) if inside(point): yield point def main(): r = 200 for func in [points_tuple, points_numpy]: t = time() for point in func(r): pass print func.__name__, time() - t, 'seconds' if __name__ == '__main__': main() **************************************************************** On my trusty old machine the output (python 2.6, numpy 1.5.1) is: points_tuple 0.36815404892 seconds points_numpy 6.20338392258 seconds I do not need C performance here, but the latter is definitely too slow. In the real program it's not so easy (but possible) to use tuples because the code is dimension-independent. Before considering writing an own "small vector" module, I'd like about other possible solutions. Other people must have stumbled across this before! Thanks, Christoph

6 12

NumPy-Discussion October 2011