Mailman 3 October 2011 - NumPy-Discussion

ndarray with double comparison
by Chao YUE Oct. 15, 2011

Oct. 15, 2011

Dear all, sorry for this stupid question but I cannot find it in numpy tutorial or google. suppose I have a=np.arange(11). In [32]: a < 8 Out[32]: array([ True, True, True, True, True, True, True, True, False, False, False], dtype=bool) In [34]: a > 4 Out[34]: array([False, False, False, False, False, True, True, True, True, True, True], dtype=bool) how can I have boolean index like 4 < a < 8 np.where(a>4 and a<8);or plainly input "a>4 and a<8" doesn't work. thanks, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

5 6

simulate AR
by Alan G Isaac Oct. 14, 2011

Oct. 14, 2011

As a simple example, if I have y0 and a white noise series e, what is the best way to produces a series y such that y[t] = 0.9*y[t-1] + e[t] for t=1,2,...? 1. How can I best simulate an autoregressive process using NumPy? 2. With SciPy, it looks like I could do this as e[0] = y0 signal.lfilter((1,),(1,-0.9),e) Am I overlooking similar (or substitute) functionality in NumPy? Thanks, Alan Isaac

4 13

statistics in python
by Radim Oct. 14, 2011

Oct. 14, 2011

Hi Rense (cross-posting to the numpy mailing list because these guys are awesome), On Oct 13, 10:01 pm, Rense Lange <rense.la...(a)gmail.com> wrote: > I have potentially millions of tuples <v1,v2,v3 ..., observation> and > I want to create frequency distributions conditional on the values of > discrete variables v1, v2, ... (e.g. the sums for boys vs. girls), or > combinations thereof (poor boys, poor girls, rich boys, rich girls). > Very few of the v1 x v2 x ... combinations might actually occur. Also, > it is sometimes necessary to combine different data sets. > > Should I just use some DB system (and if so, which one is best within > Python), or are there sparse matrix methods that are to be preferred? yes, that sounds like a job for a database. Sqlite is built-in (=part of standard Python library: `import sqlite3`). Numpy supports structured arrays (records) as well: >>> import numpy >>> dt = numpy.dtype([('name', 'a10'), ('wealth', numpy.int32), ('sex', 'a1')]) >>> x = numpy.array([('Mary', 10000, 'F'), ('John', 1000, 'M'), ('unknown', -1, '?')], dtype=dt) >>> print x[(x['wealth'] > 5000) & (x['sex'] == 'F')] # print records for all rich girls [('Mary', 10000, 'F')] so perhaps that could also fit your bill. There are no indexes but it's more pleasant to work with, imo. Note that "potentially millions of records" is not particularly big, so as long as you don't have too many variables, some in-memory db should be ok and will save you from headaches of dealing with complex db setups. I have also heard very good things about pytables, though i've never used it myself (gensim uses plain float matrices), you can have a look there: http://pytables.org HTH, Radim > > On Oct 12, 10:01 pm, Rense Lange <rense.la...(a)gmail.com> wrote: > > > > Gensim must be storing data very efficiently, and I need to do something > > > similar for another application. Can you tell me what Python programming > > > approach was used in Gensim, and are there perhaps particular sections in > > > the Gensim code that I shoGuld be looking at for inspiration and examples? > > > that question is too broad. What kind of data and application do you > > have? What sort of efficiency are you after? (fast access/little disk > > space/fast load time/...?) > > > Radim

1 0

the difference of np.nan, np.NaN, np.NAN?
by Chao YUE Oct. 14, 2011

Oct. 14, 2011

Dear all, is there any difference between np.nan, np.NaN and np.NAN? they really confuse me.... they are all Not a Number? In [75]: np.nan==np.NaN Out[75]: False In [77]: np.NaN==np.NAN Out[77]: False Thanks a lot, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 4

the axis parameter in the np.ma.concatenate is not working?
by Chao YUE Oct. 14, 2011

Oct. 14, 2011

Dear all, I use numpy version 1.5.1 which is installed by default when I do sudo apt-get install numpy on ubuntu 11.04. but it seems that for np.ma.concatenate(arrays, axis), the axis parameter is not working? In [460]: a=np.arange(10) In [461]: a=np.ma.masked_array(a,a<3) In [462]: a Out[462]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9], mask = [ True True True False False False False False False False], fill_value = 999999) In [463]: b=np.arange(10) In [464]: b=np.ma.masked_array(a,b>7) In [465]: b Out[465]: masked_array(data = [-- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False True True], fill_value = 999999) In [466]: c=np.ma.concatenate((a,b),axis=0) In [467]: c Out[467]: masked_array(data = [-- -- -- 3 4 5 6 7 8 9 -- -- -- 3 4 5 6 7 -- --], mask = [ True True True False False False False False False False True True True False False False False False True True], fill_value = 999999) In [468]: c.shape Out[468]: (20,) In [469]: c=np.ma.concatenate((a,b),axis=1) In [470]: c.shape Out[470]: (20,) cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 2

wanted: decent matplotlib alternative
by Christoph Groth Oct. 13, 2011

Oct. 13, 2011

Hello, Is it just me who thinks that matplotlib is ugly and a pain to use? So far I haven't found a decent alternative usable from within python. (I haven't tried all the packages out there.) I'm mostly interested in 2d plots. Who is happy enough with a numpy-compatible plotting package to recommend it? Thanks, Christoph A few things I can't stand about matplotlib: * It works as a state machine. (There is an OO-API, too, but it's ugly and cumbersome to use, and most examples use the state machine mode or, even worse, a mixture of OO and global state.) * It uses inches by default. (I propose to switch to nails: 1 nail = 3 digits = 2 1/4 inches = 1/16 yard.) * subplot(211) (ugh!) * Concepts are named in a confusing way. ("ax = subplot(112)" anyone?)

11 14

NumPy foundations
by Linus Jundén Oct. 13, 2011

Oct. 13, 2011

Hello everyone! I am about to make a NumPy presentation for my colleges in about a week. I want to tell them something about the history of the library and what kind of code it relies on. While researching and preparing for this presentation I found it very hard to find information about the origins of the numeric code. I might of course dive into the source code itself but thought that someone else might have the same questions. That's what the list is for, right? Is NumPy based on some external code like e.g. BLAS, LAPACK etc or is it coded from scratch? Anyone out there that can settle the question? Regards Linus Jundén Umeå University, Sweden

5 4

how to list all the values in a ndarray without repeat (like filter in excel)
by Chao YUE Oct. 13, 2011

Oct. 13, 2011

Dear all, if I have a ndarray like array([1,2,3,2,3,1,1,1,2,2,....,2,2,3]) containing some values that are flag for data quality. how can list all the values in this array, like doing a descriptive statistics. I guess I should use Scipy statistics ? Thanks for any ideas. Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 3

np.finfo().maxexp confusing
by Matthew Brett Oct. 11, 2011

Oct. 11, 2011

Hi, I realize it is probably too late to do anything about this, but: In [72]: info = np.finfo(np.float32) In [73]: info.minexp Out[73]: -126 In [74]: info.maxexp Out[74]: 128 minexp is correct, in that 2**(-126) is the minimum value for the exponent part of float32. But maxexp is not correct, because 2**(127) is the maximum value for the float32 exponent part: http://en.wikipedia.org/wiki/Single_precision_floating-point_format There is the same maxexp+1 feature for the other float types. Is this a sufficiently quiet corner of the API that it might be changed in the future with suitable warnings? Best, Matthew

1 1

Rounding to next lowest float
by Matthew Brett Oct. 11, 2011

Oct. 11, 2011

Hi, Can anyone think of a clever way to round an integer to the next lowest integer represented in a particular floating point format? For example: In [247]: a = 2**25+3 This is out of range of the continuous integers representable by float32, hence: In [248]: print a, int(np.float32(a)) 33554435 33554436 But I want to round down (floor) the integer in float32. That is, in this case I want: >>> floor_exact(a, np.float32) 33554432 I can break the float into its parts to do it: https://github.com/matthew-brett/nibabel/blob/f687bfc88d1676a09fc76c968a346… but that's obviously rather ugly... Is there a simpler way? I'm sure there is and I haven't thought of it... Best, Matthew

2 2