Mailman 3 September 2012 - NumPy-Discussion

Job opportunity working with python
by Scott Collis 25 Sep '12

25 Sep '12

Good evening fellow Numpy-ers! I sent an email a while ago about a employment opportunity at Argonne National Laboratory but I think I sent it from the wrong address so it bounced. We are seeking to hire some one to work on a python based toolkit for working with scanning weather radar. The main role for this person will be to find novel ways of working with large radar based data sets, probably writing code in C, C++ and Fortran to make advanced modules available in Python. Working with communities such as this will be encouraged in this new position. Argonne is a multidisciplinary lab in the Chicago burbs.. once you get over the cold winters you will enjoy good parks, schools and easy access to Chicago! Take a look at the description here: http://www.arm.gov/news/jobs/post/18955 and use the PD to apply here: http://web.anl.gov/jobsearch/detail.jsp?userreqid=319854+EVS&lsBrowse=ALL Basically you can read the job description as: This person will build and maintain a scikit for working with weather radars… Thanks for your time, Scott --- Dr Scott Collis ARM Precipitation Radar Translator Environmental Sciences Division Argonne National Laboratory Mb: +1 630 235 8025 Of: +1 630 252 0550 http://radar.arm.gov

1 0

Memory leak with numpy master
by Gael Varoquaux 24 Sep '12

24 Sep '12

Hi list, I think that I am hit a memory leak with numpy master. The following code enables to reproduce it: ________________________________________________________________________________ import numpy as np n = 100 m = np.eye(n) for i in range(30000): #np.linalg.slogdet(m) t, result_t = np.linalg.linalg._commonType(m) a = np.linalg.linalg._fastCopyAndTranspose(t, m) pivots = np.zeros((n,), np.linalg.linalg.fortran_int) results = np.linalg.lapack_lite.dgetrf(n, n, a, n, pivots, 0) d = np.diagonal(a) if not i % 1000: print i ________________________________________________________________________________ If you execute this code, you'll see the memory go steadily up. The reason that I came up with such a strange looking code is that in my codebase, I do repeated calls to np.linalg.slogdet. I came up with the code above by simplifying what is done in slogdet. I don't think that I can simplify any further and still reproduce the memory leak. Should I submit a bug report (in other words, can people reproduce?)? Cheers, Gaël

3 5

np.array execution path
by Sebastian Berg 24 Sep '12

24 Sep '12

Hi, I have a bit of trouble figuring this out. I would have expected np.asarray(array) to go through ctors, PyArray_NewFromArray, but it seems to me it does not, so which execution path is exactly taken here? The reason I am asking is that I want to figure out this behavior/bug, and I really am not sure which function is responsible: In [69]: o = np.ones(3) In [70]: no = np.asarray(o, order='C') In [71]: no[:] = 10 In [72]: o # OK, o was changed in place: Out[72]: array([ 10., 10., 10.]) In [73]: no.flags # But no claims to own its data! Out[73]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [74]: no = np.asarray(o, order='F') In [75]: no[:] = 11 In [76]: o # Here asarray actually returned a real copy! Out[76]: array([ 10., 10., 10.]) Thanks, Sebastian

5 8

Re: [Numpy-discussion] ZeroRank memmap behavior?
by Wim Bakker 24 Sep '12

24 Sep '12

Thanks Sebastian. Casting it to an array would certainly help. Another oddity of zero-ranked scalars is that they look iterable, but in fact are not. Because all they do is generate an error. >>> a = np.array(22) Test if iterable: >>> hasattr(a, __iter__) True Or: >>> import collections >>> isinstance(a, collections.Iterable) True >>> for e in a: print e TypeError: iteration over a 0-d array Wouldn't it be better to have a different type altogether for numpy scalars? Because scalars don't seem to quack quite like arrays... (I haven't been following the discussion around scalars, so if this is a silly remark just ignore it) Another question. What would be the preferred method of testing for zero-ranked array? Something like this? >>> if a.shape: print "array" else: print "scalar" Regards, Wim

1 0

Views of memmaps and offset
by Gael Varoquaux 23 Sep '12

23 Sep '12

Hi list, I am struggling with offsets on the view of a memmaped array. Consider the following: import numpy as np a = np.memmap('tmp.mmap', dtype=np.float64, shape=50, mode='w+') a[:] = np.arange(50) b = a[10:] Here, I have a.offset == 0 and b.offset == 0. In practice, the data in b is offset compared to the start of the file, given that it is a view computed with an offset. My goal is, given b, to find a way to open a new view on the file, e.g. in a different process. For this I need the offset. Any idea of how I can retrieve it? In the previous numpy versions, I could go from b to a using the 'base' attribute of a. This is no longer possible. Also, should the above behavior be considered as a bug? Cheers, Gaël

4 19

errors of scipy build
by 心如烛光 23 Sep '12

23 Sep '12

Hi,all. I have installed numpy and scipy step by step carefully as the instructions from website:http://www.scipy.org/Installing_SciPy/Mac_OS_X But still get many errors and warnings while it's building. (OSX lion 10.7.4 /Xcode 4.5 /clang /gfortran4.2.3) Do scipy and numpy must be built by gcc?? Such as: 1 error generated. _configtest.c:5:28: error: 'test_array' declared as an array with a negative size static int test_array [1 - 2 * !(((long) (sizeof (npy_check_sizeof_type))) == 4)]; ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. C compiler: clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes /////////////////////////////////////////////////////////// Constructing wrapper function "drotm"... getarrdims:warning: assumed shape array, using 0 instead of '*' getarrdims:warning: assumed shape array, using 0 instead of '*' x,y = drotm(x,y,param,[n,offx,incx,offy,incy,overwrite_x,overwrite_y])

2 1

Ignore axes with dimension==1 for contiguous flags
by Sebastian Berg 22 Sep '12

22 Sep '12

Hey, Numpy currently assumes that if "ndim > 1" then it is impossible for any array to be both C- and F-contiguous, however an axes of dimension 1 does have no effect on the memory layout. I think I have made most important changes (actually really very few), though I bet some parts of numpy still need adapting because of smaller quirks: https://github.com/seberg/numpy/compare/master...cflags This example sums up two advantages. On that branch: In [9]: a = np.arange(9).reshape(3,3)[::3,:] In [10]: a.flags.contiguous, a.flags.fortran Out[10]: (True, True) Note that currently _both_ are false, because numpy does not reset the strides for the first dimension. The only real problem I see is that someone who assumes that for a contiguous array strides[0] or strides[-1] is elemsize has to change the code or face segmentation faults, but maybe I am missing something big? Any comments if this is the right idea? And if where would more changes be needed? Regards, Sebastian

1 0

Regression: in-place operations (possibly intentional)
by Benjamin Root 22 Sep '12

22 Sep '12

Consider the following code: import numpy as np a = np.array([1, 2, 3, 4, 5], dtype=np.int16) a *= float(255) / 15 In v1.6.x, this yields: array([17, 34, 51, 68, 85], dtype=int16) But in master, this throws an exception about failing to cast via same_kind. Note that numpy was smart about this operation before, consider: a = np.array([1, 2, 3, 4, 5], dtype=np.int16) a *= float(128) / 256 yields: array([0, 1, 1, 2, 2], dtype=int16) Of course, this is different than if one does it in a non-in-place manner: np.array([1, 2, 3, 4, 5], dtype=np.int16) * 0.5 which yields an array with floating point dtype in both versions. I can appreciate the arguments for preventing this kind of implicit casting between non-same_kind dtypes, but I argue that because the operation is in-place, then I (as the programmer) am explicitly stating that I desire to utilize the current array to store the results of the operation, dtype and all. Obviously, we can't completely turn off this rule (for example, an in-place addition between integer array and a datetime64 makes no sense), but surely there is some sort of happy medium that would allow these sort of operations to take place? Lastly, if it is determined that it is desirable to allow in-place operations to continue working like they have before, I would like to see such a fix in v1.7 because if it isn't in 1.7, then other libraries (such as matplotlib, where this issue was first found) would have to change their code anyway just to be compatible with numpy. Cheers! Ben Root

7 29

ZeroRank memmap behavior?
by Wim Bakker 21 Sep '12

21 Sep '12

I'm deeply puzzled by the recently changed behavior of zero-rank memmaps. I think this change happened from version 1.6.0 to 1.6.1, which I'm currently using. >>> import numpy as np Create a zero-rank memmap. >>> x = np.memmap(filename='/tmp/m', dtype=float, mode='w+', shape=()) Give it a value: >>> x[...] = 22 >>> x memmap(22.0) So far so good. But now: >>> b = (x + x) / 1.5 >>> b memmap(29.333333333333332) WT.? Why is the result of this calculation a memmap? It even thinks that it's still linked to the file, but it's not: >>> b.filename '/tmp/m' If I try this with arrays then I don't get this weird behavior: >>> a = np.array(2, dtype=float) >>> (a + a) / 2.5 1.6000000000000001 which gives me a Python float, not a zero-rank array. Why does the memmap behave like that? Why do I get a memmap even though it's not connected to any file? Regards, Wim

2 1

np.delete fix
by Sebastian Berg 21 Sep '12

21 Sep '12

Hey, I have written a small PR, to fix np.delete, since it would change the behavior a little (to the better IMO) I think I should also write to the list? So here is the problem with np.delete: 1. When using slices with negative strides, it does not work (best case) or give even wrong results. 2. When using an index array, it ignores negative indexes. 3. The fact that it uses setdiff1d makes it unnecessarily slow. https://github.com/numpy/numpy/pull/452/files fixes these things. The change is that out of bounds indices would raise an Exception (even if one might say that they do not matter to deletion), however I consider that a feature. And a small example how badly its wrong with slices: In [1]: arr = np.arange(4) In [2]: set(arr) - set(arr[1::-1]) Out[2]: set([2, 3]) In [3]: np.delete(arr, np.s_[1::-1]) Out[3]: array([3, 3]) Regards, Sebastian

1 0