Mailman 3 August 2009 - NumPy-Discussion

masked index surprise
by John Hunter 14 Aug '09

14 Aug '09

I just tracked down a subtle bug in my code, which is equivalent to In [64]: x, y = np.random.rand(2, n) In [65]: z = np.zeros_like(x) In [66]: mask = x>0.5 In [67]: z[mask] = x/y I meant to write z[mask] = x[mask]/y[mask] so I can fix my code, but why is line 67 allowed In [68]: z[mask].shape Out[68]: (54,) In [69]: (x/y).shape Out[69]: (100,) it seems like broadcasting would fail In [70]: np.__version__ Out[70]: '1.4.0.dev7153' In [71]:

3 4

Re: [Numpy-discussion] ndarray subclass causes memory leak
by Jeremy Mayes 14 Aug '09

14 Aug '09

I've narrowed this down to a change between 1.0.4 and 1.1.0. Valgrind (of v1.3.0) shows the following result. The change was in setArrayFromSequence where PyArray_EnsureArray gets invoked in v1.1.0 where it did not in v1.0.4. ==10132== 4,474,768 (3,197,200 direct, 1,277,568 indirect) bytes in 39,965 blocks are definitely lost in loss record 36 of 36 ==10132== at 0x4905B65: malloc (vg_replace_malloc.c:149) ==10132== by 0x7FFFDAA: array_alloc (arrayobject.c:7387) ==10132== by 0x8003465: PyArray_NewFromDescr (arrayobject.c:5900) ==10132== by 0x802434D: PyArray_EnsureArray (multiarraymodule.c:226) ==10132== by 0x8025766: setArrayFromSequence (arrayobject.c:7938) ==10132== by 0x80256B8: setArrayFromSequence (arrayobject.c:7957) ==10132== by 0x800F5FD: PyArray_FromAny (arrayobject.c:7984) ==10132== by 0x802CC21: PyArray_CheckFromAny (arrayobject.c:9530) ==10132== by 0x8037F64: _array_fromobject (multiarraymodule.c:6329) ==10132== by 0x4AC9501: PyEval_EvalFrameEx (ceval.c:3612) ==10132== by 0x4ACA894: PyEval_EvalCodeEx (ceval.c:2875) ==10132== by 0x4ACAA11: PyEval_EvalCode (ceval.c:514) ==10132== by 0x4AEC98B: PyRun_FileExFlags (pythonrun.c:1273) ==10132== by 0x4AED612: PyRun_SimpleFileExFlags (pythonrun.c:879) ==10132== by 0x4AF88C7: Py_Main (main.c:532) ==10132== by 0x38C821C3FA: (below main) (in /lib64/tls/libc-2.3.4.so) ==10132== On Fri, Aug 14, 2009 at 6:34 AM, Jeremy Mayes <jeremy.mayes(a)gmail.com>wrote: > This simple example causes a massive leak in v1.3.0 which didn't exist in > v1.0.1. What am I doing wrong? If I replace > arr = [Array((2,2)), Array((2,2))] > > with > > arr = [numpy.ndarray((2,2,)), numpy.ndarray((2,2))] > > then I don't have the leak > > > > import numpy > import gc > > class Array(numpy.ndarray): > def __new__(subtype, shape, dtype=float, buffer=None, offset=0, > strides=None, order=None, info=None): > return numpy.ndarray.__new__(subtype, shape, dtype, buffer, offset, > strides, order) > > def __array_finalize__(self, obj): > print 'called array_finalize' > > if __name__=='__main__': > arr = [Array((2,2)), Array((2,2))] > > nbytesAllocated = 0 > for i in xrange(1000000000): > a = numpy.array(arr) > nbytesAllocated += a.nbytes > if i%1000 == 0: > print 'allocted %s'%nbytesAllocated > gc.collect() > > > -- > --jlm > > > > > -- > --jlm > -- --jlm

1 0

finding range of values below threshold in sorted array
by Mark Bakker 14 Aug '09

14 Aug '09

Hello List, I am trying to find a quick way to do the following: I have a *sorted* array of real numbers, say array A, sorted in ascending order (but easy to store descending if that would help) I want to find all numbers below a certain value, say b Sure, I can do A < b and I will get back a list with a bunch of True-s and then a bunch of False's, but all I need is the highest index for which A[i] < b, since A is sorted. Does anybody know a quick way to do this? I need to do it a lot, so the quicker the better. Thanks, Mark

2 1

ndarray subclass causes memory leak
by Jeremy Mayes 14 Aug '09

14 Aug '09

This simple example causes a massive leak in v1.3.0 which didn't exist in v1.0.1. What am I doing wrong? If I replace arr = [Array((2,2)), Array((2,2))] with arr = [numpy.ndarray((2,2,)), numpy.ndarray((2,2))] then I don't have the leak import numpy import gc class Array(numpy.ndarray): def __new__(subtype, shape, dtype=float, buffer=None, offset=0, strides=None, order=None, info=None): return numpy.ndarray.__new__(subtype, shape, dtype, buffer, offset, strides, order) def __array_finalize__(self, obj): print 'called array_finalize' if __name__=='__main__': arr = [Array((2,2)), Array((2,2))] nbytesAllocated = 0 for i in xrange(1000000000): a = numpy.array(arr) nbytesAllocated += a.nbytes if i%1000 == 0: print 'allocted %s'%nbytesAllocated gc.collect() -- --jlm -- --jlm -- --jlm

1 0

SciPy2009 BoF Wiki Page
by David Warde-Farley 13 Aug '09

13 Aug '09

I needed a short break from some heavy writing, so on Fernando's suggestion I took to the task of aggregating together mailing list traffic about the BoFs next week. So far, 4 have been proposed, and I've written down under "attendees" the names of anyone who has expressed interest (except in Perry's case, where I've only heard it via proxy). The page is at http://scipy.org/SciPy2009/BoF I've created sections below that are hyperlink targets for the topic of the session, if someone more knowledgeable of that domain can fill in those sections, please do. Edit away, and see you next week! (And if someone can forward this to the Matplotlib list, I'm not currently subscribed) David

3 2

documentation translation
by Dokuro 13 Aug '09

13 Aug '09

Hello, we have a python group in Maracaibo Venezuela and have started a translation project for as many python documentations we can find from english to spanish, making the use of python easier to non-english speakers, so in behalf of our group i would like to ask if it won't be a problem if we make this translations and keep them in our wiki http://proyectociencia.org/Wiki/index.php/Grupo_de_Usuarios_de_Python Dokuro.

2 1

adaptive sampling of an interval or plane
by John Hunter 13 Aug '09

13 Aug '09

We would like to add function plotting to mpl, but to do this right we need to be able to adaptively sample a function evaluated over an interval so that some tolerance condition is satisfied, perhaps with both a relative and absolute error tolerance condition. I am a bit out of my area of competency here, eg I do not know exactly how the tolerance condition should be specified, but I suspect some of you here may be experts on this. Does anyone have some code compatible with the BSD license, preferably based on numpy but we would consider an extension code or scipy solution, for doing this? The functionality we have in mind is provided in matlab with fplot http://www.mathworks.com/access/helpdesk/help/techdoc/index.html?/access/he… We would like 1D and 2D versions of this ideally. If anyone has some suggestions, let me know. Thanks, JDH

3 3

SciPy tutorials and talks will be recorded and posted online!
by Fernando Perez 12 Aug '09

12 Aug '09

Hi all, as you may recall, there have been recently a number of requests for videotaping the conference. I am very happy to announce that we will indeed have full video coverage this year of both tutorial tracks as well as the main talks (minus any specific talk where a speaker may object to being taped, since we'll respect such objections if they are made). Jeff Teeters and Kilian Koepsell from UC Berkeley, who have in the past made recordings like these of one of my workshops and a recent talk by Gael : http://www.archive.org/search.php?query=Fernando+Perez+scientific+python http://www.archive.org/details/ucb_py4science_2009_07_14_Gael_Varoquaux are going to perform the work. I'd like to sincerely thank: - Jeff and Kilian for offering to do this work and providing some of the recording equipment. - The Redwood Center for Theoretical Neuroscience, which provided other equipment. - Enthought, who are funding this on very short notice!!! (Especially Dave Peterson and Eric Jones, who tolerated my last minute nags very graciously). Without Enthought's last-minute support, this would simply not be happening. I really appreciate that everyone involved worked on short notice to make this possible, I hope the entire community will benefit from these resources being available. Best regards, f

2 1

memmap, write through and flush
by Tom Kuiper 12 Aug '09

12 Aug '09

There is something curious here. The second flush() fails. Can anyone explain this? Tom ------------------- code snippet ------------------------ ... # create a memmap with dtype and shape that matches the data fp = np.memmap(filename, dtype='float32', mode='w+', shape=(3,4)) print "Initial memory mapped array (mode 'w+'):\n",fp # write data to memmap array fp[:] = data[:] fp.flush() # append a row to the array fp = np.append(fp, [[12,13,14,15]], 0) print "Filled memory mapped array:\n",fp fp.flush() ... ----------------- output ----------------------- Filled memory mapped array: [[ 0. 1. 2. 3.] [ 4. 5. 6. 7.] [ 8. 9. 10. 11.] [ 12. 13. 14. 15.]] --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) ..../memmap_test.py in <module>() 21 fp = np.append(fp, [[12,13,14,15]], 0) 22 print "Filled memory mapped array:\n",fp ---> 23 fp.flush() 24 25 # deletion flushes memory changes to disk before removing the object AttributeError: 'numpy.ndarray' object has no attribute 'flush' WARNING: Failure executing file: <memmap_test.py>

3 2

Re: [Numpy-discussion] saving incrementally numpy arrays
by Juan Fiol 12 Aug '09

12 Aug '09

Thanks David, I'll look into it now. Regarding the allocation/deallocation times I think that is not an issue for me. The chunks are generated by a fortran routine that takes several minutes to run (I am collecting a few thousand points before saving to disk). They are approximately the same size but not exactly. I want them to be stored for later retrieval and analysis in a convenient way. Thanks, regards ----------------------------------------- You'll probably want the EArray. createEArray() on a new h5file, then append to it. http://www.pytables.org/docs/manual/ch04.html#EArrayMethodsDescr If your chunks are always the same size it might be best to try and do your work in-place and not allocate a new NumPy array each time. In theory 'del' ing the object when you're done with it should work but the garbage collector may not act quickly enough for your liking/the allocation step may start slowing you down. What do I mean? Well, you could clear the array when you're done with it using foo[:] = 0 (or nan, or whatever) and when you're "building it up" use the inplace augmented assignment operators as much as possible (+=, /=, -=, *=, %=, etc.). David

1 0