Mailman 3 February 2007 - NumPy-Discussion

NaN, min, and max
by Keith Goodman Feb. 8, 2007

Feb. 8, 2007

I keep running into this problem: >> import numpy.matlib as M >> x = M.rand(3,3) >> x[1,1] = M.nan >> x matrix([[ 0.94425407, 0.02216611, 0.999475 ], [ 0.40444129, nan, 0.23264341], [ 0.24202372, 0.05344269, 0.37967564]]) >> x.max() 0.379675636032 <---- Wrong (for me) >> x[1,1] = 0 >> x.max() 0.999474999444 <----- Beautiful! Look at all the tripple digits! How do I add nanmax as a method of the matrix class? How would I replace max with nanmax? Would I have to do that in every module of my package if I want to access numpy.matlib as M instead of packagename.M?

2 1

Please help with subclassing numpy.ndarray
by Jeremy Conlin Feb. 8, 2007

Feb. 8, 2007

I have subclassed the numpy.ndarray object, but need some help setting some attributes. I have read http://scipy.org/Subclasses but it doesn't provide the answer I am looking for. I create an instance of the class in my __new__ method as: import numpy class MyClass(numpy.ndarray): __new__(self,…): # Some stuff here H, edges = numpy.histogramdd(…) return H This sets H as the instance of my object. I would also like to have edges be an attribute of MyClass. I can't do: self.edges = edges because the object hasn't been instantiated yet. Can someone show me how I can also keep the information from the variable edges? Thanks, Jeremy

6 14

SciPy '07 ???
by Christopher Barker Feb. 7, 2007

Feb. 7, 2007

Hi, Does anyone know if there will be a SciPy '07 conference, and if so, when? I'd really like to try to get there this year, but need to start planning my summer schedule. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 2

Bug in numpy user-define types mechanism causes segfault on multiple calls to a ufunc
by Tom Denniston Feb. 7, 2007

Feb. 7, 2007

I am trying to register a custom type to numpy. When I do so it works and the ufuncs work but then when I invoke any ufunc twice the second time my python interpretter segfaults. I think i know what the problem is. In the select_types method in ufuncobject.c in numpy/core/src/ numpy gets a reference to the key for the loop via a call to PyInt_FromLong: key = PyInt_FromLong((long) userdef); Then it gets the actual loop via a call to PyDict_GetItem: obj = PyDict_GetItem(self->userloops, key); It later proceeds to do a decref on key: Py_DECREF(key); and later a decref on obj Py_DECREF(obj); None of this code actually runs unless you are doing an operation on a user defined type because it is all in the block with an if statement if (userdef > 0). The Py_DECREF on key is correct because it returns a new reference per python c api doc: PyObject* PyInt_FromLong( long ival) Return value: New reference. Create a new integer object with a value of ival. However the Py_DECREF on the obj looks incorrect to me because the PyDict_Getitem returns a borrowed reference and the numpy code doesn't increment the reference: PyObject* PyDict_GetItem( PyObject *p, PyObject *key) Return value: Borrowed reference. Return the object from dictionary p which has a key key. Return NULL if the key key is not present, but without setting an exception. So what seems to happen is the last reference gets decremented and the garbage collector frees up obj (the ufunc loop) on the first time through. The second time the through the ufunc loop is garbage memory and it segfaults. If I comment out the obj DECREF it works. I think one needs to either do that or add an INCREF right after retrieving key. I think either will work but the multithreading implications are different. I don't think it matters give that (I believe) numpy doesn't release the GIL but I thought someone on this list would be a better judge than I (maybe Travis) of what the correct fix should be. Am I correct in my analysis? In either case it should be a one or two line fix. --Tom

2 2

bug in numpy.equal?
by Tom Denniston Feb. 7, 2007

Feb. 7, 2007

The behavior below seems strange to me. The string array is type S3 yet it says that comparison with 'abc' is not implemented. The == operator seems to work though. Is there a subtlty I am missing or is it simply a bug? In [1]: import numpy In [2]: numpy.equal(numpy.array(['abc', 'def']), 'abc') Out[2]: NotImplemented In [3]: numpy.array(['abc', 'def']) == 'abc' Out[3]: array([ True, False], dtype=bool) In [4]: numpy.equal? Type: ufunc Base Class: <type 'numpy.ufunc'> String Form: <ufunc 'equal'> Namespace: Interactive Docstring: y = equal(x1,x2) returns elementwise x1 == x2 in a bool array

2 2

confused with apply_along_axis()
by Joris De Ridder Feb. 7, 2007

Feb. 7, 2007

Hi, I'm confused by the output of apply_along_axis() in the following very simple example: In [93]: a = arange(12.).reshape(2,2,3) In [95]: a Out[95]: array([[[ 0., 1., 2.], [ 3., 4., 5.]], [[ 6., 7., 8.], [ 9., 10., 11.]]]) In [96]: def myfunc(b): ....: print "slice:", b, " middle value:", b[1] ....: return b[1] ....: In [97]: apply_along_axis(myfunc,2,a) slice: [ 0. 1. 2.] middle value: 1.0 slice: [ 3. 4. 5.] middle value: 4.0 slice: [ 6. 7. 8.] middle value: 7.0 slice: [ 9. 10. 11.] middle value: 10.0 Out[97]: array([[ 7., 7.], [ 10., 10.]]) I expected as output array([[ 1., 4.], [ 7., 10.]]) Why is this not the case? How exactly does apply_along_axis() use the output of myfunc() to produce its result? Cheers, J. P.S. I'm working with Python 2.5 & Numpy 1.0 Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

2 2

force column vector
by Christian Feb. 7, 2007

Feb. 7, 2007

Hi, when creating an ndarray from a list, how can I force the result to be 2d *and* a column vector? So in case I pass a nested list, there will be no modification of the shape and when I pass a simple list, it will be converted to a 2d column vector. I can only think of a solution using 'if' clauses but I suppose there is a more elegant way. Thanks, Christian

8 10

importing multiarraymodule.c in Python embedding
by Steele, Greg Feb. 7, 2007

Feb. 7, 2007

I have run into an interesting issue with multiarraymodule.c with regards to embedding Python in a C application on Windows XP. In the application, the following abbreviated sequence was executed 1) Py_Initialize 2) numpy is imported 3) Py_Finalize 4) Py_Initialize 5) attempt to import numpy results in an error. I have tracked this down to two things 1) Python does not call the FreeLibrary windows API on C extension modules when shutting down 2) multiarraymodule.c has a static variable _multiarray_module_loaded which, when set, bypasses the call to Py_InitModule The error at step 5 above occurs because the multiarraymodule DLL is loaded at step 2, the _multiarray_module_loaded variable is set at step 2, and Py_InitModule is bypassed at step 5. Since Py_InitModule is not called, multiarraymodule is not placed in the module dictionary (i.e. sys.modules) and an error occurs in importdl.h which checks if the module is there. The obvious solution is to not execute the sequence above. This issue does seem to highlight a weakness in the implementation of multiarraymodule.c. Could someone comment on the need for static variable _multiarray_module_loaded? Is there a more robust way to achieve the goal? Thanks. Greg

2 1

thread safe subclasses of ndarray
by Matt Knox Feb. 7, 2007

Feb. 7, 2007

Sturla Molden brought up a point in an earlier thread ("Please help with subclassing numpy.ndarray") that I think is worth highlighting. A "common" approach to subclassing ndarray in python (http://www.scipy.org/Subclasses) results in code that is not thread safe. This approach involves setting some class level variables inside the __new__ method, and then retrieving those values inside the __array_finalize__ method. A simple work around that Sturla alluded to is to explictly set a lock inside the __new__ method, and later release it in the __array_finalize__ method. However, this would cause problems if someone invoked the __new__ method in such a way that __array_finalize__ was not called afterwards (which I think is possible, isn't it?). The two subclasses of ndarray in the core numpy distribution (that I am aware of) , chararray and matrix, don't run into this problem because they can get all the relevant info they need by directly inspecting the ndarray that comes out of the __new__ method, and so they don't need to share any extra info between __new__ and __array_finalize__ So, I am wondering if perhaps one of the experts on this list would be able to shed some light on a good way to create thread safe subclasses of ndarray. Thanks, - Matt Knox

1 0

memmap on 64bit Linux for > 2 GB files
by Sebastian Haase Feb. 6, 2007

Feb. 6, 2007

Hi, I finally tried to do the test, to memmap a large file filesize: 2.8G a memmap call gives this error: {{{ >>> N.memmap('20050622-1648-Y_DEMO-1') Traceback (most recent call last): File "<input>", line 1, in ? File "/jws30/haase/PrLinN64/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) OverflowError: memory mapped size is too large (limited by C int) }}} I'm using a recent numpy on a 64bit Linux (debian etch, kernel: 2.6.16-2-em64t-p4-smp) {{{ >>> N.__version__ '1.0.2.dev3509' >>> N.int0 <type 'numpy.int64'> }}} Is this supposed to work ? Thanks, Sebastian

3 2