Mailman 3 January 2007 - NumPy-Discussion

memmap close() and flush()
by Sebastian Haase Jan. 31, 2007

Jan. 31, 2007

Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap. In numarray these were methods called memmappedArr.flush() and memmappedArr.close() Thanks, Sebastian

5 6

pyrex c_numpy.pyx problem
by Stefan van der Walt Jan. 31, 2007

Jan. 31, 2007

Hi, Pyrex 0.9.5.1 doesn't like the following snippet out of c_numpyx.pyx: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: cdef int numiter cdef npy_intp size, index cdef int nd cdef npy_intp dimensions[NPY_MAXDIMS] cdef flatiter iters[NPY_MAXDIMS] which corresponds to typedef struct { PyObject_HEAD int numiter; /* number of iters */ npy_intp size; /* broadcasted size */ npy_intp index; /* current index */ int nd; /* number of dims */ npy_intp dimensions[NPY_MAXDIMS]; /* dimensions */ PyArrayIterObject *iters[NPY_MAXARGS]; /* iterators */ } PyArrayMultiIterObject; in ndarrayobject.h. I changed it to be ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: cdef int numiter cdef npy_intp size, index cdef int nd cdef npy_intp *dimensions cdef flatiter **iters after which it complains /tmp/pyrex/c_numpy.pxd:99:22: Pointer base type cannot be a Python object so I changed the last line to cdef flatiter iters That compiles, but it can't be right. Any advice/suggestions? Cheers Stéfan

2 1

Conversion of Numeric arrays from/to numpy arrays
by William Hughes Jan. 31, 2007

Jan. 31, 2007

Hi: Is there a easy and efficient way to convert between Numeric and numpy arrays? - William Hughes

2 1

Patch for RPy to use NumPy uploaded to www.scipy.org
by Travis Oliphant Jan. 31, 2007

Jan. 31, 2007

I've added a patch to the Porting to NumPy page so that RPy uses NumPy instead of Numeric. It would be very helpful for unification if these package authors would accept these patches. NumPy is far enough along in development that I don't think there is any reason for new releases of packages to continue to support Numeric. For numarray-support in packages the transition may take a little longer. I know that some think that supporting multiple array packages may be a good idea. I think that is a recipe for long-term headaches. We are moving to get the array interface into Python. Until that happens (hopefully by Python 2.6), third-party code should write to NumPy. You can use the attribute-based array interface if you are willing to alter that at some point in the future when the array interface is moved into Python. -Travis

4 5

Subclassing ndarray in Pyrex
by Pierre GM Jan. 31, 2007

Jan. 31, 2007

I'm still trying to subclass ndarray in Pyrex, without much success so far. I started to follow Francesc's suggestion (http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025644.ht…), but that doesn't fit what I need: the myarray class Francesc introduced is just an object, not a subclass of ndarray... The closest I came to something vaguely running is the following (called subnd.pyx later on): #------------------- from definitions cimport import_array, \ malloc, free, npy_intp, \ PyArray_GETITEM, PyArray_EMPTY, \ ndarray, dtype # NumPy must be initialized import_array() import numpy as _N cdef class Sub_2(ndarray): cdef readonly object _info cdef ndarray __mask def __new__(self, object shape=None, object dtype=None, object buffer=None, object order=None, object infodict={}): self.__mask = <ndarray>_N.zeros(shape, _N.bool) self._info = infodict return property info: def __get__(self): return self._info def __set__(self, value): self._info = value property _mask: def __get__(self): return self.__mask def subarray(obj, info={}): _obj = <ndarray>_N.asarray(obj) print "_obj is: %s" % _obj _internal = Sub_2(shape=_obj.shape, dtype=_obj.dtype) _internal.flat[:] = _obj.flat[:] _internal.info = info return _internal #---------------------- However, I get a segfault when I try to play with it in Python: >>> import numpy as N >>> import subnd >>> L = N.array([1,2,3] >>> x = subarray(L) >>> x Sub_2([1, 2, 3]) >>> x+1 Sub_2([2, 3, 4]) >>> N.log(x) crash So obviously I'm missing something, but what ? Some kind of closure ? Moreover, I'm a bit disappointed with this method: if I understand correctly the subtleties of subclassing in pyrex, the ndarray.__new__ is called before Sub_2.__new__, with the same arguments. As ndarray.__new__ doesn't take optional arguments, the subclass can't either. That's a bit limiting, I have to call the constructor function all the time... I'm completely at loss, here. Any advice/help/suggestions would be more than welcome.

1 0

Pickling ufuncs?
by James A. Bednar Jan. 30, 2007

Jan. 30, 2007

Hi, Does anyone know whether it is possible to pickle and unpickle numpy ufuncs? I can't find anything about that on scipy.org or the mailing list archives. I have several important pieces of code that accept a numpy ufunc as an argument and later apply it to some data, while keeping a copy of the ufunc in an attribute. So far I have been able to pickle this code only by doing some very ugly hacks and workarounds. The ufuncs from Numeric 24 and earlier did not even deepcopy, which caused us lots of other problems, but deepcopying them works now in numpy 1.0.1. However, they still don't seem to pickle with the regular picklers. Is this deliberately disabled for some reason, or is there some workaround? Here's an example that illustrates the problem: I have a class Test defined in test.py: class Test(object): def __init__(self,a): self.a = a If I try this code at the commandline: import numpy from test import Test t=Test(numpy.multiply) import pickle s=pickle.dumps(t) I get "TypeError: can't pickle ufunc objects" Also, I can't reproduce it in a small example yet, but pickling completes without errors in our larger program that also stores ufuncs, yet I then get a different error on unpickling: File "/disk/home/lodestar1/jbednar/topographica/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/disk/home/lodestar1/jbednar/topographica/lib/python2.4/pickle.py", line 1097, in load_newobj obj = cls.__new__(cls, *args) TypeError: object.__new__(numpy.ufunc) is not safe, use numpy.ufunc.__new__() Any ideas? Thanks, Jim _______________________________________________________________________________ Unix> cat test.py class Test(object): def __init__(self,a): self.a = a Unix> bin/python Python 2.4.4 (#5, Nov 9 2006, 22:58:03) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> from test import Test >>> t=Test(numpy.multiply) >>> import pickle >>> s=pickle.dumps(t) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/jb/lib/python2.4/pickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "/home/jb/lib/python2.4/pickle.py", line 231, in dump self.save(obj) File "/home/jb/lib/python2.4/pickle.py", line 338, in save self.save_reduce(obj=obj, *rv) File "/home/jb/lib/python2.4/pickle.py", line 433, in save_reduce save(state) File "/home/jb/lib/python2.4/pickle.py", line 293, in save f(self, obj) # Call unbound method with explicit self File "/home/jb/lib/python2.4/pickle.py", line 663, in save_dict self._batch_setitems(obj.iteritems()) File "/home/jb/lib/python2.4/pickle.py", line 677, in _batch_setitems save(v) File "/home/jb/lib/python2.4/pickle.py", line 313, in save rv = reduce(self.proto) File "/home/jb/lib/python2.4/copy_reg.py", line 69, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle ufunc objects >>>

3 3

Selected and altering a submatrix
by Steve Lianoglou Jan. 30, 2007

Jan. 30, 2007

Hi, I was just curious what the "correct" (fast) way to select and alter a submatrix. For example, say I have a 10x10 array and only want to add some number to the elements in the submatrix that consists of the [0,1,2] th rows, and [4,5,6]th colums. You can imagine that those rows/cols select a square in the top- middle of the 10x10 which I want to alter. The only way I can get this to work is if I iterate over the indices in one of the dimensions (say the rows) and use the column indices to slice out the relevant elements to add to .. is there a NumPy-thonic way to do this: === import numpy as N mat = N.zeros((10,10)) rows = [0,1,2] cols = [4,5,6] for row in rows: mat[row,cols] += 1 ==== I found something on the lists from a few years back that was in reference to numeric or numarray that suggested doing some gymnastics with take/put, but it still seemed as if there was no way to slice out this view of a matrix w/o making a copy. Thanks, -steve

5 5

Re: [Numpy-discussion] Numpy-discussion Digest, Vol 4, Issue 84
by James A. Bednar Jan. 30, 2007

Jan. 30, 2007

| Date: Mon, 29 Jan 2007 15:55:06 -0700 | From: Travis Oliphant <oliphant(a)ee.byu.edu> | | James A. Bednar wrote: | | >Hi, | > | >Does anyone know whether it is possible to pickle and unpickle numpy | >ufuncs? | > | Not directly. Ufuncs are a built-in type and do not have the required | __reduce__ method needed to be pickleable. It could be added, but | hasn't been. Thanks for the quick reply! Please consider our request that it be added. Meanwhile, we'll do a workaround. We just wanted to make sure that we were not overlooking some obvious way we were supposed to be pickling them, such as through some special pickler hidden somewhere. :-) | > I can't find anything about that on scipy.org or the mailing | >list archives. I have several important pieces of code that accept a | >numpy ufunc as an argument and later apply it to some data, while | >keeping a copy of the ufunc in an attribute. So far I have been able | >to pickle this code only by doing some very ugly hacks and | >workarounds. | | Is storing the name considered an ugly hack? Yes -- see below. | >Here's an example that illustrates the problem: | > | > | >I have a class Test defined in test.py: | > | >class Test(object): | > def __init__(self,a): | > self.a = a | | Why don't you store the name of the ufunc instead: | | def __init__(self, a): | self._a = a.__name__ | | Then, whenever you are going to use the ufunc you do | | import numpy | func = getattr(numpy,self._a) | | Then, pickle should work. For numpy ufuncs alone this would work, but in general we want to support any ufunc-like operator, including ones that our API users define, not just those defined in the numpy namespace. Specifically, we have numerous classes that have a user-selectable "operator" attribute that is *typically* set to a numpy ufunc like add or max. But a user can pass anything he or she wants as an operator, such as return_first(), where that's defined as: class return_first(object): @staticmethod def reduce(x): return x[0] Anything that supports reduce() (as ufuncs do) will work for our purposes, but if we put in specific hacks for numpy, then our users will be limited to only the numpy ufuncs. Of course, we can add more hacks to limit the hacks to being applied only for ufuncs, but we've done something vaguely like this and it's pretty complicated. We just want to be able to treat ufuncs like any other function object in Python... | Alternatively you can write your own __reduce__ function for the Test | class. We can do that too, but we actually store such operators in numerous locations in different classes, so that too turns out to be complicated. | Direct pickling of ufuncs is not a trivial issue as ufuncs contain | code-pointers at their core which cannot really be pickled. It's | not a simple problem to over-come in general. We could store the | name of the ufunc, but for user-defined ufuncs these might come | from a different package and which package the ufunc lives under is | not stored as part of the ufunc. >From a technical perspective it's worked fine for us to simply pickle the name and then reconstruct the ufunc from the name when unpickling. It's just the practical side of how to do that relatively cleanly and without having to explain it all to our API users that's been a problem. And Robert's suggestion below looks like it will help clean that up. | From: Robert Kern <robert.kern(a)gmail.com> | Subject: Re: [Numpy-discussion] Pickling ufuncs? | | Travis Oliphant wrote: | > Why don't you store the name of the ufunc instead: | | Or you can register pickler/unpickler functions for ufuncs: | | | In [24]: import copy_reg | | In [25]: import numpy | | In [26]: def ufunc_pickler(ufunc): | return ufunc.__name__ | ....: | | In [28]: def ufunc_unpickler(name): | import numpy | return getattr(numpy, name) | ....: | | In [31]: copy_reg.pickle(numpy.ufunc, ufunc_pickler, ufunc_unpickler) | | In [32]: import cPickle | | In [33]: cPickle.dumps(numpy.add) | Out[33]: 'cnumpy.core.umath\nadd\np1\n.' | | In [34]: cPickle.loads(cPickle.dumps(numpy.add)) | Out[34]: <ufunc 'add'> | | | Note that this is a hack. It won't work for the ufuncs in scipy.special, for | example. That sounds like a significantly cleaner hack than the ones we've been doing, so I think we'll try that out. Thanks! (I think we actually tried this years ago with Numeric, but from the above example it sounds worth trying again in numpy.) | We should look into including __module__ information in ufuncs. This is how | regular functions are pickled. I vote for that! Thanks, Jim

1 0

python's random.random() faster than numpy.random.rand() ???
by Mark P. Miller Jan. 29, 2007

Jan. 29, 2007

Thanks to all that responded. Yes...I need to look into pre-generating all of the random numbers that may be needed. It will not be trivial in the case of this particular application I am implementing, but is absolutely worth considering. Robert Kern's response was more along the lines of what I was trying to convey though. Does that particular bit of code require optimization? Thanks, -Mark -- Dr. Mark P. Miller Department of Biology 5305 Old Main Hill Utah State University Logan, UT 84322-5305 USA ><><><><><><><><><><><><><><><>< http://www.biology.usu.edu/people/facultyinfo.asp?username=mpmbio http://www.marksgeneticsoftware.net

2 1

Newbee question - looking to use Numpy to solve permutation problem
by Chris Toczycki Jan. 28, 2007

Jan. 28, 2007

I am looking to find a solution to a problem and believe that Numpy may help. I'd appreciate some advice and/or direction and I'll pursue from there. The problem is that I am looking for a most efficient solution where I have to select 25 solutions where the highest pair-covariance value is the lowest of all possible 25 solution combinations. There are possible 95 solutions to choose from and I have a 95 x 95 matrix of pair-covariance values. Thanks in advance. Chris

2 2