Mailman 3 April 2006 - NumPy-Discussion

Matrix and var method
by Zachary Pincus April 22, 2006

April 22, 2006

Hi folks, I just ran across an error with numpy.matrix types: the var() method does not seem to work! (I've tried all sorts of permutations on the matrix shape, and the axis parameter to var; nothing works.) Perhaps this has already been fixed -- I haven't updated my numpy in a week or so. If so, sorry; if not, I hope this helps. Zach In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.7.2335' In [3]: numpy.matrix([[1,2,3], [1,2,3]]).var() ------------------------------------------------------------------------ --- exceptions.ValueError Traceback (most recent call last) /Users/zpincus/<ipython console> /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/defmatrix.py in __mul__(self, other) 147 if isinstance(other, N.ndarray) or N.isscalar(other) or \ 148 not hasattr(other, '__rmul__'): --> 149 return N.dot(self, other) 150 else: 151 return NotImplemented ValueError: matrices are not aligned In [4]: numpy.array([[1,2,3], [1,2,3]]).var() Out[4]: 0.80000000000000004

1 0

seterr changes
by Tim Hochberg April 22, 2006

April 22, 2006

Hi Travis et al, I started looking at your seterr changes. I stared at yours for a while then I stared at mine for a while. Then I decided that mine wouldn't work right in the presence of threads. Then I decided that yours wouldn't work right in the presence of threads either. Specifically, it looks like ufunc_update_use_defaults isn't going to work. I think I know how to fix that, but I'm not sure that it's worth the trouble since I also did some benchmarking and it appears that the benefit of special casing is minimal. I looked at six cases: small (len-1), medium (len-1e4) and large (len-1e6) arrays with error checking on and error checking off. For medium and large arrays, I could discern no difference at all. For small arrays, there may be some difference, but it appears to be less than 5%. I'm not sure it's worth working through a bunch of finicky thread stuff to get just 5% back. If these benchmark numbers hold up I'd be inclined to rip out the use_default support since it's complicated enough that I know we'll end up chasing a few evil thread related bugs down through it. I'll include the benchmarking code below. If people could (a) look it over and confirm that I'm not doing something bogus and (b) try it on some different platforms and see if they see a more signifigant difference, I'd appreciate it. I'm also curious about the seterr interface. It returns ufunc_values_obj. I'm wasn't sure how one is supposed to pass that back in to seterr, so I modified seterr to instead return a dictionary. I also modified it so that the seterr function itself has no defaults (or rather they're all None). Instead, any unspecified values are taken from the current error state. Thus seterr(divide="warn") changes only the divide state, leaving the other entries alone. Regards, -tim if True: from timeit import Timer setup = """ import numpy numpy.seterr(divide="%s") a = numpy.zeros([%s], dtype=float) """ for size in [1, 10000, 1000000]: for i in range(3): for state in ['ignore', 'warn']: reps = min(100000000 / size, 100000) timer = Timer("a * a", setup % (state, size)) print "%s|%s =>" % (state, size), timer.timeit(reps) print print

4 3

Dumping record arrays
by Pierre GM April 22, 2006

April 22, 2006

Folks, I'd like to dump/pickle some record arrays. The pickling works, the unpickling raises a ValueError (on my version of numpy 0.9.6). (cf below). Is this already corrected in the svn version ? Thx ########################################################################### # x1 = array([21,32,14]) x2 = array(['my','first','name']) x3 = array([3.1, 4.5, 6.2]) r = rec.fromarrays([x1,x2,x3], names='id, word, number') r.dump('dumper') rb=load('dumper') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/backtopop/Work/workspace-python/pyflows/src/<ipython console> /usr/lib64/python2.4/site-packages/numpy/core/numeric.py in load(file) 331 if isinstance(file, type("")): 332 file = _file(file,"rb") --> 333 return _cload(file) 334 335 # These are all essentially abbreviations /usr/lib64/python2.4/site-packages/numpy/core/_internal.py in _reconstruct(subtype, shape, dtype) 251 252 def _reconstruct(subtype, shape, dtype): --> 253 return ndarray.__new__(subtype, shape, dtype) 254 255 ValueError: ('data-type with unspecified variable length', <function _reconstruct at 0x2aaaafcf1578>, (<class 'numpy.core.records.recarray'>, (0,), 'V'))

3 2

'append' array method request.
by Robert Hetland April 21, 2006

April 21, 2006

I find myself writing things like x = []; y = []; t = [] for line in open(filename).readlines(): xstr, ystr, tstr = line.split() x.append(float(xstr)) y.append(float(ystr)_ t.append(dateutil.parser.parse(tstr)) # or something similar x = asarray(x) y = asarray(y) t = asarray(t) I think it would be nice to be able to create empty arrays, and append the values onto the end as I loop through the file without creating the intermediate list. Is this reasonable? Is there a way to do this with existing methods or functions that I am missing? Is there a better way altogether? -Rob. ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland(a)tamu.edu, w: http://pong.tamu.edu

4 3

Source release of 0.9.6 on sourceforge is wrong
by cookedm＠physics.mcmaster.ca April 21, 2006

April 21, 2006

Travis, Looks like you uploaded the bdist .tar.gz of NumPy 0.9.6 to sourceforge, instead of the sdist. The one there isn't the source, it's a binary distribution of a 32-bit Linux compile. It's been over a month, with 2684 downloads, and I can't find a mention that anybody's noticed this before... Have we silently lost people who think we're on crack, or are there 2684 people who haven't looked at what they got? [On a another note, the download URL on PyPi won't work with setuptools; I've fixed the setup.py in svn to use the correct one, but if you could fix it on PyPi and set it to http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 then people can use easy_install to install numpy.] -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm(a)physics.mcmaster.ca

4 4

six-pack
by Rosalia Oneal April 20, 2006

April 20, 2006

1 0

Bypassing a[2].item()?
by Ryan Gutenkunst April 20, 2006

April 20, 2006

Hi all, I'm porting some code from old scipy to new scipy, and I've run into a rather large performance problem. The heart of the code is integrating a system of nonlinear differential equations using odeint. The function that dominates the time to run calculates the right hand side, given a current state x. (len(x) ~ 50.) Abstracted, the function looks like: def rhs(x) output = scipy.zeros(10, scipy.Float) a = x[0] b = x[1] ... output[0] = a/b + c*sqrt(d)... output[1] = b-a + 2*b... ... return output (I copy the elements of the current state to local variables to avoid the cost of repeatedly calling x.__getitem__, and to make the resulting equations easier to read.) When using numpy, a and b are now array scalars and the arithmetic is much slower, resulting in about a factor of 10 increase in runtimes from those using Numeric. I've tried doing: a = x[0].item(), which allows the arimetic be done on pure scalars. This is a little faster, but still results in a factor of 3 increase in runtime from old scipy. I imagine the slowdown comes from having to call __getitem__() followed by item() So questions: 1) I haven't followed the details of the array scalar discussions. Is it anticipated that array scalar arithmetic will eventually be as fast as arithmetic in native python types? 2) If not, is it possible to get a "pure" scalar directly from an array in one function call? Thanks for any help, Ryan -- Ryan Gutenkunst | Cornell LASSP | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)227-7914 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/

3 4

Performance problems with strided arrays in NumPy
by faltet＠xot.carabos.com April 20, 2006

April 20, 2006

Hi, I'm seeing some slowness in NumPy when dealing with strided arrays. numarray is dealing better with these situations, so I guess that something could be done in NumPy about this. Below are the situations that I've found up to now (maybe there are others). For the timings, I've used numpy 0.9.7.2278 and numarray 1.5.1. It seems that NumPy copy() method is almost 3x slower than in numarray: In [105]: npcopy=timeit.Timer('b=a.copy()','import numpy as np;a=np.arange(1000000,dtype="Float64")[::10]') In [106]: npcopy.repeat(3,10) Out[106]: [0.171913146972656, 0.175906896591186, 0.171195983886718] In [107]: nacopy=timeit.Timer('b=a.copy()','import numarray as np;a=np.arange(1000000,type="Float64")[::10]') In [108]: nacopy.repeat(3,10) Out[108]: [0.065090894699096, 0.0630550384521484, 0.0626609325408935] However, a copy without strides performs similarly in both packages In [127]: npcopy2=timeit.Timer('b=a.copy()','import numpy as np;a=np.arange(1000000,dtype="Float64")') In [128]: npcopy2.repeat(3,10) Out[128]: [0.24657797813415527, 0.24657106399536133, 0.2464911937713623] In [129]: nacopy2=timeit.Timer('b=a.copy()','import numarray as np;a=np.arange(1000000,type="Float64")') In [130]: nacopy2.repeat(3,10) Out[130]: [0.244544982910156, 0.251885890960693, 0.2419440746307373] -------------------------------------------- where() seems more than 2x slower in NumPy than in numarray: In [136]: tnpf=timeit.Timer('np.where(a + b < 10, a, b)','import numpy as np;a=np.arange(100000,dtype="float64");b=a*2') In [137]: tnpf.repeat(3,10) Out[137]: [0.225586891174316, 0.22503495216369629, 0.224209785461425] In [138]: tnaf=timeit.Timer('np.where(a + b < 2, a, b)','import numarray as np;a=np.arange(100000,type="Float64");b=a*2') In [139]: tnaf.repeat(3,10) Out[139]: [0.108436822891235, 0.1069340705871582, 0.10654377937316895] However, for where() without parameters, NumPy performs slightly better than numarray: In [143]: tnpf2=timeit.Timer('np.where(a + b < 10)','import numpy as np;a=np.arange(100000,dtype="float64");b=a*2') In [144]: tnpf2.repeat(3,10) Out[144]: [0.0759999752044677, 0.0731539726257324, 0.073034048080444336] In [145]: tnaf2=timeit.Timer('np.where(a + b < 2)','import numarray as np;a=np.arange(100000,type="Float64");b=a*2') In [146]: tnaf2.repeat(3,10) Out[146]: [0.0890851020812988, 0.0853078365325927, 0.085799932479858398] Cheers, Francesc

3 8

LAPACK question building numpy
by Tom Denniston April 20, 2006

April 20, 2006

Is there a way to pass a command line argument to setup.py for numpy that does the equivalent of a make using the flags: -L/home/tdennist/lib -lmkl_lapack -lmkl_lapack32 -lmkl_ia32 -lmkl -lguide All i can find on the subject is a page on the scipy wiki that says to use the variable LAPACK and set it to a .a file. When I do so I get undefined symbol problems. I this is probably really obvous and documented somewhere but I haven't been able to find it. I don't really know where to look. --Tom

2 2

Summer of Code ideas
by Tim Hochberg April 19, 2006

April 19, 2006

Discussing ideas for summer of code projects seems to be all the rage right now on various other Python lists, so I though I'd throw out a few that I've had. There are several different things that could be done with numexpr including: 1. Adding broadcasting. 2. Coercing arrays a chunk at a time instead of all at once when coercion is necessary. 3. Fancier syntax. I think that some variant of the following could be made to work: with deferred_evaluation: # Converts everything in local namespace to special objects # all of these math operations are deferred a = 5 + b*32 c = a + 73 # Now all objects are restored and deferred experesions are evaluated. This might be cool or it might be useless, but it sounds fun to try. I haven't talked to David Cooke about any of these and since numexpr is really his project he should be consulted before anyone tries these. There's also some stuff to be done on the basearray front. I expect I'll have the actual basearray object together in the next couple of weeks depending on my level of busyness, but there'll be a lot of other stuff to do besides just that. My general plan it to build a toolkit around basearray that can be used to build other array packages. These packages might be lighter weight than numpy or they might be specialized in some way that's not really compatible with numpy and ndarray. There's also room for potential for experimentation with protocols / generic functions. If anyones interested I suggest you read the thread (currently dormant) on python-3000.devel on this topic. There are lots of possible applications for this in numpy including using them to implement or replace: * asarray * __array_priority__ (by making the ufuncs and thus __add__, etc overloaded functions). * __array__, __array_wrap__, etc. * all the various functions that are giving us trouble with MA. * probably a bunch of other stuff. The basic basearray toolkit I mentioned above would be a good place to experiment with stuff like this, once it exists, since in theory it will be simpler than the full numpy codebase and you don't have to worry so much about backwards compatibility. Anyway, that's a bunch of random ideas that I at least find interesting. Regards, -tim

1 0