Mailman 3 July 2013 - NumPy-Discussion

what is data attribute of numpy.str_ ?
by josef.pktd＠gmail.com 23 Jul '13

23 Jul '13

python 3.3 I have a bug because we have a check for a .data attribute, that is not supposed to be available for a string. (Pdb) dir(data) ['T', '__abs__', '__add__', '__and__', '__array__', '__array_interface__', '__array_priority__', '__array_struct__', '__array_wrap__', '__bool__', '__class__', '__contains__', '__copy__', '__deepcopy__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__int__', '__invert__', '__iter__', '__le__', '__len__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__xor__', 'all', 'any', 'argmax', 'argmin', 'argsort', 'astype', 'base', 'byteswap', 'capitalize', 'casefold', 'center', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'count', 'cumprod', 'cumsum', 'data', 'diagonal', 'dtype', 'dump', 'dumps', 'encode', 'endswith', 'expandtabs', 'fill', 'find', 'flags', 'flat', 'flatten', 'format', 'format_map', 'getfield', 'imag', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'item', 'itemset', 'itemsize', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'max', 'mean', 'min', 'nbytes', 'ndim', 'newbyteorder', 'nonzero', 'partition', 'prod', 'ptp', 'put', 'ravel', 'real', 'repeat', 'replace', 'reshape', 'resize', 'rfind', 'rindex', 'rjust', 'round', 'rpartition', 'rsplit', 'rstrip', 'searchsorted', 'setfield', 'setflags', 'shape', 'size', 'sort', 'split', 'splitlines', 'squeeze', 'startswith', 'std', 'strides', 'strip', 'sum', 'swapaxes', 'swapcase', 'take', 'title', 'tofile', 'tolist', 'tostring', 'trace', 'translate', 'transpose', 'upper', 'var', 'view', 'zfill'] (Pdb) data '0' (Pdb) type(data) <class 'numpy.str_'> (Pdb) data.data *** TypeError: memoryview: numpy.str_ object does not have the buffer interface (Pdb) Josef

1 0

Bringing order to higher dimensional operations
by Nathaniel Smith 22 Jul '13

22 Jul '13

Hi all, I hadn't realized until Pauli just pointed it out that np.dot and the new gufuncs actually have different rules for how they handle extra axes: https://github.com/numpy/numpy/pull/3524#issuecomment-21117601 This is turning into a big mess, and it may take some hard decisions to fix it. Before I explain the problem, a bit of terminology: a "vectorized operation" is built by taking an "intrinsic operation" and putting a loop around it, to apply it many times. So for np.add for example, the intrinsic operation is scalar addition, and if you put a loop around this you get vectorized addition. The question then is, given some input arrays: which parts do you loop over, and which things you pass to the intrinsic operation? In the case of scalar operations (like classic ufuncs), this is pretty straightforward: we broadcast the input arrays together, and loop over them in parallel. So np.add(ones((2, 3, 4)), ones((3, 4))).shape == (2, 3, 4) The other obvious option is, instead of looping over the two arrays in parallel, instead find all combinations. This is what the .outer method on ufuncs does, so np.add.outer(ones((2, 3)), ones((4, 5))).shape == (2, 3, 4, 5) Now, that's for vectorized versions scalar operations. We also have a bunch of vectorized operations whose "intrinsic operation" is itself a function over multidimensional arrays. For example, reduction operations like 'sum' intrinsically take a 1-dim array and return a 0-dim (scalar), but they can be vectorized to apply to a single axis of a >1-dim array. Or matrix multiplication intrinsically takes two 2-dim arrays and returns a 2-dim array; it can be vectorized to apply to >2 dim inputs. (As shorthand I'll write "takes two 2-dim arrays and returns a 2-dim array" as "2,2->2"; so 'sum' is 1->0 and 'cumsum' is 1->1.) ----- Okay, now I can explain the problem. For vectorized multidimensional operations, we have four (!) different conventions deployed: Convention 1: Ufunc .reduce (1->0), .accumulate (1->1), .reduceat (1,1->1): These pick the 0th axis for the intrinsic axis and loop over the rest. (By default; can be modified by specifying axis.) np.add.reduce(np.ones((2, 3, 4))).shape == (3, 4) np.add.accumulate(np.ones((2, 3, 4))).shape == (2, 3, 4) Convention 2: Reduction (1->0) and accumulation (1->1) operations defined as top-level functions and ndarray methods (np.sum, ndarray.sum, np.mean, np.cumprod, etc.): These flatten the array and use the whole thing as the intrinsic axis. (By default; can be modified by specifying axis=.) np.sum(np.ones((2, 3, 4))).shape == () np.cumsum(np.ones((2, 3, 4))).shape == (24,) Convention 3: gufuncs (any->any): These take the *last* k axes for the intrinsic axes, and then broadcast and parallel-loop over the rest. Cannot currently be modified. gu_dot = np.linalg._umath_linalg.matrix_multiply # requires current master gu_dot(np.ones((2, 3, 4, 5)), np.ones((1, 3, 5, 6))).shape == (2, 3, 4, 6) (So what's happened here is that the gufunc pulled off the last two axes of each array, so the intrinsic operation is always going to be a matrix multiply of a (4, 5) array by a (5, 6) array, producing a (4, 6) array. Then it broadcast the remaining axes together: (2, 3) and (1, 3) broadcast to (2, 3), and did a parallel iteration over them: output[0, 0, :, :] is the result of dot(input1[0, 0, :, :], input2[0, 0, :, :]).) Convention 4: np.dot (2->2): this one is bizarre: np.dot(np.ones((1, 2, 10, 11)), np.ones((101, 102, 11, 12)).shape == (1, 2, 10, 101, 102, 12) So what it's done is picked the last two axes to be the intrinsic axes, just like the gufunc -- so it always does a bunch of matrix multiplies of a (10, 11) array with an (11, 12) array. But then it didn't do a ufunc-style parallel loop. Instead it did a ufunc.outer-style outer loop, in which it found all n^2 ways of matching up a matrix in the first input with a matrix in the second input, and took the dot of each. And then it packed these up into an array with a rather weird shape: first all the looping axes from the first input, then the first axis of the output matrix, then all the looping axes from the second input, and then finally the second axis of the output matrix. ----- There are plenty of general reasons to want to simplify this -- it'd make numpy easier to explain and understand, simplify the code, etc. -- but also a few more specific reasons that make it urgent: - gufuncs haven't really been used much yet, so maybe it'd be easy to change how they work now. But in the next release, everything in np.linalg will become a gufunc, so it'll become much harder to change after that. - we'd really like np.dot to become a gufunc -- in particular, in combination with Blake's work on ufunc overrides, this would allow np.dot() to work on scipy.sparse matrices. - pretty soon we'll want to turn things like 'mean' into gufuncs too, for the same reason. ----- Okay, what to do? The gufunc convention actually seems like the right one to me. This is unfortunate, because it's also the only one we could easily change :-(. But we obviously want our vectorized operations to do broadcasting by default, both for consistency with scalar ufuncs, and because it just seems to be the most useful convention. So that rules out the np.dot convention. Then given that we're broadcasting, the two options are to pick intrinsic axes from the right like gufuncs do, or to follow the ufunc.reduce convention and pick intrinsic axes from the left. But picking from the left seems confusing to me, because broadcasting is itself a right-to-left operation. This doesn't matter for .reduce and such because they only take one input, but for something like 'dot', it means you can have 1's inserted in the "middle" of the array, and then broadcast up to a higher dimension. Compare: gu_dot_leftwards(ones((10, 11, 4)), ones((11, 12, 3, 4))) -> (10, 12, 3, 4) versus gu_dot_rightwards(ones((4, 10, 11)), ones((3, 4, 11, 12))) -> (3, 4, 10, 12) To me, it's easier to figure out which axes end up where in the second case. Working from the right, we take two axes to be the intrinsic axes, then we match up the next axis (4 matches 4), then we append a 1 and match up the last axis (1 broadcasts to match 3). So: QUESTION 1: does that sound right: that in a perfect world, the current gufunc convention would be the only one, and that's what we should work towards, at least in the cases where that's possible? QUESTION 2: Assuming that's right, it would be *really nice* if we could at least get np.dot onto our new convention, for consistency with the rest of np.linalg, and to allow it to be overridden. I'm sort of terrified to touch np.dot's API, but the only cases where it would act differently is when *both* arguments have *3 or more dimensions*, and I guess there are very very few users who fall into that category. So maybe we could start raising some big FutureWarnings for this case in the next release, and eventually switch? (I'm even more terrified of trying to mess with np.sum or np.add.reduce, so I'll leave that alone for now -- maybe we're just stuck with them. And at least they do already go through the ufunc machinery.) -n

4 9

Mag Gam
by Mag Gam 21 Jul '13

21 Jul '13

http://houtwormbestrijding-houtwormbestrijding.nl/mlwoeh/ibivodpmj.ikuklorz… Mag Gam 7/21/2013 7:24:11 AM

2 1

Re: [Numpy-discussion] subtypes of ndarray and round()
by Matti Picus 20 Jul '13

20 Jul '13

1 0

subtypes of ndarray and round()
by Matti Picus 20 Jul '13

20 Jul '13

round() does not consistently preserve subtype of the ndarray, is this known behaviour or should I file a bug for it? Python 2.7.3 (default, Sep 26 2012, 21:51:14) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.version.version '1.7.0' >>> a=np.matrix(range(10)) >>> a.round(decimals=10) matrix([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]) >>> a.round(decimals=-10) array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) Matti

2 1

simpleselect v2.0
by Graeme B. Bell 19 Jul '13

19 Jul '13

Hi all, I've just released version 2.0 of simple select. In brief this is a drop-in replacement for numpy.select with the following qualities: - Faster! (benchmarks 2-5x faster than numpy.select depending on use case and faster than v1.0 simpleselect) - Full broadcasting. - All bugs in numpy.select fixed, tested. - Better documented code. - Improvements to the test harness. I'm also submitting this as a pull request to the main numpy distribution, since it now covers all the functionality of numpy.select, but is faster and fixed long-standing bugs. I've tested with the numpy runtests.py as well as a test harness and unit tests of my own. Hope it's useful. Drop me a mail if you have any feedback. Have a nice weekend all, Graeme Bell

1 1

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 82, Issue 34
by Colin J. Williams 19 Jul '13

19 Jul '13

1 0

the mean, var, std of non-arrays
by Yaroslav Halchenko 19 Jul '13

19 Jul '13

Hi everyone, Some of my elderly code stopped working upon upgrades of numpy and upcoming pandas: https://github.com/pydata/pandas/issues/4290 so I have looked at the code of 2481 def mean(a, axis=None, dtype=None, out=None, keepdims=False): 2482 """ ... 2489 Parameters 2490 ---------- 2491 a : array_like 2492 Array containing numbers whose mean is desired. If `a` is not an 2493 array, a conversion is attempted. ... 2555 """ 2556 if type(a) is not mu.ndarray: 2557 try: 2558 mean = a.mean 2559 return mean(axis=axis, dtype=dtype, out=out) 2560 except AttributeError: 2561 pass 2562 2563 return _methods._mean(a, axis=axis, dtype=dtype, 2564 out=out, keepdims=keepdims) here 'array_like'ness is checked by a having mean function. Then it is assumed that it has the same definition as ndarray, including dtype keyword argument. Not sure anyways if my direct numpy.mean application to pandas DataFrame is "kosher" -- initially I just assumed that any argument is asanyarray'ed first -- but I think here catching TypeError for those incompatible .mean's would not hurt either. What do you think? Similar logic applies to mean cousins (var, std, ...?) decorated around _methods implementations. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

3 4

User Guide
by Colin J. Williams 19 Jul '13

19 Jul '13

3 2

Mixed Python + C debugging support in Visual Studio
by Shahrokh Mortazavi 18 Jul '13

18 Jul '13

Hi folks, 1st time poster - apologies if I'm breaking any protocols... We were told that this would be a good alias to announce this on: a few Python & OSS enthusiasts and Microsoft have created a plug-in for Visual Studio that enables Python <-> C/C++ debugging. You may find this useful for debugging your extension modules. A quick video overview of the mixed mode debugging feature: http://www.youtube.com/watch?v=wvJaKQ94lBY&hd=1 (HD) Documentation: https://pytools.codeplex.com/wikipage?title=Mixed-mode%20debugging Python Tools for Visual Studio is free (and OSS): http://pytools.codeplex.com Cheers, s

1 0