Mailman 3 July 2014 - NumPy-Discussion

`allclose` vs `assert_allclose`
by Tony Yu 19 Jul '14

19 Jul '14

Is there any reason why the defaults for `allclose` and `assert_allclose` differ? This makes debugging a broken test much more difficult. More importantly, using an absolute tolerance of 0 causes failures for some common cases. For example, if two values are very close to zero, a test will fail: np.testing.assert_allclose(0, 1e-14) Git blame suggests the change was made in the following commit, but I guess that change only reverted to the original behavior. https://github.com/numpy/numpy/commit/f43223479f917e404e724e6a3df27aa701e6d… It seems like the defaults for `allclose` and `assert_allclose` should match, and an absolute tolerance of 0 is probably not ideal. I guess this is a pretty big behavioral change, but the current default for `assert_allclose` doesn't seem ideal. Thanks, -Tony

7 24

String type again.
by Charles R Harris 18 Jul '14

18 Jul '14

As previous posts have pointed out, Numpy's `S` type is currently treated as a byte string, which leads to more complicated code in python3. OTOH, the unicode type is stored as UCS4, which consumes a lot of space, especially for ascii strings. This note proposes to adapt the currently existing 'a' type letter, currently aliased to 'S', as a new fixed encoding dtype. Python 3.3 introduced two one byte internal representations for unicode strings, ascii and latin1. Ascii has the advantage that it is a subset of UTF-8, whereas latin1 has a few more symbols. Another possibility is to just make it an UTF-8 encoding, but I think this would involve more overhead as Python would need to determine the maximum character size. These are just preliminary thoughts, comments are welcome. Chuck

15 43

Short-hand array creation in `numpy.mat` style
by Daniel da Silva 18 Jul '14

18 Jul '14

The idea is that there be a short-hand for creating arrays as there is for matrices: np.mat('.2 .7 .1; .3 .5 .2; .1 .1 .9') It was suggested in GitHub issue #4817 <https://github.com/numpy/numpy/issues/4817> in light that it would be beneficial to beginners and to presenters during demonstrations. In GitHub pull request #484 <https://github.com/numpy/numpy/pull/4845>, I implemented this as the np.arr function. Does anyone have any feedback on the API details? Some examples from my implementation follow. >>> np.arr('3; 4; 5') array([[3], [4], [5]]) >>> np.arr('3; 4; 5', dtype=float) array([[ 3.], [ 4.], [ 5.]]) >>> np.arr('1 0 0; 0 1 0; 0 0 1') array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> np.arr('4, 5; 6, 7') array([[4, 5], [6, 7]])

15 37

Numpy BoF at SciPy 2014 - quick report
by Fernando Perez 18 Jul '14

18 Jul '14

Hi all, sorry for not posting earlier, post-conference InboxInfinity blues and all that... The BoF did go as planned, and it was a good discussion, mostly following the tentative agenda outlined here: https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 Various folks were kind enough to take notes during the conversation on an Etherpad instance: https://scipy2014.etherpad.mozilla.org/35 For the sake of completeness and future reference, below I'm including a copy of the notes in this email. Other than what's in the notes, my take home from the discussion is mostly that: - we probably needed a longer slot than 45 minutes to have a chance to dig in a little deeper. - it would have been more productive if a focused numpy sprint had been also planned, so that there could be more structured follow-up on the ideas that came up. It would be great to hear from others who were present at the conference. In particular, Chris Barker brought up a number of things regarding datetime and planned on following up during the sprints, but I'm not sure what ended up happening. Thanks to everyone who participated! Cheers f #### Copy of Etherpad notes as of 7/16/2014: Notes from BoF: 1:30, July 19, 2014 Working with topics on this page: https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 chuck: where do we go from here? -- what is the role of numpy now? Generalized ufuncs -- still some more to do -- (LA stuff - norms) - some ufuncs don't impliment array interface -- which are those -- sprint topic? - zeros_like, ones_like, more... (duplicate) github issue: https://github.com/numpy/numpy/issues/4862 Here's the original issue: https://github.com/numpy/numpy/issues/3602 Implementation of @ (matrix multiplication) - will be in 3.5 ~ 18months - no work started yet -- have to make sure we do it. - @@ was not added. - The PEP for numpy is well-defined. Not much thinking to be done. (Good for a sprint) Datetime: - Can it be done? -- too many calendars -- to many time scales, etc. - Can we cover most applications? - DynND -- higher abstraction -- convert to back end implimentation - Also look at what R and Julia do? - Maybe fix up the little issues in datetime64, first? - Pandas does not use numpy machinery - uses a array of objects: those objects are subclassed form datetime.datetime - does use int64, but gets unboxed on storage. - Root cause is using UTC, rather than a naive time. - Naive is not associated with a time zone. Can be interpreted in any way. - Ripping out the locale timezone on I/O would help. - More often than not, using the locale timezone is not desired. - For example, many experimental data do not attach time zones. (Or wrong timezone) - Consider laboratory time (stopwatch rather than a clock). (timedelta) - The C++ committee is standardizing this. - A key feature which is missing, is being able to choose your epoch. New DTypes - Example: quad float types. A solution for missing values? Adding units support. - Record & structured arrays play around with dtypes. Needs to be easier to use these. - Improve documentation. - How to extend to support things like labeled arrays? - This is orthogonal to dtypes. - Would rather access time column instead of 3rd column. - Would provide a better foundation for pandas. - Key is to keep inputs simple. - Finish the DataArray push? - We are very closely there. It has been sitting there for a while. - If interested, talk at sprints on July 10. Missing values? - maybe improve masked array. - give up for now. Inheriting ndarray - introduces many bugs. - should discourage this, but make it easier to work with it. Dynd - The issues discussed so far were motivation for starting dynd - for example, a pluggable type system - adding a categorical type in numpy (at Continuum) broke lots. Easier in dynd. - Commitment for dynd is to give it a numpy-like API - Both need to evolve together. - Find ways to make things more uniform (in numpy) - Dynd is more an experimental phase, changing quickly. - Can we import dynd as np? - Not a goal. More exploratory in this phase. - Adding a layer like that at a later time would be good. Not there, yet. - Do not want to repeat py2->py3 debacle. - Buffer protocol: - Supported, but dynd extends it. - As a pure C++ library, goal is to freeze once stable so systems beyond Python can depend on it as a stable interface for working with array data. Boost::Python - Nothing official from numpy for using numpy arrays in C++ - Not prioritized. - Numpy has gotten better about namespace pollution? - It kind of works already. Talk to Mike Droettboom -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

2 1

Re: [Numpy-discussion] Numpy BoF at SciPy 2014 - quick report
by Robert Lupton the Good 18 Jul '14

18 Jul '14

Having just re-read the PEP I'm concerned that this proposal leaves at least one major (?) trap for naive users, namely x = np.array([1, 10]) print X.T@x which will print 101, not [[1, 10], [10, 100]] Yes, I know why this is happening but it's still a problem -- the user said, "I'm thinking matrices" when they wrote @ but the x.T had done the "wrong" thing before the @ kicked in. And yes, a savvy user would have written x = np.ones([[1, 10]]) (but then np.dot(x, x.T) isn't a scalar). This is the way things are at present, but with the new @ syntax coming in I think we should consider fixing it. I can think of three possibilities: 1. Leave this as a trap for the unwary, and a reason for people to stick to np.matrix (np.matrix([1, 10]) behaves "correctly") 2. Make x.T a syntax error for 1-D arrays. It's a no-op and IMHO a trap. 3. Make x.T promote the shape == (2,) array to (1, 2) and return a (2, 1) array. This may be too magic, but it's my preferred solution. R > Implementation of @ (matrix multiplication) > - will be in 3.5 ~ 18months > - no work started yet -- have to make sure we do it. > - @@ was not added. > - The PEP for numpy is well-defined. Not much thinking to be done. (Good for a sprint)

6 5

problems with mailing list ?
by josef.pktd＠gmail.com 18 Jul '14

18 Jul '14

Are the problems with sending out the messages with the mailing lists? I'm getting some replies without original messages, and in some threads I don't get replies, missing part of the discussions. Josef

3 3

Mailing list slowdown (was Re: __numpy_ufunc__)
by Nathaniel Smith 18 Jul '14

18 Jul '14

On 17 Jul 2014 11:51, "Sebastian Berg" <sebastian(a)sipsolutions.net> wrote: > > On Mi, 2014-07-16 at 09:07 +0100, Nathaniel Smith wrote: > > Weirdly, I never received Chuck's original email in this thread. > > Should some list admin be informed? > > > > I send some mails yesterday and they never arrived... Not sure if it is > a problem on my side or not. I did eventually get Chuck's original message, but not until several days later. CC'ing postmaster(a)enthought.com in case they have some insight into what's going on! -n

2 1

__numpy_ufunc__
by Charles R Harris 18 Jul '14

18 Jul '14

Hi All, Julian has raised the question of including numpy_ufunc in numpy 1.9. I don't feel strongly one way or the other, but it doesn't seem to be finished yet and 1.10 might be a better place to work out the remaining problems along with the astropy folks testing possible uses. Thoughts? Chuck

5 6

Re: [Numpy-discussion] Bug in np.cross for 2D vectors
by Neil Hodgson 17 Jul '14

17 Jul '14

> Hi, > > We came across this bug while using np.cross on 3D arrays of 2D vectors. > > What version of numpy are you using? This should already be solved in numpy > master, and be part of the 1.9 release. Here's the relevant commit, > although the code has been cleaned up a bit in later ones: > https://github.com/numpy/numpy/commit/b9454f50f23516234c325490913224c3a69fb… > Jaime Yes, we are using 1.8 - sorry I should have checked! Thanks Neil

1 1

parallel distutils extensions build? use gcc -flto
by Julian Taylor 17 Jul '14

17 Jul '14

hi, I have been playing around a bit with gccs link time optimization feature and found that using it actually speeds up a from scratch build of numpy due to its ability to perform parallel optimization and linking. As a bonus you also should get faster binaries due to the better optimizations lto allows. As compiling with lto does require some possibly lesser know details I wanted to share it. Prerequesits are a working gcc toolchain of at least gcc-4.8 and binutils > 2.21, gcc 4.9 is better as its faster. First of all numpy checks the long double representation by compiling a file and looking at the binary, this won't work as the od -b reimplementation here does not understand lto objects, so on x86 we must short circuit that: --- a/numpy/core/setup_common.py +++ b/numpy/core/setup_common.py @@ -174,6 +174,7 @@ def check_long_double_representation(cmd): # We need to use _compile because we need the object filename src, object = cmd._compile(body, None, None, 'c') try: + return 'IEEE_DOUBLE_LE' type = long_double_representation(pyod(object)) return type finally: Next we build numpy as usual but override the compiler, linker and ar to add our custom flags. The setup.py call would look like this: CC='gcc -fno-fat-lto-objects -flto=4 -fuse-linker-plugin -O3' \ LDSHARED='gcc -fno-fat-lto-objects -flto=4 -fuse-linker-plugin -shared -O3' AR=gcc-ar \ python setup.py build_ext Some explanation: The ar override is needed as numpy builds a static library and ar needs to know about lto objects. gcc-ar does exactly that. -flto=4 the main flag tell gcc to perform link time optimizations using 4 parallel processes. -fno-fat-lto-objects tells gcc to only build lto objects, normally it builds both an lto object and a normal object for toolchain compatibilty. If our toolchain can handle lto objects this is just a waste of time and we skip it. (The flag is default in gcc-4.9 but not 4.8) -fuse-linker-plugin directs it to run its link time optimizer plugin in the linking step, the linker must support plugins, both bfd (> 2.21) and gold linker do so. This allows for more optimizations. -O3 has to be added to the linker too as thats where the optimization occurs. In general a problem with lto is that the compiler options of all steps much match the flags used for linking. If you are using c++ or gfortran you also have to override that to use lto (CXX and FF(?)) See https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html for a lot more details. For some numbers on my machine a from scratch numpy build with no caching takes 1min55s, with lto on 4 it only takes 55s. Pretty neat for a much more involved optimization process. Concerning the speed gain we get by this, I ran our benchmark suite with this build, there were no really significant gains which is somewhat expected as numpy is simple C code with most function bottlenecks already inlined. So conclusion: flto seems to work well with recent gccs and allows for faster builds using the limited distutils. While probably not useful for development where compiler caching (ccache) is of utmost importance it is still interesting for projects doing one shot uncached builds (travis like CI) and have huge objects (e.g. swig or cython) and don't want to change to proper parallel build systems like bento. PS: So far I know clang also supports lto but I never used it PPS: using NPY_SEPARATE_COMPILATION=0 crashes gcc-4.9, time for a bug report. Cheers, Julian

1 0

NumPy-Discussion July 2014