Mailman 3 July 2014 - NumPy-Discussion

Re: [Numpy-discussion] `allclose` vs `assert_allclose`
by alex 19 Jul '14

19 Jul '14

On Fri, Jul 18, 2014 at 9:47 PM, Charles R Harris <charlesr.harris(a)gmail.com> wrote: > > > > On Fri, Jul 18, 2014 at 2:32 PM, Chris Barker <chris.barker(a)noaa.gov> wrote: >> >> On Fri, Jul 18, 2014 at 12:43 PM, Pauli Virtanen <pav(a)iki.fi> wrote: >>> >>> 18.07.2014 22:13, Chris Barker kirjoitti: >>> [clip] >>> > but an appropriate rtol would work there too. If only zero testing is >>> > needed, then atol=0 makes sense as a default. (or maybe atol=eps) >>> >>> There's plenty of room below eps, but finfo(float).tiny ~ 3e-308 (or >>> some big multiple) is also reasonable in the scale-freeness sense. >> >> >> right! brain blip -- eps is the difference between 1 and then next larger >> representable number, yes? So a long way away from smallest representable >> number. So yes, zero or [something]e-308 -- making zero seem like a good >> idea again.... >> >> is it totally ridiculous to have the default be dependent on dtype? >> float32 vs float64? >> > > Whatever the final decision is, if the defaults change we should start with > a FutureWarning. How we can make that work is uncertain, because I don't > know of any reliable way to detect if we are using the default value or if a > value was passed in. There are tricks like http://stackoverflow.com/questions/12265695, not that I'm suggesting to do that.

1 0

`allclose` vs `assert_allclose`
by Tony Yu 19 Jul '14

19 Jul '14

Is there any reason why the defaults for `allclose` and `assert_allclose` differ? This makes debugging a broken test much more difficult. More importantly, using an absolute tolerance of 0 causes failures for some common cases. For example, if two values are very close to zero, a test will fail: np.testing.assert_allclose(0, 1e-14) Git blame suggests the change was made in the following commit, but I guess that change only reverted to the original behavior. https://github.com/numpy/numpy/commit/f43223479f917e404e724e6a3df27aa701e6d… It seems like the defaults for `allclose` and `assert_allclose` should match, and an absolute tolerance of 0 is probably not ideal. I guess this is a pretty big behavioral change, but the current default for `assert_allclose` doesn't seem ideal. Thanks, -Tony

7 24

String type again.
by Charles R Harris 18 Jul '14

18 Jul '14

As previous posts have pointed out, Numpy's `S` type is currently treated as a byte string, which leads to more complicated code in python3. OTOH, the unicode type is stored as UCS4, which consumes a lot of space, especially for ascii strings. This note proposes to adapt the currently existing 'a' type letter, currently aliased to 'S', as a new fixed encoding dtype. Python 3.3 introduced two one byte internal representations for unicode strings, ascii and latin1. Ascii has the advantage that it is a subset of UTF-8, whereas latin1 has a few more symbols. Another possibility is to just make it an UTF-8 encoding, but I think this would involve more overhead as Python would need to determine the maximum character size. These are just preliminary thoughts, comments are welcome. Chuck

15 43

Short-hand array creation in `numpy.mat` style
by Daniel da Silva 18 Jul '14

18 Jul '14

The idea is that there be a short-hand for creating arrays as there is for matrices: np.mat('.2 .7 .1; .3 .5 .2; .1 .1 .9') It was suggested in GitHub issue #4817 <https://github.com/numpy/numpy/issues/4817> in light that it would be beneficial to beginners and to presenters during demonstrations. In GitHub pull request #484 <https://github.com/numpy/numpy/pull/4845>, I implemented this as the np.arr function. Does anyone have any feedback on the API details? Some examples from my implementation follow. >>> np.arr('3; 4; 5') array([[3], [4], [5]]) >>> np.arr('3; 4; 5', dtype=float) array([[ 3.], [ 4.], [ 5.]]) >>> np.arr('1 0 0; 0 1 0; 0 0 1') array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> np.arr('4, 5; 6, 7') array([[4, 5], [6, 7]])

15 37

Numpy BoF at SciPy 2014 - quick report
by Fernando Perez 18 Jul '14

18 Jul '14

Hi all, sorry for not posting earlier, post-conference InboxInfinity blues and all that... The BoF did go as planned, and it was a good discussion, mostly following the tentative agenda outlined here: https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 Various folks were kind enough to take notes during the conversation on an Etherpad instance: https://scipy2014.etherpad.mozilla.org/35 For the sake of completeness and future reference, below I'm including a copy of the notes in this email. Other than what's in the notes, my take home from the discussion is mostly that: - we probably needed a longer slot than 45 minutes to have a chance to dig in a little deeper. - it would have been more productive if a focused numpy sprint had been also planned, so that there could be more structured follow-up on the ideas that came up. It would be great to hear from others who were present at the conference. In particular, Chris Barker brought up a number of things regarding datetime and planned on following up during the sprints, but I'm not sure what ended up happening. Thanks to everyone who participated! Cheers f #### Copy of Etherpad notes as of 7/16/2014: Notes from BoF: 1:30, July 19, 2014 Working with topics on this page: https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 chuck: where do we go from here? -- what is the role of numpy now? Generalized ufuncs -- still some more to do -- (LA stuff - norms) - some ufuncs don't impliment array interface -- which are those -- sprint topic? - zeros_like, ones_like, more... (duplicate) github issue: https://github.com/numpy/numpy/issues/4862 Here's the original issue: https://github.com/numpy/numpy/issues/3602 Implementation of @ (matrix multiplication) - will be in 3.5 ~ 18months - no work started yet -- have to make sure we do it. - @@ was not added. - The PEP for numpy is well-defined. Not much thinking to be done. (Good for a sprint) Datetime: - Can it be done? -- too many calendars -- to many time scales, etc. - Can we cover most applications? - DynND -- higher abstraction -- convert to back end implimentation - Also look at what R and Julia do? - Maybe fix up the little issues in datetime64, first? - Pandas does not use numpy machinery - uses a array of objects: those objects are subclassed form datetime.datetime - does use int64, but gets unboxed on storage. - Root cause is using UTC, rather than a naive time. - Naive is not associated with a time zone. Can be interpreted in any way. - Ripping out the locale timezone on I/O would help. - More often than not, using the locale timezone is not desired. - For example, many experimental data do not attach time zones. (Or wrong timezone) - Consider laboratory time (stopwatch rather than a clock). (timedelta) - The C++ committee is standardizing this. - A key feature which is missing, is being able to choose your epoch. New DTypes - Example: quad float types. A solution for missing values? Adding units support. - Record & structured arrays play around with dtypes. Needs to be easier to use these. - Improve documentation. - How to extend to support things like labeled arrays? - This is orthogonal to dtypes. - Would rather access time column instead of 3rd column. - Would provide a better foundation for pandas. - Key is to keep inputs simple. - Finish the DataArray push? - We are very closely there. It has been sitting there for a while. - If interested, talk at sprints on July 10. Missing values? - maybe improve masked array. - give up for now. Inheriting ndarray - introduces many bugs. - should discourage this, but make it easier to work with it. Dynd - The issues discussed so far were motivation for starting dynd - for example, a pluggable type system - adding a categorical type in numpy (at Continuum) broke lots. Easier in dynd. - Commitment for dynd is to give it a numpy-like API - Both need to evolve together. - Find ways to make things more uniform (in numpy) - Dynd is more an experimental phase, changing quickly. - Can we import dynd as np? - Not a goal. More exploratory in this phase. - Adding a layer like that at a later time would be good. Not there, yet. - Do not want to repeat py2->py3 debacle. - Buffer protocol: - Supported, but dynd extends it. - As a pure C++ library, goal is to freeze once stable so systems beyond Python can depend on it as a stable interface for working with array data. Boost::Python - Nothing official from numpy for using numpy arrays in C++ - Not prioritized. - Numpy has gotten better about namespace pollution? - It kind of works already. Talk to Mike Droettboom -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

2 1

Re: [Numpy-discussion] Numpy BoF at SciPy 2014 - quick report
by Robert Lupton the Good 18 Jul '14

18 Jul '14

Having just re-read the PEP I'm concerned that this proposal leaves at least one major (?) trap for naive users, namely x = np.array([1, 10]) print X.T@x which will print 101, not [[1, 10], [10, 100]] Yes, I know why this is happening but it's still a problem -- the user said, "I'm thinking matrices" when they wrote @ but the x.T had done the "wrong" thing before the @ kicked in. And yes, a savvy user would have written x = np.ones([[1, 10]]) (but then np.dot(x, x.T) isn't a scalar). This is the way things are at present, but with the new @ syntax coming in I think we should consider fixing it. I can think of three possibilities: 1. Leave this as a trap for the unwary, and a reason for people to stick to np.matrix (np.matrix([1, 10]) behaves "correctly") 2. Make x.T a syntax error for 1-D arrays. It's a no-op and IMHO a trap. 3. Make x.T promote the shape == (2,) array to (1, 2) and return a (2, 1) array. This may be too magic, but it's my preferred solution. R > Implementation of @ (matrix multiplication) > - will be in 3.5 ~ 18months > - no work started yet -- have to make sure we do it. > - @@ was not added. > - The PEP for numpy is well-defined. Not much thinking to be done. (Good for a sprint)

6 5

problems with mailing list ?
by josef.pktd＠gmail.com 18 Jul '14

18 Jul '14

Are the problems with sending out the messages with the mailing lists? I'm getting some replies without original messages, and in some threads I don't get replies, missing part of the discussions. Josef

3 3

Mailing list slowdown (was Re: __numpy_ufunc__)
by Nathaniel Smith 18 Jul '14

18 Jul '14

On 17 Jul 2014 11:51, "Sebastian Berg" <sebastian(a)sipsolutions.net> wrote: > > On Mi, 2014-07-16 at 09:07 +0100, Nathaniel Smith wrote: > > Weirdly, I never received Chuck's original email in this thread. > > Should some list admin be informed? > > > > I send some mails yesterday and they never arrived... Not sure if it is > a problem on my side or not. I did eventually get Chuck's original message, but not until several days later. CC'ing postmaster(a)enthought.com in case they have some insight into what's going on! -n

2 1

__numpy_ufunc__
by Charles R Harris 18 Jul '14

18 Jul '14

Hi All, Julian has raised the question of including numpy_ufunc in numpy 1.9. I don't feel strongly one way or the other, but it doesn't seem to be finished yet and 1.10 might be a better place to work out the remaining problems along with the astropy folks testing possible uses. Thoughts? Chuck

5 6

Re: [Numpy-discussion] Bug in np.cross for 2D vectors
by Neil Hodgson 17 Jul '14

17 Jul '14

> Hi, > > We came across this bug while using np.cross on 3D arrays of 2D vectors. > > What version of numpy are you using? This should already be solved in numpy > master, and be part of the 1.9 release. Here's the relevant commit, > although the code has been cleaned up a bit in later ones: > https://github.com/numpy/numpy/commit/b9454f50f23516234c325490913224c3a69fb… > Jaime Yes, we are using 1.8 - sorry I should have checked! Thanks Neil

1 1