Mailman 3 April 2014 - NumPy-Discussion

ANN: numexpr 2.4 RC1
by Francesc Alted 07 Apr '14

07 Apr '14

=========================== Announcing Numexpr 2.4 RC1 =========================== Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== A new `contains()` function has been added for detecting substrings in strings. Thanks to Marcin Krol. Also, there is a new version of setup.py that allows better management of the NumPy dependency during pip installs. Thanks to Aleks Bunin. This is the first release candidate before 2.4 final would be out, so please give it a go and report back any problems you may have. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

2 2

Default builds of OpenBLAS development branch are now fork safe
by Nathaniel Smith 06 Apr '14

06 Apr '14

Hey all, Just a heads up: thanks to the tireless work of Olivier Grisel, the OpenBLAS development branch is now fork-safe when built with its default threading support. (It is still not thread-safe when built using OMP for threading and gcc, but this is not the default.) Gory details: https://github.com/xianyi/OpenBLAS/issues/294 Check it out - if it works you might want to consider lobbying your favorite distro to backport it. -n

10 61

Resolving the associativity/precedence debate for @
by Nathaniel Smith 06 Apr '14

06 Apr '14

Hi all, After 88 emails we don't have a conclusion in the other thread (see [1] for background). But we have to come to some conclusion or another if we want @ to exist :-). So I'll summarize where the discussion stands and let's see if we can find some way to resolve this. The fundamental question is whether a chain like (a @ b @ c) should be evaluated left-to-right (left-associativity) or right-to-left (right-associativity). DATA SOURCE 1: This isn't a democratic vote, but it's useful to get a sense of people's intuitions. Counting messages in the other thread, opinion seems to be pretty evenly split: == "Votes" for right-associativity == Weak-right: [2] [3] [5] Tight-right: [4] [6] Same-right: [11] == "Votes" for left-associativity == Same-left: [7] [8] [14] [15] [16] Tight-left: [9] Weak-left: [12] There's also the "grouping" option (described in [10]), but that's received very little support (just [13]). DATA SOURCE 2: Several people have suggested that performance considerations mean that right-to-left evaluation is more common in practice than left-to-right evaluation. But, if we look at actual usage in Python code, that's not what we find: when people call dot() in chains, then they're about evenly split, and actually use the left-to-right, left-associative order slightly more often than the right-to-left, right-associative order: http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069578.html DATA SOURCE 3: And if we look at other languages, then we find: == "Votes" for right-associativity == <none> == "Votes" for left-associativity == Same-left: Matlab, Julia, IDL, GAUSS Tight-left: R And Mathematica uses the "grouping" approach. ARGUMENTS: The final outcome of this is that I need to write a piece of text that says what our (at least rough) consensus is, and lays out the reasons. So long as the "vote" is so evenly split, I can't really do this. But I can imagine what the different pieces of text might look like. THE CASE FOR LEFT-ASSOCIATIVITY: If I were writing this text in favor of left-associativity, I'd point out: - "Special cases aren't special enough to break the rules". Every single operator in Python besides ** is left-associative (and ** has very compelling arguments for right associativity). @ does not have similarly compelling arguments. If we were having this debate about "*", then it'd probably be much more lopsided towards left-associativity. So sure, there's something about @ that makes right-associativity *more* appealing than for most other operators. But not *much* more appealing -- left-associativity still comes out at least slightly ahead in all of the above measures. And there are a lot of benefits to avoiding special cases -- it gives fewer rules to memorize, fewer rules to remember, etc. So @ may be a special case, but it's not special enough. - Other languages with @ operators almost overwhelmingly use the "same-left" rule, and I've never heard anyone complain about this, so clearly nothing horrible will happen if we go this way. We have no comparable experience for right-associativity. - Given left-associativity, then there's good agreement about the appropriate precedence. If we choose right-associativity then it's much less clear (which will then make things harder for experts to remember, harder for non-experts to guess, etc.). Note that one of the votes for right-associativity even preferred the "same-right" rule, which is not even technically possible... This strikes me as a nice solid case. THE CASE FOR RIGHT-ASSOCIATIVITY: If I were writing this text in favor of right-associativity, I'd point out: - Because matrix multiplication has a tight conceptual association with function application/composition, many mathematically sophisticated users have an intuition that a matrix expression like R S x proceeds from right-to-left, with first S transforming x, and then R transforming the result. This isn't universally agreed, but at the least this intuition is more common than for other operations like 2 * 3 * 4 that everyone reads as going from left-to-right. - There might be some speed argument, if people often write things like "Mat @ Mat @ vec"? But no-one has found any evidence that people actually do write such things often. - There's been discussion of how right-associativity might maybe perhaps be nice for non-matmul applications? But I can't use those arguments [17] [18]. - ...... I got nothin'. I am fine with any outcome here. (I'm actually listed under *both* tight-right and same-left in the straw poll above ;-).) I'm totally happy to go back to Guido et al and argue for right-associativity. BUT if you all want me to do that then you need to give me some better arguments to use :-). One way to do this might be to go through the ((a @ b) @ c) and (a @ (b @ c)) examples I found (the scripts are available [19], and I can help modify them to spit out more details), look at the actual code, and demonstrate that the left-to-right ((a @ b) @ c) cases are mostly ones where evaluation order doesn't matter (i.e., they could have just as well been written the other way), and the right-to-left (a @ (b @ c)) ones are ones where right-to-left really is better than left-to-right. I have no idea if this is true, and it'll require some reading of the surrounding code to figure out what the matrix shapes are, but if it *is* true then at least that'd be something solid that right-associativity advocates could point to. WHAT NOW: If seeing this summary laid out caused you to change your mind one way or the other, then please reply and say so! If you think of some way to get more data that could favor one or the other option (running some kind of usability experiment on people? [20]), then please share! If you think of some other arguments in favor of left-associativity, then please share! If you think of some other arguments in favor of right-associativity, especially ones that are based on something besides your gut feeling, then PLEASE PLEASE share! Thanks, -n [1] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069444.html [2] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069446.html [3] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069450.html [4] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069452.html [5] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069455.html [6] https://mail.python.org/pipermail/python-ideas/2014-March/027124.html [7] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069512.html [8] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069513.html [9] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069467.html [10] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069530.html [11] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069537.html [12] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069540.html [13] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069571.html [14] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069514.html [15] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069531.html [16] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069567.html [17] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069527.html [18] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069584.html [19] https://gist.github.com/njsmith/9157645 [20] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069584.html -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

12 19

numpy git master requiring cython for build
by Julian Taylor 06 Apr '14

06 Apr '14

hi, numpy.random is largely built from a cython file. Up to know numpy git included generated c sources for this one file. It is troublesome to have merge 20k line changes for one line bugfixes, so it is proposed to remove the generated sources from the master branch in this PR: https://github.com/numpy/numpy/pull/4589 Like in SciPy the sources will not be contained in git anymore but built by the regular setup.py build when required. Release source tarballs (sdist) will continue contain the generated sources so users of stable release are not required to have cython installed. If you have any objects to this please speak up soon. Cheers, Julian

1 0

ANN: NumPy 1.8.1 release
by Julian Taylor 05 Apr '14

05 Apr '14

Hello, I'm happy to announce the of Numpy 1.8.1. This is a bugfix only release supporting Python 2.6 - 2.7 and 3.2 - 3.4. More than 48 issues have been fixed, the most important issues are listed in the release notes: https://github.com/numpy/numpy/blob/maintenance/1.8.x/doc/release/1.8.1-not… Compared to the last release candidate we have fixed a regression of the 1.8 series that prevented using some gufunc based linalg functions on larger matrices on 32 bit systems. This implied a few changes in the NDIter C-API which might expose insufficient checks for error conditions in third party applications. Please check the release notes for details. Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.1 Cheers, Julian Taylor

11 36

Where to put versionadded
by Charles R Harris 04 Apr '14

04 Apr '14

Hi All, Currently there are several placements of the '.. versionadded::' directive and I'd like to settle on a proper style for consistency. There are two occasions on which it is used, first, when a new function or class is added and second, when a new keyword is added to an existing function or method. The options are as follows. *New Function* 1) Originally, the directive was added in the notes section. Notes ----- .. versionadded:: 1.5.0 2) Alternatively, it is placed after the extended summary. blah, blah ..versionadded:: 1.5.0 Between these two, I vote for 2) because the version is easily found when reading the documentation either in a terminal or rendered into HTML. *New Parameter* 1) It is placed before the parameter description newoption : int, optional .. versionadded:: 1.5.0 blah. 2) It is placed after the parameter description. newoption : int, optional blah. .. versionadded:: 1.5.0 Both of these render correctly, but the first is more compact while the second puts the version after the description where it doesn't interrupt the reading. I'm tending towards 1) on account of its compactness. Thoughts? Chuck

4 6

Standard Deviation (std): Suggested change for "ddof" default value
by Haslwanter Thomas 04 Apr '14

04 Apr '14

While most other Python applications (scipy, pandas) use for the calculation of the standard deviation the default "ddof=1" (i.e. they calculate the sample standard deviation), the Numpy implementation uses the default "ddof=0". Personally I cannot think of many applications where it would be desired to calculate the standard deviation with ddof=0. In addition, I feel that there should be consistency between standard modules such as numpy, scipy, and pandas. I am wondering if there is a good reason to stick to "ddof=0" as the default for "std", or if others would agree with my suggestion to change the default to "ddof=1"? Thomas --- Prof. (FH) PD Dr. Thomas Haslwanter School of Applied Health and Social Sciences University of Applied Sciences Upper Austria FH OÖ Studienbetriebs GmbH Garnisonstraße 21 4020 Linz/Austria Tel.: +43 (0)5 0804 -52170 Fax: +43 (0)5 0804 -52171 E-Mail: Thomas.Haslwanter(a)fh-linz.at<mailto:Thomas.Haslwanter@fh-linz.at> Web: me-research.fh-linz.at<http://work.thaslwanter.at> or work.thaslwanter.at

11 16

ANN: Scipy 0.14.0 release candidate 1
by Ralf Gommers 03 Apr '14

03 Apr '14

Hi, I'm pleased to announce the availability of the first release candidate of Scipy 0.14.0. Please try this RC and report any issues on the scipy-dev mailing list. A significant number of fixes for scipy.sparse went in after the beta release, so users of that module may want to test this release carefully. Source tarballs, binaries and the full release notes can be found at https://sourceforge.net/projects/scipy/files/scipy/0.14.0rc1/. The final release will follow in one week if no new issues are found. A big thank you to everyone who contributed to this release! Ralf

1 0

PyData SV Proposals
by Leah Silen 02 Apr '14

02 Apr '14

Be sure and get your proposals in for PyData Silicon Valley 2014 by April 6! The event brings together scientists, analysts, developers, engineers, architects and others from the Python data science community to discuss new techniques and tools for management, analytics and visualization of data. Presentation content can be at a novice, intermediate or advanced level. Talks will run 30-40 min and hands-on tutorial and workshop sessions will run 90 min. or 3 hours. If you are interested in presenting a talk, tutorial or sponsor workshop on meeting the challenges of big data using Python, please submit a SHORT abstract and bio. If you don't have time to create an abstract, feel free to submit a presentation outline. To see the type of topics presented at previous PyData events, please look at our past conference sites at pydata.org or check out the videos on vimeo.com/pydata. Submit Your Proposal Registration will be complimentary for speakers. Any questions can be directed to admin(a)pydata.org. Conference dates are May 2-4. Proposal deadline has been extended through April 6, 2014. --- Leah Silen leah(a)numfocus.org leah(a)pydata.org 512.222.5449

1 0

structured array assignments
by mbyt 02 Apr '14

02 Apr '14

Hello, I am writing due to an issue in structured array assignments. Let's consider the following example: In [31]: x = np.empty(1, dtype=[('field', 'i4', 10)]) In [32]: type(x[0]) Out[32]: numpy.void In [33]: x[0] = np.ones(10, dtype='i4').view('V40') In [34]: x Out[34]: array([([1, 1, 1, 1, 1, 1, 1, 1, 1, 1],)], dtype=[('field', '<i4', (10,))]) In [42]: type(x[0]['field']) Out[42]: numpy.ndarray In [43]: x[0]['field'] = np.zeros(10, dtype='i4') In [44]: x Out[44]: array([([0, 1, 1, 1, 1, 1, 1, 1, 1, 1],)], dtype=[('field', '<i4', (10,))]) The assignments in both (33 and 43) call the C function voidtype_ass_subscript, but the second call produces an unexpected result --- it only copies the first element. 43 will work, if a slice assignment (C function array_assign_slice) is forced by a left hand [:] (that is x[0]['field'][:]). The question now is: What is the general opinion about forcing slice assignments for ndarray types and raising otherwise? References: * https://github.com/numpy/numpy/issues/3126 * https://github.com/numpy/numpy/pull/4556 Cheers, mbyt

1 0