Mailman 3 January 2019 - NumPy-Discussion

ENH: Added option to suppress stout/err capture in tests PR #12829
by Joseph Fox-Rabinovitz 22 Jan '19

22 Jan '19

Hi, I recently had some issues setting up gdb to fix some self-inflicted segfaults, so instead I ended up adding an option to suppress stdout/stderr capture by pytest in PR#12829. I think this is a useful feature to have in general, so I made this PR. The only problem with it is that there are no formal tests for it (I did verify that all the possible options work manually though). Regards, - Joe

1 0

DOC: Updates to nditer usage instructions PR#12828
by Joseph Fox-Rabinovitz 22 Jan '19

22 Jan '19

Hi, I have just added PR #12828, based off issue #12764 to clarify some of the documentation of `nditer`. While it contains most of the necessary material, it's still a bit of a rough draft, and I'd be happy to have some comments/advice on it. In particular, I didn't know how much to emphasize the fact that context management only became a thing for `nditer` in version 1.15.0. I'm also not sure about how much need there is to reiterate the C-style iteration methods vs the normal Python-style iteration. Regards, - Joe

1 0

numba: maps, and list of vectors
by Stuart Reynolds 17 Jan '19

17 Jan '19

I'm having great luck using with numba, but there are two problems I find difficult to solve with it. Often I want to algorithms that performs a lookup of a list/array of numbers. And this operation often comes in two flavors: (1) <index> => <int array> # datastructure.get(0) => array (2) <any int key> => <int array> # datastructure.get(999999999999) => array In numba, how to people deal with indexed lookup where the range of the data structure is a memory managed object (array/list) (1)? In numba, how to people deal with problems needing hashed lookups (2)? Are there extension libraries / ways of calling out to C/C++ libraries that can easily help. (I know Cython is a route, but I am asking about numba specifically). Thanks, - Stuart

2 1

Transcendental Functions
by Devulapalli, Raghuveer 16 Jan '19

16 Jan '19

Hello, Are Transcendental Functions SIMD vectorized in NumPy? Raghuveer

1 0

Reminder: weekly status meeting Jan 16 at 12:00 pacific time
by Matti Picus 16 Jan '19

16 Jan '19

The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. The BIDS team

1 0

py3k.os_fspath enforces inconsist rules with python3 os.fspath
by Stuart Reynolds 15 Jan '19

15 Jan '19

After a recent upgrade of numpy (1.13.1 -> 1.6.0), my code is failing where I provide unicode objects as filenames. Previously they were allowed. Now that are not, and I *must* provide a (py2) str or bytearray only. # str is OK $ python2.7 -c "from numpy.compat import py3k; print py3k.os_fspath('123')" 123 # unicode is not $ python -c "from numpy.compat import py3k; print py3k.os_fspath(u'123')" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/numpy/compat/py3k.py", line 237, in os_fspath "not " + path_type.__name__) TypeError: expected str, bytes or os.PathLike object, not unicode But this enforcement of "str, bytes or os.PathLike" comes from: https://docs.python.org/3/library/os.html where in Python 3 str is a unicode, and moreover, os.fspath allows $ python3 -c "import os; print(os.fspath(u'123'))" # unicode str 123 $ python3 -c "import os; print(os.fspath('123'))" # also unicode str 123 .... so... shouldn't py3k.os_fspath allow py2 unicode obejcts. - Stu

2 3

NumPy 1.16.0 released.
by Charles R Harris 14 Jan '19

14 Jan '19

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.0. This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are Experimental support for overriding numpy functions in downstream projects. - The matmul function is now a ufunc and can be overridden using __array_ufunc__. - Improved support for the ARM, POWER, and SPARC architectures. - Improved support for AIX and PyPy. - Improved interoperation with ctypes. - Improved support for PEP 3118. The supported Python versions are 2.7 and 3.5-3.7, support for 3.4 has been dropped. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.0/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.0>. *Contributors* A total of 113 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Alan Fontenot + * Allan Haldane * Alon Hershenhorn + * Alyssa Quek + * Andreas Nussbaumer + * Anner + * Anthony Sottile + * Antony Lee * Ayappan P + * Bas van Schaik + * C.A.M. Gerlach + * Charles Harris * Chris Billington * Christian Clauss * Christoph Gohlke * Christopher Pezley + * Daniel B Allan + * Daniel Smith * Dawid Zych + * Derek Kim + * Dima Pasechnik + * Edgar Giovanni Lepe + * Elena Mokeeva + * Elliott Sales de Andrade + * Emil Hessman + * Eric Larson * Eric Schles + * Eric Wieser * Giulio Benetti + * Guillaume Gautier + * Guo Ci * Heath Henley + * Isuru Fernando + * J. Lewis Muir + * Jack Vreeken + * Jaime Fernandez * James Bourbeau * Jeff VanOss * Jeffrey Yancey + * Jeremy Chen + * Jeremy Manning + * Jeroen Demeyer * John Darbyshire + * John Kirkham * John Zwinck * Jonas Jensen + * Joscha Reimer + * Juan Azcarreta + * Julian Taylor * Kevin Sheppard * Krzysztof Chomski + * Kyle Sunden * Lars Grüter * Lilian Besson + * MSeifert04 * Mark Harfouche * Marten van Kerkwijk * Martin Thoma * Matt Harrigan + * Matthew Bowden + * Matthew Brett * Matthias Bussonnier * Matti Picus * Max Aifer + * Michael Hirsch, Ph.D + * Michael James Jamie Schnaitter + * MichaelSaah + * Mike Toews * Minkyu Lee + * Mircea Akos Bruma + * Mircea-Akos Brumă + * Moshe Looks + * Muhammad Kasim + * Nathaniel J. Smith * Nikita Titov + * Paul Müller + * Paul van Mulbregt * Pauli Virtanen * Pierre Glaser + * Pim de Haan * Ralf Gommers * Robert Kern * Robin Aggleton + * Rohit Pandey + * Roman Yurchak + * Ryan Soklaski * Sebastian Berg * Sho Nakamura + * Simon Gibbons * Stan Seibert + * Stefan Otte * Stefan van der Walt * Stephan Hoyer * Stuart Archibald * Taylor Smith + * Tim Felgentreff + * Tim Swast + * Tim Teichmann + * Toshiki Kataoka * Travis Oliphant * Tyler Reddy * Uddeshya Singh + * Warren Weckesser * Weitang Li + * Wenjamin Petrenko + * William D. Irons * Yannick Jadoul + * Yaroslav Halchenko * Yug Khanna + * Yuji Kanagawa + * Yukun Guo + * @ankokumoyashi + * @lerbuke + Cheers, Charles Harris

1 0

Add guaranteed no-copy to array creation and reshape?
by Sebastian Berg 13 Jan '19

13 Jan '19

Hi all, In https://github.com/numpy/numpy/pull/11897 I am looking into the addition of a `copy=np.never_copy` argument to: * np.array * arr.reshape/np.reshape * arr.astype Which would cause an error to be raised when numpy cannot guarantee that the returned array is a view of the input array. The motivation is to easier avoid accidental copies of large data, or ensure that in-place manipulation will be meaningful. The copy flag API would be: * `copy=True` forces a copy * `copy=False` allows numpy to copy if necessary * `copy=np.never_copy` will error if a copy would be necessary * (almost) all other input will be deprecated. Unfortunately using `copy="never"` is tricky, because currently `np.array(..., copy="never")` behaves exactly the same as `np.array(..., copy=bool("never"))`. So that the wrong result would be given on old numpy versions and it would be a behaviour change. Some things that are a not so nice maybe: * adding/using `np.never_copy` is not very nice * Scalars need a copy and so will not be allowed * For rare array-likes numpy may not be able to guarantee no-copy, although it could happen (but should not). The history is that a long while ago I considered adding a copy flag to `reshape` so that it is possible to do `copy=np.never_copy` (or similar) to ensure that no copy is made. In these, you may want something like an assertion: ``` new_arr = arr.reshape(new_shape) assert np.may_share_memory(arr, new_arr) # Which is sometimes -- but should not be -- written as: arr.shape = new_shape # unnecessary container modification # Or: view = np.array(arr, order="F") assert np.may_share_memory(arr, new_arr) ``` but is more readable and will not cause an intermediate copy on error. So what do you think? Other variants would be to not expose this for `np.array` and probably limit `copy="never"` to the reshape method. Or just to not do it at all. Or to also accept "never" for `reshape`, although I think I would prefer to keep it in sync and wait for a few years to consider that. Best, Sebastian

9 32

Debugging NumPy development build in conda environment
by Lars Grueter 13 Jan '19

13 Jan '19

Dear devs, If this is the wrong place for this kind of question I apologize and feel free to ignore this or point me to the right place. I'm using the conda package manager for my development environment which used to work just fine with these steps: > conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage > source activate dev-numpy > conda uninstall --force numpy numpy-base > pip install -e . This would yield me a nice isolated playground and I could use > python setup.py build_ext --incplace -j 4 at anytime to rebuild the binaries. Unfortunately the last two commands stopped working out of the blue with my existing environment and fail with > RuntimeError: Broken toolchain: cannot link a simple C program (full output in [1]) on a clean repository. No recent changes to my tool chain come to mind which could help me pinpoint the problem. Because it used to work just fine I have the annoying feeling that the fix is something small and trivial and have the hope that it's immediately obvious to someone here or that you can suggest a way to debug this. In any case your help would be appreciated. I'm aware that this may be an exotic dev setup. Using a virtual environment instead of the conda one works just fine. The build script seems to use a different linear algebra library in that case: blas from my OS as opposed to mkl in the conda environment. This is not my preferred solution but it would do if nothing comes of this. Some additional facts and research I've done: - Using conda environments with asv has stopped working as well due to not finding any LA library at all. - If I'm understanding [1] correctly the build script finds the mkl library provided by the conda environment. - Trying earlier versions of NumPy shows the same behavior. I tested the tags v1.14.6, v1.15.4. - However creating a development environment for SciPy and scikit-image using this approach still works fine. - Searching the internet or looking at NumPy's build guide didn't yield me a solution. - I'm on linux (Manjaro) if that is of any relevance. - Miniconda was reinstalled in my home directory with the installation script provided through Anaconda's website. - Haven't yet tried this approach in a fresh virtual machine. I hope that I haven't overlooked something obvious. In any case thank you for your time and attention. Cheers, Lars [1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5

3 4

summation along a non-fast axis
by Keith Goodman 11 Jan '19

11 Jan '19

I remember back when a.sum(axis=0) was much slower than a.sum(axis=1) for something like a=np.ones((1000, 1000)). But now it runs in about the same time. How does numpy do it? Does numpy do something like for i in range(a.shape[0]): for j in range(x.shape[1]): result[j] += a[i, j]

2 1

NumPy-Discussion January 2019