Mailman 3 November 2016 - NumPy-Discussion

Re: [Numpy-discussion] NumPy 1.12.0b1 released
by Matti Picus 18 Nov '16

18 Nov '16

Congrats to all on the release.Two questions: Is there a guide to building standard wheels for NumPy? Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and OSX64, how can I get them blessed and uploaded to PyPI? Matti On 17/11/16 07:47, numpy-discussion-request(a)scipy.org wrote: > Date: Wed, 16 Nov 2016 22:47:39 -0700 > From: Charles R Harris<charlesr.harris(a)gmail.com> > To: numpy-discussion<numpy-discussion(a)scipy.org>, SciPy Users List > <scipy-user(a)scipy.org>, SciPy Developers List<scipy-dev(a)scipy.org>, > python-announce-list(a)python.org > Subject: [Numpy-discussion] NumPy 1.12.0b1 released. > > Hi All, > > I'm pleased to annouce the release of NumPy 1.12.0b1. This release > supports Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests > submitted by 133 contributors. It is quite sizeable and rather than put the > release notes inline I've attached them as a file and they may also be > viewed at Github<https://github.com/numpy/numpy/releases/tag/v1.12.0b1>. > Zip files and tarballs may also be found the Github link. Wheels and source > archives may be downloaded from PyPI, which is the recommended method. > > This release is a large collection of fixes, enhancements, and improvements > and it is difficult to select just a few as highlights. However, the > following enhancements may be of particular interest > > - Order of operations in ``np.einsum`` now can be optimized for large > speed improvements. > - New ``signature`` argument to ``np.vectorize`` for vectorizing with > core dimensions. > - The ``keepdims`` argument was added to many functions. > - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this > is a milestone for PyPy's C-API compatibility layer. > > Thanks to all, > > Chuck

6 6

ANN: xtensor and xtensor-python
by Sylvain Corlay 18 Nov '16

18 Nov '16

Hi all, We are pleased to announce the release of the xtensor <https://github.com/QuantStack/xtensor> library and its python bindings xtensor-python <https://github.com/QuantStack/xtensor-python>, by Johan Mabille (@JohanMabille) and Sylvain Corlay (@SylvainCorlay). <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#1-xt…> 1. xtensor xtensor <https://github.com/QuantStack/xtensor> is a C++ template library for manipulating multi-dimensional array expressions. It provides - an API following the idioms of the *C++ standard library*. - an extensible expression system enabling lazy broadcasting and universal functions. - python bindings for operating inplace on numpy arrays in your C++ thanks to Python's buffer protocol and pybind. More details on lazy computing with xtensor are available below. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#2-xt…> 2. xtensor-cookiecutter Besides, xtensor <https://github.com/QuantStack/xtensor> and xtensor-python <https://github.com/QuantStack/xtensor-python>, we provide a cookiecutter template project to help extension authors get started. The xtensor-cookiecutter <https://github.com/QuantStack/xtensor-python> generates a simple project for a Python C++ extension with - a working setup.py compiling the extension module - a few example of functions making use of xtensor exposed to python, and the example of a vectorized function exposed to python. - all the boilerplate for the setup of the unit testing and generation of the HTML documentation for these examples. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#3-yo…>3. You can try it now! You can try the xtensor live on the project website <http://quantstack.net/xtensor>. The Try it Now button is powered by - The Cling C++ interpreter. - The Jupyter notebook. - The Binder project. ------------------------------ ------------------------------ <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#gett…>Getting started xtensor requires a modern C++ compiler supporting C++14. The following C+ compilers are supported: - On Windows platforms, Visual C++ 2015 Update 2, or more recent - On Unix platforms, gcc 4.9 or a recent version of Clang <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#inst…> Installation xtensor and xtensor-python are header-only libraries. We provide packages for the conda package manager. conda install -c conda-forge xtensor # installs xtensor conda install -c conda-forge xtensor-python # installs xtensor and xtensor-python <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#usage> Usage <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#basi…>Basic Usage Initialize a 2-D array and compute the sum of one of its rows and a 1-D array. #include <iostream> #include "xtensor/xarray.hpp" #include "xtensor/xio.hpp" xt::xarray<double> arr1 {{1.0, 2.0, 3.0}, {2.0, 5.0, 7.0}, {2.0, 5.0, 7.0}}; xt::xarray<double> arr2 {5.0, 6.0, 7.0}; xt::xarray<double> res = xt::make_xview(arr1, 1) + arr2; std::cout << res; Outputs: {7, 11, 14} Initialize a 1-D array and reshape it inplace. #include <iostream> #include "xtensor/xarray.hpp" #include "xtensor/xio.hpp" xt::xarray<int> arr {1, 2, 3, 4, 5, 6, 7, 8, 9}; arr.reshape({3, 3}); std::cout << arr; Outputs: {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}} <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#lazy…>Lazy Broadcasting with xtensor We can operate on arrays of different shapes of dimensions in an elementwise fashion. Broadcasting rules of xtensor are similar to those of numpy <http://www.numpy.org/> and libdynd <http://libdynd.org/>. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#broa…>Broadcasting rules In an operation involving two arrays of different dimensions, the array with the lesser dimensions is broadcast across the leading dimensions of the other. For example, if A has shape (2, 3), and B has shape (4, 2, 3), the result of a broadcasted operation with A and B has shape (4, 2, 3). (2, 3) # A (4, 2, 3) # B --------- (4, 2, 3) # Result The same rule holds for scalars, which are handled as 0-D expressions. If matched up dimensions of two input arrays are different, and one of them has size 1, it is broadcast to match the size of the other. Let's say B has the shape (4, 2, 1) in the previous example, so the broadcasting happens as follows: (2, 3) # A (4, 2, 1) # B --------- (4, 2, 3) # Result <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#univ…>Universal functions, Laziness and Vectorization With xtensor, if x, y and z are arrays of *broadcastable shapes*, the return type of an expression such as x + y * sin(z) is not an array. It is an xexpression object offering the same interface as an N-dimensional array, which does not hold the result. Values are only computed upon access or when the expression is assigned to an xarray object. This allows to operate symbolically on very large arrays and only compute the result for the indices of interest. We provide utilities to vectorize any scalar function (taking multiple scalar arguments) into a function that will perform on xexpressions, applying the lazy broadcasting rules which we just described. These functions are called *xfunction*s. They are xtensor's counterpart to numpy's universal functions. In xtensor, arithmetic operations (+, -, *, /) and all special functions are *xfunction*s. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#iter…>Iterating over xexpressions and Broadcasting Iterators All xexpressions offer two sets of functions to retrieve iterator pairs (and their const counterpart). - begin() and end() provide instances of xiterators which can be used to iterate over all the elements of the expression. The order in which elements are listed is row-major in that the index of last dimension is incremented first. - xbegin(shape) and xend(shape) are similar but take a *broadcasting shape* as an argument. Elements are iterated upon in a row-major way, but certain dimensions are repeated to match the provided shape as per the rules described above. For an expression e, e.xbegin(e.shape()) and e.begin() are equivalent. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#fixe…> Fixed-dimension *and* Dynamic dimension Two container classes implementing multi-dimensional arrays are provided: xarray and xtensor. - xarray can be reshaped dynamically to any number of dimensions. It is the container that is the most similar to numpy arrays. - xtensor has a dimension set at compilation time, which enables many optimizations. For example, shapes and strides of xtensor instances are allocated on the stack instead of the heap. xarray and xtensor container are both xexpressions and can be involved and mixed in universal functions, assigned to each other etc... Besides, two access operators are provided: - The variadic template operator() which can take multiple integral arguments or none. - And the operator[] which takes a single multi-index argument, which can be of size determined at runtime. operator[] also supports access with braced initializers. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#pyth…>Python bindings The python bindings are built upon the pybind11 <https://github.com/pybind/pybind11> library, a lightweight header-only for creating bindings between the Python and C++ programming languages. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#exam…>Example 1: Use an algorithm of the C++ library on a numpy array inplace. C++ code #include <numeric> // std::accumulate #include "pybind11/pybind11.h" // pybind11 #include "xtensor/xmath.hpp" // C++ universal functions #include "xtensor-python/pyarray.hpp" // numpy bindings double sum_of_sines(xt::pyarray<double> &m) { auto sines = xt::sin(m); // sines does not hold any value return std::accumulate(sines.begin(), sines.end(), 0.0); } PYBIND11_PLUGIN(xtensor_python_test) { pybind11::module m("xtensor_python_test", "Test module for xtensor python bindings"); m.def("sum_of_sines", sum_of_sines, "Return the sum of the sines"); return m.ptr(); } Python Code import numpy as npimport xtensor_python_test as xt a = np.arange(15).reshape(3, 5) s = xt.sum_of_sines(v) s Outputs 1.2853996391883833 <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#exam…>Example 2: Create a universal function from a C++ scalar function C++ code #include "pybind11/pybind11.h" #include "xtensor-python/pyvectorize.hpp" #include <numeric> #include <cmath> namespace py = pybind11; double scalar_func(double i, double j) { return std::sin(i) - std::cos(j); } PYBIND11_PLUGIN(xtensor_python_test) { py::module m("xtensor_python_test", "Test module for xtensor python bindings"); m.def("vectorized_func", xt::pyvectorize(scalar_func), ""); return m.ptr(); } Python Code import numpy as npimport xtensor_python_test as xt x = np.arange(15).reshape(3, 5) y = [1, 2, 3] z = xt.vectorized_func(x, y) z Outputs [[-1. , 0.30116, 1.32544, 1.13111, -0.10315], [-1.95892, -0.81971, 1.07313, 1.97935, 1.06576], [-1.54402, -1.54029, -0.12042, 1.41016, 1.64425]]

1 0

Status of NAs for integer arrays
by Dmitrii Izgurskii 18 Nov '16

18 Nov '16

Hi, I would like to know where I can follow the progress of what's being done on implementing missing values in integer arrays. Some threads I found are from 2012. Is really nothing being done on this issue?

2 1

The --compilers flag is ignored for build_ext
by Andres Guzman-Ballen 17 Nov '16

17 Nov '16

Hello Numpy community! First time poster here. I was recommended by Charles Harris to present a question I have to you guys regarding using --compiler=intelem being ignored when I use build_ext. I am trying to compile Numpy v1.11.2 that I am getting from https://pypi.python.org/packages/16/f5/b432f028134dd30cfbf6f21b8264a9938e5e…, using Python 2.7.12 on my Ubuntu 10.04 LTS Machine. I specifically want to compile this with the Intel compiler. I have sourced the compilervars.sh so the Intel compiler is in the path. This is the command that is failing: 'python setup.py build_ext --inplace --debug --force --compiler=intelem --fcompiler=intelem'. The reason it fails is because I have added compiler flags to umath's configuration extension code in numpy/core/setup.py as extra_compiler_arg and they are specifically meant for Intel. Since the compiler setup is ignored (unlike in setup.py build, where config_cc is correctly called and takes care of this), it defaults to GCC. This is a snippet of what you get for 'build_ext': running build_clib running build_src build_src building py_modules sources building library "npymath" sources creating build creating build/src.linux-x86_64-2.7 customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config This is what you get for using 'build': running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.linux-x86_64-2.7 creating build/src.linux-x86_64-2.7/numpy creating build/src.linux-x86_64-2.7/numpy/distutils building library "npymath" sources Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/icc Could not locate executable ecc customize IntelEM64TFCompiler Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/ifort customize IntelEM64TFCompiler using config I do not experience this problem with 'python setup.py build --debug --force --compiler=intelem --fcompiler=intelem'. because the configuration function is correctly called here: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/bui… (that is where I got the output from above). However, I do not see anything like this in build_ext: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/bui…. This of course could be because build_ext is doing something else below in the run command on line 109. This is most likely not the right answer, but what harm does this do? diff --git a/numpy/distutils/command/build_ext.py b/numpy/distutils/command/build_ext.py index 0fa52a2..ffa88ae 100644 --- a/numpy/distutils/command/build_ext.py +++ b/numpy/distutils/command/build_ext.py @@ -79,6 +79,8 @@ class build_ext (old_build_ext): return # Make sure that extension sources are complete. + self.run_command('config_cc') + self.run_command('config_fc') self.run_command('build_src') if self.distribution.has_c_libraries(): I am able to get build_ext to behave properly by adding this snippet, but I'm pretty sure this is the wrong approach because there seems to be code below line 109 in numpy/distutils/command/build_ext.py that sets compiler-related things up. This is the ticket I created under Numpy's issues page: https://github.com/numpy/numpy/issues/8283. Any feedback is very well appreciated and I'd be more than happy to help contribute and get this fixed for the community. I don't want to just throw a problem at you guys and expect someone else to fix it for me. Thanks! Andres

1 0

NumPy 1.12.0b1 released.
by Charles R Harris 17 Nov '16

17 Nov '16

Hi All, I'm pleased to annouce the release of NumPy 1.12.0b1. This release supports Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests submitted by 133 contributors. It is quite sizeable and rather than put the release notes inline I've attached them as a file and they may also be viewed at Github <https://github.com/numpy/numpy/releases/tag/v1.12.0b1>. Zip files and tarballs may also be found the Github link. Wheels and source archives may be downloaded from PyPI, which is the recommended method. This release is a large collection of fixes, enhancements, and improvements and it is difficult to select just a few as highlights. However, the following enhancements may be of particular interest - Order of operations in ``np.einsum`` now can be optimized for large speed improvements. - New ``signature`` argument to ``np.vectorize`` for vectorizing with core dimensions. - The ``keepdims`` argument was added to many functions. - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this is a milestone for PyPy's C-API compatibility layer. Thanks to all, Chuck

1 0

ufunc for sum of squared difference
by Matthew Harrigan 17 Nov '16

17 Nov '16

I was reading this <http://notes-on-cython.readthedocs.io/en/latest/std_dev.html> and got thinking about if a ufunc could compute the sum of squared differences in a single pass without a temporary array. The python code below demonstrates a possible approach. import numpy as np x = np.arange(10) c = 1.0 def add_square_diff(x1, x2): return x1 + (x2-c)**2 ufunc = np.frompyfunc(add_square_diff, 2, 1) print(ufunc.reduce(x) - x[0] + (x[0]-c)**2) print(np.sum(np.square(x-c))) I have (at least) 4 questions: 1. Is it possible to pass run time constants to a ufunc written in C for use in its inner loop, and if so how? 2. Is it possible to pass an initial value to reduce to avoid the clean up required for the first element? 3. Does that ufunc work, or are there special cases which cause it to fall apart? 4. Would a very specialized ufunc such as this be considered for incorporating in numpy since it would help reduce time and memory of functions already in numpy? Thank you, Matt

6 12

subclassing ndarray and keeping same ufunc behavior
by Stuart Reynolds 15 Nov '16

15 Nov '16

I'm trying to subclass an ndarray so that I can add some additional fields. When I do this however, I get new odd behavior when my object is passed to a variety of numpy functions. For example nanmin returns now return an object of the type of my new array class, whereas previously I'd get a float64. Why? Is this a bug with nanmin or my class? import numpy as np class NDArrayWithColumns(np.ndarray): def __new__(cls, obj, columns=None): obj = obj.view(cls) obj.columns = tuple(columns) return obj def __array_finalize__(self, obj): if obj is None: return self.columns = getattr(obj, 'columns', None) NAN = float("nan") r = np.array([1.,0.,1.,0.,1.,0.,1.,0.,NAN, 1., 1.])print "MIN", np.nanmin(r), type(np.nanmin(r)) gives: MIN 0.0 <type 'numpy.float64'> but >>> r = NDArrayWithColumns(r, ["a"])>>> print "MIN", np.nanmin(r), type(np.nanmin(r)) MIN 0.0 <class '__main__.NDArrayWithColumns'>>>> print r.shape # ?!(11,) Note the change in type, and also that str(np.nanmin(r)) shows 1 field, not 11 as indicated by its shape. This seems wrong. Is there a way to get my subclass to behave more like an ndarray? I realize from the docs that I can override __array_wrap__, but its not clear me how how to use it to solve this issue. Or whether its the right tool. In case you're interested, I'm subclassing because I'd like to track column names in matrices of a single type. This is pretty common wish in scikit pipelines. Structured arrays and record type arrays allow for varying type. Pandas provides this functionality, but dealing with numpy arrays is easier (and more efficient) when writing cython extensions. Also, I think the structured arrays and record types are unlikely to play nice with cython because they're more freely typed -- I want to deal exclusively with arrays of doubles. Any thoughts of how to subclass ndarray and keep original behavior in ufuncs?

3 3

Ensuring one can operate on array-like argument in place
by Pavlyk, Oleksandr 14 Nov '16

14 Nov '16

Hi, In my Cython code a function processes it argument x as follows: x_arr = PyArray_CheckFromAny( x, NULL, 0, 0, cnp.NPY_ELEMENTSTRIDES | cnp.NPY_ENSUREARRAY | cnp.NPY_NOTSWAPPED, NULL) if x_arr is not x: in_place = 1 # a copy was made, so we can work in place. The logic is of the last line turns out to be incorrect, because the input x can be a class with an array interface: class FakeArray(object): def __init__(self, data): self._data = data self.__array_interface__ = data.__array_interface__ Feeding my function FakeArray(xx), x_arr will point into the content of xx, resulting in unwarranted content overwrite of xx. I am trying to replace that condition with if x_arr is not x and cnp.PyArray_Check(x): # a copy was made, so we can work in place. in_place = 1 if cnp.PyArray_CHKFLAGS(x_arr, cnp.NPY_WRITEABLE) else 0 I am wondering if I perhaps overlooked some case. Thank you, Sasha

3 2

Re: [Numpy-discussion] Numpy 1.12.x branched
by Matti Picus 10 Nov '16

10 Nov '16

On 07/11/16 10:19, numpy-discussion-request(a)scipy.org wrote: > Date: Sun, 06 Nov 2016 17:56:12 +0100 > From: Sebastian Berg<sebastian(a)sipsolutions.net> > To:numpy-discussion@scipy.org > Subject: Re: [Numpy-discussion] Numpy 1.12.x branched > Message-ID:<1478451372.3875.5.camel(a)sipsolutions.net> > Content-Type: text/plain; charset="utf-8" > > On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote: >> >Hi All, >> > >> >Numpy 1.12.x has been branched and the 1.13 development branch is >> >open. It would be helpful if folks could review the release notes as >> >it is likely I've missed something.? I'd like to make the first beta >> >release in a couple of days. >> > > Very cool, thanks for all the hard work! > > - Sebastian > > >> >Chuck Thanks for managing this. I don't know where, but it would be nice if the release notes could mention the PyPy support - we are down to only a few failures on the test suite, the only real oustanding issue is nditer using UPDATEIFCOPY which depends on refcounting semantics to trigger the copy. Other than that PyPy + NumPy 1.12 is a working thing, we (PyPy devs) will soon try to make it work faster :). Matti

3 3

Numpy 1.12.x branched
by Charles R Harris 06 Nov '16

06 Nov '16

Hi All, Numpy 1.12.x has been branched and the 1.13 development branch is open. It would be helpful if folks could review the release notes as it is likely I've missed something. I'd like to make the first beta release in a couple of days. Chuck

2 1