NumPy-Discussion August 2014

numpy-discussion@python.org

55 participants
47 discussions

ANN: DistArray 0.5 release
by Kurt Smith 11 Aug '14

11 Aug '14

=============================================== DistArray 0.5 release =============================================== **Mailing list:** distarray(a)googlegroups.com **Documentation:** http://distarray.readthedocs.org **License:** Three-clause BSD **Python versions:** 2.7, 3.3, and 3.4 **OS support:** \*nix and Mac OS X What is DistArray? ------------------ DistArray aims to bring the ease-of-use of NumPy to data-parallel high-performance computing. It provides distributed multi-dimensional NumPy arrays, distributed ufuncs, and distributed IO capabilities. It can efficiently interoperate with external distributed libraries like Trilinos. DistArray works with NumPy and builds on top of it in a flexible and natural way. 0.5 Release ----------- Noteworthy improvements in this release include: * closer alignment with NumPy's API, * support for Python 3.4 (existing support for Python 2.7 and 3.3), * a performance-oriented MPI-only mode for deployment on clusters and supercomputers, * a way to register user-defined functions to be callable locally on worker processes, * more consistent naming of sub-packages, * testing with MPICH2 (already tested against OpenMPI), * improved and expanded examples, * installed version testable via ``distarray.test()``, and * performance and scaling improvements. With this release, DistArray ready for real-world testing and deployment. The project is still evolving rapidly and we appreciate the continued input from the larger scientific-Python community. Existing features ----------------- DistArray: * supports NumPy-like slicing, reductions, and ufuncs on distributed multidimensional arrays; * has a client-engine process design -- data resides on the worker processes, commands are initiated from master; * allows full control over what is executed on the worker processes and integrates transparently with the master process; * allows direct communication between workers, bypassing the master process for scalability; * integrates with IPython.parallel for interactive creation and exploration of distributed data; * supports distributed ufuncs (currently without broadcasting); * builds on and leverages MPI via MPI4Py in a transparent and user-friendly way; * has basic support for unstructured arrays; * supports user-controllable array distributions across workers (block, cyclic, block-cyclic, and unstructured) on a per-axis basis; * has a straightforward API to control how an array is distributed; * has basic plotting support for visualization of array distributions; * separates the array’s distribution from the array’s data -- useful for slicing, reductions, redistribution, broadcasting, and other operations; * implements distributed random arrays; * supports ``.npy``-like flat-file IO and hdf5 parallel IO (via ``h5py``); leverages MPI-based IO parallelism in an easy-to-use and transparent way; and * supports the distributed array protocol [protocol]_, which allows independently developed parallel libraries to share distributed arrays without copying, analogous to the PEP-3118 new buffer protocol. Planned features and roadmap ---------------------------- Near-term features and improvements include: * array re-distribution capabilities; * lazy evaluation and deferred computation for latency hiding; * interoperation with Trilinos [Trilinos]_; and * distributed broadcasting support. The longer-term roadmap includes: * Integration with other packages [petsc]_ that subscribe to the distributed array protocol [protocol]_; * Distributed fancy indexing; * Out-of-core computations; * Support for distributed sorting and other non-trivial distributed algorithms; and * End-user control over communication and temporary array creation, and other performance aspects of distributed computations. History and funding ------------------- Brian Granger started DistArray as a NASA-funded SBIR project in 2008. Enthought picked it up as part of a DOE Phase II SBIR [SBIR]_ to provide a generally useful distributed array package. It builds on NumPy, MPI, MPI4Py, IPython, IPython.parallel, and interfaces with the Trilinos suite of distributed HPC solvers (via PyTrilinos [Trilinos]_). This material is based upon work supported by the Department of Energy under Award Number DE-SC0007699. This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. .. [protocol] http://distributed-array-protocol.readthedocs.org/en/rel-0.10.0/ .. [Trilinos] http://trilinos.org/ .. [petsc] http://www.mcs.anl.gov/petsc/ .. [SBIR] http://www.sbir.gov/sbirsearch/detail/410257

1 0

Help - numpy / scipy binary compatibility
by Matthew Brett 11 Aug '14

11 Aug '14

Hi, I would be very happy of some help trying to work out a numpy package binary incompatibility. I'm trying to work out what's happening for this ticket: https://github.com/scipy/scipy/issues/3863 which I summarized at the end: https://github.com/scipy/scipy/issues/3863#issuecomment-51669861 but basically, we're getting these errors: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility I now realize I am lost in the world of numpy / scipy etc binary compatibility, I'd really like some advice. In this case numpy == 1.8.1 scipy == 0.14.0 - compiled against numpy 1.5.1 scikit-learn == 0.15.1 compiled against numpy 1.6.0 Can y'all see any potential problem with those dependencies in binary builds? The relevant scipy Cython c files seem to guard against raising this error by doing not-strict checks of the e.g. numpy dtype, so I am confused how these errors come about. Can anyone give any pointers? Cheers, Matthew

4 5

Calculation of a hessian
by Kiko 11 Aug '14

11 Aug '14

Hi all, I am trying to calculate a Hessian. I am using numdifftools for this ( https://pypi.python.org/pypi/Numdifftools). My question is, is it possible to make it using pure numpy?. The actual code is like this: *import numdifftools as nd* *import numpy as np* *def log_likelihood(params):* * sum1 = 0; sum2 = 0* * mu = params[0]; sigma = params[1]; xi = params[2]* * for z in data:* * x = 1 + xi * ((z-mu)/sigma)* * sum1 += np.log(x)* * sum2 += x**(-1.0/xi)* * return -((-len(data) * np.log(sigma)) - (1 + 1/xi)*sum1 - sum2) # negated so we can use 'minimum'* *kk = nd.Hessian(log_likelihood)* Thanks in advance.

3 4

ANN: NumPy 1.8.2 bugfix release
by Julian Taylor 10 Aug '14

10 Aug '14

Hello, I am pleased to announce the release of NumPy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4722: Fix seg fault converting empty string to object * gh-4613: Fix lack of NULL check in array_richcompare * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 The source distributions have been uploaded to PyPI. The Windows installers, documentation and release notes can be found at: https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ Cheers, Julian Taylor

2 1

OpenBLAS and dotblas
by Charles R Harris 09 Aug '14

09 Aug '14

Hi All, It looks like numpy dot only uses BLAS if ATLAS is present, see numpy/core/setup.py. Has anyone done the mods needed to use OpenBLAS? What is the current status of using OpenBLAS with numpy? Also, I'm thinking of moving linalg/lapack_lite/blas_lite.c down into the core. Thoughts? Chuck

3 5

Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas
by Matti Picus 09 Aug '14

09 Aug '14

Hi. I am working on numpy in pypy. It would be much more challenging for me if you merged more code into the core of numpy, that means even more must be duplicated when using a different underlying ndarray implementation. If you are thinking of touching linalg/lapack_lite/blas_lite, I would prefer a simpler model that uses ndarray as a simple storage container, does not use cpython-specifc capi extentions with reference counting, and interfaces with python via cffi[0]. Matti [0] https://pypi.python.org/pypi/cffi/ On 8/08/2014 8:00 PM, numpy-discussion-request(a)scipy.org wrote: > Message: 1 > Date: Thu, 7 Aug 2014 20:32:21 -0600 > From: Charles R Harris <charlesr.harris(a)gmail.com> > Subject: [Numpy-discussion] OpenBLAS and dotblas > To: numpy-discussion <numpy-discussion(a)scipy.org> > Message-ID: > <CAB6mnxJOON2tn2-YP8wsBwyqRhNOQEe715Tek6rGGOVTuazyFw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > It looks like numpy dot only uses BLAS if ATLAS is present, see > numpy/core/setup.py. Has anyone done the mods needed to use OpenBLAS? What > is the current status of using OpenBLAS with numpy? > > Also, I'm thinking of moving linalg/lapack_lite/blas_lite.c down into the > core. Thoughts? > > Chuck >

3 2

ANN: NumPy 1.8.2 release candidate
by Julian Taylor 07 Aug '14

07 Aug '14

Hello, I am pleased to announce the first release candidate for numpy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ If no regressions show up the final release is planned this weekend. The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4613: Fix lack of NULL check in array_richcompare * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ Cheers, Julian Taylor

5 6

Preliminary thoughts on implementing __matmul__
by Charles R Harris 07 Aug '14

07 Aug '14

Hi All, I've been looking to implement the "@" operator from Python 3.5. Looking at the current implementation of the dot function, it only uses a vector inner product, which is either that defined in arraytypes.c.src or a version using cblas defined in _dotblas for the float, cfloat, double, cdouble types. I note that the versions defined in arraytypes.c.src include all the numeric types plus boolean, datetime, timedelta, and object. I'm not clear why datetime and timedelta should have dot products, except perhaps for scalar multiplication. The boolean version has the advantage that it can short circuit. I also note that all the operations proposed for "@" can easily be done with einsum except for objects. So I'm wondering if one easy way to implement the functions is to extend einsum to work with objects and make it use blas when available. Another thing that may be worth looking into would be some way to multiply by the complex conjugate, as that is easy to implement at the low level. I'd welcome any thoughts as to how that might be done. Anyway, I'm just looking for a discussion and ideas here. Any input is welcome. Chuck

8 17

Inverted indices
by Nicolas P. Rougier 07 Aug '14

07 Aug '14

Hi, I've a small problem for which I cannot find a solution and I'm quite sure there is an obvious one: I've an array Z (any dtype) with some data. I've a (sorted) array I (of integer, same size as Z) that tells me the index of Z[i] (if necessary, the index can be stored in Z). Now, I have an arbitrary sequence S of indices (in the sense of I), how do I build the corresponding data ? Here is a small example: Z = [(0,0), (1,1), (2,2), (3,3), (4,4)) I = [0, 20, 23, 24, 37] S = [ 20,20,0,24] -> Result should be [(1,1), (1,1), (0,0),(3,3)] S = [15,15] -> Wrong (15 not in I) but ideally, I would like this to be converted to [(0,0), (0,0)] Any idea ? Nicolas

3 5

How to give feedback to github
by Charles R Harris 07 Aug '14

07 Aug '14

Does anyone know how to complain about features to github? The new author selection list for PRs is practically useless as 1) it only lists authors belonging to the project and 2) it doesn't list the number of PRs for each author. The old list was far more useful. Chuck

2 1

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion August 2014