Mailman 3 August 2014 - NumPy-Discussion

Scipy.org has been unaccessible from UK and US
by Arman Eshaghi Aug. 19, 2014

Aug. 19, 2014

Hi everyone, I apologize if this is not the right place to report this, for some reason docs.scipy.org including numpy documentation have been unavailable from yesterday. I have tried to access from servers in UK and US and the problem seems to be on scipy servers. scipy.org works fine though. Just wanted to give a heads up. Thanks Arman

1 0

Docs website down?
by Joris Van den Bossche Aug. 18, 2014

Aug. 18, 2014

It seems the docs website of numpy and scipy (http://docs.scipy.org/doc/) is down. Is anyone looking at this? There is even already a stackoverflow question about it .. Best regards, Joris

1 0

What to do about vdot?
by Charles R Harris Aug. 17, 2014

Aug. 17, 2014

Hi All, I've moved the cblas implementations of the dot and inner functions from _dotblas down into multiarray. The only reason to retain the _dotblas file at this point is the vdot function. I think that vdot belongs in the linalg module, but another option is to make it part of multiarray where it sort of complements the inner function. Opinions? Chuck

1 0

Heads-up: Guido van Rossum on mypy-style type annotations...
by Fernando Perez Aug. 15, 2014

Aug. 15, 2014

Hi folks, [x-posting to numba and numpy lists, discussion should be on the python-ideas list where the python core folks actually reside] just a quick note that Guido has proposed to adopt the mypy model for type annotations in the language: http://thread.gmane.org/gmane.comp.python.ideas/28619 This same heads-up was also sent out by Stefan Behnel to the cython ML, please feel free to pass it to other communities that might be interested in this problem and can provide feedback to python-core while this is in the discussion phase. Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

2 1

Question about OpenBLAS affinity setting.
by Charles R Harris Aug. 15, 2014

Aug. 15, 2014

Hi All, Seeking some discussion about setting the OpenBLAS affinity. Currently this is set when numeric.py is imported: try: > # disables openblas affinity setting of the main thread that limits > # python threads or processes to one core > if 'OPENBLAS_MAIN_FREE' not in os.environ: > os.environ['OPENBLAS_MAIN_FREE'] = '1' > if 'GOTOBLAS_MAIN_FREE' not in os.environ: > os.environ['GOTOBLAS_MAIN_FREE'] = '1' > from ._dotblas import dot, vdot, inner > except ImportError: > ... > Note that the affinity is set whether or not the import of _dotblas fails, which it will if cblas was not found at buildtime. This all seems a bit hinky to me. If we are always going to set the affinity, it should be in the core/__init__.py file. If the setting is moved after the import, it will still happen if, say, ATLAS cblas is present. It seems to me that there should be a better, more transparent way to do this, especially as I'm in the process of moving some of the cblas down into multiarray. One option is to make this part of the multiarray module import, but without a bit of work, that would still depend only on cblas being detected during the build process rather than OpenBLAS. Also, is GotoBLAS still a viable option? Thoughts? Chuck

1 0

Fwd: We need help working on code coverage for Cython code
by Matthew Brett Aug. 14, 2014

Aug. 14, 2014

Hi, Sorry for those of you also on the scikit-image mailing list - but here again I'm asking for help to get coverage working for Cython code. Over on another mailing list, we've hit a big problem trying to work out coverage on a large amount of Cython code. As y'all probably know, there's no automated way of checking code coverage on Cython code at the moment. The Cython developers have done some work on this [1] but it is currently stalled for lack of developer time to work on it. We'd really like to get this working, and the Cython developers have offered to help, to get this started. Can anyone help us out by a) joining an interactive discussion for 15 minutes or so with the Cython developers to get us started b) helping with a short burst of coding that will follow, we estimate a few days. I think this is something many of us need, and it would also be a thank you to the Cython team for their work, which we all use so much. Cheers, Matthew [1] http://trac.cython.org/cython_trac/ticket/815

3 6

Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas
by Matti Picus Aug. 13, 2014

Aug. 13, 2014

Hi Nathaniel. Thanks for your prompt reply. I think numpy is a wonderful project, and you all do a great job moving it forward. If you ask what would my vision for maturing numpy, I would like to see a grouping of linalg matrix-operation functionality into a python level package, exactly the opposite of more tightly tying linalg into the core of numpy. The orthagonality would allow goups like PyOpenCL to reuse the matrix operations on data located off the CPU's RAM, just to give one example; and make it easier for non-numpy developers to create a complete replacement of lapack with other implementations. Much of the linalg package would of course be implemented in c or fortran, but the interface to ndarray would use the well-established idea of contiguous matrices with shapes, strides, and a single memory store, supporting only numeric number types. I suggested cffi since it provides a convienent and efficient interface to ndarray. Thus python could remain as a thin wrapper over the calls out to c-based libraries much like lapack_lite does today, but at the python level rather that the capi level. Yes, a python-based interface would slows the code down a bit, but I would argue that 1. the current state of lapack_litemodule.c and umath_linalg.c.src, with its myriad of compile-time macros and complex code paths, scares people away from contributing to the ongoing maintenance of the library while tying the code very closely to the lapack routines, and 2. matrices larger than 3x3 or so should be spending most of the computation time in the underlying lapack/blas library irregardless of whether the interface is python-based or capi-based. Matti On 10/08/2014 8:00 PM, numpy-discussion-request(a)scipy.org wrote: > > Date: Sat, 9 Aug 2014 21:11:19 +0100 > From: Nathaniel Smith <njs(a)pobox.com> > Subject: Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > On Sat, Aug 9, 2014 at 8:35 PM, Matti Picus <matti.picus(a)gmail.com> wrote: >> Hi. I am working on numpy in pypy. It would be much more challenging for >> me if you merged more code into the core of numpy, > Hi Matti, > > I can definitely see how numpy changes cause trouble for you, and > sympathize. But, can you elaborate on what kind of changes would make > your life easier *that also* help make numpy proper better in their > own right? Because unfortunately, I don't see how we can reasonably > pass up on improvements to numpy if the only justification is to make > numpypy's life easier. (I'd also love to see pypy become usable for > general numerical work, but not only is it not there now, I don't see > how numpypy will ultimately get us there even if we do help it along > -- almost none of the ecosystem can get by numpy's python-level APIs > alone.) But obviously if there are changes that are mutually > beneficial, well then, that's a lot easier to justify :-) > > -n

7 14

ANN: DistArray 0.5 release
by Kurt Smith Aug. 11, 2014

Aug. 11, 2014

=============================================== DistArray 0.5 release =============================================== **Mailing list:** distarray(a)googlegroups.com **Documentation:** http://distarray.readthedocs.org **License:** Three-clause BSD **Python versions:** 2.7, 3.3, and 3.4 **OS support:** \*nix and Mac OS X What is DistArray? ------------------ DistArray aims to bring the ease-of-use of NumPy to data-parallel high-performance computing. It provides distributed multi-dimensional NumPy arrays, distributed ufuncs, and distributed IO capabilities. It can efficiently interoperate with external distributed libraries like Trilinos. DistArray works with NumPy and builds on top of it in a flexible and natural way. 0.5 Release ----------- Noteworthy improvements in this release include: * closer alignment with NumPy's API, * support for Python 3.4 (existing support for Python 2.7 and 3.3), * a performance-oriented MPI-only mode for deployment on clusters and supercomputers, * a way to register user-defined functions to be callable locally on worker processes, * more consistent naming of sub-packages, * testing with MPICH2 (already tested against OpenMPI), * improved and expanded examples, * installed version testable via ``distarray.test()``, and * performance and scaling improvements. With this release, DistArray ready for real-world testing and deployment. The project is still evolving rapidly and we appreciate the continued input from the larger scientific-Python community. Existing features ----------------- DistArray: * supports NumPy-like slicing, reductions, and ufuncs on distributed multidimensional arrays; * has a client-engine process design -- data resides on the worker processes, commands are initiated from master; * allows full control over what is executed on the worker processes and integrates transparently with the master process; * allows direct communication between workers, bypassing the master process for scalability; * integrates with IPython.parallel for interactive creation and exploration of distributed data; * supports distributed ufuncs (currently without broadcasting); * builds on and leverages MPI via MPI4Py in a transparent and user-friendly way; * has basic support for unstructured arrays; * supports user-controllable array distributions across workers (block, cyclic, block-cyclic, and unstructured) on a per-axis basis; * has a straightforward API to control how an array is distributed; * has basic plotting support for visualization of array distributions; * separates the array’s distribution from the array’s data -- useful for slicing, reductions, redistribution, broadcasting, and other operations; * implements distributed random arrays; * supports ``.npy``-like flat-file IO and hdf5 parallel IO (via ``h5py``); leverages MPI-based IO parallelism in an easy-to-use and transparent way; and * supports the distributed array protocol [protocol]_, which allows independently developed parallel libraries to share distributed arrays without copying, analogous to the PEP-3118 new buffer protocol. Planned features and roadmap ---------------------------- Near-term features and improvements include: * array re-distribution capabilities; * lazy evaluation and deferred computation for latency hiding; * interoperation with Trilinos [Trilinos]_; and * distributed broadcasting support. The longer-term roadmap includes: * Integration with other packages [petsc]_ that subscribe to the distributed array protocol [protocol]_; * Distributed fancy indexing; * Out-of-core computations; * Support for distributed sorting and other non-trivial distributed algorithms; and * End-user control over communication and temporary array creation, and other performance aspects of distributed computations. History and funding ------------------- Brian Granger started DistArray as a NASA-funded SBIR project in 2008. Enthought picked it up as part of a DOE Phase II SBIR [SBIR]_ to provide a generally useful distributed array package. It builds on NumPy, MPI, MPI4Py, IPython, IPython.parallel, and interfaces with the Trilinos suite of distributed HPC solvers (via PyTrilinos [Trilinos]_). This material is based upon work supported by the Department of Energy under Award Number DE-SC0007699. This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. .. [protocol] http://distributed-array-protocol.readthedocs.org/en/rel-0.10.0/ .. [Trilinos] http://trilinos.org/ .. [petsc] http://www.mcs.anl.gov/petsc/ .. [SBIR] http://www.sbir.gov/sbirsearch/detail/410257

1 0

Help - numpy / scipy binary compatibility
by Matthew Brett Aug. 11, 2014

Aug. 11, 2014

Hi, I would be very happy of some help trying to work out a numpy package binary incompatibility. I'm trying to work out what's happening for this ticket: https://github.com/scipy/scipy/issues/3863 which I summarized at the end: https://github.com/scipy/scipy/issues/3863#issuecomment-51669861 but basically, we're getting these errors: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility I now realize I am lost in the world of numpy / scipy etc binary compatibility, I'd really like some advice. In this case numpy == 1.8.1 scipy == 0.14.0 - compiled against numpy 1.5.1 scikit-learn == 0.15.1 compiled against numpy 1.6.0 Can y'all see any potential problem with those dependencies in binary builds? The relevant scipy Cython c files seem to guard against raising this error by doing not-strict checks of the e.g. numpy dtype, so I am confused how these errors come about. Can anyone give any pointers? Cheers, Matthew

4 5

Calculation of a hessian
by Kiko Aug. 11, 2014

Aug. 11, 2014

Hi all, I am trying to calculate a Hessian. I am using numdifftools for this ( https://pypi.python.org/pypi/Numdifftools). My question is, is it possible to make it using pure numpy?. The actual code is like this: *import numdifftools as nd* *import numpy as np* *def log_likelihood(params):* * sum1 = 0; sum2 = 0* * mu = params[0]; sigma = params[1]; xi = params[2]* * for z in data:* * x = 1 + xi * ((z-mu)/sigma)* * sum1 += np.log(x)* * sum2 += x**(-1.0/xi)* * return -((-len(data) * np.log(sigma)) - (1 + 1/xi)*sum1 - sum2) # negated so we can use 'minimum'* *kk = nd.Hessian(log_likelihood)* Thanks in advance.

3 4

ANN: NumPy 1.8.2 bugfix release
by Julian Taylor Aug. 10, 2014

Aug. 10, 2014

Hello, I am pleased to announce the release of NumPy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4722: Fix seg fault converting empty string to object * gh-4613: Fix lack of NULL check in array_richcompare * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 The source distributions have been uploaded to PyPI. The Windows installers, documentation and release notes can be found at: https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ Cheers, Julian Taylor

2 1

OpenBLAS and dotblas
by Charles R Harris Aug. 9, 2014

Aug. 9, 2014

Hi All, It looks like numpy dot only uses BLAS if ATLAS is present, see numpy/core/setup.py. Has anyone done the mods needed to use OpenBLAS? What is the current status of using OpenBLAS with numpy? Also, I'm thinking of moving linalg/lapack_lite/blas_lite.c down into the core. Thoughts? Chuck

3 5

Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas
by Matti Picus Aug. 9, 2014

Aug. 9, 2014

Hi. I am working on numpy in pypy. It would be much more challenging for me if you merged more code into the core of numpy, that means even more must be duplicated when using a different underlying ndarray implementation. If you are thinking of touching linalg/lapack_lite/blas_lite, I would prefer a simpler model that uses ndarray as a simple storage container, does not use cpython-specifc capi extentions with reference counting, and interfaces with python via cffi[0]. Matti [0] https://pypi.python.org/pypi/cffi/ On 8/08/2014 8:00 PM, numpy-discussion-request(a)scipy.org wrote: > Message: 1 > Date: Thu, 7 Aug 2014 20:32:21 -0600 > From: Charles R Harris <charlesr.harris(a)gmail.com> > Subject: [Numpy-discussion] OpenBLAS and dotblas > To: numpy-discussion <numpy-discussion(a)scipy.org> > Message-ID: > <CAB6mnxJOON2tn2-YP8wsBwyqRhNOQEe715Tek6rGGOVTuazyFw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi All, > > It looks like numpy dot only uses BLAS if ATLAS is present, see > numpy/core/setup.py. Has anyone done the mods needed to use OpenBLAS? What > is the current status of using OpenBLAS with numpy? > > Also, I'm thinking of moving linalg/lapack_lite/blas_lite.c down into the > core. Thoughts? > > Chuck >

3 2

ANN: NumPy 1.8.2 release candidate
by Julian Taylor Aug. 7, 2014

Aug. 7, 2014

Hello, I am pleased to announce the first release candidate for numpy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ If no regressions show up the final release is planned this weekend. The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4613: Fix lack of NULL check in array_richcompare * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ Cheers, Julian Taylor

5 6

Preliminary thoughts on implementing __matmul__
by Charles R Harris Aug. 7, 2014

Aug. 7, 2014

Hi All, I've been looking to implement the "@" operator from Python 3.5. Looking at the current implementation of the dot function, it only uses a vector inner product, which is either that defined in arraytypes.c.src or a version using cblas defined in _dotblas for the float, cfloat, double, cdouble types. I note that the versions defined in arraytypes.c.src include all the numeric types plus boolean, datetime, timedelta, and object. I'm not clear why datetime and timedelta should have dot products, except perhaps for scalar multiplication. The boolean version has the advantage that it can short circuit. I also note that all the operations proposed for "@" can easily be done with einsum except for objects. So I'm wondering if one easy way to implement the functions is to extend einsum to work with objects and make it use blas when available. Another thing that may be worth looking into would be some way to multiply by the complex conjugate, as that is easy to implement at the low level. I'd welcome any thoughts as to how that might be done. Anyway, I'm just looking for a discussion and ideas here. Any input is welcome. Chuck

8 17

Inverted indices
by Nicolas P. Rougier Aug. 7, 2014

Aug. 7, 2014

Hi, I've a small problem for which I cannot find a solution and I'm quite sure there is an obvious one: I've an array Z (any dtype) with some data. I've a (sorted) array I (of integer, same size as Z) that tells me the index of Z[i] (if necessary, the index can be stored in Z). Now, I have an arbitrary sequence S of indices (in the sense of I), how do I build the corresponding data ? Here is a small example: Z = [(0,0), (1,1), (2,2), (3,3), (4,4)) I = [0, 20, 23, 24, 37] S = [ 20,20,0,24] -> Result should be [(1,1), (1,1), (0,0),(3,3)] S = [15,15] -> Wrong (15 not in I) but ideally, I would like this to be converted to [(0,0), (0,0)] Any idea ? Nicolas

3 5

How to give feedback to github
by Charles R Harris Aug. 6, 2014

Aug. 6, 2014

Does anyone know how to complain about features to github? The new author selection list for PRs is practically useless as 1) it only lists authors belonging to the project and 2) it doesn't list the number of PRs for each author. The old list was far more useful. Chuck

2 1

Array2 subset of array1
by Jurgens de Bruin Aug. 5, 2014

Aug. 5, 2014

Hi, I am new to numpy so any help would be greatly appreciated. I have two arrays: array1 = np.arange(1,100+1) array2 = np.arange(1,50+1) How can I calculate/determine if array2 is a subset of array1 (falls within array 1) Something like : array2 in array1 = TRUE for the case above. Thank

4 4

last call for numpy 1.8.2 bugfixes
by Julian Taylor Aug. 4, 2014

Aug. 4, 2014

hi, as numpy 1.9 is going to be a relative hard upgrade as indexing changes expose a couple bugs in third party packages and the large amount of small little incompatibilities I will create a numpy 1.8.2 release tomorrow with a couple of important or hard to work around bugfixes. The most important bugfix is fixing the wrong result partition with multiple selections could produce if selections ended up in an equal range, see https://github.com/numpy/numpy/issues/4836 (if the crash is still unreproducable, help appreciated). the rest of the fixes are small ones listed below. If I have missed one or you consider one of the fixes to invasive for a bugfix release please speak up now. As the number of fixes is small I will skip a release candidate. Make fftpack._raw_fft threadsafe https://github.com/numpy/numpy/issues/4656 Prevent division by zero https://github.com/numpy/numpy/issues/650 Fix lack of NULL check in array_richcompare https://github.com/numpy/numpy/issues/4613 incorrect argument order to _copyto in in np.nanmax, np.nanmin https://github.com/numpy/numpy/issues/4628 Hold GIL for types with fields, fixes https://github.com/numpy/numpy/issues/4642 svd ufunc typo https://github.com/numpy/numpy/issues/4733 check alignment of strides for byteswap https://github.com/numpy/numpy/issues/4774 add missing elementsize alignment check for simd reductions https://github.com/numpy/numpy/issues/4853 ifort has issues with optimization flag /O2 https://github.com/numpy/numpy/issues/4602

3 5

Requesting Code Review of weighted covariance ENH
by tpoole Aug. 4, 2014

Aug. 4, 2014

Hi everyone, I've added the ability to handle weighted data in a covariance calculation, in a similar manner to that already implemented for the calculation of a weighted average. https://github.com/tpoole/numpy/compare/weighted_cov Could an experienced someone please look over my changes before I submit a pull request? Validation of the formula can be found at "Exponential smoothing weighted correlations", F. Pozzi, T. Matteo, and T. Aste, Eur. Phys. J. B. 85, 175 (2012). --- though it is unfortunately paywalled and "An Analysis of WinCross, SPSS, and Mentor Procedures for Estimating the Variance of a Weighted Mean", A. Madansky and H. G. B. Alexander, www.analyticalgroup.com/download/weighted_variance.pdf for the "effective number of samples". Thanks, Tom -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Requesting-Code-Review-of-weigh… Sent from the Numpy-discussion mailing list archive at Nabble.com.

1 0

Class to experiment with '@'
by Charles R Harris Aug. 3, 2014

Aug. 3, 2014

Hi All, I've attached a subclass of ndarray that implements the new '@' operator for experimentation and comment. It is only intended for playing with that operator and may not work for other things. You will need to install python 3.5.0a1 to play with it. Chuck

1 2

It lives! Or at least is not undead
by Charles R Harris Aug. 2, 2014

Aug. 2, 2014

charris@localhost [matmul (master)]$ python3.5 Python 3.5.0a0 (default:4425024f2e01, Aug 2 2014, 10:10:31) [GCC 4.8.3 20140624 (Red Hat 4.8.3-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> import testing >>> a = np.ones(3).view(testing.marray) >>> a@3 marray([ 3., 3., 3.]) >>> 3@a marray([ 3., 3., 3.]) >>> a(a)np.eye(3) marray([ 1., 1., 1.]) >>> np.eye(3)@a marray([ 1., 1., 1.]) This was just for quick experimentation. Chuck

1 0

Remove numpy/compat/_inspect.py ?
by Charles R Harris Aug. 2, 2014

Aug. 2, 2014

Hi All, The _inspect.py function looks like a numpy version of the python inspect function. ISTR that is was a work around for problems with the early python versions, but that would have been back in 2009. Thoughts? Chuck

3 18

OSX wheels for older numpy versions on pypi
by Matthew Brett Aug. 1, 2014

Aug. 1, 2014

Hi, I took the liberty of uploading OSX wheels for some older numpy versions to pypi. These can be useful for testing, or when building your own wheels to be compatible with earlier numpy versions - see: http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-… There are currently wheels for numpy 1.5.1 py27 numpy 1.6.0 py27 numpy 1.6.1 py27 numpy 1.7.1 py27, 32, 33, 34 These are all compiled against ATLAS: https://github.com/matthew-brett/numpy-atlas-binaries install with e.g. pip install numpy==1.6.1 If anyone needs other wheels compiled, let me know, I'll try and upload them, Cheers, Matthew

2 4