Mailman 3 August 2014 - SciPy-Dev

sprint @ EuroSciPy, Aug 31
by Ralf Gommers 19 Aug '14

19 Aug '14

Hi all, Here is a reminder that on Sunday 31 August, there will be a Scipy sprint at EuroSciPy (in Cambridge, UK). Details can be found at https://www.euroscipy.org/2014/program/sprints/ Newcomers to Scipy development are very welcome; actually one of the main goals of the sprint is to help new people to get started. Last year's sprint was excellent - 20 people joined and we still have all-time highs in the commits per month and contributors per month graph to show for it: https://www.openhub.net/p/scipy If you have time and will be at EuroSciPy: please join! Cheers, Ralf

1 0

Histogram as its own class
by Johann Goetz 15 Aug '14

15 Aug '14

Hello, I'm a long-time user of scipy doing mostly multivariate big-data (several terabytes) analysis in the high-energy physics realm. One thing I've found useful was to promote the histogram to it's own class. Instead of creating yet another package, I have a mind to include it into the scipy.stats module and I would like some feed-back. I.e. is this the right place for such an object? I have some documentation, but not enough I would say, and the classes are currently buried in my "pyhep" project, but they are easily extracted out. https://bitbucket.org/theodoregoetz/pyhep/wiki/Home Here are some details: The histograms I am addressing are N-dimensional over a continuous-domain (floating-point data, no gaps - though bins can have value inf or nan if need-be) along each axis. The axes need not be uniform. There are two classes: HistogramAxis and Histogram. The Axes are always floating point, but the histogram's data can be any dtype (default: np.int, a "cast" to float is done when dividing two histograms). I make use of np.histogramdd() and store the data along with the uncertainty. Many operations are supported including adding, subtracting, multiplying, dividing, bin-merging, cutting/clipping along one or more axes, projecting along an axis, iterating over an axis, filling from a sample with or without weights. Most of power in this package is in the fitting method of the histogram which makes use of scipy.curve_fit(). It handles missing data (when a bin is inf or nan), can include the uncertainty in the fit, and calculates a goodness of fit. On top of this, I have free functions to plot 1D and 2D histograms using matplotlib, as well as functions to handle reading in large HDF5 files. These are auxiliary and may not fit into scipy directly. Thank you all, Johann.

1 0

computing pairwise distance of vectors with missing (nan) values
by Moritz Emanuel Beber 12 Aug '14

12 Aug '14

Dear all, My basic problem is that I would like to compute distances between vectors with missing values. You can find more detail in my question on SO (http://stackoverflow.com/questions/24781461/compute-the-pairwise-distance-i…). Since it seems this is not directly possible with scipy at the moment, I started to Cythonize my function. Currently, the below function is not much faster than my pure Python implementation, so I thought I'd ask the experts here. *Note that even though I'm computing the euclidean distance, I'd like to make use of different distance metrics. * So my current attempt at Cythonizing is: import numpy cimport numpy cimport cython from numpy.linalg import norm numpy.import_array() @cython.boundscheck(False) @cython.wraparound(False) def masked_euclidean(numpy.ndarray[numpy.double_t, ndim=2] data): cdef Py_ssize_t m = data.shape[0] cdef Py_ssize_t i = 0 cdef Py_ssize_t j = 0 cdef Py_ssize_t k = 0 cdef numpy.ndarray[numpy.double_t] dm = numpy.zeros(m * (m - 1) // 2, dtype=numpy.double) cdef numpy.ndarray[numpy.uint8_t, ndim=2, cast=True] mask = numpy.isfinite(data) # boolean for i in range(m - 1): for j in range(i + 1, m): curr = numpy.logical_and(mask[i], mask[j]) u = data[i][curr] v = data[j][curr] dm[k] = norm(u - v) k += 1 return dm Maybe the lack of speed-up is due to the Python function 'norm'? So my question is, how to improve the Cython implementation? Or is there a completely different way of approaching this problem? Thanks in advance, Moritz

3 4

Fastest way to multiply a sparse matrix with another numpy array
by Manoj Kumar 11 Aug '14

11 Aug '14

Hello, I was wondering what is the fastest way (format) to multiply a sparse matrix with a numpy array. Intuitively, a csr format multiplied with a numpy array which is fortran contiguous seems to be the fastest, but I have ran a few benchmarks and it seems otherwise. It is also mentioned here http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.csc… that using csr matrices "may" be faster. In [5]: X Out[5]: <11314x130107 sparse matrix of type '<type 'numpy.float64'>' with 1787565 stored elements in Compressed Sparse Row format> In [6]: _, n_features = X.shape In [9]: w_c = np.random.rand(n_features, 10) In [10]: w_f = np.asarray(w_c, order='f') In [13]: csc = sparse.csc_matrix(X) In [30]: %timeit X * w_f 10 loops, best of 3: 40.5 ms per loop In [31]: %timeit X * w_c 10 loops, best of 3: 37.3 ms per loop In [32]: %timeit csc * w_c 10 loops, best of 3: 24.3 ms per loop In [33]: %timeit csc * w_f 10 loops, best of 3: 27.3 ms per loop It seems here, using a csc matrix is faster with a C-contiguous numpy array which is completely non-intuitive to me. Are there any hard rules for this? or is it data dependent? Sorry for my noobish questions! -- Regards, Manoj Kumar, GSoC 2014, Scikit-learn Mech Undergrad http://manojbits.wordpress.com

1 1

ANN: NumPy 1.8.2 bugfix release
by Julian Taylor 10 Aug '14

10 Aug '14

Hello, I am pleased to announce the release of NumPy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4722: Fix seg fault converting empty string to object * gh-4613: Fix lack of NULL check in array_richcompare * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 The source distributions have been uploaded to PyPI. The Windows installers, documentation and release notes can be found at: https://sourceforge.net/projects/numpy/files/NumPy/1.8.2/ Cheers, Julian Taylor

2 1

ANN: NumPy 1.8.2 release candidate
by Julian Taylor 05 Aug '14

05 Aug '14

Hello, I am pleased to announce the first release candidate for numpy 1.8.2, a pure bugfix release for the 1.8.x series. https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ If no regressions show up the final release is planned this weekend. The upgrade is recommended for all users of the 1.8.x series. Following issues have been fixed: * gh-4836: partition produces wrong results for multiple selections in equal ranges * gh-4656: Make fftpack._raw_fft threadsafe * gh-4628: incorrect argument order to _copyto in in np.nanmax, np.nanmin * gh-4613: Fix lack of NULL check in array_richcompare * gh-4642: Hold GIL for converting dtypes types with fields * gh-4733: fix np.linalg.svd(b, compute_uv=False) * gh-4853: avoid unaligned simd load on reductions on i386 * gh-4774: avoid unaligned access for strided byteswap * gh-650: Prevent division by zero when creating arrays from some buffers * gh-4602: ifort has issues with optimization flag O2, use O1 Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.2rc1/ Cheers, Julian Taylor

1 0