Mailman 3 June 2016 - NumPy-Discussion

How best to turn JSON into a CSV or Pandas data frame table?
by David Shi June 25, 2016

June 25, 2016

Which are the best ways to turn a JSON object into a CSV or Pandas data frame table? Looking forward to hearing from you. Regards. David

1 0

Support of '@='?
by Hans Larsen June 24, 2016

June 24, 2016

I have Python 3-5-1 and NumPy 1-11! windows 64bits! When will by side 'M=M@P' be supported with 'M@=P'???:-( -- Hans Larsen Galgebakken Sønder 4-11A 2620 Albertslund Danmark/Danio

4 4

Integers to integer powers, let's make a decision
by Charles R Harris June 21, 2016

June 21, 2016

Hi All, I've made a new post so that we can make an explicit decision. AFAICT, the two proposals are 1. Integers to negative integer powers raise an error. 2. Integers to integer powers always results in floats. My own sense is that 1. would be closest to current behavior and using a float exponential when a float is wanted is an explicit way to indicate that desire. OTOH, 2. would be the most convenient default for everyday numerical computation, but I think would more likely break current code. I am going to come down on the side of 1., which I don't think should cause too many problems if we start with a {Future, Deprecation}Warning explaining the workaround. Chuck

15 57

scipy 0.18 release candidate 1
by Evgeni Burovski June 21, 2016

June 21, 2016

Hi, I'm pleased to announce the availability of the first release candidate for scipy 0.18.0. Please try this release and report any issues on Github tracker, https://github.com/scipy/scipy, or scipy-dev mailing list. Source tarballs and release notes are available from Github releases, https://github.com/scipy/scipy/releases/tag/v0.18.0rc1 Please note that this is a source-only release. We do not provide Windows binaries for this release. OS X and Linux wheels will be provided for the final release. The current release schedule is 27 June: rc2 (if necessary) 11 July: final release Thanks to everyone who contributed to this release! Cheers, Evgeni A part of the release notes follows: ========================== SciPy 0.18.0 Release Notes ========================== .. note:: Scipy 0.18.0 is not released yet! .. contents:: SciPy 0.18.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.19.x branch, and on adding new features on the master branch. This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.1 or greater. Highlights of this release include: - - A new ODE solver for two-point boundary value problems, `scipy.optimize.solve_bvp`. - - A new class, `CubicSpline`, for cubic spline interpolation of data. - - N-dimensional tensor product polynomials, `scipy.interpolate.NdPPoly`. - - Spherical Voronoi diagrams, `scipy.spatial.SphericalVoronoi`. - - Support for discrete-time linear systems, `scipy.signal.dlti`. New features ============ `scipy.integrate` improvements - ------------------------------ A solver of two-point boundary value problems for ODE systems has been implemented in `scipy.integrate.solve_bvp`. The solver allows for non-separated boundary conditions, unknown parameters and certain singular terms. It finds a C1 continious solution using a fourth-order collocation algorithm. `scipy.interpolate` improvements - -------------------------------- Cubic spline interpolation is now available via `scipy.interpolate.CubicSpline`. This class represents a piecewise cubic polynomial passing through given points and C2 continuous. It is represented in the standard polynomial basis on each segment. A representation of n-dimensional tensor product piecewise polynomials is available as the `scipy.interpolate.NdPPoly` class. Univariate piecewise polynomial classes, `PPoly` and `Bpoly`, can now be evaluated on periodic domains. Use ``extrapolate="periodic"`` keyword argument for this. `scipy.fftpack` improvements - ---------------------------- `scipy.fftpack.next_fast_len` function computes the next "regular" number for FFTPACK. Padding the input to this length can give significant performance increase for `scipy.fftpack.fft`. `scipy.signal` improvements - --------------------------- Resampling using polyphase filtering has been implemented in the function `scipy.signal.resample_poly`. This method upsamples a signal, applies a zero-phase low-pass FIR filter, and downsamples using `scipy.signal.upfirdn` (which is also new in 0.18.0). This method can be faster than FFT-based filtering provided by `scipy.signal.resample` for some signals. `scipy.signal.firls`, which constructs FIR filters using least-squares error minimization, was added. `scipy.signal.sosfiltfilt`, which does forward-backward filtering like `scipy.signal.filtfilt` but for second-order sections, was added. Discrete-time linear systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `scipy.signal.dlti` provides an implementation of discrete-time linear systems. Accordingly, the `StateSpace`, `TransferFunction` and `ZerosPolesGain` classes have learned a the new keyword, `dt`, which can be used to create discrete-time instances of the corresponding system representation. `scipy.sparse` improvements - --------------------------- The functions `sum`, `max`, `mean`, `min`, `transpose`, and `reshape` in `scipy.sparse` have had their signatures augmented with additional arguments and functionality so as to improve compatibility with analogously defined functions in `numpy`. Sparse matrices now have a `count_nonzero` method, which counts the number of nonzero elements in the matrix. Unlike `getnnz()` and ``nnz`` propety, which return the number of stored entries (the length of the data attribute), this method counts the actual number of non-zero entries in data. `scipy.optimize` improvements - ----------------------------- The implementation of Nelder-Mead minimization, `scipy.minimize(..., method="Nelder-Mead")`, obtained a new keyword, `initial_simplex`, which can be used to specify the initial simplex for the optimization process. Initial step size selection in CG and BFGS minimizers has been improved. We expect that this change will improve numeric stability of optimization in some cases. See pull request gh-5536 for details. Handling of infinite bounds in SLSQP optimization has been improved. We expect that this change will improve numeric stability of optimization in the some cases. See pull request gh-6024 for details. A large suite of global optimization benchmarks has been added to ``scipy/benchmarks/go_benchmark_functions``. See pull request gh-4191 for details. Nelder-Mead and Powell minimization will now only set defaults for maximum iterations or function evaluations if neither limit is set by the caller. In some cases with a slow converging function and only 1 limit set, the minimization may continue for longer than with previous versions and so is more likely to reach convergence. See issue gh-5966. `scipy.stats` improvements - -------------------------- Trapezoidal distribution has been implemented as `scipy.stats.trapz`. Skew normal distribution has been implemented as `scipy.stats.skewnorm`. Burr type XII distribution has been implemented as `scipy.stats.burr12`. Three- and four-parameter kappa distributions have been implemented as `scipy.stats.kappa3` and `scipy.stats.kappa4`, respectively. New `scipy.stats.iqr` function computes the interquartile region of a distribution. Random matrices ~~~~~~~~~~~~~~~ `scipy.stats.special_ortho_group` and `scipy.stats.ortho_group` provide generators of random matrices in the SO(N) and O(N) groups, respectively. They generate matrices in the Haar distribution, the only uniform distribution on these group manifolds. `scipy.stats.random_correlation` provides a generator for random correlation matrices, given specified eigenvalues. `scipy.linalg` improvements - --------------------------- `scipy.linalg.svd` gained a new keyword argument, ``lapack_driver``. Available drivers are ``gesdd`` (default) and ``gesvd``. `scipy.linalg.lapack.ilaver` returns the version of the LAPACK library SciPy links to. `scipy.spatial` improvements - ---------------------------- Boolean distances, `scipy.spatial.pdist`, have been sped up. Improvements vary by the function and the input size. In many cases, one can expect a speed-up of x2--x10. New class `scipy.spatial.SphericalVoronoi` constructs Voronoi diagrams on the surface of a sphere. See pull request gh-5232 for details. `scipy.cluster` improvements - ---------------------------- A new clustering algorithm, the nearest neighbor chain algorithm, has been implemented for `scipy.cluster.hierarchy.linkage`. As a result, one can expect a significant algorithmic improvement (:math:`O(N^2)` instead of :math:`O(N^3)`) for several linkage methods. `scipy.special` improvements - ---------------------------- The new function `scipy.special.loggamma` computes the principal branch of the logarithm of the Gamma function. For real input, ``loggamma`` is compatible with `scipy.special.gammaln`. For complex input, it has more consistent behavior in the complex plane and should be preferred over ``gammaln``. Vectorized forms of spherical Bessel functions have been implemented as `scipy.special.spherical_jn`, `scipy.special.spherical_kn`, `scipy.special.spherical_in` and `scipy.special.spherical_yn`. They are recommended for use over ``sph_*`` functions, which are now deprecated. Several special functions have been extended to the complex domain and/or have seen domain/stability improvements. This includes `spence`, `digamma`, `log1p` and several others. Deprecated features =================== The cross-class properties of `lti` systems have been deprecated. The following properties/setters will raise a `DeprecationWarning`: Name - (accessing/setting raises warning) - (setting raises warning) * StateSpace - (`num`, `den`, `gain`) - (`zeros`, `poles`) * TransferFunction (`A`, `B`, `C`, `D`, `gain`) - (`zeros`, `poles`) * ZerosPolesGain (`A`, `B`, `C`, `D`, `num`, `den`) - () Spherical Bessel functions, ``sph_in``, ``sph_jn``, ``sph_kn``, ``sph_yn``, ``sph_jnyn`` and ``sph_inkn`` have been deprecated in favor of `scipy.special.spherical_jn` and ``spherical_kn``, ``spherical_yn``, ``spherical_in``. The following functions in `scipy.constants` are deprecated: ``C2K``, ``K2C``, ``C2F``, ``F2C``, ``F2K`` and ``K2F``. They are superceded by a new function `scipy.constants.convert_temperature` that can perform all those conversions plus to/from the Rankine temperature scale. Backwards incompatible changes ============================== `scipy.optimize` - ---------------- The convergence criterion for ``optimize.bisect``, ``optimize.brentq``, ``optimize.brenth``, and ``optimize.ridder`` now works the same as ``numpy.allclose``. `scipy.ndimage` - --------------- The offset in ``ndimage.iterpolation.affine_transform`` is now consistently added after the matrix is applied, independent of if the matrix is specified using a one-dimensional or a two-dimensional array. `scipy.stats` - ------------- ``stats.ks_2samp`` used to return nonsensical values if the input was not real or contained nans. It now raises an exception for such inputs. Several deprecated methods of `scipy.stats` distributions have been removed: ``est_loc_scale``, ``vecfunc``, ``veccdf`` and ``vec_generic_moment``. Deprecated functions ``nanmean``, ``nanstd`` and ``nanmedian`` have been removed from `scipy.stats`. These functions were deprecated in scipy 0.15.0 in favor of their `numpy` equivalents. A bug in the ``rvs()`` method of the distributions in `scipy.stats` has been fixed. When arguments to ``rvs()`` were given that were shaped for broadcasting, in many cases the returned random samples were not random. A simple example of the problem is ``stats.norm.rvs(loc=np.zeros(10))``. Because of the bug, that call would return 10 identical values. The bug only affected code that relied on the broadcasting of the shape, location and scale parameters. The ``rvs()`` method also accepted some arguments that it should not have. There is a potential for backwards incompatibility in cases where ``rvs()`` accepted arguments that are not, in fact, compatible with broadcasting. An example is stats.gamma.rvs([2, 5, 10, 15], size=(2,2)) The shape of the first argument is not compatible with the requested size, but the function still returned an array with shape (2, 2). In scipy 0.18, that call generates a ``ValueError``. `scipy.io` - ---------- `scipy.io.netcdf` masking now gives precedence to the ``_FillValue`` attribute over the ``missing_value`` attribute, if both are given. Also, data are only treated as missing if they match one of these attributes exactly: values that differ by roundoff from ``_FillValue`` or ``missing_value`` are no longer treated as missing values. `scipy.interpolate` - ------------------- `scipy.interpolate.PiecewisePolynomial` class has been removed. It has been deprecated in scipy 0.14.0, and `scipy.interpolate.BPoly.from_derivatives` serves as a drop-in replacement. Other changes ============= Scipy now uses ``setuptools`` for its builds instead of plain distutils. This fixes usage of ``install_requires='scipy'`` in the ``setup.py`` files of projects that depend on Scipy (see Numpy issue gh-6551 for details). It potentially affects the way that build/install methods for Scipy itself behave though. Please report any unexpected behavior on the Scipy issue tracker. PR `#6240 <https://github.com/scipy/scipy/pull/6240>`__ changes the interpretation of the `maxfun` option in `L-BFGS-B` based routines in the `scipy.optimize` module. An `L-BFGS-B` search consists of multiple iterations, with each iteration consisting of one or more function evaluations. Whereas the old search strategy terminated immediately upon reaching `maxfun` function evaluations, the new strategy allows the current iteration to finish despite reaching `maxfun`. The bundled copy of Qhull in the `scipy.spatial` subpackage has been upgraded to version 2015.2. The bundled copy of ARPACK in the `scipy.sparse.linalg` subpackage has been upgraded to arpack-ng 3.3.0. The bundled copy of SuperLU in the `scipy.sparse` subpackage has been upgraded to version 5.1.1. Authors ======= * @endolith * @yanxun827 + * @kleskjr + * @MYheavyGo + * @solarjoe + * Gregory Allen + * Gilles Aouizerate + * Tom Augspurger + * Henrik Bengtsson + * Felix Berkenkamp * Per Brodtkorb * Lars Buitinck * Daniel Bunting + * Evgeni Burovski * CJ Carey * Tim Cera * Grey Christoforo + * Robert Cimrman * Philip DeBoer + * Yves Delley + * Dávid Bodnár + * Ion Elberdin + * Gabriele Farina + * Yu Feng * Andrew Fowlie + * Joseph Fox-Rabinovitz * Simon Gibbons + * Neil Girdhar + * Kolja Glogowski + * Christoph Gohlke * Ralf Gommers * Todd Goodall + * Johnnie Gray + * Alex Griffing * Olivier Grisel * Thomas Haslwanter + * Michael Hirsch + * Derek Homeier * Golnaz Irannejad + * Marek Jacob + * InSuk Joung + * Tetsuo Koyama + * Eugene Krokhalev + * Eric Larson * Denis Laxalde * Antony Lee * Jerry Li + * Henry Lin + * Nelson Liu + * Loïc Estève * Lei Ma + * Osvaldo Martin + * Stefano Martina + * Nikolay Mayorov * Matthieu Melot + * Sturla Molden * Eric Moore * Alistair Muldal + * Maniteja Nandana * Tavi Nathanson + * Andrew Nelson * Joel Nothman * Behzad Nouri * Nikolai Nowaczyk + * Juan Nunez-Iglesias + * Ted Pudlik * Eric Quintero * Yoav Ram * Jonas Rauber + * Tyler Reddy + * Juha Remes * Garrett Reynolds + * Ariel Rokem + * Fabian Rost + * Bill Sacks + * Jona Sassenhagen + * Marcello Seri + * Sourav Singh + * Martin Spacek + * Søren Fuglede Jørgensen + * Bhavika Tekwani + * Martin Thoma + * Sam Tygier + * Meet Udeshi + * Utkarsh Upadhyay * Bram Vandekerckhove + * Sebastián Vanrell + * Ze Vinicius + * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Jakub Wilk + * Josh Wilson * Phillip J. Wolfram + * Nathan Woods * Haochen Wu * G Young + A total of 99 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete.

2 2

block function
by Charles R Harris June 21, 2016

June 21, 2016

Hi All, I've updated Stefan Otte's block function enhancement at https://github.com/numpy/numpy/pull/7768. Could folks interested in that function review the proposed grammar for the creation of blocked arrays. Chuck

1 0

Deprecating silent truncation of floats when assigned to int array
by Nathaniel Smith June 17, 2016

June 17, 2016

It was recently pointed out: https://github.com/numpy/numpy/issues/7730 that this code silently truncates floats: In [1]: a = np.arange(10) In [2]: a.dtype Out[2]: dtype('int64') In [3]: a[3] = 1.5 In [4]: a[3] Out[4]: 1 The proposal is that we should deprecate this, and eventually turn it into an error. Any objections? We recently went through a similar deprecation cycle for in-place operations, i.e., this used to silently truncate but now raises an error: In [1]: a = np.arange(10) In [2]: a += 1.5 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-2-9cf893a64492> in <module>() ----> 1 a += 1.5 TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('int64') with casting rule 'same_kind' so the proposal here is to extend this to regular assignment. -n -- Nathaniel J. Smith -- https://vorpus.org

3 2

Re: [Numpy-discussion] axis parameter for count_nonzero
by G Young June 17, 2016

June 17, 2016

Hi, I have had a PR <https://github.com/numpy/numpy/pull/7177> open (first draft can be found here <https://github.com/numpy/numpy/pull/7138>) for quite some time now that adds an 'axis' parameter to *count_nonzero*. While the functionality is fully in-place, very robust, and actually higher-performing than the original *count_nonzero* function, the obstacle at this point is the implementation, as most of the functionality is now surfaced at the Python level instead of at the C level. I have made several attempts to move the code into C to no avail and have not received much feedback from maintainers unfortunately to move this forward, so I'm opening this up to the mailing list to see what you guys think of the changes and whether or not it should be merged in as is or be tabled until a more C-friendly solution can be found. Thanks!

2 4

Re: [Numpy-discussion] broadcasting for randint
by G Young June 17, 2016

June 17, 2016

Hi, I have had a PR <https://github.com/numpy/numpy/pull/6938> open for quite some time now that allows arguments to broadcast in *randint*. While the functionality is fully in-place and very robust, the obstacle at this point is the implementation. When the *dtype* parameter was added to *randint* (see here <https://github.com/numpy/numpy/pull/6910>), a big issue with the implementation was that it created so much duplicate code that it would be a huge maintenance nightmare. However, this was dismissed in the original PR message because it was believed that template-ing would be trivial, which seemed reasonable at the time. When I added broadcasting, I introduced a template system to the code that dramatically cut down on the duplication. However, the obstacle has been whether or not this template system is too *ad hoc* to be merged into the library. Implementing a template in Cython was not considered sufficient and is in fact very tricky to do, and unfortunately, I have not received any constructive suggestions from maintainers about how to proceed, so I'm opening this up to the mailing to see whether or not there are better alternatives to what I did, whether this should be merged as it, or whether this should be tabled until a better template can be found. Thanks!

1 3

how did numpy make mkl use multiple threads?
by Roger Xu June 16, 2016

June 16, 2016

Hi. I have tried a lot of things to make mkl use multiple threads. I just learned to write c extension for python yesterday. Can I know how did numpy make mkl use multiple threads? Numpy can make mkl do that on the machine I am working on now. My best try is described at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/640… two earlier attempts are described here: https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/639… and also here: http://stackoverflow.com/questions/37536106/directly-use-intel-mkl-library-… Thank you. - rxu

1 0

numpy threads crash when allocating arrays
by Burlen Loring June 16, 2016

June 16, 2016

Hi All, I'm working on a threaded pipeline where we want the end user to be able to code up Python functions to do numerical work. Threading is all done in C++11 and in each thread we've acquired gill before we invoke the user provided Python callback and release it only when the callback returns. We've used SWIG to expose bindings to C++ objects in Python. When run with more than 1 thread I get inconsistent segv's in various numpy routines, and occasionally see a *** Reference count error detected: an attempt was made to deallocate 11 (f) ***. To pass data from C++ to the Python callback as numpy array we have 206 // **************************************************************************** 207 template <typename NT> 208 PyArrayObject *new_object(teca_variant_array_impl<NT> *varrt) 209 { 210 // allocate a buffer 211 npy_intp n_elem = varrt->size(); 212 size_t n_bytes = n_elem*sizeof(NT); 213 NT *mem = static_cast<NT*>(malloc(n_bytes)); 214 if (!mem) 215 { 216 PyErr_Format(PyExc_RuntimeError, 217 "failed to allocate %lu bytes", n_bytes); 218 return nullptr; 219 } 220 221 // copy the data 222 memcpy(mem, varrt->get(), n_bytes); 223 224 // put the buffer in to a new numpy object 225 PyArrayObject *arr = reinterpret_cast<PyArrayObject*>( 226 PyArray_SimpleNewFromData(1, &n_elem, numpy_tt<NT>::code, mem)); 227 PyArray_ENABLEFLAGS(arr, NPY_ARRAY_OWNDATA); 228 229 return arr; 230 } This is the only place we create numpy objects in the C++ side. In my demo the Python callback is as follows: 33 def get_execute(rank, var_names): 34 def execute(port, data_in, req): 35 sys.stderr.write('descriptive_stats::execute MPI %d\n'%(rank)) 36 37 mesh = as_teca_cartesian_mesh(data_in[0]) 38 39 table = teca_table.New() 40 table.copy_metadata(mesh) 41 42 table.declare_columns(['step','time'], ['ul','d']) 43 table << mesh.get_time_step() << mesh.get_time() 44 45 for var_name in var_names: 46 47 table.declare_columns(['min '+var_name, 'avg '+var_name, \ 48 'max '+var_name, 'std '+var_name, 'low_q '+var_name, \ 49 'med '+var_name, 'up_q '+var_name], ['d']*7) 50 *51 **var = mesh.get_point_arrays().get(var_name).as_array()* 52 53 table << float(np.min(var)) << float(np.average(var)) \ 54 << float(np.max(var)) << float(np.std(var)) \ 55 << map(float, np.percentile(var, [25.,50.,75.])) 56 57 return table 58 return execute this callback is the only spot where numpy is used. the as_array call is implemented by new_object template above. Further, If I remove our use of PyArray_SimpleNewFromData, by replacing line 51 in the Python code above with var = np.array(range(1, 1100), 'f'), the problem disappears. It must have something to do with use of PyArray_SimpleNewFromData. I'm at a loss to see why things are going south. I'm using the GIL and I thought that would serialize the Python code. I suspect that numpy is using global or static variables some where internally and that it's inherently thread unsafe. Can anyone confirm/deny? maybe point me in the right direction?

3 6