Mailman 3 January 2015 - NumPy-Discussion

The future of ndarray.diagonal()
by Alexander Belopolsky 15 Jan '15

15 Jan '15

A discussion [1] is currently underway at GitHub which will benefit from a larger forum. In version 1.9, the diagonal() method was changed to return a read-only (non-contiguous) view into the original array instead of a plain copy. Also, it has been announced [2] that in 1.10 the view will become read/write. A concern has now been raised [3] that this change breaks backward compatibility too much. Consider the following code: x = numy.eye(2) d = x.diagonal() d[0] = 2 In 1.8, this code runs without errors and results in [2, 1] stored in array d. In 1.9, this is an error. With the current plan, in 1.10 this will become valid again, but the result will be different: x[0,0] will be 2 while it is 1 in 1.8. Two alternatives are suggested for discussion: 1. Add copy=True flag to diagonal() method. 2. Roll back 1.9 change to diagonal() and introduce an additional diagonal_view() method to return a view. [1] https://github.com/numpy/numpy/pull/5409 [2] http://docs.scipy.org/doc/numpy/reference/generated/numpy.diagonal.html [3] http://khinsen.wordpress.com/2014/09/12/the-state-of-numpy/

11 19

Memory allocation of arrays and tracemalloc
by Antoine Pitrou 15 Jan '15

15 Jan '15

Hello, I see that the PyDataMem_* APIs call malloc()/free()/etc. directly, instead of going through PyMem_Malloc, etc. This means the memory allocated by those APIs won't be seen by tracemalloc. Is it deliberate? Am I missing something? Regards Antoine.

2 1

proposed change to recarray access
by Allan Haldane 14 Jan '15

14 Jan '15

Hello all, I've submitted a pull request on github which changes how string values in recarrays are returned, which may break old code. https://github.com/numpy/numpy/pull/5454 See also: https://github.com/numpy/numpy/issues/3993 Previously, recarray fields of type 'S' or 'U' (ie, strings) would be returned as chararrays when accessed by attribute, but ndarrays when accessed by indexing: >>> arr = np.array([('abc ', 1), ('abc', 2)], dtype=[('str', 'S4'), ('id', int)]) >>> arr = arr.view(np.recarray) >>> type(arr.str) numpy.core.defchararray.chararray >>> type(arr['str']) numpy.core.records.recarray Chararray is deprecated, and furthermore this led to bugs in my code since chararrays trim trailing whitespace but but ndarrays do not (and I was not aware of conversion to chararray). For example: >>> arr.str[0] == arr.str[1] True >>> arr['str'][0] == arr['str'][1] False In the pull request I have changed recarray attribute access so ndarrays are always returned. I think this is a sensible thing to do but it may break code which depends on chararray features (including the trimmed whitespace). Does this sound reasonable? Best, Allan

1 0

linspace handling of extra return
by Jaime Fernández del Río 14 Jan '15

14 Jan '15

While working on something else, I realized that linspace is not handling requests for returning the sampling spacing consistently: >>> np.linspace(0, 1, 3, retstep=True) (array([ 0. , 0.5, 1. ]), 0.5) >>> np.linspace(0, 1, 1, retstep=True) array([ 0.]) >>> np.linspace(0, 1, 0, retstep=True) array([], dtype=float64) Basically, retstep is ignored if the number of samples is 0 or 1. One could argue that it makes sense, because those sequences do not have a spacing defined. But at the very least it should be documented as doing so, and the following inconsistency removed: >>> np.linspace(0, 1, 1, endpoint=True, retstep=True) array([ 0.]) >>> np.linspace(0, 1, 1, endpoint=False, retstep=True) (array([ 0.]), 1.0) I am personally inclined to think that if a step is requested, then a step should be returned, and if it cannot be calculated in a reasonable manner, then a placeholder such as None, nan, 0 or stop - start should be returned. What does the collective wisdom think is the best approach for this? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 6

Equality of dtypes does not imply equality of type kinds
by Alexander Belopolsky 13 Jan '15

13 Jan '15

Consider this (on a 64-bit platform): >>> numpy.dtype('q') == numpy.dtype('l') True but >>> numpy.dtype('q').char == numpy.dtype('l').char False Is that intended? Shouldn't dtype constructor "normalize" 'l' to 'q' (or 'i')?

5 5

ANN: Scipy 0.15.0 release
by Pauli Virtanen 12 Jan '15

12 Jan '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, We are pleased to announce the Scipy 0.15.0 release. The 0.15.0 release contains bugfixes and new features, most important of which are mentioned in the excerpt from the release notes below. Source tarballs, binaries, and full release notes are available at https://sourceforge.net/projects/scipy/files/scipy/0.15.0/ Best regards, Pauli Virtanen ========================== SciPy 0.15.0 Release Notes ========================== SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.5.1 or greater. New features ============ Linear Programming Interface - ---------------------------- The new function `scipy.optimize.linprog` provides a generic linear programming similar to the way `scipy.optimize.minimize` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported, allowing the user to monitor the progress of the algorithm. Differential evolution, a global optimizer - ------------------------------------------ A new `scipy.optimize.differential_evolution` function has been added to the ``optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques. ``scipy.signal`` improvements - ----------------------------- The function `scipy.signal.max_len_seq` was added, which computes a Maximum Length Sequence (MLS) signal. ``scipy.integrate`` improvements - -------------------------------- It is now possible to use `scipy.integrate` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance. ``scipy.linalg`` improvements - ----------------------------- The function `scipy.linalg.orthogonal_procrustes` for solving the procrustes linear algebra problem was added. BLAS level 2 functions ``her``, ``syr``, ``her2`` and ``syr2`` are now wrapped in ``scipy.linalg``. ``scipy.sparse`` improvements - ----------------------------- `scipy.sparse.linalg.svds` can now take a ``LinearOperator`` as its main input. ``scipy.special`` improvements - ------------------------------ Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``. New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added. ``scipy.sparse.csgraph`` improvements - ------------------------------------- Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added. ``scipy.stats`` improvements - ---------------------------- Added a Dirichlet multivariate distribution, `scipy.stats.dirichlet`. The new function `scipy.stats.median_test` computes Mood's median test. The new function `scipy.stats.combine_pvalues` implements Fisher's and Stouffer's methods for combining p-values. `scipy.stats.describe` returns a namedtuple rather than a tuple, allowing users to access results by index or by name. Deprecated features =================== The `scipy.weave` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: https://github.com/scipy/weave. It is a pure Python package, and can easily be installed with ``pip install weave``. `scipy.special.bessel_diff_formula` is deprecated. It is a private function, and therefore will be removed from the public API in a following release. ``scipy.stats.nanmean``, ``nanmedian`` and ``nanstd`` functions are deprecated in favor of their numpy equivalents. Backwards incompatible changes ============================== scipy.ndimage - ------------- The functions `scipy.ndimage.minimum_positions`, `scipy.ndimage.maximum_positions`` and `scipy.ndimage.extrema` return positions as ints instead of floats. scipy.integrate - --------------- The format of banded Jacobians in `scipy.integrate.ode` solvers is changed. Note that the previous documentation of this feature was erroneous. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlSyt/cACgkQ6BQxb7O0pWA8SACfXmpUsJcXT5espj71OYpeaj5b JJwAoL10ud3q1f51A5Ij4lgqMeZGnHlj =ZmOl -----END PGP SIGNATURE-----

5 4

[ANN] bcolz v0.8.0
by Valentin Haenel 09 Jan '15

09 Jan '15

====================== Announcing bcolz 0.8.0 ====================== What's new ========== This version adds a public API in the form of a Cython definitions file (``carray_ext.pxd``) for the ``carray`` class! This means, other libraries can use the Cython definitions to build more complex programs using the objects provided by bcolz. In fact, this feature was specifically requested and there already exists a nascent application called *bquery* (https://github.com/visualfabriq/bquery) which provides an efficient out-of-core groupby implementation for the ``ctable`` object Because this is a fairly sweeping change, the minor version number was incremented and no additional major features or bugfixes were added to this release. We kindly ask any users of bcolz to try this version carefully and report back any issues, bugs, or even slow-downs you experience. I.e. please, please be careful when deploying this version into production. Many, many kudos to Francesc Elies and Carst Vaartjes of Visualfabriq for their hard work, continued effort to push this feature and their work on bquery which makes use of it! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/) and Quantopian (https://www.quantopian.com/) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!**

1 0

Build doesn't pass tests
by Nikolay Mayorov 09 Jan '15

09 Jan '15

Hi all! I'm trying to build numpy on Windows 64 bit, Python 3.4.2 64 bit. I do environment setup by the following command: CMD /K "SET MSSdk=1 && SET DISTUTILS_USE_SDK=1 && "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x86_amd64" Then I cd to the newly cloned numpy folder and do: python setup.py build_ext --inplace It looks like the build process finishes correctly. But then python -c "import numpy; numpy.test()" crashes the interpreter (some tests pass before the crash). I found out that it is caused by numpy.fromfile function call. What might be the reason of that? Do I use wrong msvc compiler?

2 2

Proceedings of EuroSciPy 2014
by Ralf Gommers 06 Jan '15

06 Jan '15

On Tue, Dec 23, 2014 at 7:06 PM, <pdebuyl(a)ulb.ac.be> wrote: > Dear scientist using Python, > > We are glad to announce the publication of the proceedings of the 7th > European > Conference on Python in Science, EuroSciPy 2014, still in 2014! > > The proceedings cover various scientific fields in which Python and its > scientific libraries are used. You may obtain the table of contents and > all the > articles on the arXiv at http://arxiv.org/abs/1412.7030 > For convenience, the articles' titles are listed below. > > It is a useful reference to have as the publication of software-related > scientific work is not always straightforward. > > Thanks go to all authors and reviewers for their contributions. The reviews > were conducted publicly at > https://github.com/euroscipy/euroscipy_proceedings > > Pierre de Buyl & Nelle Varoquaux, editors > > PS: there was no large announcement of the proceedings of EuroSciPy 2013. > In the > hope that this can increase their visibility, here is the URL > Proceedings of EuroSciPy 2013: http://arxiv.org/abs/1405.0166 > > Pierre de Buyl, Nelle Varoquaux: Preface > Jérôme Kieffer, Giannis Ashiotis: PyFAI: a Python library for high > performance azimuthal integration on GPU > Andrew Leonard, Huw Morgan: Temperature diagnostics of the solar > atmosphere using SunPy > Bastian Venthur, Benjamin Blankertz: Wyrm, A Pythonic Toolbox for > Brain-Computer Interfacing > Christophe Pouzat, Georgios Is. Detorakis: SPySort: Neuronal Spike Sorting > with Python > Thomas Cokelaer, Julio Saez-Rodriguez: Using Python to Dive into > Signalling Data with CellNOpt and BioServices > Davide Monari, Francesco Cenni, Erwin Aertbeliën, Kaat Desloovere: > Py3DFreeHandUS: a library for voxel-array reconstruction using > Ultrasonography and attitude sensors > Esteban Fuentes, Hector E. Martinez: SClib, a hack for straightforward > embedded C functions in Python > Jamie A Dean, Liam C Welsh, Kevin J Harrington, Christopher M Nutting, > Sarah L Gulliford: Predictive Modelling of Toxicity Resulting from > Radiotherapy Treatments of Head and Neck Cancer > Rebecca R. Murphy, Sophie E. Jackson, David Klenerman: pyFRET: A Python > Library for Single Molecule Fluorescence Data Analysis > Robert Cimrman: Enhancing SfePy with Isogeometric Analysis > Steve Brasier, Fred Pollard: A Python-based Post-processing Toolset For > Seismic Analyses > Vladimír Lukeš, Miroslav Jiřík, Alena Jonášová, Eduard Rohan, Ondřej > Bublík, Robert Cimrman: Numerical simulation of liver perfusion: from CT > scans to FE model > > _______________________________________________ > euroscipy-org mailing list > euroscipy-org(a)python.org > https://mail.python.org/mailman/listinfo/euroscipy-org >

1 0

edge-cases of ellipsis indexing
by Antony Lee 06 Jan '15

06 Jan '15

While trying to reproduce various fancy indexings for astropy's FITS sections (a loaded-on-demand array), I found the following interesting behavior: >>> np.array([1])[..., 0] array(1) >>> np.array([1])[0] 1 >>> np.array([1])[(0,)] 1 The docs say "Ellipsis expand to the number of : objects needed to make a selection tuple of the same length as x.ndim.", so it's not totally clear to me how to explain that difference in the results. Antony

3 3