Mailman 3 May 2015 - NumPy-Discussion

Bug in np.nonzero / Should index returning functions return ndarray subclasses?
by Jaime Fernández del Río 12 May '15

12 May '15

There is a reported bug (issue #5837 <https://github.com/numpy/numpy/issues/5837>) regarding different returns from np.nonzero with 1-D vs higher dimensional arrays. A full summary of the differences can be seen from the following output: >>> class C(np.ndarray): pass ... >>> a = np.arange(6).view(C) >>> b = np.arange(6).reshape(2, 3).view(C) >>> anz = a.nonzero() >>> bnz = b.nonzero() >>> type(anz[0]) <type 'numpy.ndarray'> >>> anz[0].flags C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> anz[0].base >>> type(bnz[0]) <class '__main__.C'> >>> bnz[0].flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : False ALIGNED : True UPDATEIFCOPY : False >>> bnz[0].base array([[0, 1], [0, 2], [1, 0], [1, 1], [1, 2]]) The original bug report was only concerned with the non-writeability of higher dimensional array returns, but there are more differences: 1-D always returns an ndarray that owns its memory and is writeable, but higher dimensional arrays return views, of the type of the original array, that are non-writeable. I have a branch that attempts to fix this by making both 1-D and n-D arrays: 1. return a view, never the base array, 2. return an ndarray, never a subclass, and 3. return a writeable view. I guess the most controversial choice is #2, and in fact making that change breaks a few tests. I nevertheless think that all of the index returning functions (nonzero, argsort, argmin, argmax, argpartition) should always return a bare ndarray, not a subclass. I'd be happy to be corrected, but I can't think of any situation in which preserving the subclass would be needed for these functions. Since we are changing the returns of a few other functions in 1.10 (diagonal, diag, ravel), it may be a good moment to revisit the behavior for these other functions. Any thoughts? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

5 7

Proposed deprecations for 1.10: dot corner cases
by Nathaniel Smith 11 May '15

11 May '15

Hi all, I'd like to suggest that we go ahead and add deprecation warnings to the following operations. This doesn't commit us to changing anything on any particular time scale, but it gives us more options later. 1) dot(A, B) where A and B *both* have *3 or more dimensions*: currently, this does a weird "outer product" thing, where it computes all pairwise matrix products. We've had numerous discussions about why this is suboptimal, and it contradicts the PEP 465 semantics for @, which broadcast + vectorize over extra dimensions. (If you have a vectorized version, then the outer product one is easy to derive; if you have only the outer product version .) While dot() is widely used in general, this particular varient is very, very rarely used. I propose we issue a FutureWarning here, so as to lay the groundwork for someday eventually making dot() and @ the same. 2) dot(A, B) where one of the argument is a scalar: currently, this does scalar multiplication. There is no logically consistent motivation for this, it violates TOOWTDI, and again it is inconsistent with the PEP semantics for @ (which are that this case should be an error). (NB for those still using np.matrix: scalar * np.matrix will still be supported regardless; this would only affect expressions where you actually call the dot() function.) I propose to make this a DeprecationWarning. -- Nathaniel J. Smith -- http://vorpus.org

3 5

ANN: pandas 0.16.1 released
by Jeff Reback 11 May '15

11 May '15

Hello, We are proud to announce v0.16.1 of pandas, a minor release from 0.16.0. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was a release of 7 weeks with 222 commits by 57 authors encompassing 85 issues. We recommend that all users upgrade to this version. *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Highlights of this release include: - Support for *CategoricalIndex*, a category based index, see here <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0161-enh…> - New section on how-to-contribute to *pandas*, see here <http://pandas.pydata.org/pandas-docs/stable/contributing.html> - Revised "Merge, join, and concatenate" documentation, including graphical examples to make it easier to understand each operations, see here <http://pandas.pydata.org/pandas-docs/stable/merging.html> - New method *sample* for drawing random samples from Series, DataFrames and Panels. See here <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0161-enh…> - The default *Index* printing has changed to a more uniform format, see here <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0161-ind…> - *BusinessHour* datetime-offset is now supported, see here <http://pandas.pydata.org/pandas-docs/stable/timeseries.html#business-hour> - Further enhancement to the *.str* accessor to make string operations easier, see here <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0161-enh…> See the Whatsnew in v0.16.1 <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-16-1-may-11-20…> Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.8 macosx wheels are courtesy of Matthew Brett Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.16.1 release - - Alfonso MHC - Andy Hayden - Artemy Kolchinsky - Chris Gilmer - Chris Grinolds - Dan Birken - David BROCHART - David Hirschfeld - David Stephens - Dr. Leo - Evan Wright - Frans van Dunné - Hatem Nassrat - Henning Sperr - Hugo Herter - Jan Schulz - Jeff Blackburne - Jeff Reback - Jim Crist - Jonas Abernot - Joris Van den Bossche - Kerby Shedden - Leo Razoumov - Manuel Riel - Mortada Mehyar - Nick Burns - Nick Eubank - Olivier Grisel - Phillip Cloud - Pietro Battiston - Roy Hyunjin Han - Sam Zhang - Scott Sanderson - Stephan Hoyer - Tiago Antao - Tom Ajamian - Tom Augspurger - Tomaz Berisa - Vikram Shirgur - Vladimir Filimonov - William Hogman - Yasin A - Younggun Kim - behzad nouri - dsm054 - floydsoft - flying-sheep - gfr - jnmclarty - jreback - ksanghai - lucas - mschmohl - ptype - rockg - scls19fr - sinhrks

1 0

read not byte aligned records
by Gmail 10 May '15

10 May '15

Hi, I am developping a code to read binary files (MDF, Measurement Data File). In its previous version 3, data was always byte aligned. I used widely numpy.core.records module (fromstring, fromfile) showing good performance to read and unpack data on the fly. However, in the latest version 4, not byte aligned data is possible. It allows to reduce size of file, especially when raw data is not actually recorded on bytes, like 10bits for analog converter. For instance, a record structure could be: uint64, float32, uint8, unit10, padding 6bits, uint9, padding 7bits, uint24, uint24, uint24, etc. I found a way using instead of numpy.core.records the bitstring module to read these records when not aligned but performance is much worse (I did not try cython implementation though but in python like x10) ? Would there be a pure numpy way to do ? Regards Aymeric

5 5

Using gentxt to import a csv with a string class label and hundreds of integer features
by Dammy 08 May '15

08 May '15

Hi, I am trying to use numpy.gentxt to import a csv for classification using scikit-learn. The first column in the csv is a string type class label while 200+ extra columns are integer features. Please I wish to find out how I can use the gentext function to specify a dtype of string for the first column while specifying int type for all other columns. I have tried using "dtype=None" as shown below, but when I print dataset.shape, I get (number_or_rows,) i.e no columns are read in: dataset = np.genfromtxt(file,delimiter=',', skip_header=True) I also tried setting the dtypes as shown in the examples below, but I get the same error as dtype=None: a: dataset = np.genfromtxt(file,delimiter=',', skip_header=True, dtype=['S2'] + [ int for n in range(241)],) b: dataset = np.genfromtxt(file,delimiter=',', skip_header=True, dtype=['S2'] + [ int for n in range(241)],names=True ) Any thoughts? Thanks for your assistance. Dammy -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Using-gentxt-to-import-a-csv-wi… Sent from the Numpy-discussion mailing list archive at Nabble.com.

3 2

ANN: PyTables 3.2.0 (final) released!
by Francesc Alted 06 May '15

06 May '15

=========================== Announcing PyTables 3.2.0 =========================== We are happy to announce PyTables 3.2.0. ******************************* IMPORTANT NOTICE: If you are a user of PyTables, it needs your help to keep going. Please read the next thread as it contains important information about the future (or the lack of it) of the project: https://groups.google.com/forum/#!topic/pytables-users/yY2aUa4H7W4 Thanks! ******************************* What's new ========== This is a major release of PyTables and it is the result of more than a year of accumulated patches, but most specially it fixes a couple of nasty problem with indexed queries not returning the correct results in some scenarios. There are many usablity and performance improvements too. In case you want to know more in detail what has changed in this version, please refer to: http://www.pytables.org/release_notes.html You can install it via pip or download a source package with generated PDF and HTML docs from: http://sourceforge.net/projects/pytables/files/pytables/3.2.0 For an online version of the manual, visit: http://www.pytables.org/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

ANN: python-blosc 1.2.7 released
by Francesc Alted 06 May '15

06 May '15

============================= Announcing python-blosc 1.2.7 ============================= What is new? ============ Updated to use c-blosc v1.6.1. Although that this supports AVX2, it is not enabled in python-blosc because we still need a way to devise how to detect AVX2 in the underlying platform. At any rate, c-blosc 1.6.1 fixed an important bug in the blosclz codec that a release was deemed important. For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate object manipulations that are memory-bound (http://www.blosc.org/docs/StarvingCPUs.pdf). See http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on how much speed it can achieve in some datasets. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library. There is also a handy tool built on Blosc called Bloscpack (https://github.com/Blosc/bloscpack). It features a commmand line interface that allows you to compress large binary datafiles on-disk. It also comes with a Python API that has built-in support for serializing and deserializing Numpy arrays both on-disk and in-memory at speeds that are competitive with regular Pickle/cPickle machinery. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/Blosc/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://python-blosc.blosc.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/Blosc/python-blosc/blob/master/LICENSES for more details. ---- **Enjoy data!** -- Francesc Alted

1 0

Dispatch rules for binary operations on ndarrays
by Nathaniel Smith 06 May '15

06 May '15

I just wanted to draw the list's attention to a discussion happening on the tracker, about the details of how methods like ndarray.__add__ are implemented, and how this interacts with the new __numpy_ufunc__ method that will make it possible for third party libraries to override arbitrary ufuncs starting in (hopefully) 1.10: https://github.com/numpy/numpy/issues/5844 The details are somewhat arcane, but very important for anyone who implements ndarray-like objects or (to a lesser extent) anyone who subclasses ndarray. So feedback very welcome. -n

1 0

import scipy.linalg is hanging on Marvell armada 370
by Alexander Brezinov 05 May '15

05 May '15

Hello The import of scipy.linalg is hanging in DOUBLE_mutiply function (BINARY_LOOP) in umath.so. After attaching the gdb and dumping the local varibles the args are empty strings. Could you please advise if this is known issue? I just search the mailing list and could not find any solution for the problem. I am running: kernel 3.2.36 + Debian wheezy on ARMv71 armhf CPU Armada 370 Marvell python 2.7.3 scipy 0.15.1 numpy 1.9.2 The problem could be reproduced by launching python and importing scipy.linalg(import linalg) I also run the same OS on qemu and was not able to reproduce the issue. Similar architecture such as rasbery pi (ARMv7 armhf) is fine. Also if using software floating point intead of hardware floating point on the same Armada 370 (ARMv7) working just fine. Thank you for any comments or suggestions in advance, Alex

2 1

Should ndarray subclasses support the keepdims arg?
by Allan Haldane 05 May '15

05 May '15

Hello all, A question: Many ndarray methods (eg sum, mean, any, min) have a "keepdims" keyword argument, but ndarray subclass methods sometimes don't. The 'matrix' subclass doesn't, and numpy functions like 'np.sum' intentionally drop/ignore the keepdims argument when called with an ndarray subclass as first argument. This means you can't always use ndarray subclasses as 'drop in' replacement for ndarrays if the code uses keepdims (even indirectly), and it means code that deals with keepdims (eg np.sum and more) has to detect ndarray subclasses and drop keepdims even if the subclass supports it (since there is no good way to detect support). It seems to me that if we are going to use inheritance, subclass methods should keep the signature of the parent class methods. What does the list think? ---- Details: ---- This problem comes up in a PR I'm working on (#5706) to add the keepdims arg to masked array methods. In order to support masked matrices (which a lot of unit tests check), I would have to detect and drop the keepdims arg to avoid an exception. This would be solved if the matrix class supported keepdims (plus an update to np.sum). Similarly, `np.sum(mymaskedarray, keepdims=True)` does not respect keepdims, but it could work if all subclasses supported keepdims. I do not foresee immediate problems with adding keepdims to the matrix methods, except that it would be an unused argument. Modifying `np.sum` to always pass on the keepdims arg is trickier, since it would break any code that tried to np.sum a subclass that doesn't support keepdims, eg pandas.DataFrame. **kwargs tricks might work. But if it's permissible I think it would be better to require subclasses to support all the keyword args ndarray supports. Allan

4 5

NumPy-Discussion May 2015