Mailman 3 May 2015 - NumPy-Discussion

MaskedArray compatibility decorators
by J.Goutin 18 May '15

18 May '15

Hello, I created 2 decorators for improve compatibility of "numpy.ma.MaskedArray" with functions that don't support them. This simply convert MaskedArray to classical ndarray with masked value converted to NaN, use the function, and reconvert to MaskedArray. @MaArrayToNaNKeepMask : Re-use the source mask on the output. @MaArrayToNaNFixInvalid : Replace invalid values by mask on the output. Source: import numpy as np def MaArrayToNaNKeepMask(func): """ MaArray to ndArray with nan decorator. Keep mask from originale ndArray. """ def wrapper(MaArray, *args, **kwargs): try: Mask = MaArray.mask fill = MaArray.fill_value return np.ma.masked_array(func(MaArray.filled(np.NaN),*args, **kwargs), mask=Mask, fill_value=fill) except: return func(MaArray, *args, **kwargs) return wrapper def MaArrayToNaNFixInvalid(func): """ MaArray to ndArray with nan decorator. Recreate mask from invalid points. """ def wrapper(MaArray, *args, **kwargs): try: fill = MaArray.fill_value return np.ma.fix_invalid(func(MaArray.filled(np.NaN), *args, **kwargs), fill_value=fill) except: return func(MaArray, *args, **kwargs) return wrapper Exemple: import skimage.transform @MaArrayToNaNFixInvalid def maresize(image, output_shape, order=1, mode='constant', cval=0, clip=True, preserve_range=True): return skimage.transform.resize(image, output_shape, order, mode, cval, clip, preserve_range) I think it may be usefull to include this directly on numpy. There is a lot of functions that don't work directly with MaskedArray.

1 0

ANN: NumPy Developer Meeting: July 7th @ SciPy 2015 in Austin
by Nathaniel Smith 18 May '15

18 May '15

Hi all, I wanted to announce that the numpy core team will be organizing a whole-day face-to-face developer meeting on July 7 this year at the SciPy conference in Austin, TX. (This is the second day of the tutorials and the day before the conference proper starts.) This will be a working meeting to discuss and address numpy-related issues, particularly ones that are too big to fit in a github issue, like governance and release management and where we want to be in five years. (We'll be talking more between now and then about more detailed logistics and agenda and so forth, but I wanted to get this out now.) If you're reading this and interested in these issues, then you're invited :-). -n -- Nathaniel J. Smith -- http://vorpus.org

2 2

[ANN] bcolz v0.9.0
by Valentin Haenel 17 May '15

17 May '15

====================== Announcing bcolz 0.9.0 ====================== What's new ========== This is mostly a smallish feature and bugfix release. One large topic was implementing 'addcol' and 'delcol' to properly handle on-disk tables. 'addcol' now has a new keyword argument 'move' that allows you to specify if you want to move or copy the data. 'delcol' has a new keyword argument 'keep' which allows you preserve the data on disk when removing a column. Additionally, ctable now supports an 'auto_flush' keyword that makes it flush to disk automatically after any methods that may write data. Another important aspect is handling the GIL. In this release, we do keep the GIL while calling Blosc compress and decompress in order to support lock-free operation of newer Blosc versions (1.5.x and beyond) that no longer have a global state. Furthermore we now distribute the 'carray_ext.pxd' as part of the package via PyPi to ease building applications on bcolz, for example *bquery*. Finally, the Sphinx based API documentation is now autogenerated from the docstrings in the Python sources. For the full list, please check the release notes at: https://github.com/Blosc/bcolz/blob/v0.9.0/RELEASE_NOTES.rst What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!**

1 0

Create a n-D grid; meshgrid alternative
by Stefan Otte 14 May '15

14 May '15

Hey, quite often I want to evaluate a function on a grid in a n-D space. What I end up doing (and what I really dislike) looks something like this: x = np.linspace(0, 5, 20) M1, M2 = np.meshgrid(x, x) X = np.column_stack([M1.flatten(), M2.flatten()]) X.shape # (400, 2) fancy_function(X) I don't think I ever used `meshgrid` in any other way. Is there a better way to create such a grid space? I wrote myself a little helper function: def gridspace(linspaces): return np.column_stack([space.flatten() for space in np.meshgrid(*linspaces)]) But maybe something like this should be part of numpy? Best, Stefan

5 10

Help loading data into pandas
by Vincent Davis 14 May '15

14 May '15

I have a large (~400mb) csv file I am trying to open in Pandas. When I don't specify the dtype and open it with the following command It appears to work. df = pd.io.parsers.read_csv(CSVFILECLEAN2013, quotechar='"', low_memory=False, na_values='') If I try to specify the dtype for each field I get an error but no hint as to where I should look. I have "cleaned" the csv by checking that all values that should be an int for a float are either blank or can be cast as a float or a int. I guess my question is, can I get a more useful error message or is there a hint as to where the problem is that I am not seeing. Exception Traceback (most recent call last) <ipython-input-2-8715d8cbaa54> in <module>() 3 import load_data 4 import numpy as np ----> 5 df2 = load_data.load('jeffco_2013') /Users/vmd/GitHub/Jeffco-Properties/tools/load_data.py in load(data) 47 def load(data): 48 files = dict(jeffco_2013 = '/Users/vmd/GitHub/Jeffco-Properties/Data/JeffersonCo/Datasets/2013_clean_Jeffco_ATSDTA_ATSP600.csv') ---> 49 return pd.io.parsers.read_csv(files[data], quotechar='"', low_memory=False, na_values='', dtype=DATASHAPE) /Users/vmd/anaconda/envs/py34/lib/python3.4/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines) 468 skip_blank_lines=skip_blank_lines) 469 --> 470 return _read(filepath_or_buffer, kwds) 471 472 parser_f.__name__ = name /Users/vmd/anaconda/envs/py34/lib/python3.4/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds) 254 return parser 255 --> 256 return parser.read() 257 258 _parser_defaults = { /Users/vmd/anaconda/envs/py34/lib/python3.4/site-packages/pandas/io/parsers.py in read(self, nrows) 713 raise ValueError('skip_footer not supported for iteration') 714 --> 715 ret = self._engine.read(nrows) 716 717 if self.options.get('as_recarray'): /Users/vmd/anaconda/envs/py34/lib/python3.4/site-packages/pandas/io/parsers.py in read(self, nrows) 1162 1163 try: -> 1164 data = self._reader.read(nrows) 1165 except StopIteration: 1166 if nrows is None: pandas/parser.pyx in pandas.parser.TextReader.read (pandas/parser.c:7426)() pandas/parser.pyx in pandas.parser.TextReader._read_rows (pandas/parser.c:8484)() pandas/parser.pyx in pandas.parser.TextReader._convert_column_data (pandas/parser.c:9795)() pandas/parser.pyx in pandas.parser.TextReader._convert_tokens (pandas/parser.c:10403)() pandas/parser.pyx in pandas.parser.TextReader._convert_with_dtype (pandas/parser.c:11257)() Exception: Integer column has NA values Vincent Davis 720-301-3003

2 2

ANN: SciPy 2015 Talk & Poster Selections Announced Today, Early Bird Deadline 5/22
by Courtenay Godshall (Enthought) 14 May '15

14 May '15

The talks & posters for the 2015 SciPy Conference were announced today: http://scipy2015.scipy.org/ehome/115969/292868/? <http://scipy2015.scipy.org/ehome/115969/292868/?&> &. Early bird registration deadline was extended (final) to 5/22 - hope we'll see you this year!

1 0

ANN: Scipy 0.16.0 beta 1 release
by Ralf Gommers 13 May '15

13 May '15

Hi all, I'm pleased to announce the availability of the first beta release of Scipy 0.16.0. Please try this beta and report any issues on the Github issue tracker or on the scipy-dev mailing list. This first beta is a source-only release; binary installers will follow (probably next week). Source tarballs and the full release notes can be found at https://sourceforge.net/projects/scipy/files/scipy/0.16.0b1/. Part of the release notes copied below. Thanks to everyone who contributed to this release! Ralf ========================== SciPy 0.16.0 Release Notes ========================== .. note:: Scipy 0.16.0 is not released yet! SciPy 0.16.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.15.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Highlights of this release include: - A Cython API for BLAS/LAPACK in `scipy.linalg` - A new benchmark suite. It's now straightforward to add new benchmarks, and they're routinely included with performance enhancement PRs. - Support for the second order sections (SOS) format in `scipy.signal`. New features ============ Benchmark suite --------------- The benchmark suite has switched to using `Airspeed Velocity <http://spacetelescope.github.io/asv/>`__ for benchmarking. You can run the suite locally via ``python runtests.py --bench``. For more details, see ``benchmarks/README.rst``. `scipy.linalg` improvements --------------------------- A full set of Cython wrappers for BLAS and LAPACK has been added in the modules `scipy.linalg.cython_blas` and `scipy.linalg.cython_lapack`. In Cython, these wrappers can now be cimported from their corresponding modules and used without linking directly against BLAS or LAPACK. The functions `scipy.linalg.qr_delete`, `scipy.linalg.qr_insert` and `scipy.linalg.qr_update` for updating QR decompositions were added. The function `scipy.linalg.solve_circulant` solves a linear system with a circulant coefficient matrix. The function `scipy.linalg.invpascal` computes the inverse of a Pascal matrix. The function `scipy.linalg.solve_toeplitz`, a Levinson-Durbin Toeplitz solver, was added. Added wrapper for potentially useful LAPACK function ``*lasd4``. It computes the square root of the i-th updated eigenvalue of a positive symmetric rank-one modification to a positive diagonal matrix. See its LAPACK documentation and unit tests for it to get more info. Added two extra wrappers for LAPACK least-square solvers. Namely, they are ``*gelsd`` and ``*gelsy``. Wrappers for the LAPACK ``*lange`` functions, which calculate various matrix norms, were added. Wrappers for ``*gtsv`` and ``*ptsv``, which solve ``A*X = B`` for tri-diagonal matrix ``A``, were added. `scipy.signal` improvements --------------------------- Support for second order sections (SOS) as a format for IIR filters was added. The new functions are: * `scipy.signal.sosfilt` * `scipy.signal.sosfilt_zi`, * `scipy.signal.sos2tf` * `scipy.signal.sos2zpk` * `scipy.signal.tf2sos` * `scipy.signal.zpk2sos`. Additionally, the filter design functions `iirdesign`, `iirfilter`, `butter`, `cheby1`, `cheby2`, `ellip`, and `bessel` can return the filter in the SOS format. The function `scipy.signal.place_poles`, which provides two methods to place poles for linear systems, was added. The option to use Gustafsson's method for choosing the initial conditions of the forward and backward passes was added to `scipy.signal.filtfilt`. New classes ``TransferFunction``, ``StateSpace`` and ``ZerosPolesGain`` were added. These classes are now returned when instantiating `scipy.signal.lti`. Conversion between those classes can be done explicitly now. An exponential (Poisson) window was added as `scipy.signal.exponential`, and a Tukey window was added as `scipy.signal.tukey`. The function for computing digital filter group delay was added as `scipy.signal.group_delay`. The functionality for spectral analysis and spectral density estimation has been significantly improved: `scipy.signal.welch` became ~8x faster and the functions `scipy.signal.spectrogram`, `scipy.signal.coherence` and `scipy.signal.csd` (cross-spectral density) were added. `scipy.signal.lsim` was rewritten - all known issues are fixed, so this function can now be used instead of ``lsim2``; ``lsim`` is orders of magnitude faster than ``lsim2`` in most cases. `scipy.sparse` improvements --------------------------- The function `scipy.sparse.norm`, which computes sparse matrix norms, was added. The function `scipy.sparse.random`, which allows to draw random variates from an arbitrary distribution, was added. `scipy.spatial` improvements ---------------------------- `scipy.spatial.cKDTree` has seen a major rewrite, which improved the performance of the ``query`` method significantly, added support for parallel queries, pickling, and options that affect the tree layout. See pull request 4374 for more details. The function `scipy.spatial.procrustes` for Procrustes analysis (statistical shape analysis) was added. `scipy.stats` improvements -------------------------- The Wishart distribution and its inverse have been added, as `scipy.stats.wishart` and `scipy.stats.invwishart`. The Exponentially Modified Normal distribution has been added as `scipy.stats.exponnorm`. The Generalized Normal distribution has been added as `scipy.stats.gennorm`. All distributions now contain a ``random_state`` property and allow specifying a specific ``numpy.random.RandomState`` random number generator when generating random variates. Many statistical tests and other `scipy.stats` functions that have multiple return values now return ``namedtuples``. See pull request 4709 for details. `scipy.optimize` improvements ----------------------------- A new derivative-free method DF-SANE has been added to the nonlinear equation system solving function `scipy.optimize.root`. Deprecated features =================== ``scipy.stats.pdf_fromgamma`` is deprecated. This function was undocumented, untested and rarely used. Statsmodels provides equivalent functionality with ``statsmodels.distributions.ExpandedNormal``. ``scipy.stats.fastsort`` is deprecated. This function is unnecessary, ``numpy.argsort`` can be used instead. ``scipy.stats.signaltonoise`` and ``scipy.stats.mstats.signaltonoise`` are deprecated. These functions did not belong in ``scipy.stats`` and are rarely used. See issue #609 for details. ``scipy.stats.histogram2`` is deprecated. This function is unnecessary, ``numpy.histogram2d`` can be used instead. Backwards incompatible changes ============================== The deprecated global optimizer ``scipy.optimize.anneal`` was removed. The following deprecated modules have been removed: ``scipy.lib.blas``, ``scipy.lib.lapack``, ``scipy.linalg.cblas``, ``scipy.linalg.fblas``, ``scipy.linalg.clapack``, ``scipy.linalg.flapack``. They had been deprecated since Scipy 0.12.0, the functionality should be accessed as `scipy.linalg.blas` and `scipy.linalg.lapack`. The deprecated function ``scipy.special.all_mat`` has been removed. The deprecated functions ``fprob``, ``ksprob``, ``zprob``, ``randwcdf`` and ``randwppf`` have been removed from `scipy.stats`. Other changes ============= The version numbering for development builds has been updated to comply with PEP 440. Building with ``python setup.py develop`` is now supported. Authors ======= * @axiru + * @endolith * Elliott Sales de Andrade + * Anne Archibald * Yoshiki Vázquez Baeza + * Sylvain Bellemare * Felix Berkenkamp + * Raoul Bourquin + * Matthew Brett * Per Brodtkorb * Christian Brueffer * Lars Buitinck * Evgeni Burovski * Steven Byrnes * CJ Carey * George Castillo + * Alex Conley + * Liam Damewood + * Rupak Das + * Abraham Escalante + * Matthias Feurer + * Eric Firing + * Clark Fitzgerald * Chad Fulton * André Gaul * Andreea Georgescu + * Christoph Gohlke * Andrey Golovizin + * Ralf Gommers * J.J. Green + * Alex Griffing * Alexander Grigorievskiy + * Hans Moritz Gunther + * Jonas Hahnfeld + * Charles Harris * Ian Henriksen * Andreas Hilboll * Åsmund Hjulstad + * Jan Schlüter + * Janko Slavič + * Daniel Jensen + * Johannes Ballé + * Terry Jones + * Amato Kasahara + * Eric Larson * Denis Laxalde * Antony Lee * Gregory R. Lee * Perry Lee + * Loïc Estève * Martin Manns + * Eric Martin + * Matěj Kocián + * Andreas Mayer + * Nikolay Mayorov + * Robert McGibbon + * Sturla Molden * Nicola Montecchio + * Eric Moore * Jamie Morton + * Nikolas Moya + * Maniteja Nandana + * Andrew Nelson * Joel Nothman * Aldrian Obaja * Regina Ongowarsito + * Paul Ortyl + * Pedro López-Adeva Fernández-Layos + * Stefan Peterson + * Irvin Probst + * Eric Quintero + * John David Reaver + * Juha Remes + * Thomas Robitaille * Clancy Rowley + * Tobias Schmidt + * Skipper Seabold * Aman Singh + * Eric Soroos * Valentine Svensson + * Julian Taylor * Aman Thakral + * Helmut Toplitzer + * Fukumu Tsutsumi + * Anastasiia Tsyplia + * Jacob Vanderplas * Pauli Virtanen * Matteo Visconti + * Warren Weckesser * Florian Wilhelm + * Nathan Woods * Haochen Wu + * Daan Wynen + A total of 93 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete.

1 0

Integral Equation Solver
by Pierson, Oliver C 13 May '15

13 May '15

Hi All, Awhile back I had written some code to solve Volterra integral equations (integral equations where one of the integration bounds is a variable). The code is available on Github (https://github.com/oliverpierson/volterra). Just curious if there'd be any interest in adding this to Numpy? I still have some work to do on the code. However, before I invest too much time, I was trying to get a feel for the interest in this functionality. Please let me know if you have any questions. Thanks, Oliver

2 1

Re: [Numpy-discussion] python is cool
by Roland Schulz 13 May '15

13 May '15

Hi, I think the best way to solve this issue to not use a state at all. It is fast, reproducible even in parallel (if wanted), and doesn't suffer from the shared issue. Would be nice if numpy provided such a stateless RNG as implemented in Random123: www.deshawresearch.com/resources_random123.html Roland On Tue, May 12, 2015 at 2:18 PM, Neal Becker <ndbecker2(a)gmail.com> wrote: > In order to make sure all my random number generators have good > independence, it is a good practice to use a single shared instance > (because > it is already known to have good properties). A less-desirable alternative > is to used rng's seeded with different starting states - in this case the > independence properties are not generally known. > > So I have some fairly deeply nested data structures (classes) that > somewhere > contain a reference to a RandomState object. > > I need to be able to clone these data structures, producing new independent > copies, but I want the RandomState part to be the shared, singleton rs > object. > > In python, no problem: > > --- > from numpy.random import RandomState > > class shared_random_state (RandomState): > def __init__ (self, rs): > RandomState.__init__(self, rs) > > def __deepcopy__ (self, memo): > return self > --- > > Now I can copy.deepcopy the data structures, but the randomstate part is > shared. I just use > > rs = shared_random_state (random.RandomState(0)) > > and provide this rs to all my other objects. Pretty nice! > > -- > Those who fail to understand recursion are doomed to repeat it > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- ORNL/UT Center for Molecular Biophysics cmb.ornl.gov 865-241-1537, ORNL PO BOX 2008 MS6309

2 1

python is cool
by Neal Becker 12 May '15

12 May '15

In order to make sure all my random number generators have good independence, it is a good practice to use a single shared instance (because it is already known to have good properties). A less-desirable alternative is to used rng's seeded with different starting states - in this case the independence properties are not generally known. So I have some fairly deeply nested data structures (classes) that somewhere contain a reference to a RandomState object. I need to be able to clone these data structures, producing new independent copies, but I want the RandomState part to be the shared, singleton rs object. In python, no problem: --- from numpy.random import RandomState class shared_random_state (RandomState): def __init__ (self, rs): RandomState.__init__(self, rs) def __deepcopy__ (self, memo): return self --- Now I can copy.deepcopy the data structures, but the randomstate part is shared. I just use rs = shared_random_state (random.RandomState(0)) and provide this rs to all my other objects. Pretty nice! -- Those who fail to understand recursion are doomed to repeat it

1 0