Mailman 3 October 2015 - NumPy-Discussion

Cython-based OpenMP-accelerated quartic polynomial solver
by Juha Jeronen Oct. 6, 2015

Oct. 6, 2015

Hi all, I recently developed a Cython-based, OpenMP-accelerated quartic (and cubic, quadratic) polynomial solver to address a personal research need for quickly solving a very large number of independent low-degree polynomial equations for both real and complex coefficients. For example, on an Intel i7-4710MQ, running with 8 threads this code solves approximately 1.2e7 quartic polynomial equations per second. (With OMP_NUM_THREADS=4 the solver gives approximately 8.9e6 solutions per second, so it seems HyperThreading in this case helps about 30%.) The algorithms for cubics and quadratics come from Numerical Recipes (3rd ed.), and the quartic problem is internally reduced to a cubic and two quadratics, using well-known standard tricks. Since to my understanding, for solving polynomial equations NumPy only provides roots(), which works one problem at a time, I'm writing to inquire whether there would be interest to include this functionality to NumPy, if I contribute the code (and clean it up for integration)? I have some previous open source contribution experience. I have authored the IVTC and Phosphor deinterlacers for VLC, and a modular postprocessing filter framework for the Panda3D game engine (posted at the forums on panda3d.org, projected for version 1.10). Currently the polynomial solver is available in a git repository hosted by our university, the relevant files being: https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… I'm working on a research grant, which allows me to hold the copyright; license is BSD. Polysolve2 is the fast Cython+OpenMP version, while polysolve is an earlier (slower) experimental version using only NumPy array operations. The example module contains a usage example, and setup (in the standard manner) instructs distutils and Cython as to how to build polysolve2 (including setting the relevant math flags and enabling OpenMP). (The setup script includes also some stuff specific to my solver for the Ziegler problem; basically, the "tworods" module can be ignored. I apologize for the inconvenience.) Thoughts? Best regards, -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen(a)jyu.fi University of Jyväskylä Department of Mathematical Information Technology -------------------------------------------------

12 39

Let's move forward with the current governance document.
by Travis Oliphant Oct. 5, 2015

Oct. 5, 2015

Hi everyone, After some further thought and spending quite a bit of time re-reading the discussion on a few threads, I now believe that my request to be on the steering council might be creating more trouble than it's worth. Nothing matters to me more than seeing NumPy continue to grow and improve. So, I'm switching my position to supporting the adoption of the governance model outlined and just contributing as I can outside the steering council. The people on the steering council are committed to the success of NumPy and will do a great job --- they already have in contributing to the community over the past year(s). We can always revisit the question in a year if difficulties arise with the model. If my voice and other strong voices remain outside the council, perhaps we can all encourage that the intended community governance of NumPy does in fact happen, and most decisions continue to be made in the open. I had the pleasure last night of meeting one of the new NumPy core contributors, Allan Haldane. This only underscored my confidence in everyone who is contributing to NumPy today. This confidence has already been established by watching the great contributions of many talented developers who have given their time and talents to the project over the past several years. I hope that we can move on from the governance discussion and continue to promote the success of the project together. Best, -Travis

6 8

NumPy Governance Document.
by Charles R Harris Oct. 5, 2015

Oct. 5, 2015

Hi All The NumPy Governance Document <https://github.com/numpy/numpy/pull/6352> has been merged and is now in effect. Thanks to all who contributed to the discussion. And a special thanks to Nathaniel, who wrote the draft and kept it up to date as the discussion progressed. Chuck

2 1

Re: [Numpy-discussion] [pydata] ANN: pandas v0.17.0rc2 - RELEASE CANDIDATE 2
by Jeff Reback Oct. 5, 2015

Oct. 5, 2015

it should be exactly the same (they are going to release soon as well I think) - with an updated version > On Oct 5, 2015, at 2:25 PM, Big Stone <stonebig34(a)gmail.com> wrote: > > hi, > > on pypi, pandas_datareader (0.1.1) is dated from April 10th. > > Is it up-to-date with pandas 0.17rc2 ? > >> On Sunday, October 4, 2015 at 7:36:26 AM UTC+2, Matthew Brett wrote: >> Hi, >> >> On Sat, Oct 3, 2015 at 2:33 PM, Jeff Reback <jeffr...(a)gmail.com> wrote: >> > Hi, >> > >> > I'm pleased to announce the availability of the second release candidate of >> > Pandas 0.17.0. >> > Please try this RC and report any issues here: Pandas Issues >> > We will be releasing officially on October 9. >> > >> > **RELEASE CANDIDATE 2** >> > >> > From RC 1 we have: >> > >> > compat for Python 3.5 >> > compat for matplotlib 1.5.0 >> > .convert_objects is now restored to the original, and is deprecated >> > >> > This is a major release from 0.16.2 and includes a small number of API >> > changes, several new features, enhancements, and performance improvements >> > along with a large number of bug fixes. We recommend that all users upgrade >> > to this version. >> > >> > Highlights include: >> > >> > Release the Global Interpreter Lock (GIL) on some cython operations, see >> > here >> > Plotting methods are now available as attributes of the .plot accessor, see >> > here >> > The sorting API has been revamped to remove some long-time inconsistencies, >> > see here >> > Support for a datetime64[ns] with timezones as a first-class dtype, see here >> > The default for to_datetime will now be to raise when presented with >> > unparseable formats, previously this would return the original input, see >> > here >> > The default for dropna in HDFStore has changed to False, to store by default >> > all rows even if they are all NaN, see here >> > Support for Series.dt.strftime to generate formatted strings for >> > datetime-likes, see here >> > Development installed versions of pandas will now have PEP440 compliant >> > version strings GH9518 >> > Development support for benchmarking with the Air Speed Velocity library >> > GH8316 >> > Support for reading SAS xport files, see here >> > Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, >> > see here >> > Display format with plain text can optionally align with Unicode East Asian >> > Width, see here >> > Compatibility with Python 3.5 GH11097 >> > Compatibility with matplotlib 1.5.0 GH11111 >> > >> > >> > See the Whatsnew for much more information. >> > >> > Best way to get this is to install via conda from our development channel. >> > Builds for osx-64,linux-64,win-64 for Python 2.7, Python 3.4, and Python 3.5 >> > (for osx/linux) are all available. >> > >> > conda install pandas -c pandas >> >> I built OSX wheels for Pythons 2.7, 3.4, 3.5. To test: >> >> pip install --pre -f http://wheels.scipy.org pandas >> >> There were some test failures for Python 3.3 - issue here: >> >> https://github.com/pydata/pandas/issues/11232 >> >> Cheers, >> >> Matthew > > -- > You received this message because you are subscribed to the Google Groups "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

1 0

ANN: bcolz 0.11.3 released!
by Francesc Alted Oct. 5, 2015

Oct. 5, 2015

======================= Announcing bcolz 0.11.3 ======================= What's new ========== Implemented new feature (#255): bcolz.zeros() can create new ctables too, either empty or filled with zeros. (#256 @FrancescElies @FrancescAlted). Also, in previous, non announced versions (0.11.1 and 0.11.2), new dependencies were added and other fixes are there too. For a more detailed change log, see: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/model/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted

1 0

ANN: pandas v0.17.0rc2 - RELEASE CANDIDATE 2
by Jeff Reback Oct. 4, 2015

Oct. 4, 2015

Hi, I'm pleased to announce the availability of the second release candidate of Pandas 0.17.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues/10848> We will be releasing officially on October 9. **RELEASE CANDIDATE 2** >From RC 1 we have: - compat for Python 3.5 - compat for matplotlib 1.5.0 - .convert_objects is now restored to the original, and is deprecated This is a major release from 0.16.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: - Release the Global Interpreter Lock (GIL) on some cython operations, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Plotting methods are now available as attributes of the .plot accessor, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The sorting API has been revamped to remove some long-time inconsistencies, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for a datetime64[ns] with timezones as a first-class dtype, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for to_datetime will now be to raise when presented with unparseable formats, previously this would return the original input, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for dropna in HDFStore has changed to False, to store by default all rows even if they are all NaN, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for Series.dt.strftime to generate formatted strings for datetime-likes, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Development installed versions of pandas will now have PEP440 compliant version strings GH9518 <https://github.com/pydata/pandas/issues/9518> - Development support for benchmarking with the Air Speed Velocity library GH8316 <https://github.com/pydata/pandas/pull/8316> - Support for reading SAS xport files, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Display format with plain text can optionally align with Unicode East Asian Width, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Compatibility with Python 3.5 GH11097 <https://github.com/pydata/pandas/issues/11097> - Compatibility with matplotlib 1.5.0 GH11111 <https://github.com/pydata/pandas/issues/11111> See the Whatsnew <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html> for much more information. Best way to get this is to install via conda <http://pandas-docs.github.io/pandas-docs-travis/install.html#installing-pan…> from our development channel. Builds for osx-64,linux-64,win-64 for Python 2.7, Python 3.4, and Python 3.5 (for osx/linux) are all available. conda install pandas -c pandas Thanks to all who made this release happen. It is a very large release! Jeff

2 1

Numpy 1.10.0 coming Monday, 5 Oct.
by Charles R Harris Oct. 3, 2015

Oct. 3, 2015

Hi All, A heads up about the coming Numpy release. If you have discovered any problems with rc1 or rc2, please report them. Chuck

1 0