ANN: Scipy 0.15.0 beta 1 release
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, We have finally finished preparing the Scipy 0.15.0 beta 1 release. Please try it and report any issues on the scipy-dev mailing list, and/or on Github. If no surprises turn up, the final release is planned on Dec 20 in three weeks. Source tarballs and full release notes are available at https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ Binary installers should also be up soon. Best regards, Pauli Virtanen - -------------------------------------------- SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or greater. New features ============ Linear Programming Interface - - ---------------------------- The new function ``scipy.optimize.linprog`` provides a generic linear programming similar to the way ``scipy.optimize.minimize`` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported,allowing the user to monitor the progress of the algorithm. Differential_evolution, a global optimizer - - ------------------------------------------ A new ``differential_evolution`` function is available in the ``scipy.optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques. ``scipy.signal`` improvements - - ----------------------------- The function ``max_len_seq`` was added, which computes a Maximum Length Sequence (MLS) signal. ``scipy.integrate`` improvements - - -------------------------------- It is now possible to use ``scipy.integrate`` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance. ``scipy.linalg`` improvements - - ----------------------------- Add function ``orthogonal_procrustes`` for solving the procrustes linear algebra problem. ``scipy.sparse`` improvements - - ----------------------------- ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its main input. ``scipy.special`` improvements - - ------------------------------ Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``. New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added. ``scipy.sparse.csgraph`` improvements - - ------------------------------------- Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added. ``scipy.stats`` improvements - - ---------------------------- Added a Dirichlet distribution as multivariate distribution. The new function ``scipy.stats.median_test`` computes Mood's median test. The new function ``scipy.stats.combine_pvalues`` implements Fisher's and Stouffer's methods for combining p-values. ``scipy.stats.describe`` returns a namedtuple rather than a tuple, allowing users to access results by index or by name. Deprecated features =================== The ``scipy.weave`` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: `https://github.com/scipy/weave`_. It is a pure Python package, and can easily be installed with ``pip install weave``. ``scipy.special.bessel_diff_formula`` is deprecated. It is a private function, and therefore will be removed from the public API in a following release. Backwards incompatible changes ============================== scipy.ndimage - - ------------- The functions ``scipy.ndimage.minimum_positions``, ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return positions as ints instead of floats. scipy.integrate - - --------------- The format of banded Jacobians in ``scipy.integrate.ode`` solvers is changed. Note that the previous documentation of this feature was erroneous. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRyaf8ACgkQ6BQxb7O0pWC7XQCeNtdJD4ZNDXvFeNFs7N3KjQn6 8QkAoK3pFmhMrTwCrgkusl+fRNMboN2r =WSpM -----END PGP SIGNATURE-----
Hi, On Sun, Nov 23, 2014 at 3:13 PM, Pauli Virtanen <pav@iki.fi> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear all,
We have finally finished preparing the Scipy 0.15.0 beta 1 release. Please try it and report any issues on the scipy-dev mailing list, and/or on Github.
If no surprises turn up, the final release is planned on Dec 20 in three weeks.
Source tarballs and full release notes are available at https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ Binary installers should also be up soon.
Best regards, Pauli Virtanen
- --------------------------------------------
SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch.
This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or greater.
New features ============
Linear Programming Interface - - ----------------------------
The new function ``scipy.optimize.linprog`` provides a generic linear programming similar to the way ``scipy.optimize.minimize`` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported,allowing the user to monitor the progress of the algorithm.
Differential_evolution, a global optimizer - - ------------------------------------------
A new ``differential_evolution`` function is available in the ``scipy.optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques.
``scipy.signal`` improvements - - -----------------------------
The function ``max_len_seq`` was added, which computes a Maximum Length Sequence (MLS) signal.
``scipy.integrate`` improvements - - --------------------------------
It is now possible to use ``scipy.integrate`` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance.
``scipy.linalg`` improvements - - -----------------------------
Add function ``orthogonal_procrustes`` for solving the procrustes linear algebra problem.
``scipy.sparse`` improvements - - -----------------------------
``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its main input.
``scipy.special`` improvements - - ------------------------------
Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``.
New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added.
``scipy.sparse.csgraph`` improvements - - -------------------------------------
Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added.
``scipy.stats`` improvements - - ----------------------------
Added a Dirichlet distribution as multivariate distribution.
The new function ``scipy.stats.median_test`` computes Mood's median test.
The new function ``scipy.stats.combine_pvalues`` implements Fisher's and Stouffer's methods for combining p-values.
``scipy.stats.describe`` returns a namedtuple rather than a tuple, allowing users to access results by index or by name.
Deprecated features ===================
The ``scipy.weave`` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: `https://github.com/scipy/weave`_. It is a pure Python package, and can easily be installed with ``pip install weave``.
``scipy.special.bessel_diff_formula`` is deprecated. It is a private function, and therefore will be removed from the public API in a following release.
Backwards incompatible changes ==============================
scipy.ndimage - - -------------
The functions ``scipy.ndimage.minimum_positions``, ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return positions as ints instead of floats.
scipy.integrate - - ---------------
The format of banded Jacobians in ``scipy.integrate.ode`` solvers is changed. Note that the previous documentation of this feature was erroneous.
I ran against current scipy stack packages, and get these errors: https://travis-ci.org/MacPython/scipy-stack-osx-testing/jobs/42022929#L324 ====================================================================== ERROR: test_netcdf.test_byte_gatts ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py", line 268, in test_byte_gatts f = netcdf_file(filename, 'w') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py", line 226, in __init__ self.fp = open(self.filename, '%sb' % omode) IOError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/g_byte_atts.nc' ====================================================================== ERROR: test_netcdf.test_open_append ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/test_netcdf.py", line 284, in test_open_append f = netcdf_file(filename, 'w') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/netcdf.py", line 226, in __init__ self.fp = open(self.filename, '%sb' % omode) IOError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/tests/data/append_dat.nc' This turns out to be because the netcdf tests are trying to write into the scipy/io/tests/data directory: https://github.com/scipy/scipy/pull/4198 Otherwise, it looks clean. I uploaded the OSX wheels to sourceforge. Cheers, Matthew
Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker (the issue arises on scipy master as well as 0.14.1) ? On Sun, Nov 23, 2014 at 11:13 PM, Pauli Virtanen <pav@iki.fi> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear all,
We have finally finished preparing the Scipy 0.15.0 beta 1 release. Please try it and report any issues on the scipy-dev mailing list, and/or on Github.
If no surprises turn up, the final release is planned on Dec 20 in three weeks.
Source tarballs and full release notes are available at https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/ Binary installers should also be up soon.
Best regards, Pauli Virtanen
- --------------------------------------------
SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch.
This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or greater.
New features ============
Linear Programming Interface - - ----------------------------
The new function ``scipy.optimize.linprog`` provides a generic linear programming similar to the way ``scipy.optimize.minimize`` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported,allowing the user to monitor the progress of the algorithm.
Differential_evolution, a global optimizer - - ------------------------------------------
A new ``differential_evolution`` function is available in the ``scipy.optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques.
``scipy.signal`` improvements - - -----------------------------
The function ``max_len_seq`` was added, which computes a Maximum Length Sequence (MLS) signal.
``scipy.integrate`` improvements - - --------------------------------
It is now possible to use ``scipy.integrate`` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance.
``scipy.linalg`` improvements - - -----------------------------
Add function ``orthogonal_procrustes`` for solving the procrustes linear algebra problem.
``scipy.sparse`` improvements - - -----------------------------
``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its main input.
``scipy.special`` improvements - - ------------------------------
Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``.
New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added.
``scipy.sparse.csgraph`` improvements - - -------------------------------------
Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added.
``scipy.stats`` improvements - - ----------------------------
Added a Dirichlet distribution as multivariate distribution.
The new function ``scipy.stats.median_test`` computes Mood's median test.
The new function ``scipy.stats.combine_pvalues`` implements Fisher's and Stouffer's methods for combining p-values.
``scipy.stats.describe`` returns a namedtuple rather than a tuple, allowing users to access results by index or by name.
Deprecated features ===================
The ``scipy.weave`` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: `https://github.com/scipy/weave`_. It is a pure Python package, and can easily be installed with ``pip install weave``.
``scipy.special.bessel_diff_formula`` is deprecated. It is a private function, and therefore will be removed from the public API in a following release.
Backwards incompatible changes ==============================
scipy.ndimage - - -------------
The functions ``scipy.ndimage.minimum_positions``, ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return positions as ints instead of floats.
scipy.integrate - - ---------------
The format of banded Jacobians in ``scipy.integrate.ode`` solvers is changed. Note that the previous documentation of this feature was erroneous.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iEYEARECAAYFAlRyaf8ACgkQ6BQxb7O0pWC7XQCeNtdJD4ZNDXvFeNFs7N3KjQn6 8QkAoK3pFmhMrTwCrgkusl+fRNMboN2r =WSpM -----END PGP SIGNATURE----- _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
David Cournapeau <cournape@gmail.com> wrote:
Shall we consider <a href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168</a> to be a blocker (the issue arises on scipy master as well as 0.14.1) ?
It is really bad, but does anyone know what is really going on? Which changes to NumPy set this off? Sturla
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
David Cournapeau <cournape@gmail.com> wrote:
Shall we consider <a href="https://github.com/scipy/scipy/issues/4168"> https://github.com/scipy/scipy/issues/4168</a> to be a blocker (the issue arises on scipy master as well as 0.14.1) ?
It is really bad, but does anyone know what is really going on?
Yes, it is in the bug report. David
Which changes to NumPy set this off?
Sturla
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
David Cournapeau <cournape@gmail.com> wrote:
Shall we consider <a href="https://github.com/scipy/scipy/issues/4168"> https://github.com/scipy/scipy/issues/4168</a> to be a blocker (the issue arises on scipy master as well as 0.14.1) ?
It is really bad, but does anyone know what is really going on?
Yes, it is in the bug report.
Nice move. I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!? Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-) Happy hacking. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- #
On Tue, Nov 25, 2014 at 11:50 PM, Andrea Gavana <andrea.gavana@gmail.com> wrote:
On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
David Cournapeau <cournape@gmail.com> wrote:
Shall we consider <a
href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168</a> to be a blocker (the issue arises on scipy master as well as 0.14.1) ?
It is really bad, but does anyone know what is really going on?
Yes, it is in the bug report.
Nice move.
I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!?
It's a regression in 1.9, yes; 1.8 is fine. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
On 11/26/2014 12:50 AM, Andrea Gavana wrote:
On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <sturla.molden@gmail.com <mailto:sturla.molden@gmail.com>> wrote:
David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote: > Shall we consider <a > href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168</a> > to be a > blocker (the issue arises on scipy master as well as 0.14.1) ? >
It is really bad, but does anyone know what is really going on?
Yes, it is in the bug report.
Nice move.
I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!?
yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc.
Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-)
There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created. It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it.
Julian Taylor <jtaylor.debian <at> googlemail.com> writes: [clip]
There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet.
What is broken is calculating eigenvalues of complex-valued sparse matrices and iterative solution of complex-valued linear equations. I.e., nothing obscure. A likely explanation is that win32 + Numpy 1.9 is a less common platform, and users whose code started failing just infrequently do not report bugs as easily...
The majority of f2py should still be working, numpys own f2py testsuite passes on win32.
Perhaps the arrays are aligned by chance? I don't think the test suite repeats the complex valued intent(inout) parameter test many times. [clip]
I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created.
David's test case is this: n = 4 x = np.zeros(n * 3, dtype="D") _dummy.zfoo(x, n) where the argument is declared as "double complex, dimension(3*n), intent(inout)" in f2py. The ARPACK stuff in Scipy also does pretty much just this. -- Pauli Virtanen
On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 11/26/2014 12:50 AM, Andrea Gavana wrote:
On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden <sturla.molden@gmail.com <mailto:sturla.molden@gmail.com>> wrote:
David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote: > Shall we consider <a > href="https://github.com/scipy/scipy/issues/4168">
https://github.com/scipy/scipy/issues/4168</a>
> to be a > blocker (the issue arises on scipy master as well as 0.14.1) ? >
It is really bad, but does anyone know what is really going on?
Yes, it is in the bug report.
Nice move.
I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun thing on NumPy/SciPy across the whole user base of Python in the company. We will probably move to 64bit-in-any-sense soon enough, I guess before this issue is solved. Tell me, NumPy, was the array aligned enough in 1.8? Is NumPy stricter in its checking because of SPARC?
SPARC?!?
yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc.
Dozens of f2py compiled extensions are going to fail soon here - which I'll reluctantly check tomorrow. I don't want to think about other people on Win32 facing the same issue elsewhere... :-)
There haven't been any real complaints from applications yet, only testsuite failure of scipy. Either the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created.
It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it.
The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned. If anyone is to blame, it is the reviewer, namely, myself. Chuck
On 11/26/2014 02:19 PM, Charles R Harris wrote:
On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>> wrote:
On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com> > <mailto:cournape@gmail.com <mailto:cournape@gmail.com>>> wrote: > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > <sturla.molden@gmail.com <mailto:sturla.molden@gmail.com> <mailto:sturla.molden@gmail.com <mailto:sturla.molden@gmail.com>>> wrote: > > David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com> > <mailto:cournape@gmail.com <mailto:cournape@gmail.com>>> wrote: > > Shall we consider <a > > href="https://github.com/scipy/scipy/issues/4168">https://github.com/scipy/scipy/issues/4168</a> > > to be a > > blocker (the issue arises on scipy master as well as 0.14.1) ? > > > > It is really bad, but does anyone know what is really going on? > > > Yes, it is in the bug report. > > > > Nice move. > > I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun > thing on NumPy/SciPy across the whole user base of Python in the > company. We will probably move to 64bit-in-any-sense soon enough, I > guess before this issue is solved. Tell me, NumPy, was the array aligned > enough in 1.8? Is NumPy stricter in its checking because of SPARC? SPARC?!?
yes, before the change numpy accepted a factor 10 performance regression in complex indexing to satisfy sparc.
> > Dozens of f2py compiled extensions are going to fail soon here - which > I'll reluctantly check tomorrow. I don't want to think about other > people on Win32 facing the same issue elsewhere... :-) >
There haven't been any real complaints from applications yet, only testsuite failure of scipy. Eithe the one thing that is broken in scipy isn't much used or windows 32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py testsuite passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase david created.
It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at least give users a way to work around it.
The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned.
If anyone is to blame, it is the reviewer, namely, myself.
concerning actually fixing it I guess we have 3 options: 1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows. fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...)
On Wed, Nov 26, 2014 at 6:59 AM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 11/26/2014 02:19 PM, Charles R Harris wrote:
On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor <jtaylor.debian@googlemail.com <mailto:jtaylor.debian@googlemail.com>> wrote:
On 11/26/2014 12:50 AM, Andrea Gavana wrote: > > On 25 November 2014 at 19:33, David Cournapeau <cournape@gmail.com
<mailto:cournape@gmail.com>
> <mailto:cournape@gmail.com <mailto:cournape@gmail.com>>> wrote: > > > > On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden > <sturla.molden@gmail.com <mailto:sturla.molden@gmail.com> <mailto:sturla.molden@gmail.com <mailto:sturla.molden@gmail.com>>> wrote: > > David Cournapeau <cournape@gmail.com <mailto:
cournape@gmail.com>
> <mailto:cournape@gmail.com <mailto:cournape@gmail.com>>>
wrote:
> > Shall we consider <a > > href="https://github.com/scipy/scipy/issues/4168">
https://github.com/scipy/scipy/issues/4168</a>
> > to be a > > blocker (the issue arises on scipy master as well as
0.14.1) ?
> > > > It is really bad, but does anyone know what is really
going on?
> > > Yes, it is in the bug report. > > > > Nice move. > > I've now recommended to hold back any
upgrade/update/pip-crap/enpkg-fun
> thing on NumPy/SciPy across the whole user base of Python in the > company. We will probably move to 64bit-in-any-sense soon enough, I > guess before this issue is solved. Tell me, NumPy, was the array
aligned
> enough in 1.8? Is NumPy stricter in its checking because of SPARC?
SPARC?!?
yes, before the change numpy accepted a factor 10 performance
regression
in complex indexing to satisfy sparc.
> > Dozens of f2py compiled extensions are going to fail soon here -
which
> I'll reluctantly check tomorrow. I don't want to think about other > people on Win32 facing the same issue elsewhere... :-) >
There haven't been any real complaints from applications yet, only testsuite failure of scipy. Eithe the one thing that is broken in scipy isn't much used or
windows
32 users aren't using 1.9 yet. The majority of f2py should still be working, numpys own f2py
testsuite
passes on win32. I still don't know what exactly arpack is doing different but I also did not have time yet to look at the testcase
david
created.
It should of course be fixed, I have an old branch with aligned allocators lying around somewhere which should fix the issue or at
least
give users a way to work around it.
The lesson I take from this is that we need a test ;) Also, that it would be nice if we got more testing by users before releases. But things are as they are, this will get fixed, and we will move on with one more lesson learned.
If anyone is to blame, it is the reviewer, namely, myself.
concerning actually fixing it I guess we have 3 options:
1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator
I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows.
fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...)
I would also go for 1) on the general principal that alignment needs to be platform dependent. I suppose the problem here is that is might also be compiler dependent. Where do the aligned copies take place? What is the downside to that solution? Chuck
On 26 Nov 2014 13:59, "Julian Taylor" <jtaylor.debian@googlemail.com> wrote:
concerning actually fixing it I guess we have 3 options:
1. reduce maximum copy alignment from currently 16 to 8 on platforms that need it. That will automatically reduce the needed alignment of complex on win32 to 8 bytes. Do other compilers provide something similar to gccs __BIGGEST_ALIGNMENT__? 2. remove bloating of complex alignment and let sparc crash. 3. add an aligned allocator
I somewhat favor 1, it has the risk that a vectorizing compiler might wreak havoc but to my knowledge numpy does not actually have real 16 byte copies. There are some occurrences of npy_int128, but those likely are not used on windows.
fwiw I could reproduce the issue on linux i386 with -malign-double, (I could have sworn I tested that configuration too...)
I'm not sure what "maximum copy alignment" means in this context, but (1) does sound the most like a proper fix to me too. -n
On Tue, Nov 25, 2014 at 12:53 PM, David Cournapeau <cournape@gmail.com> wrote:
Shall we consider https://github.com/scipy/scipy/issues/4168 to be a blocker (the issue arises on scipy master as well as 0.14.1) ?
Do you think there's anything scipy can do about it? It looks like a pure numpy bug to me. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
participants (8)
-
Andrea Gavana -
Charles R Harris -
David Cournapeau -
Julian Taylor -
Matthew Brett -
Nathaniel Smith -
Pauli Virtanen -
Sturla Molden