Hi all, At EuroScipy Pauli, David and I sat together and drafted a roadmap for Scipy 1.0. We then discussed this offline with some of the other currently most active core devs, to get it into a state that's ready for discussion on this list. So here it is: https://github.com/scipy/scipy/pull/2908 Our aim is for this roadmap to help guide us towards a 1.0 version, which will contain only code that we consider to be "of sufficient quality". Also, it will help to communicate to new and potential developers where their contributions are especially needed. In order to discuss/review this roadmap without generating a monster thread, I propose the following: - topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread. - things in the General section (API changes, documentation/test/build guidelines, etc.), are discussed on this thread as well. - for discussion of module-specific content, start a new thread and name it "1.0 roadmap: <module_name>". - for minor things, comment on the PR. Cheers, Ralf
On Sat, Sep 21, 2013 at 8:54 PM, Ralf Gommers <ralf.gommers@gmail.com>wrote:
Hi all,
At EuroScipy Pauli, David and I sat together and drafted a roadmap for Scipy 1.0. We then discussed this offline with some of the other currently most active core devs, to get it into a state that's ready for discussion on this list. So here it is: https://github.com/scipy/scipy/pull/2908
Our aim is for this roadmap to help guide us towards a 1.0 version, which will contain only code that we consider to be "of sufficient quality". Also, it will help to communicate to new and potential developers where their contributions are especially needed.
In order to discuss/review this roadmap without generating a monster thread, I propose the following: - topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread. - things in the General section (API changes, documentation/test/build guidelines, etc.), are discussed on this thread as well. - for discussion of module-specific content, start a new thread and name it "1.0 roadmap: <module_name>". - for minor things, comment on the PR.
Cheers, Ralf
Github may not survive forever, so for the record here the full text of the draft roadmap: Roadmap to Scipy 1.0 ==================== This roadmap provides a high-level view on what is needed per scipy submodule in terms of new functionality, bug fixes, etc. before we can release a ``1.0`` version of Scipy. Things not mentioned in this roadmap are not necessarily unimportant or out of scope, however we (the Scipy developers) want to provide to our users and contributors a clear picture of where Scipy is going and where help is needed most urgently. When a module is in a 1.0-ready state, it means that it has the functionality we consider essential and has an API and code quality (including documentation and tests) that's of high enough quality. General ------- This roadmap will be evolving together with Scipy. Updates can be submitted as pull requests and, unless they're very minor, have to be discussed on the scipy-dev mailing list. API changes ``````````` In general, we want to take advantage of the major version change to fix the known warts in the API. The change from 0.x.x to 1.x.x is the chance to fix those API issues that we all know are ugly warts. Example: unify the convention for specifying tolerances (including absolute, relative, argument and function value tolerances) of the optimization functions. More API issues will be noted in the module sections below. It should be made more clear what is public and what is private in scipy. Everything private should be underscored as much as possible. Now this is done consistently when we add new code, but for 1.0 it should also be done for existing code. Test coverage ````````````` Test coverage of code added in the last few years is quite good, and we aim for a high coverage for all new code that is added. However, there is still a significant amount of old code for which coverage is poor. Bringing that up to the current standard is probably not realistic, but we should plug the biggest holes. Additionally the coverage should be tracked over time and we should ensure it only goes up. Besides coverage there is also the issue of correctness - older code may have a few tests that provide decent statement coverage, but that doesn't necessarily say much about whether the code does what it says on the box. Therefore code review of some parts of the code (``stats`` and ``signal`` in particular) is necessary. Documentation ````````````` The documentation is in decent shape. Expanding of current docstrings and putting them in the standard numpy format should continue, so the number of reST errors and glitches in the html docs decreases. Most modules also have a tutorial in the reference guide that is a good introduction, however there are a few missing or incomplete tutorials - this should be fixed. Other ````` Scipy 1.0 will likely contain more backwards-incompatible changes than a minor release. Therefore we will have a longer-lived maintenance branch of the last 0.X release. It's not clear how much functionality can be Cythonized without making the .so files too large. This needs measuring. Bento will be officially supported as the second build tool besides distutils. At the moment it still has an experimental, use-at-your-own-risk status, but that has to change. A more complete continuous integration setup is needed; at the moment we often find out right before a release that there are issues on some less-often used platform or Python version. At least needed are a Windows, Linux and OS X build, coverage of the lowest and highest Python and Numpy versions that are supported, a Bento build and a PEP8 checker. Modules ------- cluster ``````` Most of the cluster module is a candidate for a Cython rewrite; this will speed up the code and it will be more maintainable than the current C code. The code should remain (or become) simple and easy to understand. Support for the arbitrary distance metrics in ``scipy.spatial`` is probably best left to scikit-learn or other more specialized libraries. constants ````````` This module is basically done, low-maintenance and without open issues. fftpack ``````` Needed: - solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative - deprecate fftpack.convolve as public function (was not meant to be public), resolve differences between ``signal.fftconvolve`` / ``fftpack.convolve`` / ``signal.convolve`` and ``numpy.convolve`` There's a large overlap with ``numpy.fft``. This duplication has to change (both are too widely used to deprecate one); in the documentation we should make clear that ``scipy.fftpack`` is preferred over ``numpy.fft``. integrate ````````` Needed for ODE solvers: - documentation is pretty bad, needs fixing - figure out if/how to integrate scikits.odes (Sundials wrapper) - figure out what to deprecate The numerical integration functions are in good shape, not much to do here. interpolate ``````````` Needed: - Transparant B-splines and their usage in the interpolation routines is needed. - Both fitpack and fitpack2 interfaces will be kept. - splmake should go; is different spline representation --> need exactly one - interp1d/interp2d are somewhat ugly but widely used, so we keep them. - Regular grid interpolation routines needed io -- wavfile; - PCM float will be supported, for anything else use audiolab or other specialized libraries. - raise errors instead of warnings if data not understood. Other sub-modules (matlab, netcdf, idl, harwell-boeing, arff, matrix market) are in good shape. lib --- ``scipy.lib`` contains nothing public anymore, so rename to ``scipy._lib``. linalg `````` Needed: - remove functions that are duplicate with numpy.linalg - get_lapack_funcs should always use flapack - cblas, clapack are deprecated, will go away - wrap more lapack functions - one too many funcs for LU decomposition, remove one misc ```` ``scipy.misc`` will be removed as a public module. The functions in it can be moved to other modules: - pilutil, images : ndimage - comb, factorials, logsumexp, pade : special - doccer : move to scipy._lib - info, who : these are in numpy - derivative, central_diff_weight : remove, replace with more extensive functionality for numerical differentiation - likely in a new module ``scipy.diff``, as discussed in https://github.com/scipy/scipy/issues/2035 ndimage ``````` Underlying ndimage is a powerful interpolation engine. Unfortunately, it was never decided whether to use a pixel model (``(1, 1)`` elements with centers ``(0.5, 0.5)``) or a data point model (values at points on a grid). Over time, it seems that the data point model is better defined and easier to implement. We therefore propose to move to this data representation for 1.0, and to vet all interpolation code to ensure that boundary values, transformations, etc. are correctly computed. Addressing this issue will close several issues, including #1323, #1903, #2045 and #2640. odr --- Rename the module to ``regression`` or ``fitting``, include ``optimize.curve_fit``. This module will then provide a home for other fitting functionality - what exactly needs to be worked out in more detail, a discussion can be found at https://github.com/scipy/scipy/pull/448. optimize ```````` Overall this module is in reasonably good shape, however it is missing a few more good global optimizers as well as large-scale optimizers. These should be added. Other things that are needed: - deprecate ``anneal``, it just doesn't work well enough. - deprecate the ``fmin_*`` functions in the documentation, ``minimize`` is preferred. - clearly define what's out of scope for this module. signal `````` *Convolution and correlation*: (Relevant functions are convolve, correlate, fftconvolve, convolve2d, correlate2d, and sepfir2d.) Eliminate the overlap with `ndimage` (and elsewhere). From `numpy`, `scipy.signal` and `scipy.ndimage` (and anywhere else we find them), pick the "best of class" for 1-D, 2-D and n-d convolution and correlation, put the implementation somewhere, and use that consistently throughout scipy. *B-splines*: (Relevant functions are bspline, cubic, quadratic, gauss_spline, cspline1d, qspline1d, cspline2d, qspline2d, cspline1d_eval, and spline_filter.) Move the good stuff to `interpolate` (with appropriate API changes to match how things are done in `interpolate`), and eliminate any duplication. *Filter design*: merge `firwin` and `firwin2` so `firwin2` can be removed. *Continuous-Time Linear Systems*: remove `lsim2`, `impulse2`, `step2`. Make `lsim`, `impulse` and `step` "just work" for any input system. Improve performance of ltisys (less internal transformations between different representations). *Wavelets*: add proper wavelets, including discrete wavelet transform. What's there now doesn't make much sense. sparse `````` The sparse matrix formats are getting feature-complete but are slow ... reimplement parts in Cython? - Small matrices are slower than PySparse, needs fixing There are a lot of formats. These should all be kept, but improvements/optimizations should go into CSR/CSC, which are the preferred formats. Don't emulate np.matrix behavior, drop 2-D? sparse.csgraph `````````````` This module is in good shape. sparse.linalg ````````````` Arpack is in good shape. isolve: - callback keyword is inconsistent - tol keyword is broken, should be relative tol - Fortran code not re-entrant (but we don't solve, maybe re-use from PyKrilov) dsolve: - remove umfpack wrapper due to license reasons - add sparse Cholesky or incomplete Cholesky - look at CHOLMOD spatial ``````` KDTree/cKDTree and the QHull wrappers are in good shape. The distance module needs bug fixes in the distance metrics, and distance_wrap.c needs to be cleaned up (maybe rewrite in Cython). special ``````` special has a lot of functions that need improvements in precision. All functions that are also implemented in mpmath can be tested against mpmath, and should match well. Things not in mpmath: - cdflib - <Pauli checks> some others stats ````` This is a large module with by far the most open issues. It has improved a lot over the past few releases, but more cleanup and rewriting of functions is needed. The Statistics Review milestone on Github gives a reasonable overview of which functions need checking, documentation and tests. ``stats.distributions`` : - skew/kurtosis of a number of distributions needs fixing - fix generic docstring examples, they should be valid Python and make sense for each distributions - document subclassing of distributions even better, make issues with state of instances clear. All hypothesis tests should get a keyword 'alternative' where applicable (see ``stats.kstest`` for an example). ``gaussian_kde`` is in good shape but limited. It should not be expanded probably, this fits better in statsmodels (which already has a lot more KDE functionality). ``stats.mstats`` is a useful module for worked with data with missing values. One problem it has though is that in many cases the functions have diverged from their counterparts in `scipy.stats`. The ``mstats`` functions should be updated so that the two sets of functions are consistent. weave ````` This is the only module that was not ported to Python 3. Effectively it's deprecated (not recommended to use for new code). In the future it should be removed from scipy (can be made into a separate module).
For scipy stats, is there anything on the table regarding somehow unifying the sampling in numpy.random and the distributions in scipy.stats? I'm specifically thinking of two issues: (1) There's a lot of duplication between numpy.random and scipy.stats but with different interfaces. This seems like something that ideally would be reduced. (2) The interface for the distributions in scipy.stats seems to explicitly be for scalar random variables, so there's no multivariate normals, multinomials, dirichlet, wishart, etc.. Instead the sampling is in numpy.random, and pdf's aren't there. Has this been discussed elsewhere? On Sat, Sep 21, 2013 at 8:03 PM, Blake Griffith <blake.a.griffith@gmail.com> wrote:
sparse ``````
Don't emulate np.matrix behavior, drop 2-D?
What is meant by this? Emulate np.array instead?
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Sun, Sep 22, 2013 at 7:52 AM, Christopher Jordan-Squire <cjordan1@uw.edu>wrote:
For scipy stats, is there anything on the table regarding somehow unifying the sampling in numpy.random and the distributions in scipy.stats? I'm specifically thinking of two issues:
(1) There's a lot of duplication between numpy.random and scipy.stats but with different interfaces. This seems like something that ideally would be reduced.
numpy.random only provides sampling and only has about half the distributions of scipy.stats. Sampling is really only a small part of what scipy.stats provides (pdf, cdf, moments, fitting a distribution, etc.). So I'm not bothered by that duplication. If we'd want to reduce it I think it would have to be removed from numpy, which doesn't sound like a good idea.
(2) The interface for the distributions in scipy.stats seems to explicitly be for scalar random variables, so there's no multivariate normals, multinomials, dirichlet, wishart, etc.. Instead the sampling is in numpy.random, and pdf's aren't there.
Two days ago PR-2726 was merged, which adds a multivariate normal distribution. Others can be added. IIRC there has been an enhancement ticket for wishhart somewhere and there's a Python implementation floating around somewhere. Cheers, Ralf
Has this been discussed elsewhere?
On Sat, Sep 21, 2013 at 8:03 PM, Blake Griffith <blake.a.griffith@gmail.com> wrote:
sparse ``````
Don't emulate np.matrix behavior, drop 2-D?
What is meant by this? Emulate np.array instead?
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Sun, Sep 22, 2013 at 1:52 AM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, Sep 22, 2013 at 7:52 AM, Christopher Jordan-Squire <cjordan1@uw.edu> wrote:
For scipy stats, is there anything on the table regarding somehow unifying the sampling in numpy.random and the distributions in scipy.stats? I'm specifically thinking of two issues:
(1) There's a lot of duplication between numpy.random and scipy.stats but with different interfaces. This seems like something that ideally would be reduced.
numpy.random only provides sampling and only has about half the distributions of scipy.stats. Sampling is really only a small part of what scipy.stats provides (pdf, cdf, moments, fitting a distribution, etc.). So I'm not bothered by that duplication. If we'd want to reduce it I think it would have to be removed from numpy, which doesn't sound like a good idea.
Yeah, I'm not about to suggest removing sampling from numpy.random. That'd be crazy. There's still the API mismatch between the names. Numpy.random, as a rule, uses the full expansion of the name while scipy.stats, as a rule, tends to abbreviate. That often confuses me, but not as confusing as I first thought, since at least it's consistent.
(2) The interface for the distributions in scipy.stats seems to explicitly be for scalar random variables, so there's no multivariate normals, multinomials, dirichlet, wishart, etc.. Instead the sampling is in numpy.random, and pdf's aren't there.
Two days ago PR-2726 was merged, which adds a multivariate normal distribution. Others can be added. IIRC there has been an enhancement ticket for wishhart somewhere and there's a Python implementation floating around somewhere.
A multivariate normal is a great addition. Currently, dirichlet and multinomial are the only random variables you can sample from in numpy.random that aren't in scipy.stats. My $0.02 for scipy 1.0 roadmap is adding dirichlet and multinomial to scipy.stats as well as wishart/inverse-wishart. Then distributions in scipy.stats would be a superset of numpy.random, and scipy.stats would include one of the most widely used distributions currently not in it. (In addition to the implementations floating around, both scikit-learn and pymc include bits and pieces of wishart-related code.) Also, right now you can use scipy.stats.rv_discrete to create your own discrete random variable, but only for an array of integers--so [1,2,3] rather than ['apple', 'orange', 'banana']. Which is fine, but that also means a lot of code duplication/wrapper classes for everyone who wants their random variable to be over a space of fruits rather than integers. Not sure how many people that effects, though. Not sure if these belong on the roadmap or just as enhancement requests. Thanks, Chris
Cheers, Ralf
Has this been discussed elsewhere?
On Sat, Sep 21, 2013 at 8:03 PM, Blake Griffith <blake.a.griffith@gmail.com> wrote:
sparse ``````
Don't emulate np.matrix behavior, drop 2-D?
What is meant by this? Emulate np.array instead?
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
22.09.2013 06:03, Blake Griffith kirjoitti:
sparse `````` Don't emulate np.matrix behavior, drop 2-D?
What is meant by this? Emulate np.array instead?
Yes, having something like `csr_array` instead of/in addition to `csr_matrix`. I haven't completely thought out how useful this would be in the long run. -- Pauli Virtanen
Dear list, There was some discussion on how to improve integrate recently mostly focused around the ode solvers, http://comments.gmane.org/gmane.comp.python.scientific.devel/18150 I'll summarise the changes we want to make here, 1) Improve the interface to integrate.ode. - see the suggestion here, https://gist.github.com/danieljfarrell/6482713 2) Make integrate.odeint a functional wrapper around integrate.ode. - For the functional version we can use the MATLAB concept of 'events' to specify time points at which the solution can be exported. 3) Update the integrate.ode solvers (from Sundials via scikits.odes) - replace VODE with CVODE (a version written in C with many improvements) - add CVODES (CVODE + simultaneous corrector methods) - add IDA (similar to MATLAB's ode15s, also has the option for solving DAEs) Points 1 and 2 are fairly standard, however point 3 generated lots of discussion i.e. do we use sundials, scikits.odes, petsc4py... as the basis of the new module? petsc4py would allow solving ODEs in parallel and has a huge array of solvers. But I think in the end it was decided that scikits.odes would be a better starting point because of the lower complexity. Plus it already wraps the Sundials suite and is actively maintained. Best wishes, Dan
2013/9/23 boyfarrell@gmail.com <boyfarrell@gmail.com>
Dear list,
There was some discussion on how to improve integrate recently mostly focused around the ode solvers, http://comments.gmane.org/gmane.comp.python.scientific.devel/18150
I'll summarise the changes we want to make here,
1) Improve the interface to integrate.ode. - see the suggestion here, https://gist.github.com/danieljfarrell/6482713
2) Make integrate.odeint a functional wrapper around integrate.ode. - For the functional version we can use the MATLAB concept of 'events' to specify time points at which the solution can be exported.
3) Update the integrate.ode solvers (from Sundials via scikits.odes) - replace VODE with CVODE (a version written in C with many improvements) - add CVODES (CVODE + simultaneous corrector methods) - add IDA (similar to MATLAB's ode15s, also has the option for solving DAEs)
Points 1 and 2 are fairly standard, however point 3 generated lots of discussion i.e. do we use sundials, scikits.odes, petsc4py... as the basis of the new module? petsc4py would allow solving ODEs in parallel and has a huge array of solvers. But I think in the end it was decided that scikits.odes would be a better starting point because of the lower complexity. Plus it already wraps the Sundials suite and is actively maintained.
I don't know the events concept. Is there something like that already in scipy? Some doc? Sundials and hence scikits.odes also have event monitoring, in essence, checking when a condition is fullfilled, eg when a ball hits a wall, and return the solver at that point (as the physics and the equations will change). I think that was meant with 'events' in an earlier discussion on sundials. In sundials it is the RootFn functions. I could set-up a clone of scipy and integrate scikits.ode as best as possible. I'm leaning though to renaming scikit.odes.ode into deq, and rewriting scipy ode and odeint as wrappers around deq. Or would you break ode api fully? Seems like a world of hurt for the users of ode. The main programmer of scikit.odes (so not me, but Pavol) very much likes to circumvent the ode wrapper and instead use directly the sundials wrapped methods. To each it's own, but that does raise the question, do we consider that part of the API (so the methods ode or deq use to set up sundials problems)? Personally, I don't see problems with it, on the other hand, documentation might become confusing. Note that for all parallel solvers, the problem seems to be more the user who has to write the correct functions to drive the solver, and not so much the solver. Once we have sundials, people can contribute to use the parallel version. I'm sure some would be interested and contribute that. Note that once you integrate sundials, you obtain kinsol for free, which should be in the linalg package or accessible from the generic solve? If you ask ida to compute the initial condition, ida in essence calls kinsol, so you can solve nonlinear algebraic equations. It would be good to expose this in the correct scipy subpackage if it complements existing methods. As a final note, scikits.odes is maintained, but I have no fixed position, so as always, that can change quickly. Benny
Best wishes,
Dan _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK. https://github.com/dagss/libfftpack As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes. Dag Sverre
On Mon, Sep 23, 2013 at 8:51 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK.
https://github.com/dagss/libfftpack
As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes.
If this is nicely-licensed C code that provides a superset of scipy.fftpack's functionality, ought we to merge it into *numpy*.fft and deprecate scipy.fftpack? (I'm a little confused at what exactly the difference between the numpy and scipy modules is in this case, except that of course the scipy version needs a fortran compiler.) -n
On Sep 24, 2013, at 1:28 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Sep 23, 2013 at 8:51 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK.
https://github.com/dagss/libfftpack
As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes.
If this is nicely-licensed C code that provides a superset of scipy.fftpack's functionality, ought we to merge it into *numpy*.fft and deprecate scipy.fftpack? (I'm a little confused at what exactly the difference between the numpy and scipy modules is in this case, except that of course the scipy version needs a fortran compiler.)
-n
On that note … a thought for scipy 1.0 : Does scipy absolutely need Fortran in general? Or are there equivalent C or C++ packages that might be used instead? Getting rid of Fortran code would simplify life for Mac users (and maybe Windows or even Linux). I don't really know how much Fortran is used in scipy or libraries it depends on, or how much of a problem Fortran really is. I'm just asking if getting rid of Fortran is an option even worth considering for scipy 1.0. Christoph
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Tue, Sep 24, 2013 at 2:29 PM, Christoph Deil < deil.christoph@googlemail.com> wrote:
On that note … a thought for scipy 1.0 :
Does scipy absolutely need Fortran in general? Or are there equivalent C or C++ packages that might be used instead?
Not really. Not with BSD-like licenses, at any rate.
Getting rid of Fortran code would simplify life for Mac users (and maybe Windows or even Linux). I don't really know how much Fortran is used in scipy or libraries it depends on, or how much of a problem Fortran really is. I'm just asking if getting rid of Fortran is an option even worth considering for scipy 1.0.
scipy 2.0, maybe. It will be a *lot* of work to find and write replacements for everything, not to mention that the replacements will probably not work the same as the current code. -- Robert Kern
24.09.2013 16:29, Christoph Deil kirjoitti: [clip]
On that note … a thought for scipy 1.0 :
Does scipy absolutely need Fortran in general? Or are there equivalent C or C++ packages that might be used instead?
Sounds unrealistic to me. Compiler issues alone definitely won't justify the work that would involved in this. Scipy can be compiled both on OSX and Windows, if you have clue what you are doing. -- Pauli Virtanen
On Tue, Sep 24, 2013 at 2:29 PM, Christoph Deil < deil.christoph@googlemail.com> wrote:
On Sep 24, 2013, at 1:28 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Sep 23, 2013 at 8:51 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK.
https://github.com/dagss/libfftpack
As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes.
If this is nicely-licensed C code that provides a superset of scipy.fftpack's functionality, ought we to merge it into *numpy*.fft and deprecate scipy.fftpack? (I'm a little confused at what exactly the difference between the numpy and scipy modules is in this case, except that of course the scipy version needs a fortran compiler.)
-n
On that note … a thought for scipy 1.0 :
Does scipy absolutely need Fortran in general? Or are there equivalent C or C++ packages that might be used instead?
Getting rid of Fortran code would simplify life for Mac users (and maybe Windows or even Linux). I don't really know how much Fortran is used in scipy or libraries it depends on, or how much of a problem Fortran really is. I'm just asking if getting rid of Fortran is an option even worth considering for scipy 1.0.
scipy contains approximately 100k LOC of often non trivial code. Lots of them most likely don't have an equivalent in any other language (ARPACK, specfunc, etc...), and few people both have the domain expertise and the will to port those things. That + the fact that fortran is often more appropriate than C/C++ anyway for numeric work. The issue of fortran on mac is partly Apple's fault for screwing up their libraries BTW. David
On Tue, Sep 24, 2013 at 12:28 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Sep 23, 2013 at 8:51 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK.
https://github.com/dagss/libfftpack
As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes.
If this is nicely-licensed C code that provides a superset of scipy.fftpack's functionality, ought we to merge it into *numpy*.fft and deprecate scipy.fftpack?
It does not provide a superset. The FORTRAN code in scipy.fftpack does ND transforms, the DCT and DST, and FFT-based convolutions. None of that code *must* remain in FORTRAN, but you would have to rewrite it all in C using the new libfftpack C code underneath. It's not a matter of "merging", I'm afraid, but significant rewriting.
(I'm a little confused at what exactly the difference between the numpy and scipy modules is in this case, except that of course the scipy version needs a fortran compiler.)
At one time, scipy.fftpack supported several different backends like FFTW, not just the FORTRAN FFTPACK. They have since been dropped because they added more difficulties to the build process than we gained in performance. -- Robert Kern
On 09/24/2013 03:53 PM, Robert Kern wrote:
On Tue, Sep 24, 2013 at 12:28 PM, Nathaniel Smith <njs@pobox.com <mailto:njs@pobox.com>> wrote:
On Mon, Sep 23, 2013 at 8:51 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no <mailto:d.s.seljebotn@astro.uio.no>> wrote:
On 09/21/2013 08:57 PM, Ralf Gommers wrote:
fftpack ``````` Needed:
- solve issues with single precision: large errors, disabled for difficult sizes - fix caching bug - Bluestein algorithm nice to have, padding is alternative
A battle-tested Bluestein is included in Martin Reinecke's C port of FFTPACK.
https://github.com/dagss/libfftpack
As you can see in the readme there were a couple of changes to FFTPACK to improve accuracy for large primes.
If this is nicely-licensed C code that provides a superset of scipy.fftpack's functionality, ought we to merge it into *numpy*.fft and deprecate scipy.fftpack?
It does not provide a superset. The FORTRAN code in scipy.fftpack does ND transforms, the DCT and DST, and FFT-based convolutions. None of that code *must* remain in FORTRAN, but you would have to rewrite it all in C using the new libfftpack C code underneath. It's not a matter of "merging", I'm afraid, but significant rewriting.
It seems to me all the FORTRAN code there is plain vanilla Netlib FFTPACK; the extra ND and convolutions etc. are in C? But indeed, seems like Martin's C port doesn't include the DCT and DST parts of FFTPACK, only complex and real FFT. That makes it not completely trivial, but I wouldn't say "significant rewrite" is accurate either, it's just about making the C port of FFTPACK complete. Martin (on CC), how long did it take to port the complex/real FFTs from Fortran to C? Dag Sverre
It seems to me all the FORTRAN code there is plain vanilla Netlib FFTPACK; the extra ND and convolutions etc. are in C?
But indeed, seems like Martin's C port doesn't include the DCT and DST parts of FFTPACK, only complex and real FFT. That makes it not completely
[Forwarding from Martin who's not subscribed.] Hi all! trivial, but I
wouldn't say "significant rewrite" is accurate either, it's just about making the C port of FFTPACK complete.
Martin (on CC), how long did it take to port the complex/real FFTs from Fortran to C?
Difficult to say, I spent much more time tweaking and polishing the code after porting, and I also had a template C implementation to begin with ... But please have a look at http://www.netlib.org/fftpack/fft.c This is a C port by Christopher Montgomery (of Ogg Vorbis/Opus fame), which he put into public domain. This file should have DCTs/DSTs (single precision only, but changing that should be trivial). Hope this helps, Martin
On Sat, Sep 21, 2013 at 7:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
- topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread.
I would be curious what the answers are to these questions :-). This looks like a big list with many good improvements on it, but I'm not sure what makes them "1.0 changes" instead of just "good changes we should do". Does 1.0 mean we can break a lot of stuff at once and get away with it? Does it mean that after that we're not allowed to change things like this ever again so we have to get it right (and maybe keep slipping the schedule until we're certain)? Does it mean no more releases until all the below things happen? Or on the other extreme, does it mean that someone will keep an eye on this list, and at some point maybe in a few years when we notice that all of these things have happened, then the next release gets called "1.0" instead of "0.18" or whatever? I think when you start talking about "1.0" people have very strong conflicting assumptions about what this "obviously" means, so... -n
On Sat, Sep 21, 2013 at 6:35 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Sep 21, 2013 at 7:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
- topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread.
I would be curious what the answers are to these questions :-).
This looks like a big list with many good improvements on it, but I'm not sure what makes them "1.0 changes" instead of just "good changes we should do". Does 1.0 mean we can break a lot of stuff at once and get away with it? Does it mean that after that we're not allowed to change things like this ever again so we have to get it right (and maybe keep slipping the schedule until we're certain)?
I haven't been to EuroScipy so I don't know the original discussion. The main difference for me would be that I expect more stability afterwards. Currently we still need to clean up code that requires deprecation and API changes pretty regularly (or they are cumbersome to cleanup while maintaining backwards compatibility.) Several parts of scipy have "organically grown", sometimes with several competing and incompatible implementations, some parts of the code were never reviewed and tested. IMO, these are the main problems with the current code that need to be cleaned up before we can call it a scipy 1.0. In the mean time, we still get new code, missing algorithms and improvements in many sub-packages.
Does it mean no more releases until all the below things happen? Or on the other extreme, does it mean that someone will keep an eye on this list, and at some point maybe in a few years when we notice that all of these things have happened, then the next release gets called "1.0" instead of "0.18" or whatever?
I think it will be the latter, keep working as usual, with an eye or priority on the things on the list, release at a pretty regular schedule. How fast we can check off the items on the roadmap to 1.0 list will depend on volunteer work. Positive case: wrong skew and kurtosis in stats.distributions has already been fixed, thanks to Evgeni's recent work. stats.distributions are in pretty good shape now and some of the remaining issues are bonus points that don't need to hold up a 1.0. Some other parts of scipy.stats have test coverage in the 60%-70% which is also a rough estimate how many functions still need to be reviewed, tested and cleaned up (30%?) (cleanup or delete legacy code? before 1.0) Once we have these functions in 1.0 condition, I don't expect that they will change much anymore. Once a t-test has been properly written, we don't need to change it anymore, (we can still add a few more bells). My examples are stats, but the same applies (at least) to signal and interpolate. my view, Josef
I think when you start talking about "1.0" people have very strong conflicting assumptions about what this "obviously" means, so...
-n _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
22.09.2013 02:12, josef.pktd@gmail.com kirjoitti: [clip]
Several parts of scipy have "organically grown", sometimes with several competing and incompatible implementations, some parts of the code were never reviewed and tested.
IMO, these are the main problems with the current code that need to be cleaned up before we can call it a scipy 1.0.
I agree, this is the main issue to be cleaned up before 1.0. There's some code there that is "research quality" [e.g. O(N^2) algorithm in a place where there is O(N) alternative and N is often big in real use cases], incompatible implementations of similar or the same things, and implementations that are missing commonly needed things. We've slowly sorted some of this crap out in the last few years, but there remains still a number of things to fix. Those that we remembered about are listed in the roadmap proposal. -- Pauli Virtanen
On Sat, Sep 21, 2013 at 7:29 PM, Pauli Virtanen <pav@iki.fi> wrote:
22.09.2013 02:12, josef.pktd@gmail.com kirjoitti: [clip]
Several parts of scipy have "organically grown", sometimes with several competing and incompatible implementations, some parts of the code were never reviewed and tested.
IMO, these are the main problems with the current code that need to be cleaned up before we can call it a scipy 1.0.
I agree, this is the main issue to be cleaned up before 1.0.
There's some code there that is "research quality" [e.g. O(N^2) algorithm in a place where there is O(N) alternative and N is often big in real use cases], incompatible implementations of similar or the same things, and implementations that are missing commonly needed things. We've slowly sorted some of this crap out in the last few years, but there remains still a number of things to fix. Those that we remembered about are listed in the roadmap proposal.
to emphasize this point We can talk *now* of a roadmap because there have been many improvements in scipy in the last few years by old and many new contributors, so that we can almost see what still needs to be done. Josef Once there was a dark tunnel, and now we can see a glimmer.
-- Pauli Virtanen
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
22.09.2013 01:35, Nathaniel Smith kirjoitti:
On Sat, Sep 21, 2013 at 7:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
- topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread.
I would be curious what the answers are to these questions :-).
This looks like a big list with many good improvements on it, but I'm not sure what makes them "1.0 changes" instead of just "good changes we should do". Does 1.0 mean we can break a lot of stuff at once and get away with it? Does it mean that after that we're not allowed to change things like this ever again so we have to get it right (and maybe keep slipping the schedule until we're certain)? Does it mean no more releases until all the below things happen? Or on the other extreme, does it mean that someone will keep an eye on this list, and at some point maybe in a few years when we notice that all of these things have happened, then the next release gets called "1.0" instead of "0.18" or whatever?
I think when you start talking about "1.0" people have very strong conflicting assumptions about what this "obviously" means, so...
In my mind, "1.0" for Scipy means that we cover a basic set of features needed for numerical science, and do it well. A definition via negative is: at "1.0" we don't have (i) stuff that's obviously crap (ii) blank spots at commonly needed functionality (iii) awkward usage patterns It is of course a subjective matter what this means, but writing the known issues down in the roadmap is one way to define it. Regarding API deprecations: I think breaking a lot of stuff at "1.0" at once is not a good idea, and I don't think that will happen. Rather, things progress as we have done so far --- if something is horribly awkward, a new API is introduced and the old one is deprecated. We can perhaps be slightly more aggressive in the cleaning out crap at "1.0", but I don't think there'll be a qualitative difference. This is also for practical reasons, as we will not be able to fix everything in one go in any case. Rather, one day in the distant future, it turns out that the next release is in a good enough shape to be called "1.0". The "0.x" roll up to that point. -- Pauli Virtanen
I think it's great to have some rough goal posts set out for 1.0. Regardless of exactly what 1.0 means, by making a giant todo list you've made contributing to SciPy much more appealing - it now feels like there is a targeted effort to tidy the whole project up, and I for one would like to be a part of it. My name is James Booth and I'm a Comp Sci PhD student. I'm part of a team at Imperial College London that is leaning heavily on Scipy for our research - I'll take a look through the list in detail and see if I can't help out in some way. Best wishes, James On 22 September 2013 00:19, Pauli Virtanen <pav@iki.fi> wrote:
22.09.2013 01:35, Nathaniel Smith kirjoitti:
On Sat, Sep 21, 2013 at 7:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
- topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread.
I would be curious what the answers are to these questions :-).
This looks like a big list with many good improvements on it, but I'm not sure what makes them "1.0 changes" instead of just "good changes we should do". Does 1.0 mean we can break a lot of stuff at once and get away with it? Does it mean that after that we're not allowed to change things like this ever again so we have to get it right (and maybe keep slipping the schedule until we're certain)? Does it mean no more releases until all the below things happen? Or on the other extreme, does it mean that someone will keep an eye on this list, and at some point maybe in a few years when we notice that all of these things have happened, then the next release gets called "1.0" instead of "0.18" or whatever?
I think when you start talking about "1.0" people have very strong conflicting assumptions about what this "obviously" means, so...
In my mind, "1.0" for Scipy means that we cover a basic set of features needed for numerical science, and do it well. A definition via negative is: at "1.0" we don't have
(i) stuff that's obviously crap (ii) blank spots at commonly needed functionality (iii) awkward usage patterns
It is of course a subjective matter what this means, but writing the known issues down in the roadmap is one way to define it.
Regarding API deprecations: I think breaking a lot of stuff at "1.0" at once is not a good idea, and I don't think that will happen. Rather, things progress as we have done so far --- if something is horribly awkward, a new API is introduced and the old one is deprecated. We can perhaps be slightly more aggressive in the cleaning out crap at "1.0", but I don't think there'll be a qualitative difference. This is also for practical reasons, as we will not be able to fix everything in one go in any case.
Rather, one day in the distant future, it turns out that the next release is in a good enough shape to be called "1.0". The "0.x" roll up to that point.
-- Pauli Virtanen
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Sun, Sep 22, 2013 at 1:28 AM, James Booth <jabooth@gmail.com> wrote:
I think it's great to have some rough goal posts set out for 1.0. Regardless of exactly what 1.0 means, by making a giant todo list you've made contributing to SciPy much more appealing - it now feels like there is a targeted effort to tidy the whole project up, and I for one would like to be a part of it.
Great. This is exactly why we want to have a roadmap.
My name is James Booth and I'm a Comp Sci PhD student. I'm part of a team at Imperial College London that is leaning heavily on Scipy for our research - I'll take a look through the list in detail and see if I can't help out in some way.
Looking forward to your pull requests! Cheers, Ralf
Ralf Gommers <ralf.gommers <at> gmail.com> writes:
Hi all, At EuroScipy Pauli, David and I sat together and drafted a roadmap for
Scipy 1.0. We then discussed this offline with some of the other currently most active core devs, to get it into a state that's ready for discussion on this list. So here it is: https://github.com/scipy/scipy/pull/2908
Our aim is for this roadmap to help guide us towards a 1.0 version, which
will contain only code that we consider to be "of sufficient quality". Also, it will help to communicate to new and potential developers where their contributions are especially needed.
In order to discuss/review this roadmap without generating a monster
thread, I propose the following:
- topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread. - things in the General section (API changes, documentation/test/build guidelines, etc.), are discussed on this thread as well. - for discussion of module-specific content, start a new thread and name it "1.0 roadmap: <module_name>". - for minor things, comment on the PR. Cheers,Ralf
The spline clean up previously mentioned by charles/pauli (http://article.gmane.org/gmane.comp.python.scientific.devel/17349) might be nice to get in. -Dave
participants (14)
-
Benny Malengier -
Blake Griffith -
boyfarrell＠gmail.com -
Christoph Deil -
Christopher Jordan-Squire -
Dag Sverre Seljebotn -
Dave Hirschfeld -
David Cournapeau -
James Booth -
josef.pktd＠gmail.com -
Nathaniel Smith -
Pauli Virtanen -
Ralf Gommers -
Robert Kern