Following on from some of the discussion surrounding my backend proposal and the discussion on gh-10175<https://github.com/scipy/scipy/issues/10175>; I've been working on adding a new scipy.fft submodule. This is now working, tested, documented and ready for review in gh-10238<https://github.com/scipy/scipy/pull/10238>. The new submodule is almost a drop-in replacement for numpy.fft and scipy.fftpack, with a few exceptions: * Does not include NumPy's Hermitian transforms (hfft, ihfft) (but would be simple to add) * Uses numpy's conventions for rfft (complex array) instead of fftpack's (complex values packed into a real array) * Convolutions and pseudo-differential operators from fftpack are not included The new submodule adds pocketfft to implement the normal FFTs (not yet DCT and DST) which adds several new features beyond scipy.fftpack: * long double transforms * multi-dimensional real FFTs (rfftn) * Orthonormal transforms (norm='ortho' argument) * Bluestein's algorithm to avoid the worst case O(n^2) complexity of FFTPACK A few implementation details worth noting: * Unlike NumPy's version of pocketfft, this is C++ and uses templates to supporting the different floating point types * pocketfft also uses pybind11 which both adds a new build-time dependency and requires C++11. I believe this is the first use of C++11 in SciPy. * pocketfft has optional support for multithreading using OpenMP but this is currently disabled (and not compiled in at all). And finally, there is the rather awkward issue that scipy.fft already exists and is an alias for the numpy.fft.fft function. Currently I've had to add a workaround to allow the module scipy.fft to be callable, as discussed in gh-10253<https://github.com/scipy/scipy/issues/10253>. This is only relevant to code that imports scipy.fft so I don't expect it to be used often, if at all. Hopefully this is a temporary solution and the NumPy functions can be removed from the scipy namespace in some future release, or at the very least the FFT functions can be removed. Peter
On Mon, Jun 10, 2019 at 4:22 PM Peter Bell <PeterBell10@live.co.uk> wrote:
Following on from some of the discussion surrounding my backend proposal and the discussion on gh-10175 <https://github.com/scipy/scipy/issues/10175>; I’ve been working on adding a new scipy.fft submodule. This is now working, tested, documented and ready for review in gh-10238 <https://github.com/scipy/scipy/pull/10238>.
The new submodule is almost a drop-in replacement for numpy.fft and scipy.fftpack, with a few exceptions:
- Does not include NumPy’s Hermitian transforms (hfft, ihfft) (but would be simple to add) - Uses numpy’s conventions for rfft (complex array) instead of fftpack’s (complex values packed into a real array) - Convolutions and pseudo-differential operators from fftpack are not included
The new submodule adds pocketfft to implement the normal FFTs (not yet DCT and DST) which adds several new features beyond scipy.fftpack:
- long double transforms - multi-dimensional real FFTs (rfftn) - Orthonormal transforms (norm=’ortho’ argument) - Bluestein’s algorithm to avoid the worst case O(n^2) complexity of FFTPACK
A few implementation details worth noting:
- Unlike NumPy’s version of pocketfft, this is C++ and uses templates to supporting the different floating point types - pocketfft also uses pybind11 which both adds a new build-time dependency and requires C++11. I believe this is the first use of C++11 in SciPy.
I've looked into this a little more. Compiler support for pybind11 is given in https://github.com/pybind/pybind11#supported-compilers.
MSVC 2015 Update 3 seems compatible with the lowest version we need as far as I can tell (Python 3.5/3.6 require MSVC 14.0 == 2015; the "Update 3" part should be fine). GCC 4.8 is already the minimum we need today I believe. Minimum Clang versions needed are from 2013, so that's fine too. For Intel compilers a version from 2016 is needed. I think we're fine requiring a more recent version of the Intel compilers. For more exotic platforms, recent enough GCC versions will also do the job. So good to go I think. Cheers, Ralf
- pocketfft has optional support for multithreading using OpenMP but this is currently disabled (and not compiled in at all).
And finally, there is the rather awkward issue that scipy.fft already exists and is an alias for the numpy.fft.fft function. Currently I’ve had to add a workaround to allow the module scipy.fft to be callable, as discussed in gh-10253 <https://github.com/scipy/scipy/issues/10253>. This is only relevant to code that imports scipy.fft so I don’t expect it to be used often, if at all. Hopefully this is a temporary solution and the NumPy functions can be removed from the scipy namespace in some future release, or at the very least the FFT functions can be removed.
Peter
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Hi all, Given that a few people were surprised that SciPy master now has a pybind11 dependency and asked if it was discussed on this list (to which I didn't have an immediate answer), replying to this thread with a more explicit subject. On Tue, Jun 11, 2019 at 4:21 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Mon, Jun 10, 2019 at 4:22 PM Peter Bell <PeterBell10@live.co.uk> wrote:
Following on from some of the discussion surrounding my backend proposal and the discussion on gh-10175 <https://github.com/scipy/scipy/issues/10175>; I’ve been working on adding a new scipy.fft submodule. This is now working, tested, documented and ready for review in gh-10238 <https://github.com/scipy/scipy/pull/10238>.
The new submodule is almost a drop-in replacement for numpy.fft and scipy.fftpack, with a few exceptions:
- Does not include NumPy’s Hermitian transforms (hfft, ihfft) (but would be simple to add) - Uses numpy’s conventions for rfft (complex array) instead of fftpack’s (complex values packed into a real array) - Convolutions and pseudo-differential operators from fftpack are not included
The new submodule adds pocketfft to implement the normal FFTs (not yet DCT and DST) which adds several new features beyond scipy.fftpack:
- long double transforms - multi-dimensional real FFTs (rfftn) - Orthonormal transforms (norm=’ortho’ argument) - Bluestein’s algorithm to avoid the worst case O(n^2) complexity of FFTPACK
A few implementation details worth noting:
- Unlike NumPy’s version of pocketfft, this is C++ and uses templates to supporting the different floating point types - pocketfft also uses pybind11 which both adds a new build-time dependency and requires C++11. I believe this is the first use of C++11 in SciPy.
I've looked into this a little more. Compiler support for pybind11 is given in https://github.com/pybind/pybind11#supported-compilers.
MSVC 2015 Update 3 seems compatible with the lowest version we need as far as I can tell (Python 3.5/3.6 require MSVC 14.0 == 2015; the "Update 3" part should be fine).
GCC 4.8 is already the minimum we need today I believe.
Minimum Clang versions needed are from 2013, so that's fine too.
For Intel compilers a version from 2016 is needed. I think we're fine requiring a more recent version of the Intel compilers.
For more exotic platforms, recent enough GCC versions will also do the job. So good to go I think.
So far pybind11 seems to have worked fine, except that it's not yet documented in all our build instructions (there's open PRs for this though, it's just waiting for Matt's larger devdocs overhaul), and that we seem to require a fairly recent version of pybind11. If there's any other missing docs or concerns, please bring them up here. Cheers, Ralf
- pocketfft has optional support for multithreading using OpenMP but this is currently disabled (and not compiled in at all).
On Fri, 5 Jul 2019 at 04:09, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Given that a few people were surprised that SciPy master now has a pybind11 dependency and asked if it was discussed on this list (to which I didn't have an immediate answer), replying to this thread with a more explicit subject.
<snip>
So far pybind11 seems to have worked fine, except that it's not yet documented in all our build instructions (there's open PRs for this though, it's just waiting for Matt's larger devdocs overhaul), and that we seem to require a fairly recent version of pybind11. If there's any other missing docs or concerns, please bring them up here.
Sigh. Having difficulties installing pybind on macOS, Python3.7.2, with pip 19.1.1. I guess I'll have to delve into this further. I was expecting the install to be straightforward. (dev3) 192-168-1-101:scipy-articles andrew$ pip install pybind Collecting pybind Using cached https://files.pythonhosted.org/packages/39/25/357918ab3113ac886ae5a244130636... Collecting pyangbind-brcd>=0.6.14 (from pybind) Using cached https://files.pythonhosted.org/packages/2d/12/299ccb698d0bfc0da6a7fc7165e169... ERROR: Complete output from command python setup.py egg_info: ERROR: Traceback (most recent call last): File "<string>", line 1, in <module> File "/private/var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/pip-install-1enlx25j/pyangbind-brcd/setup.py", line 2, in <module> from pip.req import parse_requirements ModuleNotFoundError: No module named 'pip.req' ---------------------------------------- ERROR: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/pip-install-1enlx25j/pyangbind-brcd/
pybind11, not pybind ;) Le dim. 7 juil. 2019 à 23:11, Andrew Nelson <andyfaff@gmail.com> a écrit :
On Fri, 5 Jul 2019 at 04:09, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Given that a few people were surprised that SciPy master now has a pybind11 dependency and asked if it was discussed on this list (to which I didn't have an immediate answer), replying to this thread with a more explicit subject.
<snip>
So far pybind11 seems to have worked fine, except that it's not yet documented in all our build instructions (there's open PRs for this though, it's just waiting for Matt's larger devdocs overhaul), and that we seem to require a fairly recent version of pybind11. If there's any other missing docs or concerns, please bring them up here.
Sigh. Having difficulties installing pybind on macOS, Python3.7.2, with pip 19.1.1. I guess I'll have to delve into this further. I was expecting the install to be straightforward.
(dev3) 192-168-1-101:scipy-articles andrew$ pip install pybind
Collecting pybind
Using cached https://files.pythonhosted.org/packages/39/25/357918ab3113ac886ae5a244130636...
Collecting pyangbind-brcd>=0.6.14 (from pybind)
Using cached https://files.pythonhosted.org/packages/2d/12/299ccb698d0bfc0da6a7fc7165e169...
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/pip-install-1enlx25j/pyangbind-brcd/setup.py", line 2, in <module>
from pip.req import parse_requirements
ModuleNotFoundError: No module named 'pip.req'
----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/pip-install-1enlx25j/pyangbind-brcd/
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
-- Quantitative researcher, Ph.D. Blog: http://blog.audio-tk.com/ LinkedIn: http://www.linkedin.com/in/matthieubrucher
participants (4)
-
Andrew Nelson -
Matthieu Brucher -
Peter Bell -
Ralf Gommers