fftpack: building all backends and setting them at runtime (was dropping djbfft)
Hi, After some more work, I have modified scipy.fftpack such as backends are truely independent code-wise, and can be selected at runtime. I hope it adresses all remarks comments made in the previous thread: http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/ Concretely: - all fftw/fftw3/djbfft/mkl code has been put in scipy/fftpack/backends directory. - fftpack is always build and used by default for all functions - for each backend, if the corresponding libraries are available, the module is built (so if you have mkl, fftw, fftw3 and djbfft, all backends will always be built) - fftpack module now uses some magic to set up functions from one backend or the other (scipy/fftpack/common.py). Basically, the functions are defined at runtime depending on which backend is selected. If the function is not available, the default backend is used as a fallback. TODO: - I have not implemented any registration method, because I was not sure how to do it (config file, monkey patching, import magic ?). - The convolution module is not done yet. - I have not implemented djbfft backend yet (all the other ones should work), because it needs a bit more logic, but nothing serious. - All module .pyf files are almost duplication, but since we want to have them independent, I am not sure how to do better. It should be possible to put all the code in backends out of scipy once the registration system is in place. cheers, David
David Cournapeau wrote:
Hi,
After some more work, I have modified scipy.fftpack such as backends are truely independent code-wise, and can be selected at runtime. I hope it adresses all remarks comments made in the previous thread:
http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/
Concretely: - all fftw/fftw3/djbfft/mkl code has been put in scipy/fftpack/backends directory.
looks good to me.
- All module .pyf files are almost duplication, but since we want to have them independent, I am not sure how to do better.
I agree on the independence. On the other hand, the signatures of wrapper functions should be identical for all backends and I think it can be ensured at the build time (and so saves some unittests). The content of signature files that can be different for different backends, is the callprotoargument and callstatement sections in .pyf files. Other parts of the signature files should be identical (except the module name). So, to minimize code duplication at the expense of introducing two auxiliary files, one can organize the .pyf files as follows: ! File backends/fftw/fftw.pyf: python module _fftw include ../common/fft_part0.pyf callprotoargument complex_double*,int,int*,int,int,int callstatement {& int i,sz=1,xsz=size(x); & for (i=0;i<r;++i) sz *= s[i]; & howmany = xsz/sz; & if (sz*howmany==xsz) & (*f2py_func)(x,r,s,direction,howmany,normalize); & else {& f2py_success = 0; & PyErr_SetString(_fftw_error, & "inconsistency in x.shape and s argument"); & } & } include ../common/fft_part1.pyf end python module _fftw ! File backends/fftw3/fftw3.pyf: python module _fftw3 include ../common/fft_part0.pyf callprotoargument complex_double*,int,int*,int,int,int callstatement {& int i,sz=1,xsz=size(x); & for (i=0;i<r;++i) sz *= s[i]; & howmany = xsz/sz; & if (sz*howmany==xsz) & (*f2py_func)(x,r,s,direction,howmany,normalize); & else {& f2py_success = 0; & PyErr_SetString(_fftw3_error, & "inconsistency in x.shape and s argument"); & } & } include ../common/fft_part1.pyf end python module _fftw3 Etc for the pyf files of mkl, djbfft backends. In fact, also for the default fftpack backend. I hope it is obvious what the fft_part0.pyf, .. files should contain. Note that the wrapper function names do not need to contain the name of a backend. Pearu
Pearu Peterson wrote:
I agree on the independence. On the other hand, the signatures of wrapper functions should be identical for all backends
Yes, that's my main concern, specially if the code is put in scikits and get a life on its own, we should make sure the C code call convention is the same (the other solution would be for each backend to provide a fatter wrapper, with fft implementation at the python level, but then there is python code duplication).
The content of signature files that can be different for different backends, is the callprotoargument and callstatement sections in .pyf files.
Ok, bear with me here for being slow, but I don't know anything about f2py :) Do the callprotoargument and callstatement apply to all functions from the included functions ?
Note that the wrapper function names do not need to contain the name of a backend.
At the C level, the function should certainly have different names, and I didn't find how to wrap a C function foo_bar as a foo function in the f2py generated module. I didn't look for long, though. cheers, David
On Fri, May 16, 2008 10:08 am, David Cournapeau wrote:
Pearu Peterson wrote:
I agree on the independence. On the other hand, the signatures of wrapper functions should be identical for all backends
Yes, that's my main concern, specially if the code is put in scikits and get a life on its own, we should make sure the C code call convention is the same (the other solution would be for each backend to provide a fatter wrapper, with fft implementation at the python level, but then there is python code duplication).
(the other solution should be avoided if possible)
The content of signature files that can be different for different backends, is the callprotoargument and callstatement sections in .pyf files.
Ok, bear with me here for being slow, but I don't know anything about f2py :) Do the callprotoargument and callstatement apply to all functions from the included functions ?
No, the suggestion was to split the pyf file into 3 pieces: part 0, the call* calles, part 1. Now that I look at it, it seems that also call* parts are identical for all backends except the module name part. This should make also the split of files unnecessary. I'll take a look at the branch and see if I can figure something easier out..
Note that the wrapper function names do not need to contain the name of a backend.
At the C level, the function should certainly have different names,
Why? Isn't C functions static and so there should be no conflicts provided that all backends have their own extension modules. May be I am overlooking something..
and I didn't find how to wrap a C function foo_bar as a foo function in the f2py generated module. I didn't look for long, though.
That's easy, one should use `fortranname` statement in pyd files. Search fortranname in f2py users manual. Pearu
Pearu Peterson wrote:
No, the suggestion was to split the pyf file into 3 pieces: part 0, the call* calles, part 1. Now that I look at it, it seems that also call* parts are identical for all backends except the module name part. This should make also the split of files unnecessary. I'll take a look at the branch and see if I can figure something easier out..
Great, thanks for looking at this.
Why? Isn't C functions static and so there should be no conflicts provided that all backends have their own extension modules. May be I am overlooking something..
The functions exposed by f2py cannot be static, and contrary to what was the case before, it is possible to have several backends loaded at the same time at the python <-> C layer (this was not a problem before since the several backends were only used inside C, and as such, several backends were never present in the python <-> C layer). Actually, it is one of the nice benefit of the current approach "for free": you don't have to rebuild fftpack to use a different backend. Typically, I can now test all backends just by changing one line in the fftpack module, and once we decide on the best approach for registration, we will be able to do it purely at runtime in python.
That's easy, one should use `fortranname` statement in pyd files. Search fortranname in f2py users manual.
Thanks, I will modify this part, then, instead of doing the renaming in python. cheers, David
David Cournapeau wrote:
Thanks, I will modify this part, then, instead of doing the renaming in python.
While you're here, Pearu, is fftpack the same as dfftpack API wise ? While refactoring, I noticed nobody wrapped the float version even if the fortran code is there, and that would really nice to have (would give a nice speed boost "for free" without relying on GPL or proprietary code). cheers, David
On Fri, May 16, 2008 4:53 pm, David Cournapeau wrote:
David Cournapeau wrote:
Thanks, I will modify this part, then, instead of doing the renaming in python.
While you're here, Pearu, is fftpack the same as dfftpack API wise ?
afaik, yes. See the DFFTPACK/doc.double file for details.
While refactoring, I noticed nobody wrapped the float version even if the fortran code is there, and that would really nice to have (would give a nice speed boost "for free" without relying on GPL or proprietary code).
At the time only double version was wrapped because it was a start (and I needed only the double version for my app). Pearu
On Fri, May 16, 2008 4:45 pm, Pearu Peterson wrote:
I'll take a look at the branch and see if I can figure something easier out..
Ok, here follows a simpler solution that uses numpy.distutils pyf.src templating tools. I'll give an example using fftw3 backend only because exactly the same technique applies to other backends. Below follows the corresponding diff and it works. If you find it a reasonable approach, I can commit the diff to the refactor_fft branch. Pearu pearu@utlaan-laptop:~/svn/refactor_fft/scipy/fftpack/backends$ svn diff Index: fftw3/setup.py =================================================================== --- fftw3/setup.py (revision 4373) +++ fftw3/setup.py (working copy) @@ -13,7 +13,10 @@ sources = src, include_dirs = ["../common", info['include_dirs']]) - config.add_extension("_fftw3", sources = ["fftw3.pyf"], extra_info = info, libraries = ["fftw3_backend"]) + config.add_extension("_fftw3", sources = ["fftw3.pyf.src"], + depends = ['../fft_template.pyf.src'], + extra_info = info, + libraries = ["fftw3_backend"]) def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration Index: fftw3/fftw3.pyf.src =================================================================== --- fftw3/fftw3.pyf.src (revision 0) +++ fftw3/fftw3.pyf.src (revision 0) @@ -0,0 +1,9 @@ +! -*- f90 -*- + +! <modulename=_fftw3> + +python module _fftw3 + interface +include '../fft_template.pyf.src' + end interface +end python module _fftw3 Index: fft_template.pyf.src =================================================================== --- fft_template.pyf.src (revision 0) +++ fft_template.pyf.src (revision 0) @@ -0,0 +1,72 @@ +!%f90 -*- f90 -*- +! Author: Pearu Peterson, August 2002 +! This file is included by the files <backend>/<backend>.pyf.src + + subroutine zfft<modulename>(x,n,direction,howmany,normalize) + ! y = fft(x[,n,direction,normalize,overwrite_x]) + intent(c) zfft<modulename> + complex*16 intent(c,in,out,copy,out=y) :: x(*) + integer optional,depend(x),intent(c,in) :: n=size(x) + check(n>0) n + integer depend(x,n),intent(c,hide) :: howmany = size(x)/n + check(n*howmany==size(x)) howmany + integer optional,intent(c,in) :: direction = 1 + integer optional,intent(c,in),depend(direction) & + :: normalize = (direction<0) + end subroutine zfft<modulename> + + subroutine drfft<modulename>(x,n,direction,howmany,normalize) + ! y = drfft(x[,n,direction,normalize,overwrite_x]) + intent(c) drfft<modulename> + real*8 intent(c,in,out,copy,out=y) :: x(*) + integer optional,depend(x),intent(c,in) :: n=size(x) + check(n>0&&n<=size(x)) n + integer depend(x,n),intent(c,hide) :: howmany = size(x)/n + check(n*howmany==size(x)) howmany + integer optional,intent(c,in) :: direction = 1 + integer optional,intent(c,in),depend(direction) & + :: normalize = (direction<0) + end subroutine drfft<modulename> + +! subroutine zrfft<modulename>(x,n,direction,howmany,normalize) +! ! y = zrfft(x[,n,direction,normalize,overwrite_x]) +! intent(c) zrfft<modulename> +! complex*16 intent(c,in,out,overwrite,out=y) :: x(*) +! integer optional,depend(x),intent(c,in) :: n=size(x) +! check(n>0&&n<=size(x)) n +! integer depend(x,n),intent(c,hide) :: howmany = size(x)/n +! check(n*howmany==size(x)) howmany +! integer optional,intent(c,in) :: direction = 1 +! integer optional,intent(c,in),depend(direction) & +! :: normalize = (direction<0) +! end subroutine zrfft<modulename> + + subroutine zfftnd<modulename>(x,r,s,direction,howmany,normalize,j) + ! y = zfftnd(x[,s,direction,normalize,overwrite_x]) + intent(c) zfftnd<modulename> + complex*16 intent(c,in,out,copy,out=y) :: x(*) + integer intent(c,hide),depend(x) :: r=old_rank(x) + integer intent(c,hide) :: j=0 + integer optional,depend(r),dimension(r),intent(c,in) & + :: s=old_shape(x,j++) + check(r>=len(s)) s + integer intent(c,hide) :: howmany = 1 + integer optional,intent(c,in) :: direction = 1 + integer optional,intent(c,in),depend(direction) :: & + normalize = (direction<0) + callprotoargument complex_double*,int,int*,int,int,int + callstatement {& + int i,sz=1,xsz=size(x); & + for (i=0;i<r;++i) sz *= s[i]; & + howmany = xsz/sz; & + if (sz*howmany==xsz) & + (*f2py_func)(x,r,s,direction,howmany,normalize); & + else {& + f2py_success = 0; & + PyErr_SetString(<modulename>_error, & + "inconsistency in x.shape and s argument"); & + } & + } + end subroutine zfftnd<modulename> + +! See http://cens.ioc.ee/projects/f2py2e/
Pearu Peterson wrote:
Ok, here follows a simpler solution that uses numpy.distutils pyf.src templating tools. I'll give an example using fftw3 backend only because exactly the same technique applies to other backends.
Ok, I got it working, thanks. I have another question, though: where is the code to handle .pyf.src files handled, I don't find it in numpy.distutils (I need it for numscons build). cheers, David
On Sat, May 17, 2008 12:07 pm, David Cournapeau wrote:
Pearu Peterson wrote:
Ok, here follows a simpler solution that uses numpy.distutils pyf.src templating tools. I'll give an example using fftw3 backend only because exactly the same technique applies to other backends.
Ok, I got it working, thanks. I have another question, though: where is the code to handle .pyf.src files handled, I don't find it in numpy.distutils (I need it for numscons build).
The *.src support code is in numpy/distutils/from_template.py. Pearu
participants (2)
-
David Cournapeau -
Pearu Peterson