What does fftn take as parameters?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Note I'm a programmer type, not a math type and am doing coding directed by a matlab user.) I'm trying to do an fft on multiple columns of data at once (ultimately feeding into a correlation calculation). I can use fft() to work on one column: data=[23, 43, 53, 54, 0, 10] powtwo=8 # nearest power of two size numpy.fft.fft(data, powtwo) I want to do that but using fftn (the matlab user said it is the right function) but I can't work out from the docs or experimentation how the input data should be formatted. eg is it row major or column major. For example the above could be: data=[ [23, 43, 53, 54, 0, 10] ] or data=[ [23], [43], [53], [54], [0], [10] ] All the examples in the docs use square inputs (ie x and y axes are the same length) so that doesn't help. The documentation shows examples of the output, but not the input. I found code passing in a single int (not a list of int) as the s parameter, but that also gives me an error. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7dN2YACgkQmOOfHg372QQ4YQCg4sKmtx8UAoEOuosWzUofw/KZ B5AAoKeHzP8HgpvDrXDANj0wqll5L9MO =iRAX -----END PGP SIGNATURE-----
On Mon, Dec 5, 2011 at 4:28 PM, Roger Binns <rogerb@rogerbinns.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
(Note I'm a programmer type, not a math type and am doing coding directed by a matlab user.)
I'm trying to do an fft on multiple columns of data at once (ultimately feeding into a correlation calculation). I can use fft() to work on one column:
data=[23, 43, 53, 54, 0, 10] powtwo=8 # nearest power of two size numpy.fft.fft(data, powtwo)
I am not I understand what you are trying to do ? numpy.fft.fft will compute fft on every *row*, or every column if you say pass axis=0 argument: numpy.fft.fft(data, 8, axis=0) # conceptually equivalent to the following for i in range(data.shape[1]): numpy.fft.fft(data[:, i], 8) # apply fft to each column separately fftn is for multi-dimensional fft, which is something else than doing a fft on every column, but this is true in matlab as well. cheers, David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/12/11 14:19, David Cournapeau wrote:
I am not I understand what you are trying to do ?
I had a slight misunderstanding with the math guy and had believed that for our purposes we could feed in 16 columns and get one "column" of fft output. However we do actually need 16 columns of output, each corresponding to a column of input. It seems he is obsessed with optimisation and apparently when calculating an fft of a known size it would save some redundant calculations operating on all 16 columns at once rather than doing them one at a time. That is what he assumed fftn did from the description.
numpy.fft.fft will compute fft on every *row*, or every column if you say pass axis=0 argument:
Note that I am using regular Python lists (they were JSON at one point) and the fft documentation is incomprehensible to someone who hasn't used numpy before and only cares about fft (there are a lot of matches for Google searches about fft and python pointing to numpy). The doc doesn't actually say what axis is and doesn't have an example. Additionally a "shape" attribute is used which is peculiar to whatever numpy uses as its data representation. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7dyRwACgkQmOOfHg372QToxgCfR7IoUfgGQVZEEiElnjbtx7yx R8EAnRfDg4y7AfFeSA8sQxVCq6ucgRG1 =gg2h -----END PGP SIGNATURE-----
06.12.2011 08:49, Roger Binns kirjoitti:
Note that I am using regular Python lists (they were JSON at one point) and the fft documentation is incomprehensible to someone who hasn't used numpy before and only cares about fft (there are a lot of matches for Google searches about fft and python pointing to numpy).
The doc doesn't actually say what axis is and doesn't have an example. Additionally a "shape" attribute is used which is peculiar to whatever numpy uses as its data representation.
I think this cannot be helped --- it does not make sense to explain basic Numpy concepts in every docstring, especially `axis` and `shape` are very common. However, an example with the axis keyword could be useful. -- Pauli Virtanen
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/12/11 01:58, Pauli Virtanen wrote:
I think this cannot be helped --- it does not make sense to explain basic Numpy concepts in every docstring, especially `axis` and `shape` are very common.
They don't need to be explained on the page, but instead link to a page that does explain them. The test is that an experienced Python programmer should be able to understand what is going on from the fft doc page and every page it links to. Note that searching doesn't help: http://docs.scipy.org/doc/numpy/search.html?q=axis http://docs.scipy.org/doc/numpy/search.html?q=shape
However, an example with the axis keyword could be useful.
And examples not using "square" inputs. With an input that has a long edge (eg time) and a short edge (value(s)) it is a lot easier to understand what is going on. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7eQ5gACgkQmOOfHg372QQpzQCeNO7evHqLaKNNQIzHbDLY3RVD HmwAn1MGH6DdJND5lKINwAFXzLbDry5J =xycq -----END PGP SIGNATURE-----
On 12/6/2011 8:32 AM, Roger Binns wrote:
I think this cannot be helped --- it does not make sense to explain basic Numpy concepts in every docstring, especially `axis` and `shape` are very common.
They don't need to be explained on the page, but instead link to a page that does explain them.
That would be a lot of links -- I understand your frustration, but fft is not a stand-alone fft lib -- it is a numpy module, tehre is litle choice but to understand a bit of numpy to use it. and "shape" and "axis" are quite core to numpy.
The test is that an experienced Python programmer should be able to understand what is going on from the fft doc page and every page it links to.
that's not a reasonable expectation, sorry. I doubt you'd be able to use matlab's fft functions with no knowledge of MATLAB, either.
And examples not using "square" inputs.
yes, good idea -- when I'm testing things, I always make the point of using non-square examples, so it's clear which asix is which. By the way, if you are getting JSON (from a web service?), converting to liats, converting to numpy arrays, then fft-ing (then maybe converting all that back?) I doubt that the performance of the ffts will be your bottleneck -- I'd write it the easiest way you can -- loops will be fine. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/12/11 09:59, Chris Barker wrote:
That would be a lot of links
Huh? The current page defines axis in terms of axis so if you don't know what it is you have to look it up. The search is completely useless as I showed. You want using numpy to be some sort of secret one has to work hard at? What I was asking for was that the word axis in the doc page links to a page that explains terms like axis and shape as numpy uses them.
By the way, if you are getting JSON (from a web service?), converting to liats, converting to numpy arrays, then fft-ing (then maybe converting all that back?)
I don't know what a liat is and I'm not converting to numpy arrays. I pass in regular Python lists and treat the result as though it was a regular Python list. I need to do 16 ffts with columns of data all the same length. That was why the interest in fftn (misplaced). Someone else mentioned that fft can actually do multiple ffts in one go and the math guy mentioned that it should be more efficient that way. At the moment I do indeed use a for loop with 16 iterations. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7eXFAACgkQmOOfHg372QQlRwCdGtE7+yhBX6Ng/gQp24/M76n7 BVoAn28GtkQ5KKxENJLJwD4alAWUWhB+ =myKN -----END PGP SIGNATURE-----
On 6. des. 2011, at 17:32, Roger Binns wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/12/11 01:58, Pauli Virtanen wrote:
I think this cannot be helped --- it does not make sense to explain basic Numpy concepts in every docstring, especially `axis` and `shape` are very common.
They don't need to be explained on the page, but instead link to a page that does explain them. The test is that an experienced Python programmer should be able to understand what is going on from the fft doc page and every page it links to. Note that searching doesn't help:
First google hit, "numpy axis": http://www.scipy.org/Tentative_NumPy_Tutorial Under "Basics", third and forth sentences: """ NumPy's main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In Numpy dimensions are called axes. The number of axes is rank. """ Searching is useful indeed, just use google instead. You can do the "several column FFT" easily using the axes keyword. Have a look at that link. If you are using numpy functions like FFT, you are indeed converting to arrays. Python lists are poorly performing for numerical work and do not map well onto the C and Fortran based codes that give numpy and scipy their good performance. Hence, numpy will convert to arrays for you, and if you are treating the returned array as a list, you are probably (among other things) indexing it in an inefficient way. It would seem to me that if your obsession over the FFT must be misplaced. If you are passing multidimensional lists here and there, you must have a lot of overhead memory use. Since the FFT is so efficient, O(n log (n)), I'm guessing that it's not your bottleneck (as others have suggested). Try using the cProfile module (python stdlib) to profile your code, and verify where your bottlenecks are. Come back to this list for more advice when you have evidence for where your bottleneck actually is. As a side note: since the built-in search isn't really all that good, would it be possible to put a customized google search box there instead? Good luck Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/12/11 10:45, Paul Anton Letnes wrote:
As a side note: since the built-in search isn't really all that good, would it be possible to put a customized google search box there instead?
It is easy since Sphinx is being used. Copy searchbox.html from the sphinx basic theme into the templates directory and alter the form to use Google custom search instead.
First google hit, "numpy axis": http://www.scipy.org/Tentative_NumPy_Tutorial
It is still somewhat confusing but I guess trial and error will work it out. It isn't at all obvious that you can do multiple FFTs at once from the doc as everything is written in the singular.
If you are using numpy functions like FFT, you are indeed converting to arrays.
I have no issue with that. The data comes in over the network and is supplied to the fft function as input so it doesn't really make any difference whether I convert it into the numpy array type or the fft function does internally - it is still only done once.
and if you are treating the returned array as a list, you are probably (among other things) indexing it in an inefficient way.
I iterate over it once feeding the values into a separate calculation.
If you are passing multidimensional lists here and there, you must have a lot of overhead memory use.
The data structures are less than 10MB in Python's internal PyObject representation. The machines being used have 32GB of memory. There is only one explicit or implicit conversion between different formats.
I'm guessing that it's not your bottleneck (as others have suggested).
I'm doing work directed by a matlab/embedded developer and he is obsessed with performance. I'm trying to interpret his advice to do all the ffts at once in the context of the numpy fft apis. If you look at the fftw apis they go on about plans and reusing them so I assumed there was something to all this :-) I'm currently rewriting the non-FFT bits in C because they were far too slow. It is competing against a pure integer C algorithm that gives the same output but has an O(n^2) time. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7eefMACgkQmOOfHg372QRAwACfXFK34pCgv32Xn3eri9FJbvis +XYAoNcUCILHVcQ6ri353Kp0YPrw0EbS =Z0KE -----END PGP SIGNATURE-----
participants (5)
-
Chris Barker -
David Cournapeau -
Paul Anton Letnes -
Pauli Virtanen -
Roger Binns