custom Welch method for power spectral density
Hello, I'm trying to compute a power spectral density of a signal, using the Welch method, in the broad sense; i.e. splitting the signal into segments for deriving smoother spectra. This is well implemented in scipy.signal.welch. However, I'd like to use exponentially increasing (power 2) segment length to dampen increasing variance in spectra at higher frequencies. Before hacking the scipy.signal.spectral module for this, I'd appreciate any tips on available packages/modules that allow for this kind of binning scheme, or other suggestions. Thanks, -- Seb
On Fri, Jan 12, 2018 at 1:12 PM, Seb <spluque@gmail.com> wrote:
Hello,
I'm trying to compute a power spectral density of a signal, using the Welch method, in the broad sense; i.e. splitting the signal into segments for deriving smoother spectra. This is well implemented in scipy.signal.welch. However, I'd like to use exponentially increasing (power 2) segment length to dampen increasing variance in spectra at higher frequencies. Before hacking the scipy.signal.spectral module for this, I'd appreciate any tips on available packages/modules that allow for this kind of binning scheme, or other suggestions.
Not entirely sure about this kind of binning scheme per se, but you may want to look at multitaper spectral estimation methods. The Welch method can be viewed as a poor-man's multitaper. Multitaper methods give you better control over the resolution/variance tradeoff that may help with your problem. Googling for "python multitaper" gives you several options; I haven't used any of them in anger, so I don't have a single recommendation for you. The nitime documentation provides more information about multitaper methods that may be useful to you: http://nipy.org/nitime/examples/multi_taper_spectral_estimation.html -- Robert Kern
On Fri, 12 Jan 2018 13:32:11 -0800, Robert Kern <robert.kern@gmail.com> wrote: [...]
Not entirely sure about this kind of binning scheme per se, but you may want to look at multitaper spectral estimation methods. The Welch method can be viewed as a poor-man's multitaper. Multitaper methods give you better control over the resolution/variance tradeoff that may help with your problem. Googling for "python multitaper" gives you several options; I haven't used any of them in anger, so I don't have a single recommendation for you. The nitime documentation provides more information about multitaper methods that may be useful to you:
http://nipy.org/nitime/examples/multi_taper_spectral_estimation.html
Very interesting documentation and suggestion. A test application of this looks promising. Thanks, -- Seb
Hello, how I could dynamically handle the dtype of a structured array when reading an array of C structs with np.frombuffer (regarding the member padding in the struct). So far I manually adjusted the dtype of the structured array and added a field for the padding, this works on a small scale. The structs are not defined within my code, but within a third party and basically I am looking for no-worry hassle free way to handle this, because there are a lot of structs Is there some smart way to do this in Numpy? So far the best approach seems to parse the struct with the cffi functions ffi.sizeof(), ffi.offsetof() and maybe ffi.alignof() to find out where the padding happens and add it dynamically to the dtype. But maybe someone has a smarter idea how to solve this. You can find a more detailed description and a working example here: https://stackoverflow.com/questions/48423725/how-to-handle-member-padding-in... Kind regards, Joe
Sent from my iPhone
On Jan 25, 2018, at 6:51 AM, Joe <solarjoe@posteo.org> wrote:
Hello,
how I could dynamically handle the dtype of a structured array when reading an array of C structs with np.frombuffer (regarding the member padding in the struct).
So far I manually adjusted the dtype of the structured array and added a field for the padding, this works on a small scale. The structs are not defined within my code, but within a third party and basically I am looking for no-worry hassle free way to handle this, because there are a lot of structs
Is there some smart way to do this in Numpy?
The numpy dtype constructor takes an “align” keyword that will pad it for you. However, if these strict are coming from a lib compiled by a third party, I’m not sure you can count on the alignment rules being the same. So maybe you will need to use the cffi functions :-( -CHB
So far the best approach seems to parse the struct with the cffi functions ffi.sizeof(), ffi.offsetof() and maybe ffi.alignof() to find out where the padding happens and add it dynamically to the dtype. But maybe someone has a smarter idea how to solve this.
You can find a more detailed description and a working example here:
https://stackoverflow.com/questions/48423725/how-to-handle-member-padding-in...
Kind regards, Joe
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
The numpy dtype constructor takes an “align” keyword that will pad it for you. https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html -CHB
There is a new section discussing alignment in the numpy 1.14 structured array docs, which has some hints about interfacing with C structs. These new 1.14 docs are not online yet on scipy.org, but in the meantime you can view them here: https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-a... (That links specifically to the discussion of alignments and padding). Allan On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
The numpy dtype constructor takes an “align” keyword that will pad it for you.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html
-CHB
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Thanks for your help on this! This solved my issue. Am 25.01.2018 um 19:01 schrieb Allan Haldane:
There is a new section discussing alignment in the numpy 1.14 structured array docs, which has some hints about interfacing with C structs.
These new 1.14 docs are not online yet on scipy.org, but in the meantime you can view them here: https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-a...
(That links specifically to the discussion of alignments and padding).
Allan
On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
The numpy dtype constructor takes an “align” keyword that will pad it for you.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html
-CHB
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Does someone know of a function or a convenient way to automatically derive a dtype object from a C typedef struct string or a cffi.typeof()? Am 27.01.2018 10:30 schrieb Joe:
Thanks for your help on this! This solved my issue.
Am 25.01.2018 um 19:01 schrieb Allan Haldane:
There is a new section discussing alignment in the numpy 1.14 structured array docs, which has some hints about interfacing with C structs.
These new 1.14 docs are not online yet on scipy.org, but in the meantime you can view them here: https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-a...
(That links specifically to the discussion of alignments and padding).
Allan
On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
The numpy dtype constructor takes an “align” keyword that will pad it for you.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html
-CHB
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On 01/31/2018 02:06 AM, Joe wrote:
Does someone know of a function or a convenient way to automatically derive a dtype object from a C typedef struct string or a cffi.typeof()?
I remember when I wrote those docs over a year ago, I searched for an established way to do this but didn't find one. If you find/come up with one, let us know and I might add a pointer or description of it in the docs. Searching just now, I came across this recent gist: https://gist.github.com/inactivist/4ef7058c2132fa16759d So it looks like we can do something like: # typemap from C type to numpy type may need some work... typemap = {'char': 'byte', 'short': 'short', 'int': 'intc', 'long long': 'longlong', 'size_t': 'intp', 'float': 'float', 'double': 'double'} def extract_field_info(tp): fields = [] for field, fieldtype in tp.fields: if fieldtype.type.kind == 'primitive': format = typemap[fieldtype.type.cname] else: format = extract_field_info(fieldtype.type) fields.append((field, format, fieldtype.offset)) return fields from cffi import FFI ffi = FFI() ffi.cdef("struct foo { int a; char b; double d;};") tp = ffi.typeof('struct foo') names, formats, offsets = zip(*extract_field_info(tp)) np.dtype({'names': names, 'formats': formats, 'offsets': offsets}) # compare to: np.dtype('i4,i1,f8', align=True) I just made that up now and it may be buggy and have missing corner cases. But if you agree it works, maybe we can mention it in the structured array docs instead of the vague sentence there now, that "some work may be needed to obtain correspondence with the C struct". Allan
Am 27.01.2018 10:30 schrieb Joe:
Thanks for your help on this! This solved my issue.
Am 25.01.2018 um 19:01 schrieb Allan Haldane:
There is a new section discussing alignment in the numpy 1.14 structured array docs, which has some hints about interfacing with C structs.
These new 1.14 docs are not online yet on scipy.org, but in the meantime you can view them here: https://ahaldane.github.io/user/basics.rec.html#automatic-byte-offsets-and-a...
(That links specifically to the discussion of alignments and padding).
Allan
On 01/25/2018 11:33 AM, Chris Barker - NOAA Federal wrote:
The numpy dtype constructor takes an “align” keyword that will pad it for you.
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.dtype.html
-CHB
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (5)
-
Allan Haldane -
Chris Barker - NOAA Federal -
Joe -
Robert Kern -
Seb