Adding algorithm for automatic differentiation of noisy data to SciPy
Hello, Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results. I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas. Link to the pull request. https://github.com/scipy/scipy/pull/9004 The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output. The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms. Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data. Best Regards, Siva Sankar Kannan.
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication? Cheers, Ralf
Hello, Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv. The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning. The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values. Best Regards, Siva Sankar Kannan. On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv.
Can you share a link to the paper?
The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning.
Your PR adds code to scipy.misc, which is not the right place (we'll get rid of misc soon). It sounds more like it belongs in scipy.signal. That said, don't do a lot of work to move things now - first we need to decide whether this makes sense to include in SciPy. At the moment I'd say it's too early; we probably want to wait a year or two until it's clear that the paper has been accepted and gets citations that show the algorithm is valuable. Cheers, Ralf
The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values.
Best Regards, Siva Sankar Kannan.
On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
The paper can be found here. https://arxiv.org/abs/1610.04397 Ok, I will wait before I do anything more. BR, Siva. On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv.
Can you share a link to the paper?
The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning.
Your PR adds code to scipy.misc, which is not the right place (we'll get rid of misc soon). It sounds more like it belongs in scipy.signal. That said, don't do a lot of work to move things now - first we need to decide whether this makes sense to include in SciPy. At the moment I'd say it's too early; we probably want to wait a year or two until it's clear that the paper has been accepted and gets citations that show the algorithm is valuable.
Cheers, Ralf
The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values.
Best Regards, Siva Sankar Kannan.
On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Fri, Jul 6, 2018 at 2:17 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
The paper can be found here. https://arxiv.org/abs/1610.04397
Thanks Siva. Given that that paper is quite technical and terse, that it's 2 years old with only one self-citation, and that it doesn't fit nicely in a scipy submodule, my opinion is that we unfortunately cannot accept your PR. Please don't let that discourage you though - I would suggest to release your algorithm standalone - if it matures and gets update we can reconsider at some point. Cheers, Ralf
Ok, I will wait before I do anything more.
BR, Siva.
On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv.
Can you share a link to the paper?
The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning.
Your PR adds code to scipy.misc, which is not the right place (we'll get rid of misc soon). It sounds more like it belongs in scipy.signal. That said, don't do a lot of work to move things now - first we need to decide whether this makes sense to include in SciPy. At the moment I'd say it's too early; we probably want to wait a year or two until it's clear that the paper has been accepted and gets citations that show the algorithm is valuable.
Cheers, Ralf
The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values.
Best Regards, Siva Sankar Kannan.
On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Ok, thanks for the response. I will release it as a standalone algorithm somewhere. BR, Siva. On Sat, Jul 7, 2018 at 7:56 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jul 6, 2018 at 2:17 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
The paper can be found here. https://arxiv.org/abs/1610.04397
Thanks Siva. Given that that paper is quite technical and terse, that it's 2 years old with only one self-citation, and that it doesn't fit nicely in a scipy submodule, my opinion is that we unfortunately cannot accept your PR. Please don't let that discourage you though - I would suggest to release your algorithm standalone - if it matures and gets update we can reconsider at some point.
Cheers, Ralf
Ok, I will wait before I do anything more.
BR, Siva.
On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv.
Can you share a link to the paper?
The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning.
Your PR adds code to scipy.misc, which is not the right place (we'll get rid of misc soon). It sounds more like it belongs in scipy.signal. That said, don't do a lot of work to move things now - first we need to decide whether this makes sense to include in SciPy. At the moment I'd say it's too early; we probably want to wait a year or two until it's clear that the paper has been accepted and gets citations that show the algorithm is valuable.
Cheers, Ralf
The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values.
Best Regards, Siva Sankar Kannan.
On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Siva Some years ago I created "scikit-datasmooth": https://github.com/jjstickel/scikit-datasmooth It has contained smoothing (and differentiation) by the regularization method, but I always hoped it would generate interest and that more methods would be added. You would be welcome to add your method there. We can discuss off list. Regards, Jonathan On 7/8/18 06:14 , Siva Sankar wrote:
Ok, thanks for the response. I will release it as a standalone algorithm somewhere.
BR, Siva.
On Sat, Jul 7, 2018 at 7:56 PM Ralf Gommers <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>> wrote:
On Fri, Jul 6, 2018 at 2:17 PM, Siva Sankar <sivsankar977@gmail.com <mailto:sivsankar977@gmail.com>> wrote:
The paper can be found here. https://arxiv.org/abs/1610.04397
Thanks Siva. Given that that paper is quite technical and terse, that it's 2 years old with only one self-citation, and that it doesn't fit nicely in a scipy submodule, my opinion is that we unfortunately cannot accept your PR. Please don't let that discourage you though - I would suggest to release your algorithm standalone - if it matures and gets update we can reconsider at some point.
Cheers, Ralf
Ok, I will wait before I do anything more.
BR, Siva.
On Fri, Jul 6, 2018 at 7:20 PM Ralf Gommers <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>> wrote:
On Fri, Jul 6, 2018 at 2:07 AM, Siva Sankar <sivsankar977@gmail.com <mailto:sivsankar977@gmail.com>> wrote:
Hello,
Thanks for the swift response. Yes, I have based this on a paper that the professor I am working for was working on. The paper, however, is not yet published in a Journal yet but is available in arXiv.
Can you share a link to the paper?
The algorithm was tested with real-world data and synthetic data, comparing against other algorithms such as Woltring's B-Spline (GCVSPL) and Savitzky Golay with manual parameter tuning.
Your PR adds code to scipy.misc, which is not the right place (we'll get rid of misc soon). It sounds more like it belongs in scipy.signal. That said, don't do a lot of work to move things now - first we need to decide whether this makes sense to include in SciPy. At the moment I'd say it's too early; we probably want to wait a year or two until it's clear that the paper has been accepted and gets citations that show the algorithm is valuable.
Cheers, Ralf
The real world test data included using an XSENS IMU to get inertial data readings and then comparing them against those from a VectorNAV imu which has significantly accurate readings. The XSENS IMU that was used not being very accurate simulated the noisy measurements, while the VectorNAV unit measurements were considered as the reference measurements. The algorithm was then used to estimate the derivates and then results were compared with the reference values.
Best Regards, Siva Sankar Kannan.
On Fri, Jul 6, 2018 at 8:43 AM Ralf Gommers <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>> wrote:
On Thu, Jul 5, 2018 at 3:24 PM, Siva Sankar <sivsankar977@gmail.com <mailto:sivsankar977@gmail.com>> wrote:
Hello,
Automatic differentiation of noisy data has not been very accurate and does not have a lot of different algorithms that give consistent and accurate results.
I have added an algorithm that estimates the derivative of noisy data using linear Gaussian state-space smoothing and square root formulas.
Link to the pull request. https://github.com/scipy/scipy/pull/9004
The algorithm in its current state provides the smoothed signal from the noisy measurements, estimation of the first and second order derivatives and optionally also provides the dense output for the same. It accepts non-equally spaced data abscissas and is able to compute the state parameters between the abscissas, hence being able to provide the dense output.
The algorithm was tested with data from bioanalytics and provides equal or better accuracy of the derivatives compared to the other automatic derivative algorithms.
Real life readings and measurements of data in any field are prone to noise, thereby making the normal differentiation algorithms less reliable. Having an algorithm like this can significantly help users from a variety of fields where they need a good differentiation estimation without having to tweak with the parameters to differentiate data.
Hi Siva, thank you for trying to improve the situation with differentiation functionality. In SciPy we aim to include algorithms that are well known and have good performance - that means ideally a paper with enough citations showing real-world value. Have you based this on a publication?
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (3)
-
Jonathan Stickel -
Ralf Gommers -
Siva Sankar