[SciPy-Dev] SciPy-Dev Digest, Vol 188, Issue 9

Mon Jun 17 14:58:05 EDT 2019

Hello,

Thank you for your response.

> 1. Why do you want to move your whole package into SciPy? Do you plan to
> keep maintaining mcgpy? Would you maintain the code included in SciPy?

Our main motivation for moving the package into SciPy is that we feel that more people would to have access to these useful independence tests. This is important because it appears that currently, most of those in SciPy operate on one-dimensional data vectors or have the assumption that the data is sampled from a normal distribution. We plan on deprecating mgcpy and helping to maintain the code included within SciPy. This includes porting mgcpy into SciPy and conforming mgcpy to the SciPy API where necessary.

> 2. Your code currently depends on pandas, scikit-learn and seaborn. Those
> are not dependencies of SciPy, and we'd prefer not to add new dependencies.
> I haven't looked in more detail at this - would it be easy to remove those
> dependencies?

These are used when we were generating our demos and power curves to compare the independence tests to help make plots more visually appealing. None of the core independence tests nor their unit tests import anything from those packages. As such, it is not necessary to add them dependencies.

Thank you,

Sambit Panda

> On Jun 16, 2019, at 5:59 AM, scipy-dev-request at python.org wrote:
> 
> 
> Send SciPy-Dev mailing list submissions to
> 	scipy-dev at python.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://mail.python.org/mailman/listinfo/scipy-dev
> or, via email, send a message with subject or body 'help' to
> 	scipy-dev-request at python.org
> 
> You can reach the person managing the list at
> 	scipy-dev-owner at python.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-Dev digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Overlap-add convolution implementation
>      (3ukip0s02 at sneakemail.com)
>   2. Re: Overlap-add convolution implementation (Freddy Rietdijk)
>   3. Re: Request to add functionality to scipy.stats (Ralf Gommers)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Sat, 15 Jun 2019 13:01:41 -0400
> From: 3ukip0s02 at sneakemail.com
> To: scipy-dev at python.org
> Subject: Re: [SciPy-Dev] Overlap-add convolution implementation
> Message-ID: <21025-1560618124-553550 at sneakemail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote:
> 
>> I am currently working an an implementation of overlap-add convolution
>> [0].  I have a 1D implementation working and I am going to start expanding
>> it to ND.  Overlap-add convolution can provide significant performance
>> benefits over FFT-based convolution.  Does this make sense for scipy?
>> 
> 
> Yeah!
> 
> 
>> First is the name.  We have "convolve" and "fftconvolve" already.  A few
>> options:
>> 
>> overall_add_conv
>> oaconvolve
>> oadconvolve
>> 
> 
> fftconvolve has been folded into convolve and can either be selected
> manually, or will automatically choose direct convolution, whichever is
> likely faster.
> 
> Does it make sense for the OLA implementation to also be added as a
> convolve() option, using the "method" parameter?
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190615/4b2a1c16/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Sat, 15 Jun 2019 19:47:38 +0200
> From: Freddy Rietdijk <freddyrietdijk at fridh.nl>
> To: SciPy Developers List <scipy-dev at python.org>
> Subject: Re: [SciPy-Dev] Overlap-add convolution implementation
> Message-ID:
> 	<CAOQtOH1t4sc+bDG+pSeGPW=76Z7teHwr2_TsqjSX33eoq6S_sg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> I suppose whether it can be part of `convolve()` depends on whether any
> additional options are needed. Unless a working heuristic is found the
> block size will have to be a parameter. In case convolution with a variant
> signal should be supported the hop size should be considered as well. In my
> experience you will want to use an iterative solution though when doing
> convolution with a variant signal.
> 
> On Sat, Jun 15, 2019 at 7:22 PM <3ukip0s02 at sneakemail.com> wrote:
> 
>> 
>> 
>> On Sat, Jun 15, 2019 at 12:01 PM Todd wrote:
>> 
>>> I am currently working an an implementation of overlap-add convolution
>>> [0].  I have a 1D implementation working and I am going to start expanding
>>> it to ND.  Overlap-add convolution can provide significant performance
>>> benefits over FFT-based convolution.  Does this make sense for scipy?
>>> 
>> 
>> Yeah!
>> 
>> 
>>> First is the name.  We have "convolve" and "fftconvolve" already.  A few
>>> options:
>>> 
>>> overall_add_conv
>>> oaconvolve
>>> oadconvolve
>>> 
>> 
>> fftconvolve has been folded into convolve and can either be selected
>> manually, or will automatically choose direct convolution, whichever is
>> likely faster.
>> 
>> Does it make sense for the OLA implementation to also be added as a
>> convolve() option, using the "method" parameter?
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190615/558742ad/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 3
> Date: Sun, 16 Jun 2019 12:01:28 +0200
> From: Ralf Gommers <ralf.gommers at gmail.com>
> To: SciPy Developers List <scipy-dev at python.org>
> Subject: Re: [SciPy-Dev] Request to add functionality to scipy.stats
> Message-ID:
> 	<CABL7CQhSrexZjkiwvD3VKWkO_R2ow383jyE1V9fdjqhZ6t9LeQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hi Sambit,
> 
> This sounds interesting, thanks for bringing it up. I hadn't heard of mgcpy
> before - it looks good.
> 
> 
> On Thu, Jun 6, 2019 at 6:24 PM Sambit Panda <spanda3 at jhu.edu> wrote:
> 
>> Request for project components' inclusion in scipy.stats
>> 
>> - Project name: "mgcpy"
>> - Authors: Satish Palaniappan (https://github.com/tpsatish95), Sambit
>> Panda (https://github.com/sampan501), Junhao Xiong (
>> https://github.com/junhaobearxiong), Ananya Swaminathan (
>> https://github.com/ananyas713), Sandhya Ramachandran (
>> https://github.com/sundaysundya), Richard Guo (https://github.com/rguo123)
>> - Current repository: https://github.com/neurodata/mgcpy
>> 
>> "mgcpy" is a Python package containing tools for independence testing and
>> k-sample testing. Looking through the "scipy.stats" module, The module
>> contains a host of independence and other hypothesis tests, but are limited
>> by assumptions of normality, linearity, unidimensionality, etc. While this
>> may be appropriate in a host of circumstances, it is increasingly important
>> to analyze nonlinear and high dimensional trends, which is where the
>> implementations in "mgcpy" could be very useful. Independence tests
>> included can operate on multidimensional and nonlinear data. In addition,
>> functionality has been extended to k-sample testing (with capabilities of
>> operating on the same kinds of data). The tests included can not only be
>> used for classification, but also for regression.
>> 
> 
> The papers on which your implementations are based do meet our criteria for
> inclusion in SciPy. At this point my main questions are about maintenance
> and dependencies:
> 1. Why do you want to move your whole package into SciPy? Do you plan to
> keep maintaining mcgpy? Would you maintain the code included in SciPy?
> 2. Your code currently depends on pandas, scikit-learn and seaborn. Those
> are not dependencies of SciPy, and we'd prefer not to add new dependencies.
> I haven't looked in more detail at this - would it be easy to remove those
> dependencies?
> 
> Cheers,
> Ralf
> 
> 
> 
>> Below is a list of some of the integrated tests contained within "mgcpy"
>> and citations for relevant papers about it.
>> - RV: P. Robert and Y. Escoufier, "A unifying tool for linear multivariate
>> statistical methods: the rv-coefficient," Journal of the Royal Statistical
>> Society: Series C (Applied Statistics), vol. 25, no. 3, pp. 257?265, 1976. 3
>> - CCA: D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical
>> correlation analysis: An overview with application to learning methods,"
>> Neural computation, vol. 16, no. 12, pp. 2639?2664, 2004.
>> - HHG: R. Heller, Y. Heller, and M. Gorfine, "A consistent multivariate
>> test of association based on ranks of distances," Biometrika, vol. 100, no.
>> 2, pp. 503?510, 2012.
>> - MDMR: N. J. Schork and M. A. Zapala, "Statistical properties of
>> multivariate distance matrix regression for high-dimensional data
>> analysis," Frontiers in Genetics, vol. 3, p. 190, 2012.
>> - Biased Dcorr, Unbiased Dcorr**: G. J. Sz?kely, M. L. Rizzo, N. K.
>> Bakirov et al., "Measuring and testing dependence by correlation of
>> distances," The Annals of Statistics, vol. 35, no. 6, pp. 2769?2794, 2007.
>> - Mantel: N. Mantel, "The detection of disease clustering and a
>> generalized regression approach," Cancer research, vol. 27, no. 2 Part 1,
>> pp. 209?220, 1967.
>> - MANOVA: Warne, R. T. (2014). "A primer on multivariate analysis of
>> variance (MANOVA) for behavioral scientists". Practical Assessment,
>> Research & Evaluation. 19 (17): 1?10.
>> - k-sample tests: Mart?nez-Camblor, P., & de U?a-?lvarez, J. (2009).
>> Non-parametric k-sample tests: Density functions vs distribution functions.
>> Computational Statistics & Data Analysis, 53(9), 3344-3357.
>> 
>> Not included tests, but related useful readings:
>> - Equivalency of Dcorr, HSIC, Energy, and MMD: C. Shen and J. T.
>> Vogelstein, "The exact equivalence of distance and kernel methods for
>> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018.
>> - Formulating k-sample tests as independence tests: C. Shen and J. T.
>> Vogelstein, "The exact equivalence of distance and kernel methods for
>> hypothesis testing," arXiv preprint arXiv:1806.05514, 2018.
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190616/de48bf44/attachment.html>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
> 
> 
> ------------------------------
> 
> End of SciPy-Dev Digest, Vol 188, Issue 9
> *****************************************
> 