[SciPy-User] scipy.stats one-sided two-sided less, greater, signed ?

Wed Jun 8 06:56:42 EDT 2011

On Tue, Jun 7, 2011 at 10:37 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> On Tue, Jun 7, 2011 at 4:40 PM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>>
>>
>> On Mon, Jun 6, 2011 at 9:34 PM, <josef.pktd at gmail.com> wrote:
>>>
>>> On Mon, Jun 6, 2011 at 2:34 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>>> > On 06/05/2011 02:43 PM, josef.pktd at gmail.com wrote:
>>> >> What should be the policy on one-sided versus two-sided?
>>> > Yes :-)
>>> >
>>> >> The main reason right now for looking at this is
>>> >> http://projects.scipy.org/scipy/ticket/1394 which specifies a
>>> >> "one-sided" alternative and provides both lower and upper tail.
>>> > That refers to the Fisher's test rather than the more 'traditional'
>>> > one-sided tests. Each value of the Fisher's test has special meanings
>>> > about the value or probability of the 'first cell' under the null
>>> > hypothesis.  So it is necessary to provide those three values.
>>> >
>>> >> I would prefer that we follow the alternative patterns similar to R
>>> >>
>>> >> currently only kstest has    alternative : 'two_sided' (default),
>>> >> 'less' or 'greater'
>>> >> but this should be added to other tests where it makes sense
>>> > I think that these Kolmogorov-Smirnov  tests are not the traditional
>>> > meaning either. It is a little mind-boggling to try to think about cdfs!
>>> >
>>> >> R fisher.exact
>>> >> """alternative        indicates the alternative hypothesis and must be
>>> >> one
>>> >> of "two.sided", "greater" or "less". You can specify just the initial
>>> >> letter. Only used in the 2 by 2 case."""
>>> >>
>>> >> mannwhitneyu reports a one-sided test without actually specifying
>>> >> which alternative is used  (I thought I remembered other cases like
>>> >> this but don't find any right now)
>>> >>
>>> >> related:
>>> >> in many cases in the two-sided tests the test statistic has a sign
>>> >> that indicates in which tail the test-statistic falls.
>>> >> This is useful in ttests for example, because the one-sided tests can
>>> >> be backed out from the two-sided tests. (With symmetric distributions
>>> >> one-sided p-value is just half of the two-sided pvalue)
>>> >>
>>> >> In the discussion of https://github.com/scipy/scipy/pull/8  I argued
>>> >> that this might mislead users to interpret a two-sided result as a
>>> >> one-sided result. However, I doubt now that this is a strong argument
>>> >> against not reporting the signed test statistic.
>>> > (I do not follow pull requests so is there a relevant ticket?)
>>> >
>>> >> After going through scipy.stats.stats, it looks like we always report
>>> >> the signed test statistic.
>>> >>
>>> >> The test statistic in ks_2samp is in all cases defined as a max value
>>> >> and doesn't have a sign in R either, so adding a sign there would
>>> >> break with the standard definition.
>>> >> one-sided option for ks_2samp would just require to find the
>>> >> distribution of the test statistics D+, D-
>>> >>
>>> >> ---
>>> >>
>>> >> So my proposal for the general pattern (with exceptions for special
>>> >> reasons) would be
>>> >>
>>> >> * add/offer alternative : 'two_sided' (default), 'less' or 'greater'
>>> >> http://projects.scipy.org/scipy/ticket/1394  for now,
>>> >> and adjustments of existing tests in the future (adding the option can
>>> >> be mostly done in a backwards compatible way and for symmetric
>>> >> distributions like ttest it's just a convenience)
>>> >> mannwhitneyu seems to be the only "weird" one
>>
>> This would actually make the fisher_exact implementation more consistent,
>> since only one p-value is returned in all cases. I just don't like the R
>> naming much; alternative="greater" does not convey to me that this is a
>> one-sided test using the upper tail. How about:
>>     test : {"two-tailed", "lower-tail", "upper-tail"}
>> with two-tailed the default?

I think matlab uses (in general) larger and smaller, the advantage of
less/smaller and greater/larger is that it directly refers to the
alternative hypothesis, while the meaning in terms of tails is not
always clear (in kstest and I guess some others the test statistics is
just reversed and uses the same tail in both cases)

so greater smaller is mostly "future proof" across tests, while
reference to the tail can only be used where this is an unambiguous
statement. but see below

>>
>> Ralf
>>
>>
>>>
>>> >>
>>> >> * report signed test statistic for two-sided alternative (when a
>>> >> signed test statistic exists):  which is the status quo in
>>> >> stats.stats, but I didn't know that this is actually pretty consistent
>>> >> across tests.
>>> >>
>>> >> Opinions ?
>>> >>
>>> >> Josef
>>> >> _______________________________________________
>>> >> SciPy-User mailing list
>>> >> SciPy-User at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> > I think that there is some valid misunderstanding here (as I was in the
>>> > same situation) regarding what is meant here. My understanding is that
>>> > under a one-sided hypothesis, all the values of the null hypothesis only
>>> > exist in one tail of the test distribution. In contrast the values of
>>> > null distribution exist in both tails with a two-sided hypothesis. Yet
>>> > that interpretation does not have the same meaning as the tails in the
>>> > Fisher or Kolmogorov-Smirnov tests.
>>>
>>> The tests have a clear Null Hypothesis (equality) and Alternative
>>> Hypothesis (not equal or directional, less or greater).
>>> So the "alternative" should be clearly specified in the function
>>> argument, as in R.
>>>
>>> Whether this corresponds to left and right tails of the distribution
>>> is an "implementation detail" which holds for ttests but not for
>>> kstest/ks_2samp.
>>>
>>> kstest/ks2sample   H0: cdf1 == cdf2  and H1:  cdf1 != cdf2 or H1:
>>> cdf1 < cdf2 or H1:  cdf1 > cdf2
>>> (looks similar to comparing two survival curves in Kaplan-Meier ?)
>>>
>>> fisher_exact (2 by 2)  H0: odds-ratio == 1 and H1: odds-ratio != 1 or
>>> H1: odds-ratio < 1 or H1: odds-ratio > 1
>>>
>>> I know the kolmogorov-smirnov tests, but for fisher exact and
>>> contingency tables I rely on R
>>>
>>> from R-help:
>>> For 2 by 2 tables, the null of conditional independence is equivalent
>>> to the hypothesis that the odds ratio equals one. <...> The
>>> alternative for a one-sided test is based on the odds ratio, so
>>> alternative = "greater" is a test of the odds ratio being bigger than
>>> or.
>>> Two-sided tests are based on the probabilities of the tables, and take
>>> as ‘more extreme’ all tables with probabilities less than or equal to
>>> that of the observed table, the p-value being the sum of such
>>> probabilities.
>>>
>>> Josef
>>>
>>>
>>> >
>>> > I never paid much attention to the frequency based tests but it does not
>>> > surprise if there are no one-sided tests. Most are rank-based so it is
>>> > rather hard to do in a simply manner - actually I am not even sure how
>>> > to use a permutation test.
>>> >
>>> > Bruce
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > SciPy-User mailing list
>>> > SciPy-User at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>>> >
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> But that is NOT the correct interpretation  here!
> I tried to explain to you that this is the not the usual idea
> one-sided vs two-sided tests.
> For example:
> http://www.msu.edu/~fuw/teaching/Fu_ch10_2_categorical.ppt
> "The test holds the marginal totals fixed and computes the
> hypergeometric probability that n11 is at least as large as the
> observed value"

this still sounds like a less/greater test to me

> "The output consists of three p-values:
> Left: Use this when the alternative to independence is that there is
> negative association between the variables.  That is, the observations
> tend to lie in lower left and upper right.
> Right: Use this when the alternative to independence is that there is
> positive association between the variables. That is, the observations
> tend to lie in upper left and lower right.
> 2-Tail: Use this when there is no prior alternative.
> "
> There is also the book "Categorical data analysis: using the SAS
> system  By Maura E. Stokes, Charles S. Davis, Gary G. Koch" that came
> up via Google that also refers to the n11 cell.
>
> http://www.langsrud.com/fisher.htm

I was trying to read the Agresti paper referenced there but it has too
much detail to get through in 15 minutes :)

> "The output consists of three p-values:
>
>    Left: Use this when the alternative to independence is that there
> is negative association between the variables.
>    That is, the observations tend to lie in lower left and upper right.
>    Right: Use this when the alternative to independence is that there
> is positive association between the variables.
>    That is, the observations tend to lie in upper left and lower right.
>    2-Tail: Use this when there is no prior alternative.
>
> NOTE: Decide to use Left, Right or 2-Tail before collecting (or
> looking at) the data."
>
> But you will get a different p-value if you switch rows and columns
> because of the dependence on the n11 cell. If you do that then the
> p-values switch between left and right sides as these now refer to
> different hypotheses regarding that first cell.

switching row and columns doesn't change the p-value in R
reversing columns changes the definition of less and greater, reverses them

The problem with 2 by 2 contingency tables with given marginals, i.e.
row and column totals, is that we only have one free entry. Any test
on one entry, e.g. element 0,0, pins down all the other ones and
(many) tests then become equivalent.

http://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_a0000000658.htm
some math got lost
"""
For <2 by 2> tables, one-sided -values for Fisher’s exact test are
defined in terms of the frequency of the cell in the first row and
first column of the table, the (1,1) cell. Denoting the observed (1,1)
cell frequency by , the left-sided -value for Fisher’s exact test is
the probability that the (1,1) cell frequency is less than or equal to
. For the left-sided -value, the set includes those tables with a
(1,1) cell frequency less than or equal to . A small left-sided -value
supports the alternative hypothesis that the probability of an
observation being in the first cell is actually less than expected
under the null hypothesis of independent row and column variables.

Similarly, for a right-sided alternative hypothesis, is the set of
tables where the frequency of the (1,1) cell is greater than or equal
to that in the observed table. A small right-sided -value supports the
alternative that the probability of the first cell is actually greater
than that expected under the null hypothesis.

Because the (1,1) cell frequency completely determines the table when
the marginal row and column sums are fixed, these one-sided
alternatives can be stated equivalently in terms of other cell
probabilities or ratios of cell probabilities. The left-sided
alternative is equivalent to an odds ratio less than 1, where the odds
ratio equals (). Additionally, the left-sided alternative is
equivalent to the column 1 risk for row 1 being less than the column 1
risk for row 2, . Similarly, the right-sided alternative is equivalent
to the column 1 risk for row 1 being greater than the column 1 risk
for row 2, . See Agresti (2007) for details.
R C Tables
"""

I'm not a user of Fisher's exact test (and I have a hard time keeping
the different statements straight), so if left/right or lower/upper
makes more sense to users, then I don't complain.

To me they are all just independence tests with possible one-sided
alternatives that one distribution dominates the other. (with the same
pattern as ks_2samp or ttest_2samp)

Josef

>
>
> Bruce
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>