[Numpy-discussion] Standard Deviation (std): Suggested change for "ddof" default value

[josef.pktd at gmail.com]

On Thu, Apr 3, 2014 at 2:21 PM, Bago <mrbago at gmail.com> wrote:
>
>
>
>>
>> Sturla
>>
>> P.S. Personally I am not convinced "unbiased" is ever a valid argument, as
>> the biased estimator has smaller error. This is from experience in
>> marksmanship: I'd rather shoot a tight series with small systematic error
>> than scatter my bullets wildly but "unbiased" on the target. It is the
>> total error that counts. The series with smallest total error gets the
>> best
>> score. It is better to shoot two series and calibrate the sight in between
>> than use a calibration-free sight that don't allow us to aim. That's why I
>> think classical statistics got this one wrong. Unbiased is never a virtue,
>> but the smallest error is. Thus, if we are to repeat an experiment, we
>> should calibrate our estimator just like a marksman calibrates his sight.
>> But the aim should always be calibrated to give the smallest error, not an
>> unbiased scatter. Noone in their right mind would claim a shotgun is more
>> precise than a rifle because it has smaller bias. But that is what
>> applying
>> the Bessel correction implies.
>>
>
> I agree with the point, and what makes it even worse is that ddof=1 does not
> even produce an unbiased standard deviation estimate. I produces an unbiased
> variance estimate but the sqrt of this variance estimate is a biased
> standard deviation estimate,
> http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation.

But ddof=1 still produces a smaller bias than ddof=0

I think the main point in stats is that without ddof, the variance
will be too small and t-test or similar will be liberal in small
samples, or confidence intervals will be too short.
(for statisticians that prefer to have tests that maintain their level
and prefer to err on the "conservative" side.)

Josef


>
> Bago
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>