[SciPy-Dev] scipy.stats

[Skipper Seabold]

On Mon, May 31, 2010 at 1:43 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Mon, May 31, 2010 at 8:32 AM, Skipper Seabold <jsseabold at gmail.com>
> wrote:
>>
>> On Mon, May 31, 2010 at 10:38 AM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> > On Mon, May 31, 2010 at 8:23 AM, Charles R Harris
>> > <charlesr.harris at gmail.com> wrote:
>> >>
>> >>
>> >> On Mon, May 31, 2010 at 8:16 AM, <josef.pktd at gmail.com> wrote:
>> >>>
>> >>> Since Travis seems to want to take back control of scipy.stats, I am
>> >>> considering my role as inofficial maintainer as ended.
>> >>>
>> >>> I would have appreciated his help almost 3 years ago, when I started
>> >>> to learn numpy, scipy, and started to submit patches for
>> >>> scipy.stats.distributions.
>> >>>
>> >>> But by now, I have pretty strong opinions about statistics in python,
>> >>> after almost  three years, I'm a bit tired of cleaning up the mess of
>> >>> others (and want to clean up my own mess), and there are obviously big
>> >>> philosophical differences for the development process between me and
>> >>> Travis (no discussion, no review, no tests).
>> >>> http://projects.scipy.org/scipy/log/trunk/scipy/stats/tests
>> >>>
>> >>> Watching the scipy changelog and checking any function that Travis
>> >>> quietly commits is no fun (see mailing list for the introduction of
>> >>> curve_fit or ask Stefan).
>> >>>
>> >>> I said early on that I would like to trust the results that
>> >>> scipy.stats produces (although I don't find the mailing list thread
>> >>> any more).
>> >>>
>> >>> I considered scipy to go into a stable direction like Python is,
>> >>> kitchen sink for scientific programming, which might be slow-moving
>> >>> but with high standards, and not a sandbox.
>> >>>
>> >>> Details are at
>> >>> http://mail.scipy.org/pipermail/scipy-dev/2010-April/014058.html
>> >>>
>> >>> After my initial scipy.stats.distributions cleanup, test coverage was
>> >>> at 91%, I have no idea where it is after this weekend.
>> >>>
>> >>> This is more about the process then the content, distributions was
>> >>> Travis's baby (although unfinished), and most of his changes are very
>> >>> good, but I don't want to look for the 5-10% (?) typos anymore.
>> >>>
>> >>
>> >> Ah Josef, there are easier ways to lodge complaints than resignation ;)
>> >> I
>> >> agree that it was rude of Travis to make those changes without running
>> >> them
>> >> through the list, and he does tend to toss stuff in that others have to
>> >> clean up, the same with c-code. But maybe we can manage to get him
>> >> housebroken without all moving out.
>> >>
>> >
>> > I think a policy of mandatory review will solve these sorts of problems,
>> > and
>> > that is probably a good argument for moving to github where review is
>> > much
>> > easier. On stats, we probably need an additional policy of rigorous
>> > testing
>> > to make sure that things are working right, the stat tests are more
>> > difficult by their very nature. I think Travis is amenable to such
>> > processes, but we do need to start a discussion. If you do feel strongly
>> > about the recent changes maybe they can be reverted and added back in
>> > after
>> > review.
>> >
>>
>> I am perhaps wading out of my depth here, but I agree with the
>> concerns and having the proposed dialogue, as I think having Josef's
>> input on the direction of scipy.stats is important.
>>
>> This does dovetail with the move to DVCS/github and having a review
>> and discussion policy in place before things go into trunk.  I don't
>> recall there being a time frame set up for the move (?) though there
>> was little dissent in actually making the move.  Perhaps we could
>> start hashing out concrete plans for review and a renewed policy for
>> testing standards so that the discussions can focus more on design and
>> as little energy as possible is spent uncovering precision errors,
>> typos, and niggling bugs.  Does it make sense to do this before the
>> move maybe as part of the docs marathon?  Of course there were also
>> those in favor of shoot first, sort it out as we go along because this
>> is a problem that has been solved before.
>>
>> Re:testing, the things that go into stats must be as test driven as
>> possible
>
> As in Test Driven Development?  Hear hear!  This would force (stats)
> developers to think first about developing tests to verify the correctness
> of the more esoteric aspects of the desired results of the algorithm they're
> working on, and would make it much less likely that code would be submitted
> w/out tests (if you had to write the tests first, why are you submitting
> code w/out tests?)  Obviously, this would need to be "strongly advised," as
> we have no way of enforcing how people actually write code, but we certainly
> could enforce that code w/out tests (and/or without Standards compliant
> docstrings) won't even be reviewed.
>

TDD is one of the habits that I try to follow working on statsmodels,
and I thought it was numpy/scipy policy.  The bottom line is that
writing tests is not very much fun and it's time consuming
(*especially* when you didn't write the code in the first place), but
the heavily outweighing benefits are that it helps with code reviews
for new code and makes for a more mature and stable project in the
long run.

The way that I try to work on statsmodels forces me to write
docstrings with examples and tests and I think it helps in eliciting
code reviews.

It is not strict TDD, but I make changes first in a branch, document,
and write some examples with my output versus expected results from a
statistical package (R, SAS, Stata, etc.).  I view these as pseudo
tests that become docstring examples, tutorials, and tests.  This
helps in eliciting a code review because you don't need all the
details before running the code.  You can work backwards from
input/results and then get into implementation details and discuss API
design if necessary.  The merge to trunk doesn't happen until tests
are written.  In this way it becomes "official" and is ensured to stay
correct as optimizations are made later.

This is probably far from best practice (strict TDD should probably
continue to be policy for numpy/scipy, especially incremental
improvements) and may be more idiosyncratic than anything, but I find
this to work for me.

Skipper