[SciPy-Dev] scipy.stats

Tue Jun 1 04:43:54 EDT 2010

On Jun 1, 2010, at 3:12 AM, josef.pktd at gmail.com wrote:

> On Tue, Jun 1, 2010 at 12:54 AM, Travis Oliphant <oliphant at enthought.com> wrote:
>> 
>> On May 31, 2010, at 9:16 AM, josef.pktd at gmail.com wrote:
>> 
>>> Since Travis seems to want to take back control of scipy.stats, I am
>>> considering my role as inofficial maintainer as ended.
>> 
>> Obviously I've offended you.   That has never been my intent.   I apologize if my enthusiasm for getting some changes that I wanted to see into SciPy stepped on an area you felt ownership of.     I do not mind if people add changes to code that I've written and I assume that others feel the same.   That has always been the development mode of SciPy.   We clearly have different development styles.    I think we can find a way to work together.   I think the move to github will help.
>> 
>> I did not understand that you felt such ownership of scipy.stats.  I have certainly appreciated your input.
>> 
>> I do like a more "free-wheeling" style to code development than one that is bogged down with "rules" and "procedures".     This clearly is not your style.   For me, it comes down to time to spend.   I love working on SciPy and NumPy.    I don't have a lot of time to do it.   When I see quick changes I can make that add value I like to be able to do it.   I think we both want the same thing while we may disagree about the best way to get there.
>> In my mind, discussion doesn't end when a check-in is made --- it just begins.   You should never interpret my checking something in as the final word.   We clearly have a different view of "trunk"
>> 
>> I certainly don't want my approach to open source development to offend others or chase them away.  If I check in something you don't like, then tell me and let's talk about it.    If you need to vent and call me names, a private email to me or others can go a long way.
>> 
>> What do we need to do to keep you around?   Is there specifically something you didn't like about my recent check-ins?
>> 
>> In this case, the features added were not terribly extensive.   The current unit tests helped ferret out major problems.  Yes, I could write more tests and documentation, and you have been a model of writing tests and documentation.   I have been particularly impressed by the amount of quality documentation you have written.
>> 
>> While you seem to dismiss the episode as problematic, I actually think curve_fit was a good example of how something very positive can emerge quickly when people are open and willing to work together.
>> 
>> While formal, strict test-driven development is easy to point to for salvation -- it does have its costs.   I've always used informal test-driven development.   Just because I don't *always* add formal unit tests for every piece of code written does not mean the code that is currently in SciPy is un-tested and useless.   Such an approach leaves me open to criticism, which I acknowledge.  But, I think there have been far too many dismissive comments about the state of the code.
>> 
>> I would argue that the problem with scipy.stats does not lie mainly in distributions.py or the lack of test-driven-development --- but in the lack of certain easy to use features.    Quality code comes out of people who care --- not out of procedure.
>> 
>> I think you are someone who cares and your code reflects that.    We would all benefit from your staying part of the main development.
> 
> (not answering inline to keep thoughts together)
> 
> I think the main disagreements are about the quality control of the
> trunk and whether scipy development is a community effort or not.

I certainly think scipy development is a community effort.   I'm very sorry for making you feel "dumped" on.   That has never been my intent.  I was simply hoping to contribute a little where I could.   

> As Skipper described, in statsmodels almost all development occurs in
> the sandbox and in branches, and it is only included in the "official"
> core of statsmodels after it has been verified and tests have been
> added. sandbox code is everything from first draft version to almost
> finished code.
> And one of Skippers task in his gsoc is to clean out the sandbox.
> Once it is in trunk (core) any further refactoring follows very strict rules.

This has not been SciPy's process.   I can understand people may want it to become SciPy's process, but it has not been.  There are dangers of this process --- there is a reason that the mantra of "release early and release often".  It can also prevent progress when you are dealing with people's spare time because all of that process takes time and man-power and effort.   There is some value in it, I'm just not sure the extent of that value in contrast to other uses of that time.  

For example.  I would love to see statsmodels get more use.   I think there is much code there that is usable.  Yet, it remains outside of SciPy.   

If we agree to change the SciPy process will you agree to put statsmodels into SciPy? 

> 
> Specific to stats: I want a reference for any function where the
> explanation cannot be found with a Wikipedia search with one of the
> terms in the docstring. One or a few weeks ago, scipy.stats gained a
> new function, my asking on the mailing list what it is supposed to be,
> didn't receive any reply. (besides the problem that the function had
> the same name as an existing function).

I did not see your message.   I changed the name of the function and didn't know you were concerned about the addition.   It is a convenience function for bayes_mvs that returns the distribution objects from which the other numbers can be obtained instead of just the numbers.     The paper is already referenced in bayes_mvs.   

> Dumping new code into scipy trunk, without any review and tests,
> hoping that someone else looks for the problems is not an approach
> that I find acceptable.

That was never my "hope".  I planned to and have fixed all problems that I saw later and that others have pointed out.   You can never test for all possible failures.  

> 
> Asking me if I have commit rights, shows at least some disconnect from
> the development of scipy in the last three years, since I have been
> pretty (too) noisy about it on the mailing lists.

I know you have been noisy on the lists --- that's why I spoke to you about _logpdf and friends.  It also appears that you don't commit that often.   This is your process.   But, it made me wonder if permissions were an issue.    I was pretty sure you had been given commit rights, but I could not remember.   I'm sorry if that offended you.

-Travis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100601/05c4cb0b/attachment.html>