Re: [SciPy-dev] Scipy workflow (and not tools).
-----Original Message----- From: scipy-dev-bounces@scipy.org [mailto:scipy-dev-bounces@scipy.org] On Behalf Of scipy-dev-request@scipy.org Sent: 26 February 2009 10:36 To: scipy-dev@scipy.org Subject: Scipy-dev Digest, Vol 64, Issue 56
Message: 1 Date: Wed, 25 Feb 2009 23:21:40 -0800 From: Brian Granger <ellisonbg.net@gmail.com> Subject: Re: [SciPy-dev] Scipy workflow (and not tools). To: SciPy Developers List <scipy-dev@scipy.org> Message-ID: <6ce0ac130902252321j6139238by634364acd2bd07b2@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1
Neil,
Thanks for speaking up! I think there are *many* people in your situation, including myself - I too am mostly a silent watcher of SciPy and I would be much more likely to contribute if the things you list were a part of the Scipy development culture:
* Tests * Code review * Documentation * Good tools and workflow.
I think it is an unproven myth that these things are "barriers" for people who want to write code. In most cases that I have seen, these things *encourage* new people to contribute to a project and greatly improve the quality of the code being written by newbies and veteran's alike.
Cheers,
Brian
I'm another long-time lurker who's been thinking about contributing for a while but who's been a bit uncertain about where to start. These discussions, plus the git branches Pauli has created, are encouraging me greatly to roll up my sleeves and delve in. Cheers, Gareth. ------------------------------------------------------------- IMPORTANT NOTICE. This E-Mail and any files transmitted with it, are confidential and may be privileged and are for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of distribution, copying or use of this communication or the information in it, is strictly prohibited and may be unlawful. If you have received this E-Mail in error please return it to the sender. We should be grateful if you would also copy the communication to postmaster@doosanbabcock.com then delete the E-Mail and destroy any copies of it. It is your responsibility to scan any attachments for viruses. For further information, visit us at WWW.DOOSANBABCOCK.COM -------------------------------------------------------------
I'm another long-time lurker who's been thinking about contributing for a while but who's been a bit uncertain about where to start. These discussions, plus the git branches Pauli has created, are encouraging me greatly to roll up my sleeves and delve in.
Cheers, Gareth.
Hi I'm also mostly a lurker. Here's a story about the time I tried to contribute to scipy. Maybe a year ago I wanted a procedure to attempt to optimize some aspect of a multidimensional function. I found scipy.optimize, but the procedure that I tried failed by dividing by zero. Undaunted, I started digging in the scipy source code, finding the error in the broyden2 function. The problem was that the algorithm was finding the correct solution in fewer than the default number of iterations and was dividing by an error term that was zero. I sent an email to a scipy mailing list saying that this function needed some kind of check to see if it was done (error near zero) so that it could stop iterating so that it would not divide by zero. I got a reply saying that I needed to write a patch that included the fix to the function and a new test that used to fail but that now passes. So I simplified my failing code, I changed the broyden2 function in a way that I thought would fix the problem, and I sent the code to the mailing list. I got a reply saying that what they meant by a patch was an svn diff. So I installed svn and I checked out the scipy code with the idea that I could change my local copy, test my changes, and send the diff to the mailing list. After spending some time trying to get this working, I stopped for the following reasons: 1) Binary format and path problems were causing me grief. 2) I already had a scipy that worked, and I didn't want to break this. 3) Neil's comment: """ If I want to make a change, I have to check out the trunk and then develop my change *completely without the benefit of version control*. I am not allowed to make any intermediate commits while I learn my way through the coding process. I must submit a fully formed patch without ever being able to checkpoint my own progress. This is basically a deal-breaker for me. """ I think I still use code that works around this bug by using the iteration keyword argument to specify a fewer-than-default number of iterations so that the error never reaches zero. Alex
On Thu, Feb 26, 2009 at 5:55 AM, Alex Griffing <argriffi@ncsu.edu> wrote:
I'm another long-time lurker who's been thinking about contributing for a while but who's been a bit uncertain about where to start. These discussions, plus the git branches Pauli has created, are encouraging me greatly to roll up my sleeves and delve in.
Cheers, Gareth.
Hi I'm also mostly a lurker. Here's a story about the time I tried to contribute to scipy.
Maybe a year ago I wanted a procedure to attempt to optimize some aspect of a multidimensional function. I found scipy.optimize, but the procedure that I tried failed by dividing by zero. Undaunted, I started digging in the scipy source code, finding the error in the broyden2 function. The problem was that the algorithm was finding the correct solution in fewer than the default number of iterations and was dividing by an error term that was zero.
I sent an email to a scipy mailing list saying that this function needed some kind of check to see if it was done (error near zero) so that it could stop iterating so that it would not divide by zero. I got a reply saying that I needed to write a patch that included the fix to the function and a new test that used to fail but that now passes. So I simplified my failing code, I changed the broyden2 function in a way that I thought would fix the problem, and I sent the code to the mailing list. I got a reply saying that what they meant by a patch was an svn diff.
So I installed svn and I checked out the scipy code with the idea that I could change my local copy, test my changes, and send the diff to the mailing list. After spending some time trying to get this working, I stopped for the following reasons:
1) Binary format and path problems were causing me grief. 2) I already had a scipy that worked, and I didn't want to break this. 3) Neil's comment: """ If I want to make a change, I have to check out the trunk and then develop my change *completely without the benefit of version control*. I am not allowed to make any intermediate commits while I learn my way through the coding process. I must submit a fully formed patch without ever being able to checkpoint my own progress. This is basically a deal-breaker for me. """
Hi Alex, it was my who was trying to help you with this. The problem was, that people who have svn access to scipy (e.g. I don't) are very busy, and do you must submit your patches in a way so that it is easy for them to apply it. I agree that git is much easier to use for that, but imho it's not difficult to create a patch with svn as well, especially if you have a working scipy on your computer. Just checkout the svn, copy your working scipy files over it, do "svn di" and send us the patch.
I think I still use code that works around this bug by using the iteration keyword argument to specify a fewer-than-default number of iterations so that the error never reaches zero.
If you find time, please do send us the patch with the fix+tests, it will be applied. Thanks a lot, Ondrej
On Thu, Feb 26, 2009 at 9:31 AM, Ondrej Certik <ondrej@certik.cz> wrote:
On Thu, Feb 26, 2009 at 5:55 AM, Alex Griffing <argriffi@ncsu.edu> wrote:
I'm another long-time lurker who's been thinking about contributing for a while but who's been a bit uncertain about where to start. These discussions, plus the git branches Pauli has created, are encouraging me greatly to roll up my sleeves and delve in.
Cheers, Gareth.
Hi I'm also mostly a lurker. Here's a story about the time I tried to contribute to scipy.
Maybe a year ago I wanted a procedure to attempt to optimize some aspect of a multidimensional function. I found scipy.optimize, but the procedure that I tried failed by dividing by zero. Undaunted, I started digging in the scipy source code, finding the error in the broyden2 function. The problem was that the algorithm was finding the correct solution in fewer than the default number of iterations and was dividing by an error term that was zero.
I sent an email to a scipy mailing list saying that this function needed some kind of check to see if it was done (error near zero) so that it could stop iterating so that it would not divide by zero. I got a reply saying that I needed to write a patch that included the fix to the function and a new test that used to fail but that now passes. So I simplified my failing code, I changed the broyden2 function in a way that I thought would fix the problem, and I sent the code to the mailing list. I got a reply saying that what they meant by a patch was an svn
diff.
So I installed svn and I checked out the scipy code with the idea that I could change my local copy, test my changes, and send the diff to the mailing list. After spending some time trying to get this working, I stopped for the following reasons:
1) Binary format and path problems were causing me grief. 2) I already had a scipy that worked, and I didn't want to break this. 3) Neil's comment: """ If I want to make a change, I have to check out the trunk and then develop my change *completely without the benefit of version control*. I am not allowed to make any intermediate commits while I learn my way through the coding process. I must submit a fully formed patch without ever being able to checkpoint my own progress. This is basically a deal-breaker for me. """
Hi Alex,
it was my who was trying to help you with this. The problem was, that people who have svn access to scipy (e.g. I don't) are very busy, and do you must submit your patches in a way so that it is easy for them to apply it. I agree that git is much easier to use for that, but imho it's not difficult to create a patch with svn as well, especially if you have a working scipy on your computer.
Do you want SVN access? Chuck
On Thu, Feb 26, 2009 at 8:35 AM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Thu, Feb 26, 2009 at 9:31 AM, Ondrej Certik <ondrej@certik.cz> wrote:
On Thu, Feb 26, 2009 at 5:55 AM, Alex Griffing <argriffi@ncsu.edu> wrote:
I'm another long-time lurker who's been thinking about contributing for a while but who's been a bit uncertain about where to start. These discussions, plus the git branches Pauli has created, are encouraging me greatly to roll up my sleeves and delve in.
Cheers, Gareth.
Hi I'm also mostly a lurker. Here's a story about the time I tried to contribute to scipy.
Maybe a year ago I wanted a procedure to attempt to optimize some aspect of a multidimensional function. I found scipy.optimize, but the procedure that I tried failed by dividing by zero. Undaunted, I started digging in the scipy source code, finding the error in the broyden2 function. The problem was that the algorithm was finding the correct solution in fewer than the default number of iterations and was dividing by an error term that was zero.
I sent an email to a scipy mailing list saying that this function needed some kind of check to see if it was done (error near zero) so that it could stop iterating so that it would not divide by zero. I got a reply saying that I needed to write a patch that included the fix to the function and a new test that used to fail but that now passes. So I simplified my failing code, I changed the broyden2 function in a way that I thought would fix the problem, and I sent the code to the mailing list. I got a reply saying that what they meant by a patch was an svn diff.
So I installed svn and I checked out the scipy code with the idea that I could change my local copy, test my changes, and send the diff to the mailing list. After spending some time trying to get this working, I stopped for the following reasons:
1) Binary format and path problems were causing me grief. 2) I already had a scipy that worked, and I didn't want to break this. 3) Neil's comment: """ If I want to make a change, I have to check out the trunk and then develop my change *completely without the benefit of version control*. I am not allowed to make any intermediate commits while I learn my way through the coding process. I must submit a fully formed patch without ever being able to checkpoint my own progress. This is basically a deal-breaker for me. """
Hi Alex,
it was my who was trying to help you with this. The problem was, that people who have svn access to scipy (e.g. I don't) are very busy, and do you must submit your patches in a way so that it is easy for them to apply it. I agree that git is much easier to use for that, but imho it's not difficult to create a patch with svn as well, especially if you have a working scipy on your computer.
Do you want SVN access?
No: I had it in the past, but I forgot how to login, or maybe it stopped working, I don't know. In any case I much prefer to send patches that are properly reviewed and then applied by someone who is working with the scipy tree on a daily basis (as opposed to me who only occasionally sends a patch). And Stefan did a great job with this. So this is a hint to Alex ---- just send a nice patch that works and Stefan will apply it. The only thing that doesn't work with this scenario is that my name will not appear on the nice plot that Travis was showing at euroscipy (and I think at scipy2008 too). But once we move to git, all the names of the contributors will be preserved. Then I don't see any reason why people like me should have push access --- I just publish my branch at github, Stefan pulls it and pushes it in. Ondrej
Ondrej Certik wrote:
[...]
Just checkout the svn, copy your working scipy files over it, do "svn di" and send us the patch.
Here is a bug-exposing patch that adds three tests to test_nonlin.py. One of them currently fails, and the fact that the other two pass helps to show the nature of the problem (if you use a small enough number of iterations or start your guess far enough from the true answer, then the bug is not triggered). This is probably fixed by Pauli Virtanen's work, but I haven't checked this. Alex Index: scipy/optimize/tests/test_nonlin.py =================================================================== --- scipy/optimize/tests/test_nonlin.py (revision 5597) +++ scipy/optimize/tests/test_nonlin.py (working copy) @@ -3,12 +3,13 @@ May 2007 """ +import numpy + from numpy.testing import * from scipy.optimize import nonlin from numpy import matrix, diag - def F(x): def p3(y): return float(y.T*y)*y @@ -24,6 +25,40 @@ return tuple(f.flat) +def G(x): + return [v**3 - 1 for v in x] + +class TestNonlinEasy(TestCase): + """ Test case for a stupidly easy function optimization problem. + """ + def broyden_helper(self, initial_guess, expected_result, iterations=None): + if iterations: + x = nonlin.broyden2(G, initial_guess, iter=iterations) + else: + x = nonlin.broyden2(G, initial_guess) + x_array = numpy.array(x) + xout_array = numpy.array(expected_result) + eps = 1e-9 + errmsg = 'got %s but expected %s' % (str(x_array), str(xout_array)) + difference = numpy.array(x_array - xout_array) + assert nonlin.norm(difference) < eps, errmsg + assert nonlin.norm(G(x)) < eps + + def test_broyden2_near_default_iterations(self): + initial_guess = [1.1, 1.1, 1.1] + expected_result = [1, 1, 1] + self.broyden_helper(initial_guess, expected_result) + + def test_broyden2_near_fewer_iterations(self): + initial_guess = [1.1, 1.1, 1.1] + expected_result = [1, 1, 1] + self.broyden_helper(initial_guess, expected_result, 8) + + def test_broyden2_far(self): + initial_guess = [2, 2, 2] + expected_result = [1, 1, 1] + self.broyden_helper(initial_guess, expected_result) + class TestNonlin(TestCase): """ Test case for a simple constrained entropy maximization problem (the machine translation example of Berger et al in
participants (5)
-
alex -
Alex Griffing
-
Charles R Harris -
Elston, Gareth R -
Ondrej Certik