Hi, I've committed the following change to test_print.py to fix one of the tests. Index: test_print.py =================================================================== --- test_print.py (revision 6302) +++ test_print.py (working copy) @@ -154,7 +154,7 @@ else: locale.setlocale(locale.LC_NUMERIC, 'FRENCH') - assert_equal(str(tp(1.2)), str(float(1.2)), + assert_equal(locale.format("%f",tp(1.2)), locale.format("%f",float(1.2)), err_msg='Failed locale test for type %s' % tp) finally: locale.setlocale(locale.LC_NUMERIC, locale=curloc) Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Hi Christopher,
Please do not modify those tests - they are supposed to fail,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hi David, Sorry. Should these tests be generating a "known failures" then? ====================================================================== FAIL: test_print.test_locale_single ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 165, in test_locale_single return _test_locale_independance(np.float32) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 158, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float32'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_double ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 170, in test_locale_double return _test_locale_independance(np.double) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 158, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float64'> ACTUAL: '1,2' DESIRED: '1.2' ====================================================================== FAIL: test_print.test_locale_longdouble ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/chanley/dev/site-packages/lib/python/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/decorators.py", line 82, in skipper return f(*args, **kwargs) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 175, in test_locale_longdouble return _test_locale_independance(np.longdouble) File "/Users/chanley/dev/site-packages/lib/python/numpy/core/tests/test_print.py", line 158, in _test_locale_independance err_msg='Failed locale test for type %s' % tp) File "/Users/chanley/dev/site-packages/lib/python/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: Failed locale test for type <type 'numpy.float128'> ACTUAL: '1,2' DESIRED: '1.2' Thanks, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338
On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley@stsci.edu> wrote:
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Hi Christopher,
Please do not modify those tests - they are supposed to fail,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hi David,
Sorry. Should these tests be generating a "known failures" then?
No. The problem are known, and are being fixed (in a branch). Since the problem is only in the development trunk, I don't see any problem with having failures for some time, David
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley@stsci.edu> wrote:
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Hi Christopher,
Please do not modify those tests - they are supposed to fail,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Hi David,
Sorry. Should these tests be generating a "known failures" then?
No. The problem are known, and are being fixed (in a branch). Since the problem is only in the development trunk, I don't see any problem with having failures for some time,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
I would disagree. If you were to attempt the following: n = numpy.test() n.wasSuccessful() You expect the result to be 'True'. If not it is necessary to find out why. Right now the following occurs:
n.wasSuccessful() False
I have no way of knowing that you wanted those tests to fail unless you have them marked as KNOWNFAIL. Since we use numpy in our production systems I need to determine why numpy is failing. We track the changes on the trunk because we need to know how changes will effect our code prior to our customers downloading the latest numpy release. This 'False' return value from wasSuccessful() means that our automated systems tell us that numpy is broken. The common assumption is that tests are not suppose to fail. If they are failing then there is a problem. If you have tests that you want to fail, either make the failure a condition of the test passing or move the tests to the branch where you are doing your development. Thanks, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338
On Fri, Jan 9, 2009 at 5:11 AM, Christopher Hanley <chanley@stsci.edu> wrote:
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley@stsci.edu> wrote:
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Hi Christopher,
Please do not modify those tests - they are supposed to fail,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Hi David,
Sorry. Should these tests be generating a "known failures" then?
No. The problem are known, and are being fixed (in a branch). Since the problem is only in the development trunk, I don't see any problem with having failures for some time,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
I would disagree. If you were to attempt the following:
n = numpy.test() n.wasSuccessful()
You expect the result to be 'True'. If not it is necessary to find out why. Right now the following occurs:
n.wasSuccessful() False
I have no way of knowing that you wanted those tests to fail unless you have them marked as KNOWNFAIL. Since we use numpy in our production systems I need to determine why numpy is failing. We track the changes on the trunk because we need to know how changes will effect our code prior to our customers downloading the latest numpy release.
I don't understand: you can't expect the trunk to always work. We try not to break it - but sometimes it does not work. Personally, I don't like knownfailure much anyway: I feel like it is too easy to tag one test known failure, and then nobody cares about it anymore. Those formatting problems were already problems before - the tests only show the problem, it does not cause the problem, so I don't understand why it is so important: a 100 % running test suite with a problem which is not shown or a 95 % running test suite with the problem is the same thing; the code in numpy itself is exactly the same. David
On Thu, Jan 8, 2009 at 15:26, David Cournapeau <cournape@gmail.com> wrote:
Personally, I don't like knownfailure much anyway: I feel like it is too easy to tag one test known failure, and then nobody cares about it anymore. Those formatting problems were already problems before - the tests only show the problem, it does not cause the problem, so I don't understand why it is so important: a 100 % running test suite with a problem which is not shown or a 95 % running test suite with the problem is the same thing; the code in numpy itself is exactly the same.
Don't check in failing tests without using knownfailure. Unit tests are used by others to determine whether or not *they* broke things or whether their installation failed. By checking in a failing test, you are sending others on a wild goose chase trying to figure out what they did wrong when they didn't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 5:11 AM, Christopher Hanley <chanley@stsci.edu> wrote:
David Cournapeau wrote:
David Cournapeau wrote:
On Fri, Jan 9, 2009 at 1:37 AM, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Hi Christopher,
Please do not modify those tests - they are supposed to fail,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Hi David,
Sorry. Should these tests be generating a "known failures" then? No. The problem are known, and are being fixed (in a branch). Since
On Fri, Jan 9, 2009 at 4:29 AM, Christopher Hanley <chanley@stsci.edu> wrote: the problem is only in the development trunk, I don't see any problem with having failures for some time,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion I would disagree. If you were to attempt the following:
n = numpy.test() n.wasSuccessful()
You expect the result to be 'True'. If not it is necessary to find out why. Right now the following occurs:
n.wasSuccessful() False
I have no way of knowing that you wanted those tests to fail unless you have them marked as KNOWNFAIL. Since we use numpy in our production systems I need to determine why numpy is failing. We track the changes on the trunk because we need to know how changes will effect our code prior to our customers downloading the latest numpy release.
I don't understand: you can't expect the trunk to always work. We try not to break it - but sometimes it does not work.
Personally, I don't like knownfailure much anyway: I feel like it is too easy to tag one test known failure, and then nobody cares about it anymore. Those formatting problems were already problems before - the tests only show the problem, it does not cause the problem, so I don't understand why it is so important: a 100 % running test suite with a problem which is not shown or a 95 % running test suite with the problem is the same thing; the code in numpy itself is exactly the same.
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
I do not expect the trunk to always work. I even expect it to have bugs. However, I do not expect there to be test failures for known reasons that result in wasSuccessful() returning false. This is a bad programming practice. It creates work for people trying to figure out what is wrong when the answer is already know. Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338
On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley@stsci.edu> wrote:
I do not expect the trunk to always work. I even expect it to have bugs. However, I do not expect there to be test failures for known reasons that result in wasSuccessful() returning false. This is a bad programming practice. It creates work for people trying to figure out what is wrong when the answer is already know.
Well, I don't agree it is bad practice: it is not ideal, yes, but I don't think using KnownFailure is much better. My rationale being that known failures are almost never worked on because it does not bug anyone anymore, and it is very easy to forget about them - AFAICS, most numpy/scipy known failures have never been worked on after being tagged as such. I don't think we have a good system for those cases, be it known failure - or just failing. I will tag them as known failure, since I am the only one against it, though :) David
On Fri, Jan 9, 2009 at 2:21 PM, David Cournapeau <cournape@gmail.com> wrote:
On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley@stsci.edu> wrote:
I do not expect the trunk to always work. I even expect it to have bugs. However, I do not expect there to be test failures for known reasons that result in wasSuccessful() returning false. This is a bad programming practice. It creates work for people trying to figure out what is wrong when the answer is already know.
Well, I don't agree it is bad practice: it is not ideal, yes, but I don't think using KnownFailure is much better. My rationale being that known failures are almost never worked on because it does not bug anyone anymore, and it is very easy to forget about them - AFAICS, most numpy/scipy known failures have never been worked on after being tagged as such. I don't think we have a good system for those cases, be it known failure - or just failing.
I will tag them as known failure, since I am the only one against it, though :)
Done in r6308 - please tell me if something still does not work as expected, David
2009/1/9 David Cournapeau <cournape@gmail.com>:
On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley@stsci.edu> wrote:
I do not expect the trunk to always work. I even expect it to have bugs. However, I do not expect there to be test failures for known reasons that result in wasSuccessful() returning false. This is a bad programming practice. It creates work for people trying to figure out what is wrong when the answer is already know.
Well, I don't agree it is bad practice: it is not ideal, yes, but I don't think using KnownFailure is much better. My rationale being that known failures are almost never worked on because it does not bug anyone anymore, and it is very easy to forget about them - AFAICS, most numpy/scipy known failures have never been worked on after being tagged as such. I don't think we have a good system for those cases, be it known failure - or just failing.
I agree with you point of view, but I also have sympathy for Cristopher's situation. I thought a solution to both problems would be if we could find an easy way of executing all skipped tests as if they were never decorated. Turns out nose already has this functionality: nosetests numpy --no-skip I think we should urge developers to run the test suite this way, so that we remain aware of failures, even if they are decorated. Hope that helps, Stéfan
Stéfan van der Walt wrote:
2009/1/9 David Cournapeau <cournape@gmail.com>:
On Fri, Jan 9, 2009 at 5:36 AM, Christopher Hanley <chanley@stsci.edu> wrote:
I do not expect the trunk to always work. I even expect it to have bugs. However, I do not expect there to be test failures for known reasons that result in wasSuccessful() returning false. This is a bad programming practice. It creates work for people trying to figure out what is wrong when the answer is already know.
Well, I don't agree it is bad practice: it is not ideal, yes, but I don't think using KnownFailure is much better. My rationale being that known failures are almost never worked on because it does not bug anyone anymore, and it is very easy to forget about them - AFAICS, most numpy/scipy known failures have never been worked on after being tagged as such. I don't think we have a good system for those cases, be it known failure - or just failing.
I agree with you point of view, but I also have sympathy for Cristopher's situation.
Yes, it is not a black and white situation - I first misunderstood Christopher situation because of the given context of tracking numpy changes. I can see why it is annoying - but it gives me important information (like for example the fact that solaris does not have the same formatting issues than linux and mac os X thanks to recent bug reports). As Robert said, BTS is supposedly a better system for this for this kind of things - but at least for me, trac is so slow and painful to use that I try to avoid it as much as possible. David
2009/1/9 David Cournapeau <david@ar.media.kyoto-u.ac.jp>:
As Robert said, BTS is supposedly a better system for this for this kind of things - but at least for me, trac is so slow and painful to use that I try to avoid it as much as possible.
We are running Trac 10.2 from November 2006, so it is quite possible that some of the speed issues have been addressed in the meantime. Cheers Stéfan
On Fri, Jan 9, 2009 at 01:19, Stéfan van der Walt <stefan@sun.ac.za> wrote:
I think we should urge developers to run the test suite this way, so that we remain aware of failures, even if they are decorated.
I don't think we should use unit tests as a bug tracker. We have Trac. Each known-failing test (or group of related tests) should have a ticket prioritized and scheduled appropriately. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
2009/1/9 Robert Kern <robert.kern@gmail.com>:
On Fri, Jan 9, 2009 at 01:19, Stéfan van der Walt <stefan@sun.ac.za> wrote:
I think we should urge developers to run the test suite this way, so that we remain aware of failures, even if they are decorated.
I don't think we should use unit tests as a bug tracker. We have Trac. Each known-failing test (or group of related tests) should have a ticket prioritized and scheduled appropriately.
Yup. Point is, an implementation comes with tests, and sometimes those tests break. We then have two choices: remove the tests or mark them as known failures. Since they are meant to indicate regressions, it would make little sense to remove them, hence these decorators. My suggestion was to use the --no-skip flag to make these decorators invisible to the developers, lest failures be forgotten. Stéfan
On Thu, Jan 8, 2009 at 11:37, Christopher Hanley <chanley@stsci.edu> wrote:
Hi,
I've committed the following change to test_print.py to fix one of the tests.
Index: test_print.py =================================================================== --- test_print.py (revision 6302) +++ test_print.py (working copy) @@ -154,7 +154,7 @@ else: locale.setlocale(locale.LC_NUMERIC, 'FRENCH')
- assert_equal(str(tp(1.2)), str(float(1.2)), + assert_equal(locale.format("%f",tp(1.2)), locale.format("%f",float(1.2)),
Note that this does not test anything. %f coerces the input to a Python float anyways. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
participants (5)
-
Christopher Hanley -
David Cournapeau -
David Cournapeau -
Robert Kern -
Stéfan van der Walt