On Mon, Feb 20, 2012 at 11:28 AM, Barry Warsaw <barry@python.org> wrote:
I think is an example of (mal)adapting to an incomplete module, rather than fixing it. I think doctest can handle all the points you're making. See clarification pointers below...
1. Execution context determined by outer-scope doctest defintions.
Can you explain this one?
I gave an example in a prior message on this thread, dated Feb 17. I think it's clear there but let me know. Basically, the idea is that since the class def can also have a docstring, where better would setup and teardown code go to provide the execution context of the inner method docstrings? Now the question: is it useful or appropriate to put setup and teardown code in a classdef docstring? Well, I think this requires a committment on the behalf of the coder/documentor to concoct useful (didactic) example that could go there. For example, (as in the prior-referenced message) I imagine putting example of defining a variable of the classes type (">>> g = Graph({some complex, interesting initialization})"), which might return a (testable) value upon creation. Now this could, logically, be put in the classes __init__ method, but that doesn't make sense for defining an execution context, and *in addition*, that can be saved for those complex corner cases you mentioned earlier.
I usually put all this in an additional_tests() method, such as:
Yes, I do the same for my modules with doctests. A dummy function which can catch all the non-interesting tests. This, still superior, in my opinion, than unittest. It is easier syntactically, as well as for casual users of your code (It has no leaning curve like understanding unittest). This superiority to unittest, by the way, is only realized if the second suggestion (smart comparisons) is implemented into doctest.
Yes, but you see you're destroying the very intent and spirit of doctest. The point is to make literate documentation. If you adapt to it's incompleteness, you reduce the power of it.
Well, hopefully, I've convinced you a little that the limitations in doctests over unittests are almost, if not entirely due, to the incompleteness of the module. If the two items I mentioned were implemented I think it would be far superior to unittest. (Corner cases, etc can all find a place, because every corner case should be documented somewhere anyway!!) Cheers!! mark santa fe, nm
On Mon, Feb 27, 2012 at 12:28:17PM -0700, Mark Janssen wrote:
I second what Barry says here. Doctests are for documentation, or at least, doctests in docstrings are for documentation, which means they should be simple and minimal, and only cover the most important parts of your function. Certainly they should only cover the interface, and never the implementation, so not used for regression testing or coverage of odd corner cases. I have no problem with extensive doctests if they are put in an external document. I do this myself. But when I call help(func), I want to learn how to use func, not see seven pages of tests that don't help me understand how to use the function.
And again, +1 with what Barry says here, which means I disagree with your response:
I don't accept this argument. Doctest is designed for including example code in documentation, and ensuring that the examples are correct. For that, it does a very good job. It makes a great hammer. Don't use it when you need a spanner. It's not that doctest can't handle regression tests, but that regression tests shouldn't be put inside the the function docstring. Why should people see a test case for some bug that occurred three versions back in the documentation? Put it in a separate test suite, either unit tests, or a literate programming doc using doctest. Don't pollute the docstring with tests that aren't useful documentation. When people read your docstring, you have to expect that they are reading it in isolation. They want to know "How do I use function spam?", and any example code should show them how to use function spam: >>> data = (23, 42, 'foo', 9) >>> collector = [5] >>> spam(data, collector) >>> collector [5, 28, 70, None, 79] That works as documentation first, and as a test second. This does not: >>> spam(data, collector) # data and collector defined elsewhere >>> collector [5, 28, 70, None, 79] The fact that each docstring sees a fresh execution context is a good thing, not a bug.
Is that a trick question? I don't want docstrings to have automatic setup and teardown code at all, and if they did, I certainly don't want them to be in some other object's docstring (related or not).
Now the question: is it useful or appropriate to put setup and teardown code in a classdef docstring?
In my opinion, no, neither useful nor appropriate. It would be counter-productive, by reducing the value of documentation as documentation, while still being insufficiently powerful to replace unit tests. A big -1 on this. [...]
I already think that doctest is far superior to unittest, for testing executable examples in documentation. I don't think it is superior to unittest for unit testing, or regression testing. Nor is it inferior -- its just different.
I think you have a different idea of "corner case" than I do. Corner cases, in my experience, refer to the implementation: does the function work correctly when the input is in the corner? Since this is testing the implementation, it shouldn't be in the documentation. The classic example is, does this list-function work when the list is empty? So I would expect that the unit tests for, say, the sorted() built-in will include a test case for sorted([]). (This applies regardless of whether you use unittest, doctest, nose, or some other testing framework.) But the documentation for sorted() don't need to explicitly state that sorting an empty list returns an empty list. That's a given from the accepted meaning of sorting -- if there's nothing to sort, you get nothing. Nor does it need to explicitly state that sorting a list with one item returns a list with one item. A single example of sorting a list of (say) four items is sufficient to document the purpose of sorted(), but it would be completely insufficient for unit testing purposes. -- Steven
Steven D'Aprano <steve@pearwood.info> writes:
I don't think ‘doctest’ is incomplete. It comprehensively covers the use case for which it is designed. The ‘unittest’ module is limited in usefulness at testing code examples in documentation. That doesn't make it incomplete, either.
+1 QotW. -- \ “The fact of your own existence is the most astonishing fact | `\ you'll ever have to confront. Don't dare ever see your life as | _o__) boring, monotonous, or joyless.” —Richard Dawkins, 2010-03-10 | Ben Finney
On 2/28/2012 2:59 AM, Steven D'Aprano wrote:
It depends on the function. fact(0) = 1 could go in a doc string. So, I think, could combo(0,0) = 1. But sometimes implementations introduce a special case that is an artifact of the implementation and definitely not belong in a doc string. An real example is approximating the normal integral (cumlative normal distribution) with different approximations for [0,a] and (a,infinity). Unit tests should test f(a) and f(a+epsilonl) and the difference (should be >= 0) to make sure the two approximations 'join' properly so as to be transparent to the user. If the join point changes, so does the special unit test. -- Terry Jan Reedy
On Mon, Feb 27, 2012 at 12:28:17PM -0700, Mark Janssen wrote:
I second what Barry says here. Doctests are for documentation, or at least, doctests in docstrings are for documentation, which means they should be simple and minimal, and only cover the most important parts of your function. Certainly they should only cover the interface, and never the implementation, so not used for regression testing or coverage of odd corner cases. I have no problem with extensive doctests if they are put in an external document. I do this myself. But when I call help(func), I want to learn how to use func, not see seven pages of tests that don't help me understand how to use the function.
And again, +1 with what Barry says here, which means I disagree with your response:
I don't accept this argument. Doctest is designed for including example code in documentation, and ensuring that the examples are correct. For that, it does a very good job. It makes a great hammer. Don't use it when you need a spanner. It's not that doctest can't handle regression tests, but that regression tests shouldn't be put inside the the function docstring. Why should people see a test case for some bug that occurred three versions back in the documentation? Put it in a separate test suite, either unit tests, or a literate programming doc using doctest. Don't pollute the docstring with tests that aren't useful documentation. When people read your docstring, you have to expect that they are reading it in isolation. They want to know "How do I use function spam?", and any example code should show them how to use function spam: >>> data = (23, 42, 'foo', 9) >>> collector = [5] >>> spam(data, collector) >>> collector [5, 28, 70, None, 79] That works as documentation first, and as a test second. This does not: >>> spam(data, collector) # data and collector defined elsewhere >>> collector [5, 28, 70, None, 79] The fact that each docstring sees a fresh execution context is a good thing, not a bug.
Is that a trick question? I don't want docstrings to have automatic setup and teardown code at all, and if they did, I certainly don't want them to be in some other object's docstring (related or not).
Now the question: is it useful or appropriate to put setup and teardown code in a classdef docstring?
In my opinion, no, neither useful nor appropriate. It would be counter-productive, by reducing the value of documentation as documentation, while still being insufficiently powerful to replace unit tests. A big -1 on this. [...]
I already think that doctest is far superior to unittest, for testing executable examples in documentation. I don't think it is superior to unittest for unit testing, or regression testing. Nor is it inferior -- its just different.
I think you have a different idea of "corner case" than I do. Corner cases, in my experience, refer to the implementation: does the function work correctly when the input is in the corner? Since this is testing the implementation, it shouldn't be in the documentation. The classic example is, does this list-function work when the list is empty? So I would expect that the unit tests for, say, the sorted() built-in will include a test case for sorted([]). (This applies regardless of whether you use unittest, doctest, nose, or some other testing framework.) But the documentation for sorted() don't need to explicitly state that sorting an empty list returns an empty list. That's a given from the accepted meaning of sorting -- if there's nothing to sort, you get nothing. Nor does it need to explicitly state that sorting a list with one item returns a list with one item. A single example of sorting a list of (say) four items is sufficient to document the purpose of sorted(), but it would be completely insufficient for unit testing purposes. -- Steven
Steven D'Aprano <steve@pearwood.info> writes:
I don't think ‘doctest’ is incomplete. It comprehensively covers the use case for which it is designed. The ‘unittest’ module is limited in usefulness at testing code examples in documentation. That doesn't make it incomplete, either.
+1 QotW. -- \ “The fact of your own existence is the most astonishing fact | `\ you'll ever have to confront. Don't dare ever see your life as | _o__) boring, monotonous, or joyless.” —Richard Dawkins, 2010-03-10 | Ben Finney
On 2/28/2012 2:59 AM, Steven D'Aprano wrote:
It depends on the function. fact(0) = 1 could go in a doc string. So, I think, could combo(0,0) = 1. But sometimes implementations introduce a special case that is an artifact of the implementation and definitely not belong in a doc string. An real example is approximating the normal integral (cumlative normal distribution) with different approximations for [0,a] and (a,infinity). Unit tests should test f(a) and f(a+epsilonl) and the difference (should be >= 0) to make sure the two approximations 'join' properly so as to be transparent to the user. If the join point changes, so does the special unit test. -- Terry Jan Reedy
participants (4)
-
Ben Finney -
Mark Janssen -
Steven D'Aprano -
Terry Reedy