Unit testing (again)
I was pleased to see that the 2.1 release schedule lists "unit testing" as one of the open issues. How is this going to be decided? Voting? BDFL fiat? --amk
I was pleased to see that the 2.1 release schedule lists "unit testing" as one of the open issues. How is this going to be decided? Voting? BDFL fiat?
BDFL fiat: most likely we'll be integrating PyUnit, whose author thinks this is a great idea. We'll be extending it to reduce the amount of boilerplate you have to type for new tests, and to optionally support the style of testing that Quixote's unit test package favors. This style (where the tests are given as string literals) seems to be really impopular with most people I've spoken to, but we're going to support it anyhow because there may also be cases where it's appropriate. I'm not sure however how much we'll get done for 2.1; maybe we'll just integrate the current PyUnit CVS tree. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, Feb 12, 2001 at 04:37:00PM -0500, Guido van Rossum wrote:
I'm not sure however how much we'll get done for 2.1; maybe we'll just integrate the current PyUnit CVS tree.
I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list? --amk
I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list?
I'm just reporting what I've heard on our group meetings. Fred Drake and Jeremy Hylton are in charge of getting this done. You can catch their ear on python-dev; I'm not sure about the PyUnit list. --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@digicool.com> writes:
[Andrew writes:]
I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list?
GvR> I'm just reporting what I've heard on our group meetings. Fred GvR> Drake and Jeremy Hylton are in charge of getting this done. GvR> You can catch their ear on python-dev; I'm not sure about the GvR> PyUnit list. I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit. Jeremy
On Mon, Feb 12, 2001 at 04:59:06PM -0500, Jeremy Hylton wrote:
I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit.
* Useful shorthands for common asserts (testing that two sequences are the same ignoring order, for example) * A way to write test cases that doesn't bring the test method to a halt if something raises an unexpected exception * Coverage support (though that would also entail Skip's coverage code getting into 2.1) --amk
"AMK" == Andrew Kuchling <akuchlin@cnri.reston.va.us> writes:
AMK> On Mon, Feb 12, 2001 at 04:59:06PM -0500, Jeremy Hylton wrote:
I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit.
AMK> * Useful shorthands for common asserts (testing that two AMK> sequences are the same ignoring order, for example) We can write a collection of helper functions for this, right? self.verify(sequenceElementsThatSame(l1, l2)) AMK> * A way to write test cases that doesn't bring the test method AMK> to a halt if something raises an unexpected exception I'm not sure how to achieve this or why you would want the test to continue. I know that Quixote uses test cases in strings, but it's the thing I like the least about Quixote unittest. Can you think of an alternate mechanism? Maybe I'd be less opposed if I could understand why it's desirable to continue executing a method where something has already failed unexpectedly. After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work. AMK> * Coverage support (though that would also entail Skip's AMK> coverage code getting into 2.1) Shouldn't be hard. Skip's coverage code was in 2.0; we might need to move it from Tools/script to the library, though. Jeremy
On Mon, Feb 12, 2001 at 06:16:19PM -0500, Jeremy Hylton wrote:
We can write a collection of helper functions for this, right? self.verify(sequenceElementsThatSame(l1, l2))
Pretty much; nothing too difficult.
Maybe I'd be less opposed if I could understand why it's desirable to continue executing a method where something has already failed unexpectedly. After the first exception, something is broken and
In this style of unit test, you have setup() and shutdown() methods that create and destroy the test objects afresh for each test case, so cases aren't big long skeins of assertions that will all break given a single failure. Instead they're more like 1) call a method that changes an object's state, 2) call accessors or get attributes to check invariants are what you expect. It can be useful to know that .get_parameter_value() raises an exception while .get_parameter_type() doesn't, or whatever. --amk
Andrew, Here's a sample of PyUnit stuff that I think illustrates what you're asking for... from unittest import TestCase, makeSuite, TextTestRunner class Test(TestCase): def setUp(self): self.t = {2:2} def tearDown(self): del self.t def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail) def _getitemfail(self): return self.t[1] def testGetItemSucceeds(self): assert self.t[2] == 2 def main(): suite = makeSuite(Test, 'test') runner = TextTestRunner() runner.run(suite) if __name__ == '__main__': main() Execution happens like this: call setUp() call testGetItemFails() print test results call tearDown() call setUp() call testGetItemSucceeds() print test results call tearDown() end Isn't this almost exactly what you want? Or am I completely missing the point? ----- Original Message ----- From: "Andrew Kuchling" <akuchlin@cnri.reston.va.us> To: <python-dev@python.org> Sent: Monday, February 12, 2001 10:52 PM Subject: Re: [Python-Dev] Unit testing (again)
On Mon, Feb 12, 2001 at 06:16:19PM -0500, Jeremy Hylton wrote:
We can write a collection of helper functions for this, right? self.verify(sequenceElementsThatSame(l1, l2))
Pretty much; nothing too difficult.
Maybe I'd be less opposed if I could understand why it's desirable to continue executing a method where something has already failed unexpectedly. After the first exception, something is broken and
In this style of unit test, you have setup() and shutdown() methods that create and destroy the test objects afresh for each test case, so cases aren't big long skeins of assertions that will all break given a single failure. Instead they're more like 1) call a method that changes an object's state, 2) call accessors or get attributes to check invariants are what you expect. It can be useful to know that .get_parameter_value() raises an exception while .get_parameter_type() doesn't, or whatever.
--amk
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev
Jeremy wrote:
I know that Quixote uses test cases in strings, but it's the thing I like the least about Quixote unittest
like whitespace indentation, it's done that way for a reason.
I'm not sure how to achieve this or why you would want the test to continue.
same reason you want your compiler to report more than just the first error -- so you can see patterns in the test script's behaviour, so you can fix more than one bug at a time, or fix the bugs in an order that suits you and not the framework, etc. (for some of our components, we're using a framework that can continue to run the test even if the tested program dumps core. trust me, that has saved us a lot of time...)
After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.
jeremy, that's the kind of comment I would have expected from a manager, not from a programmer who has done lots of testing. Cheers /F
"FL" == Fredrik Lundh <fredrik@effbot.org> writes:
FL> Jeremy wrote:
I know that Quixote uses test cases in strings, but it's the thing I like the least about Quixote unittest
FL> like whitespace indentation, it's done that way for a reason. Whitespace indentation is natural and makes code easier to read. Putting little snippets of Python code in string literals passed to exec has the opposite effect. doctest is a nice middle ground, because the code snippets are in a natural setting -- an interactive interpreter setting.
I'm not sure how to achieve this or why you would want the test to continue.
FL> same reason you want your compiler to report more than just the FL> first error -- so you can see patterns in the test script's FL> behaviour, so you can fix more than one bug at a time, or fix FL> the bugs in an order that suits you and not the framework, etc. Python's compiler only reports one syntax error for a source file, regardless of how many it finds <0.5 wink>.
After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.
FL> jeremy, that's the kind of comment I would have expected from a FL> manager, not from a programmer who has done lots of testing. I don't think there's any reason to be snide. The question is one of granularity: At what level of granularity should the test framework catch exceptions and continue? I'm satisfied with the unit of testing being a method. Jeremy
"JH" == Jeremy Hylton <jeremy@alum.mit.edu> writes:
JH> Whitespace indentation is natural and makes code easier to JH> read. Putting little snippets of Python code in string JH> literals passed to exec has the opposite effect. Especially because requiring the snippets to be in strings means editing them with a Python-aware editor becomes harder. JH> doctest is a nice middle ground, because the code snippets are JH> in a natural setting -- an interactive interpreter setting. And at least there, I can for the most part just cut-and-paste the output of my interpreter session into the docstrings. -Barry
barry wrote:
Especially because requiring the snippets to be in strings means editing them with a Python-aware editor becomes harder.
well, you don't have to put *all* your test code inside the test calls... try using them as asserts instead: do something do some calculations do some more calculations self.test_bool("result == 10")
And at least there, I can for the most part just cut-and-paste the output of my interpreter session into the docstrings.
cutting and pasting from the interpreter into the test assertion works just fine... Cheers /F
jeremy wrote:
FL> like whitespace indentation, it's done that way for a reason.
Whitespace indentation is natural and makes code easier to read. Putting little snippets of Python code in string literals passed to exec has the opposite effect.
Only if you're using large snippets. ...just like whitespace indentation makes things harder it you're mixing tabs and spaces, or prints a file with the wrong tab setting, or cuts and pastes code between editors with different tab settings. In both cases, the solution is simply "don't do that"
doctest is a nice middle ground, because the code snippets are in a natural setting -- an interactive interpreter setting.
They're still inside a string...
Python's compiler only reports one syntax error for a source file, regardless of how many it finds <0.5 wink>.
Sure, but is that because user testing has shown that Python programmers (unlike e.g. C programmers) prefer to see only one bug at a time, or because it's convenient to use exceptions also for syntax errors? Would a syntax-checking editor be better if it only showed one syntax error, even if it found them all?
After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.
FL> jeremy, that's the kind of comment I would have expected from a FL> manager, not from a programmer who has done lots of testing.
I don't think there's any reason to be snide.
Well, I first wrote "taken out of context, that's the kind of comment" but then I noticed that it wasn't really taken out of context. And in full context, it still looks a bit arrogant: why would Andrew raise this issue if *he* didn't want more granularity? ::: But having looked everything over one more time, and having ported a small test suite to doctest.py, I'm now -0 on adding more test frame- works to 2.1. If it's good enough for tim... (and -1 if adding more frameworks means that I have to use them ;-). Cheers /F
"FL" == Fredrik Lundh <fredrik@pythonware.com> writes:
After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.
FL> jeremy, that's the kind of comment I would have expected from a FL> manager, not from a programmer who has done lots of testing.
I don't think there's any reason to be snide.
FL> Well, I first wrote "taken out of context, that's the kind of FL> comment" but then I noticed that it wasn't really taken out of FL> context. FL> And in full context, it still looks a bit arrogant: why would FL> Andrew raise this issue if *he* didn't want more granularity? I hope it's simple disagreement and not arrogance. I do not agree with him (or you) on a particular technical issue -- whether particular expressions should be stuffed into string literals in order to recover from exceptions they raise. There's a tradeoff between readability and granularity and I prefer readability. I am surprised that you are impugning my technical abilities (manager, not programmer) or calling me arrogant because I don't agree. I think I am should be entitled to my wrong opinion. Jeremy
On Tue, Feb 13, 2001 at 12:29:35AM -0500, Jeremy Hylton wrote:
I hope it's simple disagreement and not arrogance. I do not agree
I trust not. :) My primary concern is that the tests are quickly readable, because they're also a form of documentation (hopefully not the only one though). I have enough problems debugging the actual code without having to debug a test suite. Consider the example Chris posted, which features the snippet: def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail) def _getitemfail(self): return self.t[1] I don't think this form, requiring an additional small helper method, is any clearer than self.test_exc('self.t[1]', KeyError); two extra lines and the loss of locality. Put tests for 3 or 4 different exceptions into testGetItemFails and you'd have several helper functions to trace through. For simple value tests, this is less important; the difference between test_val( 'self.db.get_user("FOO")', None ) and test_val(None, self.db.get_user, "foo") is less. /F's observation that doctest seems suitable for his work is interesting and surprising; I'll spend some more time looking at it. --amk
On Tue, 13 Feb 2001, Andrew Kuchling wrote:
Consider the example Chris posted, which features the snippet:
def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail)
def _getitemfail(self): return self.t[1]
I don't think this form, requiring an additional small helper method, is any clearer than self.test_exc('self.t[1]', KeyError); two extra lines and the loss of locality. Put tests for 3 or 4 different exceptions into testGetItemFails and you'd have several helper functions to trace through.
I'm not sure what the purpose of using a unit test to test a different unit in the same suite is. I've never used assertRaises in this way, and the small helper method seems just to illustrate your point, not an often used feature of asserting an Exception condition. More often the method you are checking for an exception comes from the thing you are testing, not the test. Maybe you have different usage patterns than I... -Michel
[/F]
But having looked everything over one more time, and having ported a small test suite to doctest.py, I'm now -0 on adding more test frameworks to 2.1. If it's good enough for tim...
I'm not sure that it is, but I have yet to make time to look at the others. It's no secret that I love doctest, and, indeed, in 20+ years of industry pain, it's the only testing approach I didn't drop ASAP. I still use it for all my stuff, and very happily. But! I don't do anything with the web or GUIs etc -- I'm an algorithms guy. Most of the stuff I work with has clearly defined input->output relationships, and capturing an interactive session is simply perfect both for documenting and testing such stuff. It's also the case that I weight the "doc" part of "doctest" more heavily than the "test" part, and when Peter or Guido say that, e.g., the reliance on exact output match is "a problem", I couldn't disagree more strongly. It's obvious to Guido that dict output may come in any order, but a doc *reader* in a hurry is at best uneasy when documented output doesn't match actual output exactly. That's not something I'll yield on. [Andrew]
def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail)
def _getitemfail(self): return self.t[1]
[vs]
self.test_exc('self.t[1]', KeyError)
My brain doesn't grasp either of those at first glance. But everyone who has used Python a week grasps this: class C: def __getitem__(self, i): """Return the i'th item. i==1 raises KeyError. For example, >>> c = C() >>> c[0] 0 >>> c[1] Traceback (most recent call last): File "x.py", line 20, in ? c[1] File "x.py", line 14, in __getitem__ raise KeyError("bad i: " + `i`) KeyError: bad i: 1 >>> c[-1] -1 """ if i != 1: return i else: raise KeyError("bad i: " + `i`) Cute: Python changed the first line of its traceback output (used to say "Traceback (innermost last):"), and current doctest wasn't expecting that. For *doc* purposes, it's important that the examples match what Python actually does, so that's a bug in doctest.
Cute: Python changed the first line of its traceback output (used to say "Traceback (innermost last):"), and current doctest wasn't expecting that.
which reminds me... are there any chance of getting a doctest that can survives its own test suite under 1.5.2, 2.0, and 2.1? the latest version blows up under 1.5.2 and 2.0: ***************************************************************** Failure in example: 1/0 from line #155 of doctest Expected: ZeroDivisionError: integer division or modulo by zero Got: ZeroDivisionError: integer division or modulo 1 items had failures: 1 of 8 in doctest ***Test Failed*** 1 failures. Cheers /F
Fredrik Lundh wrote:
Cute: Python changed the first line of its traceback output (used to say "Traceback (innermost last):"), and current doctest wasn't expecting that.
which reminds me... are there any chance of getting a doctest that can survives its own test suite under 1.5.2, 2.0, and 2.1?
the latest version blows up under 1.5.2 and 2.0:
***************************************************************** Failure in example: 1/0 from line #155 of doctest Expected: ZeroDivisionError: integer division or modulo by zero Got: ZeroDivisionError: integer division or modulo 1 items had failures: 1 of 8 in doctest ***Test Failed*** 1 failures.
Since exception message are not defined anywhere I'd suggest to simply ignore them in the output. About the traceback output format: how about adding some re support instead of using string.find() ?! -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[MAL]
Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.
Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.
About the traceback output format: how about adding some re support instead of using string.find() ?!
Why? I never use regexps where simple string matches work, and neither should you <wink>.
Tim Peters wrote:
[MAL]
Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.
Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.
But what it does write to the console changes with every release (e.g. just take the repr() changes for strings with non-ASCII data)... this simply breaks you test suite every time Writing Python programs which work on Python 1.5-2.1 which at the same time pass the doctest unit tests becomes impossible. The regression suite (and most other Python software) catches exceptions based on the exception class -- why isn't this enough for your doctest.py checks ? nit-pickling-ly, -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[MAL]
Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.
[Tim]
Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.
[MAL]
But what it does write to the console changes with every release (e.g. just take the repr() changes for strings with non-ASCII data)...
So now you don't want to test exception messages *or* non-exceptional output either. That's fine, but you're never going to like doctest if so.
this simply breaks you test suite every time
I think you're missing the point: it breaks your *docs*, if they contain any examples that rely on such stuff. doctest then very helpfully points out that your docs-- no matter how good they were before --now suck, because they're now *wrong*. It's not interested in assigning blame for that, it's enough to point out that they're now broken (else they'll never get fixed!).
Writing Python programs which work on Python 1.5-2.1 which at the same time pass the doctest unit tests becomes impossible.
Not true. You may need to rewrite your examples, though, so that your *docs* are accurate under all the releases you care about. I don't care if that drives you mad, so long as it prevents you from screwing your users with inaccurate docs.
The regression suite (and most other Python software) catches exceptions based on the exception class -- why isn't this enough for your doctest.py checks ?
Because doctest is primarily interested in ensuring that non-exceptional cases continue to work exactly as advertised. Checking that, e.g.,
fac(5) 120
still works is at least 10x easier to live with than writing crap like if fac(5) != 120: raise TestFailed("Something about fac failed but it's too " "painful to build up some string here " "explaining exactly what -- try single-" "stepping through the test by hand until " "this raise triggers.") That's regrtest.py-style testing, and if you think that's pleasant to work with you must never have seen a std test break <0.7 wink>. When a doctest'ed module breaks, the doctest output pinpoints the failure precisely, without any work on your part beyond simply capturing an interactive session that shows the results you expected.
nit-pickling-ly,
Na, you're trying to force doctest into a mold it was designed to get as far away from as possible. Use it for its intended purpose, then gripe. Right now you're complaining that the elephant's eyes are the wrong color while missing that it's actually a leopard <wink>.
[/F]
which reminds me... are there any chance of getting a doctest that can survives its own test suite under 1.5.2, 2.0, and 2.1?
the latest version blows up under 1.5.2 and 2.0:
***************************************************************** Failure in example: 1/0 from line #155 of doctest Expected: ZeroDivisionError: integer division or modulo by zero Got: ZeroDivisionError: integer division or modulo 1 items had failures: 1 of 8 in doctest ***Test Failed*** 1 failures.
Not to my mind. doctest is intentionally picky about exact matches, for reasons explained earlier. If the docs for a thing say "integer division or modulo by zero" is expected, but running it says something else, the docs are wrong and doctest's primary *purpose* is to point that out loudly. I could change the exception example to something where Python didn't gratuitously change what it prints, though <wink>. OK, I'll do that.
Not to my mind. doctest is intentionally picky about exact matches, for reasons explained earlier. If the docs for a thing say "integer division or modulo by zero" is expected, but running it says something else, the docs are wrong and doctest's primary *purpose* is to point that out loudly.
Of course, this is means that *if* you use doctest, all authoritative docs should be in the docstring, and not elsewhere. Which brings us back to the eternal question of how to indicate mark-up in docstrings. Is everything connected to everything? --Guido van Rossum (home page: http://www.python.org/~guido/)
[Tim]
Not to my mind. doctest is intentionally picky about exact matches, for reasons explained earlier. If the docs for a thing say "integer division or modulo by zero" is expected, but running it says something else, the docs are wrong and doctest's primary *purpose* is to point that out loudly.
[Guido]
Of course, this is means that *if* you use doctest, all authoritative docs should be in the docstring, and not elsewhere.
I don't know why you would reach that conclusion. My own Python work in years past had overwhelmingly little to do with anything in the Python distribution, and I surely did put all my docs in my modules. It was my only realistic choice, and doctest grew in part out of that "gotta put everything in one file, cuz one file is all I got" way of working. By allowing to put the docs for a thing right next to the tests for a thing right next to the code for a thing, doctest changed the *nature* of that compromise from a burden to a relative joy. Doesn't mean the docs couldn't or shouldn't be elsewhere, though, unless you assume that only the "authoritative docs" need to be accurate (I prefer that all docs tell the truth <wink>). I know some people have adapted the guts of doctest to ensuring that their LaTeX and/or HTML Python examples work as advertised too. Cool! The Python Tutorial is eternally out of synch in little ways with what the matching release actually does.
Which brings us back to the eternal question of how to indicate mark-up in docstrings.
I announced a few years ago I was done waiting for mark-up to reach consensus, and was going to just go ahead and write useful docstrings regardless. Never had cause to regret that -- mark-up is the tail wagging the dog, and I don't know why people tolerate it (well, yes I do: "but there's no mark-up defined!" is an excuse to put off writing decent docs! but you really don't need six levels of nested lists-- or even one --to get 99% of the info across).
Is everything connected to everything?
when-it's-convenient-to-believe-it-and-a-few-times-even-when-not-ly y'rs - tim
On Tue, 13 Feb 2001 16:45:56 -0500, Guido van Rossum <guido@digicool.com> wrote:
Of course, this is means that *if* you use doctest, all authoritative docs should be in the docstring, and not elsewhere. Which brings us back to the eternal question of how to indicate mark-up in docstrings. Is everything connected to everything?
No, but a lot of things are connected to documentation. As someone who works primarily in Perl nowadays, and hates it, I must say that as horrible and unaesthetic pod is, having perldoc package::module Just work is worth everything -- I've marked everything I wrote that way, and I can't begin to explain how much it helps. I'm slowly starting to think that the big problem with Python documentation is that you didn't pronounce. So, if some publisher needs to work harder to make dead-trees copies, it's fine by me, and even if the output looks a bit less "professional" it's also fine by me, as long as documentation is always in the same format, and always accessible by the same command. Consider this an offer to help to port (manually, if needs be) Python's current documentation. We had a DevDay, we have a sig, we have a PEP. None of this seems to help -- what we need is a BDFL's pronouncement, even if it's on the worst solution possibly imaginable. -- For public key: finger moshez@debian.org | gpg --import "Debian -- What your mother would use if it was 20 times easier" LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive.
On Wed, 14 Feb 2001, Moshe Zadka wrote:
As someone who works primarily in Perl nowadays, and hates it, I must say that as horrible and unaesthetic pod is, having
perldoc package::module
Just work is worth everything -- I've marked everything I wrote that way, and I can't begin to explain how much it helps.
I agree that this is important.
We had a DevDay, we have a sig, we have a PEP. None of this seems to help --
What are you talking about? There is an implementation and it works. I demonstrated the HTML one back at Python 8, and now there is a text-generating one in the CVS tree. -- ?!ng
[Moshe Zadka]
We had a DevDay, we have a sig, we have a PEP. None of this seems to help --
[Ka-Ping Yee]
What are you talking about? There is an implementation and it works.
There are many implementations "that work". But we haven't picked one. What's the standard markup for Python docstrings? There isn't! That's what he's talking about. This is especially bizarre because it's been clear for *years* that some variant of structured text would win in the end, but nobody playing the game likes all the details of anyone else's set of (IMO, all overly elaborate) conventions, so the situation for users is no better now than it was the day docstrings were added. Tibs's latest (and ongoing) attempt to reach a consensus can be found here: http://www.tibsnjoan.demon.co.uk/docutils/STpy.html The status of its implementation here: http://www.tibsnjoan.demon.co.uk/docutils/status.html Not close yet. In the meantime, Perlers have been "suffering" with a POD spec about 3% the size of the proposed Python spec; I guess their only consolation is that POD docs have been in universal use for years <wink>. while-ours-is-that-we'll-get-to-specify-non-breaking-spaces-someday- despite-that-not-1-doc-in-100-needs-them-ly y'rs - tim
On Fri, 16 Feb 2001, Tim Peters wrote:
[Moshe Zadka]
We had a DevDay, we have a sig, we have a PEP. None of this seems to help --
[Ka-Ping Yee]
What are you talking about? There is an implementation and it works.
There are many implementations "that work". But we haven't picked one. What's the standard markup for Python docstrings? There isn't! That's what he's talking about.
That's exactly the point i'm trying to make. There isn't any markup format enforced by pydoc, precisely because it isn't worth the strife. Moshe seemed to imply that the set of deployable documentation tools was empty, and i take issue with that. His post also had an tone of hopelessness about the topic that i wanted to counter immediately. The fact that pydoc doesn't have a way to italicize doesn't make it a non-solution -- it's a perfectly acceptable solution! Fancy formatting features can come later.
This is especially bizarre because it's been clear for *years* that some variant of structured text would win in the end, but nobody playing the game likes all the details of anyone else's set of (IMO, all overly elaborate) conventions, so the situation for users is no better now than it was the day docstrings were added.
Tibs's latest (and ongoing) attempt to reach a consensus can be found here:
http://www.tibsnjoan.demon.co.uk/docutils/STpy.html
The status of its implementation here:
http://www.tibsnjoan.demon.co.uk/docutils/status.html
Not close yet.
The design and implementation of a standard structured text syntax is emphatically *not* a prerequisite for a useful documentation tool. I agree that it may be nice, and i certainly applaud Tony's efforts, but we should not wait for it. -- ?!ng
[Ka-Ping Yee]
That's exactly the point i'm trying to make. There isn't any markup format enforced by pydoc, precisely because it isn't worth the strife. Moshe seemed to imply that the set of deployable documentation tools was empty, and i take issue with that. His post also had an tone of hopelessness about the topic that i wanted to counter immediately.
Most programmers are followers in this matter, and I agree with Moshe on this point: until something is Officially Blessed, Python programmers will stay away from every gimmick in unbounded droves. I personally don't care whether markup is ever defined, because I already gave up on it. But I-- like you --won't wait forever for anything. We're not the norm. the-important-audience-isn't-pythondev-it's-pythonlist-ly y'rs - tim
On Mon, 12 Feb 2001, Andrew Kuchling wrote:
* A way to write test cases that doesn't bring the test method to a halt if something raises an unexpected exception
I'm not sure what you mean by this, but Jim F. recently sent this email around internally: """ Unit tests are cool. One problem is that after you find a problem, it's hard to debug it, because unittest catches the exceptions. I added debug methods to TestCase and TestSuite so that you can run your tests under a debugger. When you are ready to debug a test failure, just call debug() on your test suite or case under debugger control. I checked this change into our CVS and send the auther of PyUnit a message. Jim """ I don't think it adressed your comment, but it is an interesting related feature. -Michel
Jeremy Hylton wrote:
"GvR" == Guido van Rossum <guido@digicool.com> writes:
[Andrew writes:]
I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list?
GvR> I'm just reporting what I've heard on our group meetings. Fred GvR> Drake and Jeremy Hylton are in charge of getting this done. GvR> You can catch their ear on python-dev; I'm not sure about the GvR> PyUnit list.
I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit.
Fine by private e-mail, though it would be nice if some of the discussions are seen by the PyUnit list because it's a representative community of regular users who probably have a good idea of what makes sense for them. If somebody would like to suggest changes, I can look into how they might get done. Also, I'd love to see what I can do to allay AMK's 'clunkiness' complaints! :-) Best wishes, -Steve -- Steve Purcell, Pythangelist "Life must be simple if *I* can do it" -- me
Note that doctest.py is part of the 2.1 std library. If you've never used it, pretend I didn't tell you that, and look at the new std library module difflib.py. Would you even guess there *are* unit tests in there? Here's the full text of the new std test test_difflib.py: import doctest, difflib doctest.testmod(difflib, verbose=1) I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use (and, yes, if an unexpected exception is raised, it doesn't stop the rest of the tests from running -- it's in the very nature of "unit tests" that an error in one unit should not prevent other unit tests from running). practicing-for-a-marketing-career-ly y'rs - tim
Hi, Tim Peters:
Note that doctest.py is part of the 2.1 std library. If you've never used [...] I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use (and, yes, if an unexpected exception is raised, it doesn't stop the rest of the tests from running -- it's in the very nature of "unit tests" that an error in one unit should not prevent other unit tests from running).
practicing-for-a-marketing-career-ly y'rs - tim
[a satisfied customer reports:] I like doctest very much. I'm using it for our company projects a lot. This is a very valuable tool. However Pings latest changes, which turned 'foobar\012' into 'foobar\n' and '\377\376\345' into '\xff\xfe\xe5' has broken some of the doctests in our software. Since we have to keep our code compatible with Python 1.5.2 for at least one, two or may be three more years, it isn't obvious to me how to fix this. I've spend some thoughts about a patch to doctest fooling the string printing output back to the 1.5.2 behaviour, but didn't get around to it until now. :-( Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)
[a satisfied customer reports:] I like doctest very much. I'm using it for our company projects a lot. This is a very valuable tool.
However Pings latest changes, which turned 'foobar\012' into 'foobar\n' and '\377\376\345' into '\xff\xfe\xe5' has broken some of the doctests in our software.
Since we have to keep our code compatible with Python 1.5.2 for at least one, two or may be three more years, it isn't obvious to me how to fix this.
This is a general problem with doctest, and a general solution exists. It's the same when you have a function that returns a dictionary: you can't include the dictionary in the output, because the key order isn't guaranteed. So, instead of writing your example like this: >>> foo() {"Hermione": "hippogryph", "Harry": "broomstick"} >>> you write it like this: >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"} 1 >>> I'll leave it as an exercise to the reader to apply this to string literals. --Guido van Rossum (home page: http://www.python.org/~guido/)
tim wrote:
I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use
you know, just having taken another look at doctest and the unit test options, I'm tempted to agree. except for the "if sufficient" part, that is -- given that you can easily run doctest on a test harness instead of the original module, it's *always* sufficient. (cannot allow tim to be 100% correct every time ;-) Cheers /F
On Tue, 13 Feb 2001, Fredrik Lundh wrote:
tim wrote:
I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use
you know, just having taken another look at doctest and the unit test options, I'm tempted to agree.
I also agree that doctest is the bee's knees, but I don't think it is quite as useful for us as PyUnit (for other people, I'm sure it's very useful). One of the goals of our interface work is to associate unit tests with interfaces. I don't see how doctest can work well with that. I probably need to look at it more, but one of our end goals is to walk up to a component, push a button, and have that components interfaces test the component while the system is live. I immagine this involving a bit of external framework at the interface level that would be pretty easy with PyUnit, I've only seen one example of doctest and it looks like you run it against an imported module. I don't see how this helps us with our (DC's) definition of components. A personal issue for me is that it overloads the docstring, no biggy, but it's just a personal nit I don't particularly like about doctest. Another issue is documentation. I don't know how much documentation doctest has, but PyUnit's documentation is *superb* and there are no suprises, which is absolutely +1. Quixote's documentation seems very thin (please correct me if I'm wrong). PyUnit's documentation goes beyond just explaning the software into explaining common patterns and unit testing philosophies. -Michel
[Michel Pelletier]
... A personal issue for me is that it overloads the docstring, no biggy, but it's just a personal nit I don't particularly like about doctest.
No. The docstring remains documentation. But documentation without examples generally sucks, due to the limitations of English in being precise. A sharp example can be worth 1,000 words. doctest is being used as *intended* to the extent that the embedded examples are helpful for documentation purposes. doctest then guarantees the examples continue to work exactly as advertised over time (and they don't! examples *always* get out of date, but without (something like) doctest they never get repaired). As I suggested at the start, read the docstrings for difflib.py: the examples are an integral part of the docs, and you shouldn't get any sense that they're there "just for testing" (if you do, the examples are poorly chosen, or poorly motivated in the surrounding commentary). Beyond that, doctest will also execute any code it finds in the module.__test__ dict, which maps arbitrary names to arbitrary strings. Anyone using doctest primarily as a testing framework should stuff their test strings into __test__ and leave the docstrings alone.
Another issue is documentation. I don't know how much documentation doctest has,
Look at its docstrings -- they not only explain it in detail, but contain examples of use that doctest can check <wink>.
No. The docstring remains documentation. But documentation without examples generally sucks, due to the limitations of English in being precise. A sharp example can be worth 1,000 words. doctest is being used as *intended* to the extent that the embedded examples are helpful for documentation purposes. doctest then guarantees the examples continue to work exactly as advertised over time (and they don't! examples *always* get out of date, but without (something like) doctest they never get repaired).
You're lucky that doctest doesn't return dictionaries! For functions that return dictionaries, it's much more natural *for documentation purposes* to write
book() {'Fred': 'mom', 'Ron': 'Snape'}
than the necessary work-around. You may deny that's a problem, but once we've explained dictionaries to our users, we can expect them to understand that if they see instead
book() {'Ron': 'Snape', 'Fred': 'mom'}
they will understand that that's the same thing. Writing it this way is easier to read than
book() == {'Ron': 'Snape', 'Fred': 'mom'} 1
I always have to look twice when I see something like that.
As I suggested at the start, read the docstrings for difflib.py: the examples are an integral part of the docs, and you shouldn't get any sense that they're there "just for testing" (if you do, the examples are poorly chosen, or poorly motivated in the surrounding commentary).
They are also more voluminous than I'd like the docs for difflib to be... --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
You're lucky that doctest doesn't return dictionaries! For functions that return dictionaries, it's much more natural *for documentation purposes* to write
book() {'Fred': 'mom', 'Ron': 'Snape'}
than the necessary work-around. You may deny that's a problem, but once we've explained dictionaries to our users, we can expect them to understand that if they see instead
book() {'Ron': 'Snape', 'Fred': 'mom'}
they will understand that that's the same thing. Writing it this way is easier to read than
book() == {'Ron': 'Snape', 'Fred': 'mom'} 1
I always have to look twice when I see something like that.
sortdict(book()) {'Fred': 'mom', 'Ron': 'Snape'}
Explicit is better etc. If I have a module that's going to be showing a lot of dict output, I'll write a little "sortdict" function at the top of the docs and explain why it's there. It's clear from c.l.py postings over the years that lots of people *don't* grasp that dicts are "unordered". Introducing a sortdict() function serves a useful pedagogical purpose for them too. More subtle than dicts for most people is examples showing floating-point output. This isn't reliable across platforms (and, e.g., it's no coincidence that most of the .ratio() etc examples in difflib.py are contrived to return values exactly representable in binary floating-point; but simple fractions like 3/4 are also easiest for people to visualize, so that also makes for good examples).
They [difflib.py docstring docs] are also more voluminous than I'd like the docs for difflib to be...
Not me -- there's nothing in them that I as a potential user don't need to know. But then I think the Library docs are too terse in general. Indeed, Fredrick makes part of his living selling a 300-page book supplying desperately needed Library examples <0.5 wink>. WRT difflib.py, it's OK by me if Fred throws out the examples when LaTeXing the module docstring, because a user can still get the info *from* the docstrings. For that matter, he may as well throw out everything except the first line or two of each method description, if you want bare-bones minimal docs for the manual. no-denying-that-examples-take-space-but-what's-painful-to-include- in-the-latex-docs-is-trivial-to-maintain-in-the-code-ly y'rs - tim
On Tue, 13 Feb 2001 20:24:00 -0500, "Tim Peters" <tim.one@home.com> wrote:
Not me -- there's nothing in them that I as a potential user don't need to know. But then I think the Library docs are too terse in general. Indeed, Fredrick makes part of his living selling a 300-page book supplying desperately needed Library examples <0.5 wink>.
I'm sorry, Tim, that's just too true. I want to explain my view about how it happened (I wrote some of them, and if you find a particularily terse one, just assume it's me) -- I write tersely. My boss yelled at me when doing this at work, and I redid all my internal documentation -- doubled the line count, beefed up with examples, etc. He actually submitted a bug in the internal bug tracking system to get me to do that ;-) So, I suggest you do the same -- there's no excuse for terseness, other then not-having-time, so it's really important that bugs like that are files. Something like "documentation for xxxlib is too terse". I can't promise I'll fix all these bugs, but I can try ;-) -- For public key: finger moshez@debian.org | gpg --import "Debian -- What your mother would use if it was 20 times easier" LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive.
Moshe Zadka writes:
so it's really important that bugs like that are files. Something like "documentation for xxxlib is too terse". I can't promise I'll fix all these bugs, but I can try ;-)
It would also be useful to tell what additional information you were looking for. We can probably find additional stuff to write on a lot of these, but that doesn't mean we'll interpret "too terse" in the most useful way. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Digital Creations
On Wed, Feb 14, 2001 at 06:57:35PM +0200, Moshe Zadka wrote:
On Tue, 13 Feb 2001 20:24:00 -0500, "Tim Peters" <tim.one@home.com> wrote:
Not me -- there's nothing in them that I as a potential user don't need to know. But then I think the Library docs are too terse in general. Indeed, Fredrick makes part of his living selling a 300-page book supplying desperately needed Library examples <0.5 wink>.
I'm sorry, Tim, that's just too true.
You should be appologizing to Fred, not Tim :) While I agree with the both of you, I'm not sure if expanding the library reference is going to help the problem. I think what's missing is a library *tutorial*. The reference is exactly that, a reference, and if we expand the reference we'll end up cursing it ourself, should we ever need it. (okay, so noone here needs the reference anymore <wink> except me, but when looking at the reference, I like the terse descriptions of the modules. They're just reminders anyway.) I remember when I'd finished the Python tutorial and wondered where to go next. I tried reading the library reference, but it was boring and most of it not interesting (since it isn't built up to go from useful/common -> rare, but just a list of all modules ordered by 'service'.) I ended up doing the slow and cheap version of Fredrik's book: reading python-list ;) I'll write the library tutorial once I finish the 'from-foo-import-* considered harmful' chapter ;-) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
[Thomas Wouters]
... I think what's missing is a library *tutorial*.
How would that differ from the effbot guide (to the std library)? The Python (language) Tutorial can be pretty small, because the Python language is pretty small. But the libraries are massive, and growing, and are increasingly in the hands of people with no Unix experience, or even programming experience. So I suppose "tutorial" can mean many things.
The reference is exactly that, a reference,
In part. In other parts (a good example is the profile docs) it's a lot of everything; in others it's so much "a reference" you can't figure out what it's saying unless you study the code (the pre-2.1 "random" docs sure come to mind). It's no more consistent in content level than anything else with umpteen authors.
and if we expand the reference we'll end up cursing it ourself, should we ever need it.
If the people who wanted "just a reference" were happy, I don't think David Beazley would have found an audience for his "Python Essential Reference". I can't argue about this, though, because nobody will ever agree. Guido doesn't want leisurely docs in the Reference Manual, nor does he like leisurely docs in docstrings. OTOH, those are what average and sub-average programmers *need*, and I write docs for them first, sneaking in examples when possible that I hope even experts will find pleasure in pondering. A good compromise by my lights-- and perhaps because I only care about the HTML docs, where "size" isn't apparent or a problem for navigation --would be to follow a terse but accurate reference with as many subsections as felt needed, with examples and rationale and tutorial material (has anyone ever figured how to use rexec or bastion from the docs? heh). But since nobody will agree with that either, I stick everything into docstrings and leave it to Fred to throw away the most useful parts for the "real docs" <wink>.
... I remember when I'd finished the Python tutorial and wondered where to go next. I tried reading the library reference, but it was boring and most of it not interesting (since it isn't built up to go from seful/common -> rare, but just a list of all modules ordered by service'.)
Excellent point! I had the same question when I first learned Python, but at that time the libraries were maybe 10% of what's there now. I *still* didn't know where to go next. But I was pretty sure I didn't need the SGI multimedia libraries that occupied half the docs <wink>.
... I'll write the library tutorial once I finish the 'from-foo-import-* considered harmful' chapter ;-)
Hmm. Feel free to finish the listcomp PEP too <wink>.
On Fri, Feb 16, 2001 at 04:24:41AM -0500, Tim Peters wrote:
[Thomas Wouters]
... I think what's missing is a library *tutorial*.
How would that differ from the effbot guide (to the std library)?
Not much, I bet, though I have to admit I haven't actually read the effbot guide ;-) It's just that going from the tutorial to the effbot guide (or any other book) is a fair-sized step, given that there are no pointers to them from the tutorial. I can't even *get* to the effbot guide from the documentation page (not with a decent number of clicks, anyway), not even through the PSA bookstore.
If the people who wanted "just a reference" were happy, I don't think David Beazley would have found an audience for his "Python Essential Reference".
Well, I never bought David's reference :) I only ever bought Programming Python, mostly because I saw it in a bookshop while I was in a post-tutorial, pre-usenet state ;) I'm also semi-permanently attached to the 'net, so the online docs at www.python.org are my best friend (next to docstrings, of course.)
A good compromise by my lights-- and perhaps because I only care about the HTML docs, where "size" isn't apparent or a problem for navigation --would be to follow a terse but accurate reference with as many subsections as felt needed, with examples and rationale and tutorial material (has anyone ever figured how to use rexec or bastion from the docs? heh).
Definately +1 on that idea, well received or not it might be by others :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
On Fri, Feb 16, 2001 at 04:24:41AM -0500, Tim Peters wrote:
be to follow a terse but accurate reference with as many subsections as felt needed, with examples and rationale and tutorial material (has anyone ever figured how to use rexec or bastion from the docs? heh).
Thomas Wouters writes:
Definately +1 on that idea, well received or not it might be by others :)
So what sections can I expect you two to write for the Python 2.1 documentation? -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Digital Creations
I must admit to being unfamiliar with all the options available. How well does doctest work if the output of an example or test doesn't lend itself to execution at an interactive prompt? Skip
[Skip]
I must admit to being unfamiliar with all the options available. How well does doctest work if the output of an example or test doesn't lend itself to execution at an interactive prompt?
If an indication of success/failure can't be produced on stdout, doctest is useless. OTOH, if you have any automatable way whatsoever to test a thing, I'm betting you could dream up a way to print yes or no to stdout accordingly. If not, you probably need to work on something other than your testing strategy first <wink>.
michel wrote:
One of the goals of our interface work is to associate unit tests with interfaces. I don't see how doctest can work well with that. I probably need to look at it more, but one of our end goals is to walk up to a component, push a button, and have that components interfaces test the component while the system is live.
My revised approach to unit testing is to use doctest to test the test harness, not the module itself. To handle your case, design the test to access the component via a module global, let the "onclick" code set up that global, and run the test script under doctest. (I did that earlier today, and it sure worked just fine)
Another issue is documentation. I don't know how much documentation doctest has, but PyUnit's documentation is *superb* and there are no suprises, which is absolutely +1.
No surprises? I don't know -- my brain kind of switched off when I came to the "passing method names as strings to the constructor" part. Now, how Pythonic is that on a scale? On the other hand, I also suffer massive confusion whenever I try to read Zope docs, so it's probably just different mind- sets ;-) Cheers /F
"FL" == Fredrik Lundh <fredrik@effbot.org> writes:
Another issue is documentation. I don't know how much documentation doctest has, but PyUnit's documentation is *superb* and there are no suprises, which is absolutely +1.
FL> No surprises? I don't know -- my brain kind of switched off FL> when I came to the "passing method names as strings to the FL> constructor" part. Now, how Pythonic is that on a scale? I think this is one of the issues where there is widespread argeement that a feature is needed. The constructor should assume, in the absence of some other instruction, that any method name that starts with 'test' should be considered a test method. That's about as Pythonic as it gets. Jeremy
participants (18)
-
Andrew Kuchling -
Andrew Kuchling -
barry＠digicool.com -
Chris McDonough -
Fred L. Drake, Jr. -
Fredrik Lundh -
Fredrik Lundh -
Guido van Rossum -
Jeremy Hylton -
Ka-Ping Yee -
M.-A. Lemburg -
Michel Pelletier -
Moshe Zadka -
pf＠artcom-gmbh.de -
Skip Montanaro -
Steve Purcell -
Thomas Wouters -
Tim Peters