I was pleased to see that the 2.1 release schedule lists "unit testing" as one of the open issues. How is this going to be decided? Voting? BDFL fiat?

BDFL fiat: most likely we'll be integrating PyUnit, whose author thinks this is a great idea. We'll be extending it to reduce the amount of boilerplate you have to type for new tests, and to optionally support the style of testing that Quixote's unit test package favors. This style (where the tests are given as string literals) seems to be really impopular with most people I've spoken to, but we're going to support it anyhow because there may also be cases where it's appropriate. I'm not sure however how much we'll get done for 2.1; maybe we'll just integrate the current PyUnit CVS tree. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Mon, Feb 12, 2001 at 04:37:00PM -0500, Guido van Rossum wrote:

I'm not sure however how much we'll get done for 2.1; maybe we'll just integrate the current PyUnit CVS tree.

I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list? --amk

On Mon, Feb 12, 2001 at 04:59:06PM -0500, Jeremy Hylton wrote:

I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit.

* Useful shorthands for common asserts (testing that two sequences are the same ignoring order, for example) * A way to write test cases that doesn't bring the test method to a halt if something raises an unexpected exception * Coverage support (though that would also entail Skip's coverage code getting into 2.1) --amk

Note that doctest.py is part of the 2.1 std library. If you've never used it, pretend I didn't tell you that, and look at the new std library module difflib.py. Would you even guess there *are* unit tests in there? Here's the full text of the new std test test_difflib.py: import doctest, difflib doctest.testmod(difflib, verbose=1) I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use (and, yes, if an unexpected exception is raised, it doesn't stop the rest of the tests from running -- it's in the very nature of "unit tests" that an error in one unit should not prevent other unit tests from running). practicing-for-a-marketing-career-ly y'rs - tim

...

...

...

...

"FL" == Fredrik Lundh writes:

FL> Jeremy wrote:

...

I know that Quixote uses test cases in strings, but it's the thing I like the least about Quixote unittest

FL> like whitespace indentation, it's done that way for a reason. Whitespace indentation is natural and makes code easier to read. Putting little snippets of Python code in string literals passed to exec has the opposite effect. doctest is a nice middle ground, because the code snippets are in a natural setting -- an interactive interpreter setting.

...

I'm not sure how to achieve this or why you would want the test to continue.

FL> same reason you want your compiler to report more than just the FL> first error -- so you can see patterns in the test script's FL> behaviour, so you can fix more than one bug at a time, or fix FL> the bugs in an order that suits you and not the framework, etc. Python's compiler only reports one syntax error for a source file, regardless of how many it finds <0.5 wink>.

...

After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.

FL> jeremy, that's the kind of comment I would have expected from a FL> manager, not from a programmer who has done lots of testing. I don't think there's any reason to be snide. The question is one of granularity: At what level of granularity should the test framework catch exceptions and continue? I'm satisfied with the unit of testing being a method. Jeremy

Andrew, Here's a sample of PyUnit stuff that I think illustrates what you're asking for... from unittest import TestCase, makeSuite, TextTestRunner class Test(TestCase): def setUp(self): self.t = {2:2} def tearDown(self): del self.t def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail) def _getitemfail(self): return self.t[1] def testGetItemSucceeds(self): assert self.t[2] == 2 def main(): suite = makeSuite(Test, 'test') runner = TextTestRunner() runner.run(suite) if __name__ == '__main__': main() Execution happens like this: call setUp() call testGetItemFails() print test results call tearDown() call setUp() call testGetItemSucceeds() print test results call tearDown() end Isn't this almost exactly what you want? Or am I completely missing the point? ----- Original Message ----- From: "Andrew Kuchling" To: Sent: Monday, February 12, 2001 10:52 PM Subject: Re: [Python-Dev] Unit testing (again)

On Mon, Feb 12, 2001 at 06:16:19PM -0500, Jeremy Hylton wrote:

...

We can write a collection of helper functions for this, right? self.verify(sequenceElementsThatSame(l1, l2))

Pretty much; nothing too difficult.

...

Maybe I'd be less opposed if I could understand why it's desirable to continue executing a method where something has already failed unexpectedly. After the first exception, something is broken and

In this style of unit test, you have setup() and shutdown() methods that create and destroy the test objects afresh for each test case, so cases aren't big long skeins of assertions that will all break given a single failure. Instead they're more like 1) call a method that changes an object's state, 2) call accessors or get attributes to check invariants are what you expect. It can be useful to know that .get_parameter_value() raises an exception while .get_parameter_type() doesn't, or whatever.

--amk

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev

Hi, Tim Peters:

Note that doctest.py is part of the 2.1 std library. If you've never used [...] I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use (and, yes, if an unexpected exception is raised, it doesn't stop the rest of the tests from running -- it's in the very nature of "unit tests" that an error in one unit should not prevent other unit tests from running).

practicing-for-a-marketing-career-ly y'rs - tim

[a satisfied customer reports:] I like doctest very much. I'm using it for our company projects a lot. This is a very valuable tool. However Pings latest changes, which turned 'foobar\012' into 'foobar\n' and '\377\376\345' into '\xff\xfe\xe5' has broken some of the doctests in our software. Since we have to keep our code compatible with Python 1.5.2 for at least one, two or may be three more years, it isn't obvious to me how to fix this. I've spend some thoughts about a patch to doctest fooling the string printing output back to the 1.5.2 behaviour, but didn't get around to it until now. :-( Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)

Jeremy Hylton wrote:

...

...

...

...

...

"GvR" == Guido van Rossum writes:

[Andrew writes:]

...

...

I'd really like to have unit testing in 2.1 that I can actually use. PyUnit as it stands is clunky enough that I'd still use the Quixote framework in my code; the advantage of being included with Python would not overcome its disadvantages for me. Have you got a list of desired changes? And should the changes be discussed on python-dev or the PyUnit list?

GvR> I'm just reporting what I've heard on our group meetings. Fred GvR> Drake and Jeremy Hylton are in charge of getting this done. GvR> You can catch their ear on python-dev; I'm not sure about the GvR> PyUnit list.

I'm happy to discuss on either venue, or to hash it in private email. What specific features do you need? Perhaps Steve will be interested in including them in PyUnit.

Fine by private e-mail, though it would be nice if some of the discussions are seen by the PyUnit list because it's a representative community of regular users who probably have a good idea of what makes sense for them. If somebody would like to suggest changes, I can look into how they might get done. Also, I'd love to see what I can do to allay AMK's 'clunkiness' complaints! :-) Best wishes, -Steve -- Steve Purcell, Pythangelist "Life must be simple if *I* can do it" -- me

...

...

...

...

"FL" == Fredrik Lundh writes:

...

Another issue is documentation. I don't know how much documentation doctest has, but PyUnit's documentation is *superb* and there are no suprises, which is absolutely +1.

FL> No surprises? I don't know -- my brain kind of switched off FL> when I came to the "passing method names as strings to the FL> constructor" part. Now, how Pythonic is that on a scale? I think this is one of the issues where there is widespread argeement that a feature is needed. The constructor should assume, in the absence of some other instruction, that any method name that starts with 'test' should be considered a test method. That's about as Pythonic as it gets. Jeremy

...

...

...

...

"JH" == Jeremy Hylton writes:

JH> Whitespace indentation is natural and makes code easier to JH> read. Putting little snippets of Python code in string JH> literals passed to exec has the opposite effect. Especially because requiring the snippets to be in strings means editing them with a Python-aware editor becomes harder. JH> doctest is a nice middle ground, because the code snippets are JH> in a natural setting -- an interactive interpreter setting. And at least there, I can for the most part just cut-and-paste the output of my interpreter session into the docstrings. -Barry

jeremy wrote:

FL> like whitespace indentation, it's done that way for a reason.

Whitespace indentation is natural and makes code easier to read. Putting little snippets of Python code in string literals passed to exec has the opposite effect.

Only if you're using large snippets. ...just like whitespace indentation makes things harder it you're mixing tabs and spaces, or prints a file with the wrong tab setting, or cuts and pastes code between editors with different tab settings. In both cases, the solution is simply "don't do that"

doctest is a nice middle ground, because the code snippets are in a natural setting -- an interactive interpreter setting.

They're still inside a string...

Python's compiler only reports one syntax error for a source file, regardless of how many it finds <0.5 wink>.

Sure, but is that because user testing has shown that Python programmers (unlike e.g. C programmers) prefer to see only one bug at a time, or because it's convenient to use exceptions also for syntax errors? Would a syntax-checking editor be better if it only showed one syntax error, even if it found them all?

...

After the first exception, something is broken and needs to be fixed, regardless of whether subsequent lines of code work.

FL> jeremy, that's the kind of comment I would have expected from a FL> manager, not from a programmer who has done lots of testing.

I don't think there's any reason to be snide.

Well, I first wrote "taken out of context, that's the kind of comment" but then I noticed that it wasn't really taken out of context. And in full context, it still looks a bit arrogant: why would Andrew raise this issue if *he* didn't want more granularity? ::: But having looked everything over one more time, and having ported a small test suite to doctest.py, I'm now -0 on adding more test frame- works to 2.1. If it's good enough for tim... (and -1 if adding more frameworks means that I have to use them ;-). Cheers /F

[/F]

But having looked everything over one more time, and having ported a small test suite to doctest.py, I'm now -0 on adding more test frameworks to 2.1. If it's good enough for tim...

I'm not sure that it is, but I have yet to make time to look at the others. It's no secret that I love doctest, and, indeed, in 20+ years of industry pain, it's the only testing approach I didn't drop ASAP. I still use it for all my stuff, and very happily. But! I don't do anything with the web or GUIs etc -- I'm an algorithms guy. Most of the stuff I work with has clearly defined input->output relationships, and capturing an interactive session is simply perfect both for documenting and testing such stuff. It's also the case that I weight the "doc" part of "doctest" more heavily than the "test" part, and when Peter or Guido say that, e.g., the reliance on exact output match is "a problem", I couldn't disagree more strongly. It's obvious to Guido that dict output may come in any order, but a doc *reader* in a hurry is at best uneasy when documented output doesn't match actual output exactly. That's not something I'll yield on. [Andrew]

def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail)

def _getitemfail(self): return self.t[1]

[vs]

self.test_exc('self.t[1]', KeyError)

My brain doesn't grasp either of those at first glance. But everyone who has used Python a week grasps this: class C: def __getitem__(self, i): """Return the i'th item. i==1 raises KeyError. For example, >>> c = C() >>> c[0] 0 >>> c[1] Traceback (most recent call last): File "x.py", line 20, in ? c[1] File "x.py", line 14, in __getitem__ raise KeyError("bad i: " + `i`) KeyError: bad i: 1 >>> c[-1] -1 """ if i != 1: return i else: raise KeyError("bad i: " + `i`) Cute: Python changed the first line of its traceback output (used to say "Traceback (innermost last):"), and current doctest wasn't expecting that. For *doc* purposes, it's important that the examples match what Python actually does, so that's a bug in doctest.

Fredrik Lundh wrote:

...

Cute: Python changed the first line of its traceback output (used to say "Traceback (innermost last):"), and current doctest wasn't expecting that.

which reminds me... are there any chance of getting a doctest that can survives its own test suite under 1.5.2, 2.0, and 2.1?

the latest version blows up under 1.5.2 and 2.0:

***************************************************************** Failure in example: 1/0 from line #155 of doctest Expected: ZeroDivisionError: integer division or modulo by zero Got: ZeroDivisionError: integer division or modulo 1 items had failures: 1 of 8 in doctest ***Test Failed*** 1 failures.

Since exception message are not defined anywhere I'd suggest to simply ignore them in the output. About the traceback output format: how about adding some re support instead of using string.find() ?! -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[MAL]

Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.

Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.

About the traceback output format: how about adding some re support instead of using string.find() ?!

Why? I never use regexps where simple string matches work, and neither should you <wink>.

[Michel Pelletier]

... A personal issue for me is that it overloads the docstring, no biggy, but it's just a personal nit I don't particularly like about doctest.

No. The docstring remains documentation. But documentation without examples generally sucks, due to the limitations of English in being precise. A sharp example can be worth 1,000 words. doctest is being used as *intended* to the extent that the embedded examples are helpful for documentation purposes. doctest then guarantees the examples continue to work exactly as advertised over time (and they don't! examples *always* get out of date, but without (something like) doctest they never get repaired). As I suggested at the start, read the docstrings for difflib.py: the examples are an integral part of the docs, and you shouldn't get any sense that they're there "just for testing" (if you do, the examples are poorly chosen, or poorly motivated in the surrounding commentary). Beyond that, doctest will also execute any code it finds in the module.__test__ dict, which maps arbitrary names to arbitrary strings. Anyone using doctest primarily as a testing framework should stuff their test strings into __test__ and leave the docstrings alone.

Another issue is documentation. I don't know how much documentation doctest has,

Look at its docstrings -- they not only explain it in detail, but contain examples of use that doctest can check <wink>.

Tim Peters wrote:

[MAL]

...

Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.

Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.

But what it does write to the console changes with every release (e.g. just take the repr() changes for strings with non-ASCII data)... this simply breaks you test suite every time Writing Python programs which work on Python 1.5-2.1 which at the same time pass the doctest unit tests becomes impossible. The regression suite (and most other Python software) catches exceptions based on the exception class -- why isn't this enough for your doctest.py checks ? nit-pickling-ly, -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

On Tue, 13 Feb 2001, Fredrik Lundh wrote:

tim wrote:

...

I will immodestly claim that if doctest is sufficient for your testing purposes, you're never going to find anything easier or faster or more natural to use

you know, just having taken another look at doctest and the unit test options, I'm tempted to agree.

I also agree that doctest is the bee's knees, but I don't think it is quite as useful for us as PyUnit (for other people, I'm sure it's very useful). One of the goals of our interface work is to associate unit tests with interfaces. I don't see how doctest can work well with that. I probably need to look at it more, but one of our end goals is to walk up to a component, push a button, and have that components interfaces test the component while the system is live. I immagine this involving a bit of external framework at the interface level that would be pretty easy with PyUnit, I've only seen one example of doctest and it looks like you run it against an imported module. I don't see how this helps us with our (DC's) definition of components. A personal issue for me is that it overloads the docstring, no biggy, but it's just a personal nit I don't particularly like about doctest. Another issue is documentation. I don't know how much documentation doctest has, but PyUnit's documentation is *superb* and there are no suprises, which is absolutely +1. Quixote's documentation seems very thin (please correct me if I'm wrong). PyUnit's documentation goes beyond just explaning the software into explaining common patterns and unit testing philosophies. -Michel

[MAL]

Since exception message are not defined anywhere I'd suggest to simply ignore them in the output.

[Tim]

Virtually nothing about Python's output is clearly defined, and for doc purposes I want to capture what Python actually does.

[MAL]

But what it does write to the console changes with every release (e.g. just take the repr() changes for strings with non-ASCII data)...

So now you don't want to test exception messages *or* non-exceptional output either. That's fine, but you're never going to like doctest if so.

this simply breaks you test suite every time

I think you're missing the point: it breaks your *docs*, if they contain any examples that rely on such stuff. doctest then very helpfully points out that your docs-- no matter how good they were before --now suck, because they're now *wrong*. It's not interested in assigning blame for that, it's enough to point out that they're now broken (else they'll never get fixed!).

Writing Python programs which work on Python 1.5-2.1 which at the same time pass the doctest unit tests becomes impossible.

Not true. You may need to rewrite your examples, though, so that your *docs* are accurate under all the releases you care about. I don't care if that drives you mad, so long as it prevents you from screwing your users with inaccurate docs.

The regression suite (and most other Python software) catches exceptions based on the exception class -- why isn't this enough for your doctest.py checks ?

Because doctest is primarily interested in ensuring that non-exceptional cases continue to work exactly as advertised. Checking that, e.g.,

...

...

fac(5) 120

still works is at least 10x easier to live with than writing crap like if fac(5) != 120: raise TestFailed("Something about fac failed but it's too " "painful to build up some string here " "explaining exactly what -- try single-" "stepping through the test by hand until " "this raise triggers.") That's regrtest.py-style testing, and if you think that's pleasant to work with you must never have seen a std test break <0.7 wink>. When a doctest'ed module breaks, the doctest output pinpoints the failure precisely, without any work on your part beyond simply capturing an interactive session that shows the results you expected.

nit-pickling-ly,

Na, you're trying to force doctest into a mold it was designed to get as far away from as possible. Use it for its intended purpose, then gripe. Right now you're complaining that the elephant's eyes are the wrong color while missing that it's actually a leopard <wink>.

I must admit to being unfamiliar with all the options available. How well does doctest work if the output of an example or test doesn't lend itself to execution at an interactive prompt? Skip

[Skip]

I must admit to being unfamiliar with all the options available. How well does doctest work if the output of an example or test doesn't lend itself to execution at an interactive prompt?

If an indication of success/failure can't be produced on stdout, doctest is useless. OTOH, if you have any automatable way whatsoever to test a thing, I'm betting you could dream up a way to print yes or no to stdout accordingly. If not, you probably need to work on something other than your testing strategy first <wink>.

On Tue, 13 Feb 2001 20:24:00 -0500, "Tim Peters" wrote:

Not me -- there's nothing in them that I as a potential user don't need to know. But then I think the Library docs are too terse in general. Indeed, Fredrick makes part of his living selling a 300-page book supplying desperately needed Library examples <0.5 wink>.

I'm sorry, Tim, that's just too true. I want to explain my view about how it happened (I wrote some of them, and if you find a particularily terse one, just assume it's me) -- I write tersely. My boss yelled at me when doing this at work, and I redid all my internal documentation -- doubled the line count, beefed up with examples, etc. He actually submitted a bug in the internal bug tracking system to get me to do that ;-) So, I suggest you do the same -- there's no excuse for terseness, other then not-having-time, so it's really important that bugs like that are files. Something like "documentation for xxxlib is too terse". I can't promise I'll fix all these bugs, but I can try ;-) -- For public key: finger moshez@debian.org | gpg --import "Debian -- What your mother would use if it was 20 times easier" LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive.

On Wed, Feb 14, 2001 at 06:57:35PM +0200, Moshe Zadka wrote:

On Tue, 13 Feb 2001 20:24:00 -0500, "Tim Peters" wrote:

...

Not me -- there's nothing in them that I as a potential user don't need to know. But then I think the Library docs are too terse in general. Indeed, Fredrick makes part of his living selling a 300-page book supplying desperately needed Library examples <0.5 wink>.

I'm sorry, Tim, that's just too true.

You should be appologizing to Fred, not Tim :) While I agree with the both of you, I'm not sure if expanding the library reference is going to help the problem. I think what's missing is a library *tutorial*. The reference is exactly that, a reference, and if we expand the reference we'll end up cursing it ourself, should we ever need it. (okay, so noone here needs the reference anymore <wink> except me, but when looking at the reference, I like the terse descriptions of the modules. They're just reminders anyway.) I remember when I'd finished the Python tutorial and wondered where to go next. I tried reading the library reference, but it was boring and most of it not interesting (since it isn't built up to go from useful/common -> rare, but just a list of all modules ordered by 'service'.) I ended up doing the slow and cheap version of Fredrik's book: reading python-list ;) I'll write the library tutorial once I finish the 'from-foo-import-* considered harmful' chapter ;-) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

[Moshe Zadka]

We had a DevDay, we have a sig, we have a PEP. None of this seems to help --

[Ka-Ping Yee]

What are you talking about? There is an implementation and it works.

There are many implementations "that work". But we haven't picked one. What's the standard markup for Python docstrings? There isn't! That's what he's talking about. This is especially bizarre because it's been clear for *years* that some variant of structured text would win in the end, but nobody playing the game likes all the details of anyone else's set of (IMO, all overly elaborate) conventions, so the situation for users is no better now than it was the day docstrings were added. Tibs's latest (and ongoing) attempt to reach a consensus can be found here: http://www.tibsnjoan.demon.co.uk/docutils/STpy.html The status of its implementation here: http://www.tibsnjoan.demon.co.uk/docutils/status.html Not close yet. In the meantime, Perlers have been "suffering" with a POD spec about 3% the size of the proposed Python spec; I guess their only consolation is that POD docs have been in universal use for years <wink>. while-ours-is-that-we'll-get-to-specify-non-breaking-spaces-someday- despite-that-not-1-doc-in-100-needs-them-ly y'rs - tim

[Ka-Ping Yee]

That's exactly the point i'm trying to make. There isn't any markup format enforced by pydoc, precisely because it isn't worth the strife. Moshe seemed to imply that the set of deployable documentation tools was empty, and i take issue with that. His post also had an tone of hopelessness about the topic that i wanted to counter immediately.

Most programmers are followers in this matter, and I agree with Moshe on this point: until something is Officially Blessed, Python programmers will stay away from every gimmick in unbounded droves. I personally don't care whether markup is ever defined, because I already gave up on it. But I-- like you --won't wait forever for anything. We're not the norm. the-important-audience-isn't-pythondev-it's-pythonlist-ly y'rs - tim

On Fri, Feb 16, 2001 at 04:24:41AM -0500, Tim Peters wrote:

[Thomas Wouters]

...

... I think what's missing is a library *tutorial*.

How would that differ from the effbot guide (to the std library)?

Not much, I bet, though I have to admit I haven't actually read the effbot guide ;-) It's just that going from the tutorial to the effbot guide (or any other book) is a fair-sized step, given that there are no pointers to them from the tutorial. I can't even *get* to the effbot guide from the documentation page (not with a decent number of clicks, anyway), not even through the PSA bookstore.

If the people who wanted "just a reference" were happy, I don't think David Beazley would have found an audience for his "Python Essential Reference".

Well, I never bought David's reference :) I only ever bought Programming Python, mostly because I saw it in a bookshop while I was in a post-tutorial, pre-usenet state ;) I'm also semi-permanently attached to the 'net, so the online docs at www.python.org are my best friend (next to docstrings, of course.)

A good compromise by my lights-- and perhaps because I only care about the HTML docs, where "size" isn't apparent or a problem for navigation --would be to follow a terse but accurate reference with as many subsections as felt needed, with examples and rationale and tutorial material (has anyone ever figured how to use rexec or bastion from the docs? heh).

Definately +1 on that idea, well received or not it might be by others :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

[Fred L. Drake, Jr.]

So what sections can I expect you two to write for the Python 2.1 documentation?

I'm waiting for you to clear the backlog of the ones I've already written <wink>.