On 05/12/2013 08:15 PM, Stephen J. Turnbull wrote:

Ethan Furman writes:

...

I will certainly ask for advice on which modules to spend my time on. I know enums are not a cure-all, but they are great for debugging and interactive work.

Especially in new code where they are used throughout. Not so in the existing stdlib, I expect.

Perhaps not to somebody who is already well versed in it. It would be very helpful to me. ;)

The concrete limitation on that theory that I envision with retrofitting the stdlib is that cooperative modules (those that call into and are called from the module being converted to use enums) are going to be expecting values, not enums. So you need to convert return values and arguments, and not only do you *not* get the benefit of enum reprs in the cooperating modules, but you introduce additional complexity in the converted module. Nor can you say "OK, it's more than I expected but I'll do the whole stdlib," because you don't know who is calling into or supplying callbacks to the stdlib modules.

Well, somebody else might, but I know how much (little?) time I have. It'll be great to have new modules use Enums; retrofitted modules should use Psuedonums (okay, I made that word up -- it's supposed to be an Enum but with some other type mixed in so it's no longer a pure Enum, more like a psuedo enum). As I was saying, if tkinter was up for conversion it would just be to StrEnum, and that would mostly consist of adding the enumeration at the top, exporting it to global, then browsing for locations where the string value was used and removing the quotes. Of course, having said that I'm sure somebody will chime in with "yes, but..."

In many cases you could convert code to use IntEnum instead of Enum, preserving the old semantics, and probably not needing to convert return values, but again I expect the benefits of Enum-ness would attenuate quickly as cooperating code converts them to int internally.

Hmmm... yeah, that would suck. Well, I'm sure I can help in others ways if this doesn't pan out. Maybe some other new module that Raymond objects to. ;) -- ~Ethan~

On May 12, 2013, at 8:26 PM, Eli Bendersky wrote:

Thanks for the insights, Raymond. I don't think anyone is planning on rushing anything. We still have to get the enum module itself committed and a serious review process has just started for that, so it will take time.

There's no general "let's replace all constants with enums" TODO item that I know of. It's my hope that such changes will happen very gradually and only when deemed important and useful by core developers.

Ethan's email suggests that against my advice he is in-fact going to go through the standard library, applying enums quite broadly. That is somewhat at odds with the notions of holistic refactoring and gradual change. Nor does it reflect sufficient appreciation for concerns about maintenance issues, code stability, the effect on 2-to-3 migration, doctests, performance, the wishes of the module authors, or whether users will see any actual benefits (particularly for internal constants). I fully understand the enthusiasm to take the car out for a spin, but the standard library isn't really a great place for experimentation. And "trying to learn the standard library" isn't a good rationale for making extensive changes to it. So, please do help make sure there is some restraint and careful consideration. Raymond

Le Mon, 13 May 2013 00:06:52 -0700, Raymond Hettinger a écrit :

On May 12, 2013, at 8:26 PM, Eli Bendersky wrote:

...

Thanks for the insights, Raymond. I don't think anyone is planning on rushing anything. We still have to get the enum module itself committed and a serious review process has just started for that, so it will take time.

There's no general "let's replace all constants with enums" TODO item that I know of. It's my hope that such changes will happen very gradually and only when deemed important and useful by core developers.

Ethan's email suggests that against my advice he is in-fact going to go through the standard library, applying enums quite broadly.

It probably won't go in without reviews, so there's no need to be too concerned IMHO. The fact that one of the enum classes is an int subclass should make the behaviour changes minimal, if any. Regards Antoine.

Bad: doctests. On Tue, May 14, 2013 at 5:08 AM, Steven D'Aprano wrote:

On 14/05/13 16:51, Gregory P. Smith wrote: [...]

This sounds like a feature request for doctest. doctest could be educated

...

about enums and automatically compare to the integer value for such cases.

Please no. Enums are not special enough to break the rules.

Good: "Doctests look at the object's repr."

Bad: "Doctests look at an object's repr, unless the object is an Enum, when it will look at the enum's value."

If I want a test that checks the enum's value, then I will write a doctest that explicitly checks the enum's value.

-- Steven

______________________________**_________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** greg%40krypto.orghttp://mail.python.org/mailman/options/python-dev/greg%40krypto.org

On Sun, May 19, 2013 at 4:41 PM, Raymond Hettinger wrote:

nicer repr" is worth "Sorry, I broke your tests, made your published examples out of date, and slowed down your code."

While the first two considerations are always potentially applicable when using enums, the latter should only be true for code that uses str() and repr() a lot. For other operations, int-based enums shouldn't add any more overhead than namedtuple does for tuples. I agree with basically everything you said, but I don't want "enums are slower than normal integers" to become a meme - there really shouldn't be a speed difference for any arithmetic operations when using IntEnum. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/19/2013 10:48 AM, Guido van Rossum wrote:

Anyway, if you're doing arithmetic on enums you're doing it wrong.

Hmm, bitwise operations, even? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGZMoAACgkQ+gerLs4ltQ79qwCgq6FWTl6ZDIDctBg69In47YB2 +FkAnj5cEyw1szQ8GCl6aQ9+aGKcwp3y =d/xt -----END PGP SIGNATURE-----

On 20 May 2013 06:25, "Terry Jan Reedy" wrote:

On 5/19/2013 4:13 PM, Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 05/19/2013 10:48 AM, Guido van Rossum wrote:

...

Anyway, if you're doing arithmetic on enums you're doing it wrong.

Hmm, bitwise operations, even?

Those are logic, not arithmetic as usually understood. (The fact that one

can do each with the other is beside the point.) I consider those to be binary arithmetic, but it's a fair point. The word I really wanted was "comparison" anyway, since the main intended uses of enums are as flags, lookup keys and marker values. Cheers, Nick.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 5/19/2013 7:22 PM, Mark Janssen wrote:

On Sun, May 19, 2013 at 1:13 PM, Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 05/19/2013 10:48 AM, Guido van Rossum wrote:

...

Anyway, if you're doing arithmetic on enums you're doing it wrong. Hmm, bitwise operations, even? I think it's rather pointless to do bitwise operations on python enums. We're not that close to the machine.

It makes sense if the enums represent bit-oriented values that will be used close to the machine. Python is used in many disciplines. --Ned.

MarkJ Tacoma, Washington _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com

On Mon, May 20, 2013 at 10:14 AM, Antoine Pitrou wrote:

On Sun, 19 May 2013 20:04:03 -0400 Ned Batchelder wrote:

...

On 5/19/2013 7:22 PM, Mark Janssen wrote:

...

On Sun, May 19, 2013 at 1:13 PM, Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 05/19/2013 10:48 AM, Guido van Rossum wrote:

...

Anyway, if you're doing arithmetic on enums you're doing it wrong. Hmm, bitwise operations, even? I think it's rather pointless to do bitwise operations on python enums. We're not that close to the machine.

It makes sense if the enums represent bit-oriented values that will be used close to the machine. Python is used in many disciplines.

Then it's up to the library writer to not use enums in that case. (assuming the performance of bitwise operations is critical here, which I doubt)

This is the point I was trying to make: once you use IntEnum (as you would in any case where you need bitwise operators), Enum gets out of the way for everything other than __str__, __repr__, and one other slot (that escapes me for the moment...). The metaclass does extra work at definition time so there shouldn't be any runtime overhead - the slots should be inherited directly from the non-Enum parent. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5/19/2013 9:08 PM, Ethan Furman wrote:

On 05/19/2013 05:24 PM, Nick Coghlan wrote:

...

This is the point I was trying to make: once you use IntEnum (as you would in any case where you need bitwise operators), Enum gets out of the way for everything other than __str__, __repr__, and one other slot (that escapes me for the moment...).

__getnewargs__ and __new__

But if you do math, the result is no longer an Enum of any type.

And thus completely loses the debugging benefits of having a nice __repr__. IntEnum isn't useful for bitfields.

On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

On May 14, 2013, at 9:39 AM, Gregory P. Smith wrote:

Bad: doctests.

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

Don't doctests intended for CPython not work on Jython, Pypy, IronPython... I've been avoiding them for this reason.

On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

On May 14, 2013, at 9:39 AM, Gregory P. Smith wrote:

Bad: doctests.

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

So long as doctests insist on comparing the repr of things being the number one practice that people use when writing them there is no other position I can hold on the matter. reprs are not stable and never have been. ordering changes, hashes change, ids change, pointer values change, wording and presentation of things change. none of those side effect behaviors were ever part of the public API to be depended on. That one can write doctests that don't depend on such things as the repr doesn't ultimately matter because the easiest thing to do, as encouraged by examples that are pasted from an interactive interpreter session into docs, is to have the interactive interpreter show the repr and not add code to check things in a accurate-for-testing manner that would otherwise make the documentation harder for a human to read. Instead, we should be clear about their primary purpose which is to test

the examples given in docstrings. In many cases, there is a great deal of benefit to docstrings that have worked-out examples (see the docstrings in the decimal module for example). In such cases it is also worthwhile to make sure those examples continue to match reality. Doctests are a vehicle for such assurance. In other words, doctests have a perfectly legitimate use case.

I really do applaud the goal of keeping examples in documentation up to date. But doctest as it is today is the wrong approach to that. A repr mismatch does not mean the example is out of date. We should continue to encourage users to make thorough unit tests

and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

Not quite, they at least tested something (yay!) but it is uncomfortably close to a worst practice. It means someone else needs to come understand the body of code containing this doctest when they make an unrelated change that triggered a behavior change as a side effect that the doctested code may or may not actually depend on but does not actually declare its intent one way or another for the purposes of being a readable example rather than accurate test. bikeshed colors: If doctest were never called a test but instead were called docchecker to not imply any testing aspect that might've helped (too late? the cat's out of the bag). Or if it never compared anything but simply ran the example code to generate and update the doc examples from the statements with the current actual results of execution instead of doing string comparisons... (ie: more of an documentation example "keep up to date" tool) Another meme that I hope dispel is the notion that the core developers

are free to break user code (such as doctests) if they believe the users aren't coding in accordance with best practices. Our goal is to improve their lives with our modifications, not to make their lives more difficult.

Educating users how to apply best practices and making that easier for them every step of the way is a primary goal. Occasionally we'll have to do something annoying in the process but we do try to limit that. In my earlier message I suggested that someone improve doctest to not do dumb string comparisons of reprs. I still think that is a good goal if doctest is going to continue to be promoted. It would help alleviate many of the issues with doctests and bring them more in line with the issues many people's regular unittests have. As Tres already showed in an example, individual doctest using projects jump through hoops to do some of that today; centralizing saner repr comparisons for less false failures as an actual doctest feature just makes sense. Successful example: We added a bunch of new comparison methods to unittest in 2.7 that make it much easier to write tests that don't depend on implementation details such as ordering. Many users prefer to use those new features; even with older Python's via unittest2 on pypi. It doesn't mean users always write good tests, but a higher percentage of tests written are more future proof than they were before because it became easier. Currently, we face an adoption problem with Python 3. At PyCon,

an audience of nearly 2500 people said they had tried Python 3 but weren't planning to convert to it in production code. All of the coredevs are working to make Python 3 more attractive than Python 2, but we also have to be careful to not introduce obstacles to conversion. Breaking tests makes it much harder to convert (especially because people need to rely on their tests to see if the conversion was successful).

Idea: I don't believe anybody has written a fixer for lib2to3 that applies fixers to doctests. That'd be an interesting project for someone. Now you've got me wondering what Python would be like if repr, `` and __repr__ never existed as language features. Upon first thoughts, I actually don't see much downside (no, i'm not advocating making that change). Something to ponder. -gps

On May 19, 2013 4:31 PM, "Benjamin Peterson" wrote:

2013/5/19 Gregory P. Smith :

...

Idea: I don't believe anybody has written a fixer for lib2to3 that

applies

...

fixers to doctests. That'd be an interesting project for someone.

2to3 can operate on doctests, though it doesn't do anything different to them than it does to normal sourcecode.

Oh cool. I didn't realize that already existed!

-- Regards, Benjamin

On 20/05/13 09:27, Gregory P. Smith wrote:

On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

...

On May 14, 2013, at 9:39 AM, Gregory P. Smith wrote:

Bad: doctests.

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

So long as doctests insist on comparing the repr of things being the number one practice that people use when writing them there is no other position I can hold on the matter. reprs are not stable and never have been.

I think this *massively* exaggerates the "problem" with doc tests. I make heavy use of them, and have no problem writing doc tests that work in code running over multiple versions, including from 2.4 through 3.3. Objects that I write myself, I control the repr and can make it as stable as I wish. Many built-in types also have stable reprs. The repr for small ints is not going to change, the repr for floats like 0.5, 0.25, 0.125 etc. are stable and predictable, lists and tuples and strings all have stable well-defined reprs. Dicts are a conspicuous counter-example, but there are trivial work-arounds. Doc tests are not limited to a simple-minded "compare the object's repr". You can write as much, or as little, scaffolding around the test as you need. If the scaffolding becomes too large, that's a sign that the test doesn't belong in documentation and should be moved out, perhaps into a unit test, or perhaps into a separate "literate testing" document that can be as big as necessary without overwhelming the doc string.

ordering changes, hashes change, ids change, pointer values change, wording and presentation of things change. none of those side effect behaviors were ever part of the public API to be depended on.

Then don't write doctests that depend on those things. It really is that simple. There's no rule that says doctests have to test the entire API. Doctests in docstrings are *documentation first*, so you write tests that make good documentation. The fact that things that are not stable parts of the API can be tested is independent of the framework you use to do the testing. If I, as an ignorant and foolish developer, wrote a unit test like this: class MyDumbTest(unittest.TestCase): def testSpamRepr(self): x = Spam(arg) self.assertEquals(repr(x), "") we shouldn't conclude that "unit tests are bad", but that MyDumbTest is bad and needs to be fixed. Perhaps the fix is to re-write the test to care less about the exact repr. (Doctest's ellipsis directive is excellent for that.) Perhaps the fix is to give the Spam object a stable repr that doesn't suck. Or perhaps the fix is to just say, this doesn't need to be a test at all. (And doctest has a directive for that too.) They are all good solutions to the "problem" of unit testing things that aren't part of the API, and they are also good solutions to the same problem when it comes to doctests. [...]

I really do applaud the goal of keeping examples in documentation up to date. But doctest as it is today is the wrong approach to that. A repr mismatch does not mean the example is out of date.

No, it means that either the test was buggy, or the test has failed. I must admit that I don't understand what you think happens with doc testing in practice. You give the impression that there are masses of doc tests being written that look like this:

...

...

x = Spam(arg) print(x)

and therefore the use of doc tests are bad because it leads to broken tests. But I don't understand why you think that nobody has noticed that this test will have failed right from the start, and will have fixed it immediately. I suppose it is possible that some people write doc tests but never run them, not even once, but that's no different from those who write unit tests but never run them. They're hardly representative of the average developer, who either doesn't write tests at all, or who both writes and runs them and will notice if they fail. [...]

In my earlier message I suggested that someone improve doctest to not do dumb string comparisons of reprs. I still think that is a good goal if doctest is going to continue to be promoted. It would help alleviate many of the issues with doctests and bring them more in line with the issues many people's regular unittests have. As Tres already showed in an example, individual doctest using projects jump through hoops to do some of that today; centralizing saner repr comparisons for less false failures as an actual doctest feature just makes sense.

If a test needs to jump through hoops to work, then it doesn't belong as a test in the doc string. It should be a unit test, or possibly a separate test file that can be as big and complicated as needed. If you want to keep it as an example, but not actually run it, doctest has a skip directive. There's no need to complicate doctest by making it "smarter" (and therefore more likely to be buggy, harder to use, or both).

Successful example: We added a bunch of new comparison methods to unittest in 2.7 that make it much easier to write tests that don't depend on implementation details such as ordering. Many users prefer to use those new features; even with older Python's via unittest2 on pypi.

And that's great, it really is, I'm not being sarcastic. But unit testing is not in competition to doc testing, they are complimentary, not alternatives. If you're not using both, then you're probably missing out on something. -- Steven

Gregory P. Smith writes:

I really do applaud the goal of keeping examples in documentation up to date. But doctest as it is today is the wrong approach to that. A repr mismatch does not mean the example is out of date.

Of course it does. The user sees something in the doc that's different from what his interpreter tells him. That may not bother a long-time user of the module, or one who hangs out on python-commits (uh-huh, uh-huh, yeah, right), but it worries new ones, and it should. "What else may have changed?" is what they should be thinking. Also, there are many cases where the output of a function is defined by some external protocol: XML, JSON, RFC xxxx, etc. Here doctests are very valuable. There are lots of testing applications where doctests suck. There are lots of testing applications where doctests are pretty much the optimal balance between ease of creation and ease of maintenance. I wouldn't be surprised if there are applications (RAD?) where *creating* tests as doctests and *converting* to a more precisely-specified framework in maintenance is best practice. Maybe somebody (not me, I do far too little testing even with doctests :-( ) should write an informational PEP.

Hi ! :) I'll be replying some individual messages in this thread in spite of putting my replies in the right context . Sorry if I repeat something , or this makes the thread hard to read . Indeed , IMHO this is a subject suitable to discuss in TiP ML . On 5/19/13, Gregory P. Smith wrote:

On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

...

On May 14, 2013, at 9:39 AM, Gregory P. Smith wrote:

Bad: doctests.

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

So long as doctests insist on comparing the repr of things being the number one practice that people use when writing them there is no other position I can hold on the matter. reprs are not stable and never have been. ordering changes, hashes change, ids change, pointer values change, wording and presentation of things change. none of those side effect behaviors were ever part of the public API to be depended on.

«Bad doctests» slogan is not positive because the subliminal message for new users is «there's something wrong with that ... let's better not use it» . IMHO that's not true ; doctest is an incredibly awesome testing framework for delta assertions and there is nothing wrong with the philosophy behind that module and its intent . This surfaces an issue I've noticed years ago wrt doctest module (so, yes , it's obvious there's an issue ;) . The way I see it this is more about the fact that module frontend does not offer the means to benefit from all the possibilities of doctest classes in the backend (e.g. output checkers , doctest runners, ...)

That one can write doctests that don't depend on such things as the repr doesn't ultimately matter because the easiest thing to do, as encouraged by examples that are pasted from an interactive interpreter session into docs, is to have the interactive interpreter show the repr and not add code to check things in a accurate-for-testing manner that would otherwise make the documentation harder for a human to read.

This is something that could be easily mitigated by a custom output checker . In the end , in docs there is no difference between output messages like '' or '' (i.e. some deterministic label like computed hex number or anything else ...) . You might also avoid printing repr(s)

...

Instead, we should be clear about their primary purpose which is to test the examples given in docstrings. In many cases, there is a great deal of benefit to docstrings that have worked-out examples (see the docstrings in the decimal module for example). In such cases it is also worthwhile to make sure those examples continue to match reality. Doctests are a vehicle for such assurance. In other words, doctests have a perfectly legitimate use case.

I really do applaud the goal of keeping examples in documentation up to date. But doctest as it is today is the wrong approach to that. A repr mismatch does not mean the example is out of date.

... and I confess I never use doctest «as it is today» in stdlib . So , you are right .

We should continue to encourage users to make thorough unit tests

...

and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

Not quite, they at least tested something (yay!) but it is uncomfortably close to a worst practice.

I disagree . IMO what is a bad practice is to spread the rumor that «doctests are evil» rather than saying «doctest module has limitations»

It means someone else needs to come understand the body of code containing this doctest when they make an unrelated change that triggered a behavior change as a side effect that the doctested code may or may not actually depend on but does not actually declare its intent one way or another for the purposes of being a readable example rather than accurate test.

I see no problem in keeping both these aspects .

bikeshed colors: If doctest were never called a test but instead were called docchecker to not imply any testing aspect

No way ! ( IMHO ) I just wrote dutest [1]_ framework , built on top of doctest and unittest , that does the following (among other things) : 1. Implements unittest loaders for doctests 2. Allows for customizing output checkers , doctest runners , ... anything you might find in the backend * For instance , replacing default test runner and output checkers might be useful to write delta assertions for command-line scripts 3. Tightly integrated with unittest (e.g. custom TestSuite(s) ...) 4. Access to unittest test case in special __tc__ variable , so all known assertion methods are handy ootb 5. Encapsulate doctest setup code (setUp , tearDown for doctests) e.g. to make doctests like the following work without actually writing in the docs all steps needed to load """Provided that «complex scenario» is prepared and satisfies some preconditions (performed in hidden unittest setup methods)

...

...

do_something() do_something_else() ... """

I can report usage of dutest module in practice to test non-trivial web apps . I've even started to write a micro-framework on top of it to test Trac plugins (based on 3. , 4. , 5. above + twill ) , and it's possible to do the same thing for other web frameworks . Use cases might not be restricted to web apps ; like I mentioned above , custom output checkers + doctest runners will make it possible to test cli . ... so , in a few words , delta assertions (like doctests) are about testing and are much more than a doc checker.

that might've helped (too late? the cat's out of the bag).

just to create more confusion ... afaict

Or if it never compared anything but simply ran the example code to generate and update the doc examples from the statements with the current actual results of execution instead of doing string comparisons... (ie: more of an documentation example "keep up to date" tool)

Considering what I mentioned above , I disagree ... [...]

In my earlier message I suggested that someone improve doctest to not do dumb string comparisons of reprs.

FWIW , that's possible with dutest [1]_ , indeed one of the main goals it was created for .

I still think that is a good goal if doctest is going to continue to be promoted. It would help alleviate many of the issues with doctests and bring them more in line with the issues many people's regular unittests have.

As Tres already showed in an example, individual doctest using projects jump through hoops to do some of that today; centralizing saner repr comparisons for less false failures as an actual doctest feature just makes sense.

+1 Besides , IMHO doctest is very useful for APIs and lowers the barrier for writing testing code for people not used to XUnit philosophy .

Successful example: We added a bunch of new comparison methods to unittest in 2.7 that make it much easier to write tests that don't depend on implementation details such as ordering. Many users prefer to use those new features; even with older Python's via unittest2 on pypi. It doesn't mean users always write good tests, but a higher percentage of tests written are more future proof than they were before because it became easier.

all this is possible with dutest [1]_ . Assertion methods of the underlying unittest test case are available in __tc__ variable e.g. """

...

...

__tc__.assertEqual(x, y) """

so both approaches to testing may be combined ... and everybody *should* be happy . [...] .. [1] dutest module @ PyPI (https://pypi.python.org/pypi/dutest) -- Regards, Olemis. Apache™ Bloodhound contributor http://issues.apache.org/bloodhound Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article:

On May 19, 2013, at 07:28 PM, Tim Peters wrote:

But more than just one ;-) Another great use has nothing to do with docstrings: using an entire file as "a doctest". This encourages writing lots of text explaining what you're doing,. with snippets of code interspersed to illustrate that the code really does behave in the ways you've claimed.

Agreed! I love separate-file doctests, and the marriage of doctests and reST/Sphinx is just fantastic. It's a pleasure to write usage documentation that contains code samples that are guaranteed to work, because they pass their doctest. (I personally don't like long-winded docstring doctests because they are harder to edit and distract from the code, but YMMV.) Several years ago, I spent some time experimenting with using doctest for *everything*. I deliberately wanted to go that extreme in order to better explore where doctests are good and where they're not so good. A general rule of thumb I came up with is that reST-style doctests are great for explanations involving mostly good-path usage of a library, or IOW "this is how you're supposed to use this API, and see it works!". IME, doctests are not so good at testing all the failure modes, odd corner cases, and the perhaps less-common good-path use cases. Fortunately, we have another useful tool for testing that stuff <wink>.

I'd rather encourage users to turn their brains on when writing doctest files - and when writing unit tests. I've lost count of how many times I've seen a unit test fail, then stared helplessly at the unit test code just trying to figure out what the author thought they were doing. A lot of comments in the test code could have helped that, but - outside of doctest-based tests - there's typically very little explanatory text in testing code.

+1 A rule-of-thumb I use is what I call the FORTH rule[1]. If you should be able to understand what your own test is trying to accomplish a week later, otherwise you're not writing very good tests. ;) -Barry [1] or PERL rule maybe, depending on the unit of time. :)

On 20 May 2013, at 18:26, Mark Janssen wrote:

...

...

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

Instead, we should be clear about their primary purpose which is to test the examples given in docstrings. In other words, doctests have a perfectly legitimate use case.

But more than just one ;-) Another great use has nothing to do with docstrings: using an entire file as "a doctest". This encourages writing lots of text explaining what you're doing,. with snippets of code interspersed to illustrate that the code really does behave in the ways you've claimed.

+1, very true. I think doctest excel in almost every way above UnitTests. I don't understand the popularity of UnitTests, except perhaps for GUI testing which doctest can't handle. I think people just aren't very *imaginative* about how to create good doctests that are *also* good documentation.

Doc tests have lots of problems for unit testing. * Every line is a test with *all* output part of the test - in unit tests you only assert the specific details you're interested in * Unordered types are a pain with doctest unless you jump through hoops * Tool support for editing within doctests is *generally* worse * A failure on one line doesn't halt execution, so you can get many many reported errors from a single failure * Try adding diagnostic prints and then running your doctests! * Tools support in terms of test discovery and running individual tests is not as smooth * Typing >>> and ... all the time is really annoying * Doctests practically beg you to write your code first and then copy and paste terminal sessions - they're the enemy of TDD * Failure messages are not over helpful and you lose the benefit of some of the new asserts (and their diagnostic output) in unittest * Tests with non-linear code paths (branches) are more painful to express in doctests and so on... However doctests absolutely rock for testing documentation / docstring examples. So I'm with Raymond on this one. All the best, Michael

That serves two very good purposes in one. How can you beat that? The issues of teardown and setup are fixable and even more beautifully solved with doctests -- just use the lexical scoping of the program to determine the execution environment for the doctests.

...

picking-your-poison-ly y'rs - tim

Cheers,

Mark _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Hi ! ... sorry , I could not avoid to reply this message ... On 5/20/13, Michael Foord wrote:

On 20 May 2013, at 18:26, Mark Janssen wrote:

...

...

...

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

Instead, we should be clear about their primary purpose which is to test the examples given in docstrings. In other words, doctests have a perfectly legitimate use case.

But more than just one ;-) Another great use has nothing to do with docstrings: using an entire file as "a doctest". This encourages writing lots of text explaining what you're doing,. with snippets of code interspersed to illustrate that the code really does behave in the ways you've claimed.

+1, very true. I think doctest excel in almost every way above UnitTests. I don't understand the popularity of UnitTests, except perhaps for GUI testing which doctest can't handle. I think people just aren't very *imaginative* about how to create good doctests that are *also* good documentation.

With enhanced doctests solution in mind ...

Doc tests have lots of problems for unit testing.

* Every line is a test with *all* output part of the test - in unit tests you only assert the specific details you're interested in

custom output checkers

* Unordered types are a pain with doctest unless you jump through hoops

( custom output checkers + doctest runner ) | (dutest __tc__ global var)

* Tool support for editing within doctests is *generally* worse

this is true , let's do it !

* A failure on one line doesn't halt execution, so you can get many many reported errors from a single failure

it should if REPORT_ONLY_FIRST_FAILURE option [1]_ is set .

* Try adding diagnostic prints and then running your doctests!

I have ... dutest suites for my Trac plugins do so . However logging information is outputted to /path/to/trac/env/log/trac.log ... so a tail -f is always handy .

* Tools support in terms of test discovery and running individual tests is not as smooth

dutest offers two options since years ago MultiTestLoader combines multiple test loaders to *load* different kinds of tests at once from a module , whereas a package loader performs test discovery . These loader objects are composable , so if an instance of MultiTestLoader is supplied in to the package test loader then multiple types of tests are loaded out of modules all over across the package hierarchy . Indeed , in +10 years of Python development I've never used unittest(2) discovery, and even recently implemented the one that's used in Apache™ Bloodhound test suite . Unfortunately I've had no much time to spend on improving all this support in dutest et al.

* Typing >>> and ... all the time is really annoying

... I have faith ... there should be something like this for vim ... I have faith ... ;)

* Doctests practically beg you to write your code first and then copy and paste terminal sessions - they're the enemy of TDD

Of course , not , all the opposite . If the approach is understood correctly then the first thing test author will do is to write the code «expected» to get something done . When everything is ok with API code style then write the code . Many problems in the API and inconsistencies are thus detected early .

* Failure messages are not over helpful and you lose the benefit of some of the new asserts (and their diagnostic output) in unittest

(custom ouput checkers) | ( dutest __tc__ variable )

* Tests with non-linear code paths (branches) are more painful to express in doctests

that's a fact , not just branches , but also exceptions Beyond this ... My really short answer is that I do not agree with this . Like I just said in previous messages with enhanced support like the one offered by dutest (i.e. __tc__ global var bound to an instance of unittest.TestCase) it's possible to invoke each and every unittest assertion method . So this may be seen all the other way round «unittest machinery is already used without even declaring a single test class» ... and so on ... ... so , in concept , there is no real benefit in using unittest over doctest *if* doctest module is eventually upgraded . [...]

However doctests absolutely rock for testing documentation / docstring examples.

FWIW , +1 [...] .. [1] doctest.REPORT_ONLY_FIRST_FAILURE (http://docs.python.org/2/library/doctest.html#doctest.REPORT_ONLY_FIRST_FAIL...) -- Regards, Olemis. Apache™ Bloodhound contributor http://issues.apache.org/bloodhound Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article:

...

* Doctests practically beg you to write your code first and then copy and paste terminal sessions - they're the enemy of TDD

Of course , not , all the opposite . If the approach is understood correctly then the first thing test author will do is to write the code «expected» to get something done . When everything is ok with API code style then write the code . Many problems in the API and inconsistencies are thus detected early .

Now all we need is a test() built-in, a companion to help() and we have the primo platform for doctest-code-test cycle for TDD and agile development. --Mark

On Sat, 18 May 2013 23:41:59 -0700 Raymond Hettinger wrote:

We should continue to encourage users to make thorough unit tests and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

There are other reasons to hate doctest, such as the obnoxious error reporting. Having to wade through ten pages of output to find what went wrong is no fun. Also the difficulty of editing them. For some reason, my editor doesn't offer me facilities to edit interactive prompt session snippets. All in all, I try hard to ignore any doctest present in the Python test suite. Regards Antoine.

On Mon, 20 May 2013 23:32:10 +1000 Steven D'Aprano wrote:

On 20/05/13 20:45, Antoine Pitrou wrote:

...

On Sat, 18 May 2013 23:41:59 -0700 Raymond Hettinger wrote:

...

We should continue to encourage users to make thorough unit tests and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

There are other reasons to hate doctest, such as the obnoxious error reporting. Having to wade through ten pages of output to find what went wrong is no fun.

Ten pages of broken unit tests are no picnic either.

You didn't understand the objection. I'm talking about *one* broken doctest in a sea of non-broken ones. For some reason doctest (or its unittest driver) insists on either displaying everything, or nothing. It doesn't only print the errors and leave the rest silent.

...

Also the difficulty of editing them. For some reason, my editor doesn't offer me facilities to edit interactive prompt session snippets.

Your text editor doesn't allow you to edit text? Even Notepad allows that!

Seriously, what editor are you using that doesn't allow you to edit pasted snippets?

I don't know if you're intentionally being stupid. Of course I can edit them *by hand*. But I'll have to re-create by hand the various artifacts of an interpreter session, e.g. the prompts. Regards Antoine.

On Tue, 21 May 2013 02:00:32 +1000 Steven D'Aprano wrote:

On 20/05/13 23:38, Antoine Pitrou wrote:

...

On Mon, 20 May 2013 23:32:10 +1000 Steven D'Aprano wrote:

...

On 20/05/13 20:45, Antoine Pitrou wrote:

...

On Sat, 18 May 2013 23:41:59 -0700 Raymond Hettinger wrote:

...

We should continue to encourage users to make thorough unit tests and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

There are other reasons to hate doctest, such as the obnoxious error reporting. Having to wade through ten pages of output to find what went wrong is no fun.

Ten pages of broken unit tests are no picnic either.

You didn't understand the objection. I'm talking about *one* broken doctest in a sea of non-broken ones. For some reason doctest (or its unittest driver) insists on either displaying everything, or nothing. It doesn't only print the errors and leave the rest silent.

It sounds like you are inadvertently calling doctest with the verbose option. It is not standard behaviour to display "everything or nothing".

Well, I never run doctest directly, I use regrtest (there are some doctests in the standard library). So perhaps the blame lies on regrtest or on the unittest adapter, my bad. Regards Antoine.

On Mon, 20 May 2013 09:37:32 -0400 "R. David Murray" wrote:

On Mon, 20 May 2013 12:45:57 +0200, Antoine Pitrou wrote:

...

On Sat, 18 May 2013 23:41:59 -0700 Raymond Hettinger wrote:

...

We should continue to encourage users to make thorough unit tests and to leave doctests for documentation. That said, it should be recognized that some testing is better than no testing. And doctests may be attractive in that regard because it is almost effortless to cut-and-paste a snippet from the interactive prompt. That isn't a best practice, but it isn't a worst practice either.

There are other reasons to hate doctest, such as the obnoxious error reporting. Having to wade through ten pages of output to find what went wrong is no fun.

That's why I added the 'failfast' option to doctest.

I didn't know that. Is it propagated by regrtest? I never use doctest standalone.

...

Also the difficulty of editing them. For some reason, my editor doesn't offer me facilities to edit interactive prompt session snippets.

I don't have much problem with lacking tailored facilities for this in vim. I suppose that is a matter of personal style. I *would* like to teach it the proper indentation, but I haven't been bothered enough yet to do it. (After all, weren't you the one who told me the lack of tab key indentation at the interactive prompt after you enabled completion by default wasn't an issue because one could just use space to indent? :)

An interpreter prompt session is throwaway, so you can pretty much indent as you like (which may not be very pretty in a tests file). Besides, I was thinking about the prompts ('>>> ' and '... '), not the indentation itself. Regards Antoine.

On May 18, 2013, at 11:41 PM, Raymond Hettinger wrote:

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

Thanks for your message Raymond. I know that doctests are controversial, but I do firmly believe that when used correctly, they have value and should not be broken without careful consideration. You make excellent points about Python 3 adoption and the "canary-like" nature of doctests. Cheers, -Barry