<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Sat, May 18, 2013 at 11:41 PM, Raymond Hettinger <span dir="ltr"><<a href="mailto:raymond.hettinger@gmail.com" target="_blank">raymond.hettinger@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div>On May 14, 2013, at 9:39 AM, Gregory P. Smith <<a href="mailto:greg@krypto.org" target="_blank">greg@krypto.org</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr" style="font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Bad: doctests.</div></blockquote></div><div><br></div><div>I'm hoping that core developers don't get caught-up in the "doctests are bad meme".</div></div></blockquote><div><br></div><div style>So long as doctests insist on comparing the repr of things being the number one practice that people use when writing them there is no other position I can hold on the matter. reprs are not stable and never have been. ordering changes, hashes change, ids change, pointer values change, wording and presentation of things change. none of those side effect behaviors were ever part of the public API to be depended on.</div>
<div style><br></div><div style>That one can write doctests that don't depend on such things as the repr doesn't ultimately matter because the easiest thing to do, as encouraged by examples that are pasted from an interactive interpreter session into docs, is to have the interactive interpreter show the repr and not add code to check things in a accurate-for-testing manner that would otherwise make the documentation harder for a human to read.</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>
Instead, we should be clear about their primary purpose which is to test</div><div>the examples given in docstrings. In many cases, there is a great deal </div><div>of benefit to docstrings that have worked-out examples (see the docstrings</div>
<div>in the decimal module for example). In such cases it is also worthwhile </div><div>to make sure those examples continue to match reality. Doctests are</div><div>a vehicle for such assurance. In other words, doctests have a perfectly</div>
<div>legitimate use case.</div></div></blockquote><div><br></div><div><div>I really do applaud the goal of keeping examples in documentation up to date. But doctest as it is today is the wrong approach to that. A repr mismatch does not mean the example is out of date.</div>
<div><br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>
We should continue to encourage users to make thorough unit tests</div><div>and to leave doctests for documentation. That said, it should be</div><div>recognized that some testing is better than no testing. And doctests</div>
<div>may be attractive in that regard because it is almost effortless to</div><div>cut-and-paste a snippet from the interactive prompt. That isn't a</div><div>best practice, but it isn't a worst practice either.</div>
</div></blockquote><div><br></div><div style>Not quite, they at least tested something (yay!) but it is uncomfortably close to a worst practice.</div><div style><br></div><div style>It means someone else needs to come understand the body of code containing this doctest when they make an unrelated change that triggered a behavior change as a side effect that the doctested code may or may not actually depend on but does not actually declare its intent one way or another for the purposes of being a readable example rather than accurate test.</div>
<div style><br></div><div style>bikeshed colors: If doctest were never called a test but instead were called docchecker to not imply any testing aspect that might've helped (too late? the cat's out of the bag). Or if it never compared anything but simply ran the example code to generate and update the doc examples from the statements with the current actual results of execution instead of doing string comparisons... (ie: more of an documentation example "keep up to date" tool)</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>
Another meme that I hope dispel is the notion that the core developers</div><div>are free to break user code (such as doctests) if they believe the</div><div>users aren't coding in accordance with best practices. Our goal is to</div>
<div>improve their lives with our modifications, not to make their lives </div><div>more difficult.</div></div></blockquote><div><br></div><div style>Educating users how to apply best practices and making that easier for them every step of the way is a primary goal. Occasionally we'll have to do something annoying in the process but we do try to limit that.</div>
<div style><br></div><div style>In my earlier message I suggested that someone improve doctest to not do dumb string comparisons of reprs. I still think that is a good goal if doctest is going to continue to be promoted. It would help alleviate many of the issues with doctests and bring them more in line with the issues many people's regular unittests have. As Tres already showed in an example, individual doctest using projects jump through hoops to do some of that today; centralizing saner repr comparisons for less false failures as an actual doctest feature just makes sense.</div>
<div style><br></div><div style>Successful example: We added a bunch of new comparison methods to unittest in 2.7 that make it much easier to write tests that don't depend on implementation details such as ordering. Many users prefer to use those new features; even with older Python's via unittest2 on pypi. It doesn't mean users always write good tests, but a higher percentage of tests written are more future proof than they were before because it became easier.</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>
Currently, we face an adoption problem with Python 3. At PyCon,</div><div>an audience of nearly 2500 people said they had tried Python 3 </div><div>but weren't planning to convert to it in production code. All of the</div>
<div>coredevs are working to make Python 3 more attractive than Python 2,</div><div>but we also have to be careful to not introduce obstacles to conversion.</div><div>Breaking tests makes it much harder to convert (especially because</div>
<div>people need to rely on their tests to see if the conversion was successful).</div></div></blockquote><div><br></div><div style>Idea: I don't believe anybody has written a fixer for lib2to3 that applies fixers to doctests. That'd be an interesting project for someone.</div>
<div style><br></div><div style>Now you've got me wondering what Python would be like if repr, `` and __repr__ never existed as language features. Upon first thoughts, I actually don't see much downside (no, i'm not advocating making that change). Something to ponder.</div>
<div style><br></div><div style>-gps</div></div></div></div>