<br><br><div class="gmail_quote">On 27 February 2012 23:23, Mark Janssen <span dir="ltr"><<a href="mailto:dreamingforward@gmail.com">dreamingforward@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div class="im">On Mon, Feb 27, 2012 at 3:59 PM, Michael Foord <span dir="ltr"><<a href="mailto:fuzzyman@gmail.com" target="_blank">fuzzyman@gmail.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>As well as fundamental problems, the particular implementation of doctest suffers from these potentially resolvable problems:</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>Execution of an individual testing section continues after a failure. So a single failure results in the *reporting* of potentially many failures.<br>
<br></div></div></blockquote></div></div></blockquote></div><div>Hmm, perhaps I don't understand you. doctest reports how many failures occur, without blocking on any single failure.</div></div></blockquote><div><br>
<br>Right. But you typically group a bunch of actions into a single "test". If a doctest fails in an early action then every line after that will probably fail - a single test failure will cause multiple *reported* failures.<br>
</div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div>The problem of being dependent on order of unorderable types (actually very difficult to solve).</div>
</div></blockquote></div></div></blockquote></div><div><br>Well, a crude solution is just to lift any output text that denotes an non-ordered type and pass it through an "eval" operation.</div></div></blockquote>
<div><br><br>Not a general solution - not all reprs are reversible (in fact very few are as a proportion of all objects).<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail_quote"><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div>Things like shared fixtures and mocking become *harder* (although by no means impossible) in a doctest environment.<br>
<br></div></div></blockquote></div></div></blockquote><div><br></div></div><div>This, I think, what I was suggesting with doctest "scoping" where the execution environment is a matter of how nested the docstring is in relation to the "python semantic environment", with a final scope of "globs" that can be passed into the test environment, for anything with global scope.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>Another thing I dislike is that it encourages a "test last" approach, as by far the easiest way of generating doctests is to copy and paste from the interactive interpreter. The alternative is lots of annoying typing of '>>>' and '...', and as you're editing text and not code IDE support tends to be worse (although this is a tooling issue and not a problem with doctest itself).<br>
</div></div></blockquote></div></div></blockquote><div><br></div></div><div>This is where I think the idea of having a test() built-in, like help(), would really be nice. One could run test(myClass.mymethod) iterively while one codes, encouraging TDD and writing tests *along with* your code. My TDD sense says it couldn't get any better.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div>More fundamental-ish problems:</div><div><br></div><div> Putting debugging prints into a function can break a myriad of tests (because they're output based).</div>
</div></blockquote><div><br></div></div><div>That's a good point. But then it's a fairly simple matter of adding the output device: 'print >> stderr, 'here I am'", another possibility, if TDD were to become more of part of the language, is a special debug exception: "raise Debug("Am at the test point, ", x)" Such special exceptions could be caught and ignored by doctest.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div> With multiple doctest blocks in a test file running an individual test can be difficult (impossible?).</div>
<div><br></div></div></blockquote></div><div>This again solved with the test() built-in an making TDD something that is a feature of the language itself.</div></div></blockquote><div><br>I don't fully follow you, but it shouldn't be hard to add this to doctest and see if it is really useful.<br>
</div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div></div><div> I may be misremembering, but I think debugging support is also problematic because of the stdout redirection</div></div></blockquote><div><br></div></div><div>Interesting, I try to pre-conceive tests well enough so I never need to invoke the debugger.</div>
</div></blockquote><div><br>Heh. When I'm adding new features to existing code it is very common for me to write a test that drops into the debugger after setting up some state - and potentially using the test infrastructure (fixtures, django test client perhaps, etc). So not being able to run a single test or drop into a debugger puts the kybosh on that.<br>
<br>Michael<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div>So yeah. Not a huge fan.</div><div><br></div></div></blockquote></div><div>That's good feedback. Thanks.</div>
<span class="HOEnZb"><font color="#888888">
<div><br></div><div>Mark</div></font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><pre cols="72"><a href="http://www.voidspace.org.uk/" target="_blank">http://www.voidspace.org.uk/</a><br><br>May you do good and not evil<br>May you find forgiveness for yourself and forgive others<br>
May you share freely, never taking more than you give.<br>-- the sqlite blessing <a href="http://www.sqlite.org/different.html" target="_blank">http://www.sqlite.org/different.html</a></pre>
<br>