Metatest 0.1.0

[Jonathan Fine]

Kay Schluehr wrote:

>>     http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html

>From the HTML slides:
> 
>    Assertion tests are easy to write but report and run poorly.
> 
> I tend to think this is a prejudice that leads to ever more ways to
> write tests perform test discoveries, invent test frameworks etc.
> 
> When I thought about organizing my tests for EasyExtend I was seeking
> for a strategy being adapted to my own behaviour. The very first step
> I made when creating a new language, modifying a grammar rule,
> changing the Transformer etc. was starting an interactive shell and
> typing some commands. 

Yes, Python is really good for that.  I do it a lot also.

> This is just checking out or smoke testing my
> application and I wanted to reproduce it. So I recorded the
> interactive shell session and replayed it as another shell session. I
> enabled to set breakpoints in the logged output and implemented a
> command for proceeding the replay.

This is similar, I think, to what Metatest does.  See 
http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide5

> Then I discovered I actually wrote an interactive test runner. I don't
> have to care for all the exceptions being thrown in the session and
> don't care for cleanups but just assign parts of the session as tests
> using assert for getting an overview at the end. The only additonal
> command I implemented was raises to capture the side-effect of raising
> an exception.

Sounds interesting.  Is this code, or examples of its use, available?

> So I'm going to argue here that there isn't anything particular about
> writing/coding a test ( planning tests, specifying tests, reviewing a
> testspecification etc. are another issue ). Instead you can keep a
> seemingly unrelated practice and turn it into a test by a tiny
> signification.

I'm all in favour of making tests easier to write, easier to run, and 
the outputs easier to understand.

I've submitted a feature request for command line use of Metatest 
(something I've been thinking of a while):
http://sourceforge.net/tracker/index.php?func=detail&aid=1797187&group_id=204046&atid=988038

Here's how I'd like the feature to look (from the above URL):
===
Python 2.4.1a0 (#2, Feb 9 2005, 12:50:04)
[GCC 3.3.5 (Debian 1:3.3.5-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> from metatest.py.mymod import plus
 >>> import metatest
 >>> plus(2, 3) == 6
<MyStub 4>
 >>> metatest.immediate = True # New command
 >>> plus(2, 3) == 6
TestError: Expected '6', got '5'
 >>>
===

OK.  So now here's a problem.  We can create stuff like the above that 
states clearly (I hope) what is required.  How can we write a test for 
this sort of behaviour?  So that's another feature request.

While I'm in favour of using Metatest to write tests for Metatest 
(eating your own dogfood), I'm more interested in real world examples. 
But I've added second feature request, that Metatest be able to test 
Metatest.
http://sourceforge.net/tracker/index.php?func=detail&aid=1797202&group_id=204046&atid=988038

-- 
Jonathan