Metatest 0.1.0

[Kay Schluehr]

On Sep 18, 7:11 pm, Jonathan Fine <jf... at pytex.org> wrote:
Jonathan Fine schrieb:
> Kay Schluehr wrote:
>
>>>     http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html
>
>>> From the HTML slides:
>>
>>    Assertion tests are easy to write but report and run poorly.
>>
>> I tend to think this is a prejudice that leads to ever more ways to
>> write tests perform test discoveries, invent test frameworks etc.
>>
>> When I thought about organizing my tests for EasyExtend I was seeking
>> for a strategy being adapted to my own behaviour. The very first step
>> I made when creating a new language, modifying a grammar rule,
>> changing the Transformer etc. was starting an interactive shell and
>> typing some commands.
>
> Yes, Python is really good for that.  I do it a lot also.
>
>> This is just checking out or smoke testing my
>> application and I wanted to reproduce it. So I recorded the
>> interactive shell session and replayed it as another shell session. I
>> enabled to set breakpoints in the logged output and implemented a
>> command for proceeding the replay.
>
> This is similar, I think, to what Metatest does.  See http://metatest.sourceforge.net/doc/pyconuk2007/metatest.html#slide5
>
>> Then I discovered I actually wrote an interactive test runner. I don't
>> have to care for all the exceptions being thrown in the session and
>> don't care for cleanups but just assign parts of the session as tests
>> using assert for getting an overview at the end. The only additonal
>> command I implemented was raises to capture the side-effect of raising
>> an exception.
>
> Sounds interesting.  Is this code, or examples of its use, available?

Sure, it's part of EasyExtend. See also www.fiber-space.de

Checkout the documentation for consoletest. I guess in the current
release recording and replaying can't be done in the same
run. I've got a corrected version on my disk but I didn't uploaded it
yet and corrected the docs.

The reason why consoletest is bundled with EasyExtend is the use of a
different, more flexible and somewhat simpler InteractiveConsole
implementation than available in the stdandard library that can
execute code of arbitrary languages. This was a projects requirement
but
there is no fundamental reason why it can't be separated from
eeconsole.py.

>> So I'm going to argue here that there isn't anything particular about
>> writing/coding a test ( planning tests, specifying tests, reviewing a
>> testspecification etc. are another issue ). Instead you can keep a
>> seemingly unrelated practice and turn it into a test by a tiny
>> signification.
>
> I'm all in favour of making tests easier to write, easier to run, and the outputs easier to understand.
>
> I've submitted a feature request for command line use of Metatest (something I've been thinking of a while):
> http://sourceforge.net/tracker/index.php?func=detail&aid=1797187&group_id=204046&atid=988038
>
> Here's how I'd like the feature to look (from the above URL):
> ===
> Python 2.4.1a0 (#2, Feb 9 2005, 12:50:04)
> [GCC 3.3.5 (Debian 1:3.3.5-8)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from metatest.py.mymod import plus
> >>> import metatest
> >>> plus(2, 3) == 6
> <MyStub 4>
> >>> metatest.immediate = True # New command
> >>> plus(2, 3) == 6
> TestError: Expected '6', got '5'
> >>>
> ===
>
> OK.  So now here's a problem.  We can create stuff like the above that states clearly (I hope) what is required.  How can we write a test for this sort of behaviour?  So that's another feature request.
How does metatest analyze the tested expression? Lets say I try to
check this expression

>>> plus( f(1) == True, f(2) != False) == plus( f(2) == True, f(1) != False)

metatest must have decent knowledge about the expression structure of
Python expressions to detect the top level comparison and separate it
from the right and left hand side in order to generate the message of
the example. Since consoletest uses Pythons assert it is obviously
primitive in this respect and lacks expressivity of error messages. So
it fails with respect of one of your criteria. But note that
EasyExtend has the full power of a python parser in the background and
is a framework for analyzing and transforming expressions. I refused
to filter assert here or nest transformers since they might interfere
with more general purposes of EasyExtend.

Regards, Kay