[pypy-dev] Re: Scripttest (was: [pypy-svn] r5369)

[Rocco Moretti]

holger krekel wrote:

 >>I hacked together a little tool for running a script at app level, 
and checking
 >>its output against an expected output file. (Sort of like 
regrtest.py) I added
 >>a feature where the files can be sectioned off into multiple parts, 
to reduce
 >>disk clutter.
 >>
 >>* scripttest directory
 >>The start of using app_level_diff as a testing tool - I've converted the
 >>CPython regression test test_grammar to it's use. Sectioning is 
incomplete -
 >>the parts that are sectioned are mainly to isolate failures
 >>
 >>I've also started to do a script made from the examples given in Guido's
 >>tutorial. Incomplete as of now.
 >
 > Can you give some examples of usage?  I am a bit of a loss what
 > exactly you are trying to do here.

Sorry if it's unclear...

It started out as a way to run a script at applevel, capture its output, 
and compare it with a known good result (presumably generated from 
CPython). I was inspired by the original parts of the CPython regression 
test framework. The thought was to make a simple way of testing app 
level functionality, without having to support a whole unittest 
framework at applevel.

app_level_diff.py is the tool that does the running and comparing. The 
scripts in scripttest/test are unittests that fit in with the general 
testing framework to run the (applevel) scripts in the scripttest 
directory proper and compare the results to the pregenerated output 
files found there.

So the usage examples are in the scripttest/test directory. - It's 
another way of adding tests to pypy, and checking that pypy works like 
CPython does.

> I guess this at least should not live directly in src/pypy but in some
> subdirectory (e.g.  tool) or at 'src/scripttest' or something similar.

Hmmm ... you're probably right that it's a poor place for it to live. 
But I'm not sure it should be in any particular subdirectory - the tests 
  exercise the whole of pypy, not just a particular part. As for putting 
them in scr/scripttest or so, I was hoping that this would be able to be 
a part of the regression testing framework, and would be hit by the 
test_all.py script.

But they are self contained, and can be moved and/or deleted without any 
consequence, if people so desire.

-Rocco

P.S. In retrospect, I probably rushed checking in the code. Upon 
reexamination, it is rather rough, and not as extensible as I would 
like. If I have time this weekend, I'll see if I can improve it. (This 
includes adding better documentation.)