[Distutils] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them

[P.J. Eby]

At 11:19 AM 2/26/2010 -0500, Olemis Lang wrote:
>On Fri, Feb 26, 2010 at 10:52 AM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 10:16 AM 2/26/2010 -0500, Olemis Lang wrote:
> >>
> >> Well not as happily as I thought in first place. The fact is that
> >> `setuptools` test command creates the runner by calling the class
> >> specified in `test_runner` arg but supplies no parameters. In this
> >> case I'd like to specify `xmlrunner:XMLTestRunner` (is that the
> >> correct syntax ?) to obtain a JUnit XML report, but I need to specify
> >> the path where those reports will be outputted, so that the CI tool
> >> (Hudson in this case) can publish'em and generate the nice graphs we
> >> all love . Considering the current implementation, I still need to
> >> override `run_tests` method as a whole, and repeat 99% of what is
> >> implemented in there, in order to get the exact runner I need
> >>
> >> :(
> >>
> >> My suggestion (hint, $0.02 ... whatever ;o) is
> >>
> >>   1. use a factory method (e.g `create_runner` ) returning an
> >>     instance of  test runner. This method could be overridden by
> >>     subclasses if further options or whatever needs to be considered
> >>     to instanciate the runner
> >
> > I'm confused.  Why don't you just set test_runner to point to your factory
> > function, then?  That's what nose does.  (Well, in its case it's the
> > test_loader, but same basic idea.)
> >
>
>Well in theory it is possible (and did not think about that in the
>first place, thnx ;o) but :
>
>   1. How could I know about extra command line
>       params (without parsing them once again of course ;o) ?
>   2. In the case of the build script, I'm writing a setup script rather than
>       a framework (e.g. nose).
>      2.1 The XML runner is not always used, but only when running at
>            the CI buildenv (I supposed I could just use
>            `-r xmlrunner:XMLTestRunner` but I need an extra arg)
>            How could I specify output path in cmdline ?

An environment variable would solve these problems, at least if you 
have a decent shell.  ;-)

>      2.2 So I suppose I should write a function to do that but
>             - Under tests pkg? ... (Unit | functional | ...) tests
>don't need it
>             - In setup.py itself ? ... What should I supply in to -r arg ?

If you are running setup.py from the command line (as opposed to 
programmatically via easy_install), then it's __main__:whatever.  (It 
won't work if setup.py isn't the "true" __main__, though, e.g., if 
the script is being execfile'd.)

>               How could I specify output path in cmdline ?
>   3. In the case of `dutest` well ... that could help (depending on
>       the answer to 1 ;o)
>      3.1 How could I reuse features already stored in `Distribution`
>             (especially values set after finalize_xxx ) ?

You can't, unless you...


>   4. (Hypothetical so far, so I'm prepared to receive tomatoes ;o)
>       What if I want to reuse (subclass) `test` command ?

Which is perfectly allowable.  You'll just have to duplicate that one 
method.  Of course, if you write a suitable patch to refactor it to 
call another method, I'll happily put it in the trunk, since you're 
working from trunk to get the test_runner feature anyway.

("Suitable" = tested by you, doesn't break or remove existing 
functionality, & other "I'll know it if I see it" criteria ;-) )