[Distutils] Distribute testing

[ssteinerX@gmail.com]

On Dec 20, 2009, at 6:54 AM, Tarek Ziadé wrote:

> On Sat, Dec 19, 2009 at 6:33 PM, ssteinerX at gmail.com
> <ssteinerx at gmail.com> wrote:
> [..]
>> 
>>        While this approach will be fine when we move it to building on-demand buildbots, it is much too time consuming both in development time and real-time processing to use for this release cycle.  It would be much better (for now) to just make sure you're running it on a machine that can do the job.
> 
> *or* to update an existing test environment. e.g. check if python != trunk is already present.

Yes, but what I mean is that I'm not going to be downloading and building all supported versions of Python itself in this version as I will in the buildbot version.

Updating the dev versions of 2.7 and 3.2 to latest trunk will be in this version as will updating any peripheral tools required for testing (pip etc.).

>>        First, we need to decide where to put the testing sub-project.  I'm asuming it can just live in the /tests subdirectory, maybe with a subdir to hold this whole project as its own module.
> 
> Sounds good

Ok, I'll do that.  I'm going to put this in one working chunk at a time so that you can use whatever's there right away.  As time goes on, more tests will be added but my intention is to keep what's checked into the main branch always working.

I'll make a new branch

How, exactly, do you run the tests that you run before you post to pypi so that I can integrate it properly?

>>        Here's what I'd like to see it do:
>> 
>> [snip...]
>>                for each product to be tested:
>>                        for each Python version this product is to be tested on:
>>                                install the product
>>                                run its test suite
>>                                record any failures
>> 
> 
> Sounds good too ! Notice that we also need to try those tests in
> various environments, besides python versions: virtualenv, no virtualenv, setuptools present before
> we start, not present, etc

Yes, I understand what might be possible at some point in the future but, for now, I just want to get the machinery running and in place to support all of those different scenarios at some point in the future.

While I'd love to do a comprehensive test suite, I just don't have the time to devote to getting every combination and permutation done it in one shot so my intention is to set up the hooks, get some useful tests into the process that cover recent issues, and make it so that anyone can add new tests with a minimum of configuration.

So...what command should I hook it into so I can quickly get this into the testing process before the next release?

Thanks,

S