On May 24, 2010, at 10:26 PM, Tarek Ziadé wrote:
1- a test command is going to be added in distutils2, and discussions have started on this feature, which is part of one GSOC project for Distutils2. See a summary of tasks here: http://wiki.python.org/moin/SummerOfCode/Distutils2 , and the GSOC who's who here: http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams. Michael, Titus and other ppl have started to discuss this test command, looking at setuptools' one and thinking about having "python setup.py test" as a standard.
Cool. I really like 'python setup.py test' as *a* standard interface for running a package's tests. This is available today in distribute, which for Ubuntu is what you actually get when you install setuptools. So today on Ubuntu I can run 'python setup.py test' if my setup.py has a 'test_suite' entry. You know all this of course. :) One of the other things I hope to work on soon is a Quickly[1] template for an opinionated creation of Python libraries and applications. By that I mean, we decide what best practices are for Python libraries, and we can define a template for the initial layout of such packages. I've talked about this to some people (though I really need to write up my ideas), but what I want is for a Python developer to be able to very easy and quickly run one command that gives them a layout which: * is a namespace package * 'python setup.py test' works (though initially does nothing) * has a base setup.py, README, license files * 'python setup.py docs' works (read: build_sphinx) * 'python setup.py publish' works (read: upload to pypi, upload docs, etc) * already has hooks for pylint, pyflakes, coverage * pre-loaded with 2to3 hooks * easy to add packaging for .deb, .rpm, etc. (extensible) * etc Being opinionated helps, with the goal of being appropriate for let's say 80-90% of Python packages out there. Complex stuff will always be complicated. Make the easy stuff stupid simple. So several ideas are involved: * One command from idea to fully fleshed out package * One command publishing * Common interface for automated testing 'health' of package * linting * test suite * easy to hook into CI * Cheeseshop can publish health statistics * Lots of carrots for package authors to DTRT
2- there's a GSOC project I have proposed, and that is being mentored by Jesse Noller, that will work on creating a CI system that will run various tests on packages uploaded at PyPI. The hardest part here is to run in a protected environment that is "cleaned up" everytime a new package is tested. So we'll script Virtual Machines, and get reports back, and we will probably use the PubHubSomething interface for this. The goal of this project is to come up with a CI system over PyPI, and eventually ask the PSF to host it.
This sounds excellent. We might be able to use some of the same infrastructure that e.g. Launchpad's build system uses for this. Running tests dovetails with the goal above of defining a standard way of testing the health of a package (test suite, plus lint/flakes, plus coverage). My current thinking for now is that 'python setup.py <cmd>' is the standard interface, where <cmd> can be 'test', 'publish', 'lint', etc. I think this can be eventually improved with the appropriate 'python -m' commands.
Any help on 1/ or 2/ is welcome !
Yes, I am very interested in helping with both!
Now for distutils2 itself, what we said at the GSOC meeting is that we wanted to setup up a CI for its code, before the GSOC starts. It has a runtests.py script right now that looks like test_distutils from the stdlib, and use the test command interface as soon as it is added in distutils2.
Yannick wants to have coverage, so he added an option to that script. Maybe, somehow, coverage and other metrics could be part of the test standard calling interface we want. I am not sure how this could look though, if we want to stay generic and avoid a dependency on a specific test tool (e.g. unittest vs nose vs ..)
Again, for the thing I'm describing above, I am allowing myself to be opinionated. What I care about most is the command line interface, and well defined standards. I'd like it to be flexible enough so that experts can extend and customize, but most people and packages would be happy with the defaults. Anyway, I look forward to seeing the fruits of the GSoC work and participating where I can! -Barry