Script for easy access to the coverage report on distutils2
Greetings Packagers, as part of our sprinting effort at Montréal-Python, we found it relevant to give an easy access to the test coverage reports to sprinters because it really helps them to see what needs more testing and when kind of breakage then can expect the tests to catch. I tried to use nose but the naming convention used seem to trigger way too many false positives so I'm afraid that we have to rule distutils2 as nose-incompatible. I could add a series of commands to our wiki, which is good for us but not very helpful to other distutils2 hackers, or I could integrate coverage in the current `runtests.py` test runner. Is there anyone one against adding a --with coverage option to `runtests.py`? Cheers, -- Yannick Gingras http://ygingras.net http://montrealpython.org -- lead organizer
Is there anyone one against adding a --with coverage option to `runtests.py`?
Please see the proof of concept patch in attachment. -- Yannick Gingras http://ygingras.net http://montrealpython.org -- lead organizer
On Sun, May 23, 2010 at 3:46 AM, Yannick Gingras
Greetings Packagers, as part of our sprinting effort at Montréal-Python, we found it relevant to give an easy access to the test coverage reports to sprinters because it really helps them to see what needs more testing and when kind of breakage then can expect the tests to catch.
I tried to use nose but the naming convention used seem to trigger way too many false positives so I'm afraid that we have to rule distutils2 as nose-incompatible.
I could add a series of commands to our wiki, which is good for us but not very helpful to other distutils2 hackers, or I could integrate coverage in the current `runtests.py` test runner. Is there anyone one against adding a --with coverage option to `runtests.py`?
Good idea, let me know when it's up in your clone, and I'll review/pull it Notice that this script is the basis for the soon-to-be-set Hudson CI server, so we will be able to run a daily coverage report of some sort we can publish -- if you take care of making it possible to output HTML. Thanks, Tarek
Cheers,
-- Yannick Gingras http://ygingras.net http://montrealpython.org -- lead organizer
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
On May 23, 2010, at 06:41 PM, Tarek Ziadé wrote:
Notice that this script is the basis for the soon-to-be-set Hudson CI server, so we will be able to run a daily coverage report of some sort we can publish -- if you take care of making it possible to output HTML.
Tarek, Can you describe this a bit more? Did I miss some discussion on distutils-sig about it? What I really want is a CI system that could run over all the packages on the Cheeseshop (or any other collection of Python packages, e.g. say in a Debian/Ubuntu repository), and run a standard test suite, providing some feedback as to the general health of the package on various versions of Python. I think a standard interface should be 'python setup.py test'. Packages don't have adhere to this standard but if they did, we could perhaps offer some nice carrots, like a star rating on Cheeseshop, detailed reports, etc. If something like this doesn't exist, I'm willing and available to work on it. -Barry
On Mon, May 24, 2010 at 10:05 PM, Barry Warsaw
On May 23, 2010, at 06:41 PM, Tarek Ziadé wrote:
Notice that this script is the basis for the soon-to-be-set Hudson CI server, so we will be able to run a daily coverage report of some sort we can publish -- if you take care of making it possible to output HTML.
Tarek,
Can you describe this a bit more? Did I miss some discussion on distutils-sig about it?
What I really want is a CI system that could run over all the packages on the Cheeseshop (or any other collection of Python packages, e.g. say in a Debian/Ubuntu repository), and run a standard test suite, providing some feedback as to the general health of the package on various versions of Python. I think a standard interface should be 'python setup.py test'. Packages don't have adhere to this standard but if they did, we could perhaps offer some nice carrots, like a star rating on Cheeseshop, detailed reports, etc.
If something like this doesn't exist, I'm willing and available to work on it.
Hello Barry, Several things here : 1- a test command is going to be added in distutils2, and discussions have started on this feature, which is part of one GSOC project for Distutils2. See a summary of tasks here: http://wiki.python.org/moin/SummerOfCode/Distutils2 , and the GSOC who's who here: http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams. Michael, Titus and other ppl have started to discuss this test command, looking at setuptools' one and thinking about having "python setup.py test" as a standard. 2- there's a GSOC project I have proposed, and that is being mentored by Jesse Noller, that will work on creating a CI system that will run various tests on packages uploaded at PyPI. The hardest part here is to run in a protected environment that is "cleaned up" everytime a new package is tested. So we'll script Virtual Machines, and get reports back, and we will probably use the PubHubSomething interface for this. The goal of this project is to come up with a CI system over PyPI, and eventually ask the PSF to host it. Any help on 1/ or 2/ is welcome ! Now for distutils2 itself, what we said at the GSOC meeting is that we wanted to setup up a CI for its code, before the GSOC starts. It has a runtests.py script right now that looks like test_distutils from the stdlib, and use the test command interface as soon as it is added in distutils2. Yannick wants to have coverage, so he added an option to that script. Maybe, somehow, coverage and other metrics could be part of the test standard calling interface we want. I am not sure how this could look though, if we want to stay generic and avoid a dependency on a specific test tool (e.g. unittest vs nose vs ..) Regards Tarek
-Barry
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
On May 24, 2010, at 10:26 PM, Tarek Ziadé wrote:
1- a test command is going to be added in distutils2, and discussions have started on this feature, which is part of one GSOC project for Distutils2. See a summary of tasks here: http://wiki.python.org/moin/SummerOfCode/Distutils2 , and the GSOC who's who here: http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams. Michael, Titus and other ppl have started to discuss this test command, looking at setuptools' one and thinking about having "python setup.py test" as a standard.
Cool. I really like 'python setup.py test' as *a* standard interface for running a package's tests. This is available today in distribute, which for Ubuntu is what you actually get when you install setuptools. So today on Ubuntu I can run 'python setup.py test' if my setup.py has a 'test_suite' entry. You know all this of course. :) One of the other things I hope to work on soon is a Quickly[1] template for an opinionated creation of Python libraries and applications. By that I mean, we decide what best practices are for Python libraries, and we can define a template for the initial layout of such packages. I've talked about this to some people (though I really need to write up my ideas), but what I want is for a Python developer to be able to very easy and quickly run one command that gives them a layout which: * is a namespace package * 'python setup.py test' works (though initially does nothing) * has a base setup.py, README, license files * 'python setup.py docs' works (read: build_sphinx) * 'python setup.py publish' works (read: upload to pypi, upload docs, etc) * already has hooks for pylint, pyflakes, coverage * pre-loaded with 2to3 hooks * easy to add packaging for .deb, .rpm, etc. (extensible) * etc Being opinionated helps, with the goal of being appropriate for let's say 80-90% of Python packages out there. Complex stuff will always be complicated. Make the easy stuff stupid simple. So several ideas are involved: * One command from idea to fully fleshed out package * One command publishing * Common interface for automated testing 'health' of package * linting * test suite * easy to hook into CI * Cheeseshop can publish health statistics * Lots of carrots for package authors to DTRT
2- there's a GSOC project I have proposed, and that is being mentored by Jesse Noller, that will work on creating a CI system that will run various tests on packages uploaded at PyPI. The hardest part here is to run in a protected environment that is "cleaned up" everytime a new package is tested. So we'll script Virtual Machines, and get reports back, and we will probably use the PubHubSomething interface for this. The goal of this project is to come up with a CI system over PyPI, and eventually ask the PSF to host it.
This sounds excellent. We might be able to use some of the same infrastructure that e.g. Launchpad's build system uses for this. Running tests dovetails with the goal above of defining a standard way of testing the health of a package (test suite, plus lint/flakes, plus coverage). My current thinking for now is that 'python setup.py <cmd>' is the standard interface, where <cmd> can be 'test', 'publish', 'lint', etc. I think this can be eventually improved with the appropriate 'python -m' commands.
Any help on 1/ or 2/ is welcome !
Yes, I am very interested in helping with both!
Now for distutils2 itself, what we said at the GSOC meeting is that we wanted to setup up a CI for its code, before the GSOC starts. It has a runtests.py script right now that looks like test_distutils from the stdlib, and use the test command interface as soon as it is added in distutils2.
Yannick wants to have coverage, so he added an option to that script. Maybe, somehow, coverage and other metrics could be part of the test standard calling interface we want. I am not sure how this could look though, if we want to stay generic and avoid a dependency on a specific test tool (e.g. unittest vs nose vs ..)
Again, for the thing I'm describing above, I am allowing myself to be opinionated. What I care about most is the command line interface, and well defined standards. I'd like it to be flexible enough so that experts can extend and customize, but most people and packages would be happy with the defaults. Anyway, I look forward to seeing the fruits of the GSoC work and participating where I can! -Barry
participants (3)
-
Barry Warsaw -
Tarek Ziadé -
Yannick Gingras