Ok, I've been quiet for a little while fixing few things on the pyvm project... so sorry for this following very long email.
Pyvm is my (personal at the moment) attempt to use what opensuse team has come up with: the opensuse build server.
This service (free) provides a continuous integration system as many commercial projects are actively using: having some first hand experience I can tell you is quite an effective method.
The basics are very simple:
1. the user/developer puts some files in a repository (actually a "svn" repository)
2. The scheduler builds a xen system image in a well known status (so guaranteed repeatable)
3. The scheduler then runs the build using the provided files including the smoke tests, collects the resulting rpms and publish them on line
The advantage on buildbot are many fold:
1. There is a guarantee the resulting binaries are untampered (using the standard libraries provided by the linux distributions not the one the developer has installed on his box)
2. Hands off build, so no funny hacks that work on some machines/distributions but not on other (yes, it enforces a development discipline)
3. It supports a variety of linux distribution in parallel (so far I've addressed the lsb standard so using the rpm)
4. It make possible for an administrator to have many python interpreter versions side by side (believe or not there're companies still using python 2.3 because is the one shipped with red hat enterprise linux 4!)
5. The system images are network disconnected
While build bot is fine for personal use, opensuse is a step ahead providing build that can be used by the users and, by the way, this system is the one that builds opensuse.
Osc (the susebuild client) can provide also a local build in the same spirit of buildbot: the only missing bit (at the moment) is the reporting part.
Now the pyvm project. --------------------------------
There are really two different subprojects: the intepreters and the modules.
Now the interpreters are taken from the svn trunk and rebuild for a pletora of linux distributions and architectures. There's a minimal smoke test and few patches applied. A user can point its yum/yast package manager and download the python intepreter for each subversion checkout (or doing it by downloading the rpms).
Then there is the more interesting part for developers: the modules. Basically they are few python packages build against each python subversion check out so the building chain is:
opt-python-svn73450 -> setuptools-latest-svn73450 -> sphinx-0.6.1-svn73450
If something is broken everything will fail: this process is automatic(-ish) so I can manage a failry large set of package already.
My use case for this is:
A module developer creates his masterpiece then, if it is worth, gets an account on opensuse. He sends the tarball, a spec file and the eventual patch files and waits for the build to happen. If fails he has to repeat it again: the first time will be painful, but with time it will improve and the overall quality will be much better.
There are many more use cases pyvm address, but it will bore people too much.
I hope this clarifies a little bit what is my aim with pyvm. I wish this approach will be adopted because I can see clearly the benefits of it in terms of quality control (I'm not promoting here pyvm but the service opensuse is providing for free).
On Thu, Jun 18, 2009 at 5:02 PM, Leonardo Santagada
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
I mentioned that need a while ago and work on a project see ( http://tarekziade.wordpress.com/2009/05/28/smoke-testing-in-distutils), and I made a buildbot doing smoke tests : I have it running for a few packages here http://buildbot.ziade.org/waterfall on one slave so far.
I helps me on making sure I don't break distutils :)
I know Titus wants to do something similar as well, and I think Antonio project is adressing another need.
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
I have a white list of packages so far. A complete system needs to run in a virtual environment and scratch it everytime a package is tested + cut the network access.
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
I am calling "python setup.py sdist" and "python setup.py bdist" so far and compare the output on various python versions, but any command would work. If you are interested in writing some scripts, you are welcome to join, it's an open project located here http://bitbucket.org/tarek/distutils-builbot/src/