[Distutils] Autobuild packages using snakebite

[A. Cavallo]

Ok,
I've been quiet for a little while fixing few things on the pyvm project...
so sorry for this following very long email.

Pyvm is my (personal at the moment) attempt to use what opensuse team has come 
up with: the opensuse build server.

This service (free) provides a continuous integration system as many 
commercial projects are actively using: having some first hand experience I 
can tell you is quite an effective method.

The basics are very simple:

1. the user/developer puts some files in a repository (actually a "svn" 
repository)

2. The scheduler builds a xen system image in a well known status (so 
guaranteed repeatable) 

3. The scheduler then runs the build using the provided files including the 
smoke tests, collects the resulting rpms and publish them on line

The advantage on buildbot are many fold:

1. There is a guarantee the resulting binaries are untampered (using the 
standard libraries provided by the linux distributions not the one the 
developer has installed on his box) 

2. Hands off build, so no funny hacks that work on some machines/distributions 
but not on other (yes, it enforces a development discipline)

3. It supports a variety of linux distribution in parallel (so far I've 
addressed the lsb standard so using the rpm)

4. It make possible for an administrator to have many python interpreter 
versions side by side (believe or not there're companies still using python 
2.3 because is the one shipped with red hat enterprise linux 4!)

5. The system images are network disconnected

While build bot is fine for personal use, opensuse is a step ahead providing 
build that can be used by the users and, by the way, this system is the one 
that builds opensuse.

Osc (the susebuild client) can provide also a local build in the same spirit 
of buildbot: the only missing bit (at the moment) is the reporting part.


Now the pyvm project.
--------------------------------

There are really two different subprojects: the intepreters and the modules.

Now the interpreters are taken from the svn trunk and rebuild for a pletora of 
linux distributions and architectures. There's a minimal smoke test and few 
patches applied.
A user can point its yum/yast package manager and download the python 
intepreter for each subversion checkout (or doing it by downloading the rpms).

Then there is the more interesting part for developers: the modules.
Basically they are few python packages build against each python subversion 
check out so the building chain is:

 opt-python-svn73450 ->  setuptools-latest-svn73450 -> sphinx-0.6.1-svn73450

If something is broken everything will fail: this process is automatic(-ish) 
so I can manage a failry large set of package already.

My use case for this is:

A module developer creates his masterpiece then, if it is worth, gets an 
account on opensuse.
He sends the tarball, a spec file and the eventual patch files and waits for 
the build to happen. 
If fails he has to repeat it again: the first time will be painful, but with 
time it will improve and the overall quality will be much better.

There are many more use cases pyvm address, but it will bore people too much.

I hope this clarifies a little bit what is my aim with pyvm.
I wish this approach will be adopted because I can see clearly the benefits of 
it in terms of quality control (I'm not promoting here pyvm but the service 
opensuse is providing for free).

Regards,
Antonio













> On Thu, Jun 18, 2009 at 5:02 PM, Leonardo Santagada 
<santagada at gmail.com>wrote:
> > I remember when snakebite was announced someone had the idea to have a
> > kind of build bot go into pypi and get all packages sent to it and
> > automatically compile them on snakebite for all major platforms and
> > automatically run tests and submit the compiled packages to pypi. Is
> > anyone working on this?
>
> I mentioned that need a while ago and work on a project see (
> http://tarekziade.wordpress.com/2009/05/28/smoke-testing-in-distutils), and
> I made a buildbot doing smoke tests : I have it running for a few packages
> here http://buildbot.ziade.org/waterfall on one slave so far.
>
> I helps me on making sure I don't break distutils :)
>
> I know Titus wants to do something similar as well, and I think Antonio
> project is adressing another need.
>
> > The biggest problem I see is security, but if people are really
> > interested in this we could at least try it no?
>
> I have a white list of packages so far. A complete system needs to run in a
> virtual environment and scratch it everytime
> a package is tested + cut the network access.
>
> > Automatic testing in many python versions, conversion using 2to3 and
> > building packages for windows and macosx seems like a very interesting
> > service.
>
> I am calling "python setup.py sdist" and "python setup.py bdist" so far and
> compare the output on various python versions,
> but any command would work. If you are interested in writing some scripts,
> you are welcome to join, it's an open project
> located here http://bitbucket.org/tarek/distutils-builbot/src/
>
> Tarek