[Python-Dev] Automatic installer builds (was Re: Fwd: Broken link to download (Mac OS X))

[David Bolen]

Ronald Oussoren <ronaldoussoren at mac.com> writes:

> Speaking of which... I have a mac-mini that could be used for a
> buildbot. How much work is needed to kickstart a buildbot, taking
> into account that I'd prefer to have a buildbot with different
> configure-flags that the default unix build (that is, I want to test
> a framework build that is a universal binary).

Sort of picking this message to enter the thread ...

Interestingly enough, I had made an offer to Martin to host an OSX
build slave last week (before this thread), since I had a Mini whose
disk crashed and would be reasonably free after I rebuilt it, and I
had noticed there were no longer any OSX build slaves around.

It's online now, though it certainly need not be the only one.
There's not much to setting up a build slave - just easy_install
buildbot on top of a basic Python install (you'll need Twisted), use
the "build-slave" command to create a tree, edit the files in "info",
and "buildbot start <tree-root>" to get it started.  Of course,
there's some overhead to monitoring things over time.

My new slave is a Tiger box, since that's what I still need for my own
application builds. So it won't help with building/testing x64 builds.

Per the "startup on reboot" part of the thread, I typically use the
LaunchAgents setup on my Macs for that purpose, although to be honest,
not many of my build slaves start automatically anyway, as the boxes
don't tend to get restarted other than manually.

In the first builds, I have noticed that the build master seems to
execute the builds as a Unix system, so isn't building with the
universalsdk option or as a framework, though the latter would
probably need a bit of glue to make the framework available just
locally to the buildbot process.

On Windows, the buildbot tasks use scripts that keep the third party
stuff in the buildbot branch tree itself.  I wonder if perhaps with a
little TLC, we could come up with some Mac-specific buildbot scripts
to better have an OSX build slave mimic the final build process.  It
might be as simple as using build-installer with a specified directory
within the build tree, though utilitizing the built framework for the
tests probably needs some support.  The build master could then be
set to execute those scripts (much as it does on Windows) rather than
the stock Unix approach.

I anticipate some tuning to do with respect to external libraries.
Rather than use system libraries or my own (or finks or MacPorts)
version of libraries, I suspect it would be better to download the
same releases of the third party stuff that the installer script
downloads and install them locally for normal buildbot builds, to best
match what would be packaged up with a final binary.  Of course, if
so, then there's the question of keeping the buildbot environment up
to date with the installer script.

As I believe Martin mentioned elsewhere in the thread, he used to have
my Windows slave generate nightly MSI builds and upload them, but the
process became fragile over time, and didn't seem to be missed when we
discontinued them.  But something similar could probably be worked out
for building nightly DMGs for OSX if it were deemed useful.

> Creating the Mac installer is easy: just run
> Mac/BuildScript/build-installer.py on an OSX 10.5 system where a
> local version of Tcl/Tk 8.4 is installed in /Library/Frameworks. The
> system should also not have fink or darwinports and a clean
> /usr/local tree to avoid contaminating the build.

Yes, I've successfully used the script to create a DMG of the latest
2.7 out of trunk, and at least the basic sniff test (install it
locally, run a few things) seems fine.  Did run into one quirk where
if you have already built Python within the tree you run the script
from, it doesn't rebuild everything beneath /tmp/_py/_bld - such as
the pgen stuff.  So you want a clean source tree too.

To the extent that the output (DMG) of the build-installer script is
what is currently needed for OSX, I can generate one if it is still
needed, though I can't promise much beyond just executing the script.
The new build slave can also be made available for RMs (or whomever)
to generate the DMG when needed if that might be helpful.  Though
there should probably be some basic installation test on other systems
prior to publishing any such generated files.

-- David