[Distutils] Deployment with setuptools: a basket-of-eggs approach

[Mars]

Hello all,

I was reading through my backlog of Daily Python URL's and saw that
the topics of deployment and configuration managment seem to be
getting some attention lately.  We have been having issues similar to
this at my company, and I was hoping for some feedback on a solution I
devised for these problems.

In our situation:
 - we have only a handful of developers, maintaining a very large
legacy code base.
 - different applications use different versions of the same library
on the same machine
 - developers still want to push bug fixes out without visiting every
application
 - developers want to write new (non-compatible) versions of their libraries
 - developers want other developers to use their new libraries instead
of the old ones

The plan I have borrows a bit from the Java and Python development
communities, and relies heavily on setuptools, policy, and some
developer dicipline (we are professionals, after all).

*Outline*
- Developers outline Families of modules for common enterprise-wide tasks
- Developers build Packages (python .egg files) from Families
- Developers build Applications which are composed of Packages

*Packages*
- Packages are deployed to a SharedDirectory
- Package versions have three number parts, Major.Minor.Bugfix
- New package features, or *any* change in the package functionality,
changes _at least_ the Minor version number.
- Bugfixes should not change any functionality, and thus they only
update the Bugfix version number
- Developers write ChangeLogs for Packages before deployment

*SharedDirectory*
- Servers mount the packages SharedDirectory

*ChangeLogs*
- ChangeLogs outline any new features or Bugfixes in a Package
- ChangeLogs are sent to all other developers after deployment

*Applications*
- Applications find their dependancies in the SharedDirectory
- Applications have startup scripts that 'require' the
developer-specified packages (for example, setuptools entry_points and
automatic script generation do this)
- Applications 'require' only the first two numbers of the package
version, Major.Minor

Some observations for deployers:
 - All applications should automatically upgrade if there is a bugfix version.
 - Changes in functionality are isolated from legacy applications.
 - Because applications are self-contained in setuptools .egg files,
we can specify the shared directory as a global installation source
when using easy_install.
 - Applications can be installed locally in a fashion similar to Java
applications, which are often a collection of .jar files and data
thrown into a common directory.  We can do the same with "easy_install
-f /opt/eggs -zmad /opt/myapp".
 - The application script itself could be shared from the central
drive, if we write a custom script that does not use the shebang line.
 This has the side-effect of automatically upgrading *everyone's*
version, which can be a blessing or a curse when you take versioning
of run-time data into account.

Some observations for developers:
 - The version numbering policy is flexible, but one *must* end up
with a deployment policy where business-critical applications do not
have even the most minor of functionality changes foisted upon them
(for one solution see Java-style installations, outlined above).
 - Developers know what is happening to a package at a grain higher
than that presented by Subversion, thanks to a published ChangeLog.
 - Developers are free to chose any version of the available packages,
because they know that they are all available at install-time.  They
can upgrade and downgrade versions at will.
 - Developers are still responsible for upgrading legacy applications
with more functional library versions, but that is what unit tests are
for (you /do/ write unit tests, don't you?)
 - It helps to use a setuptools alias to easy_install when building
packages for deployment.  You can then type "python setup.py deploy"
in the source directory as a quick and easy shortcut for building the
required .egg.

There are a few concerns that cross into release handling, and code
library maintenance and care:
 - Regular library reviews and a comfortable package end-of-life
schedule should be used to help prune the "supported packages" tree.
 - A relaxed package release schedule, with more features per-release,
should help slow things down to a comfortable balance between time
spent on upgrades, bugfixes, and features.
 - If the version numbers are climbing too fast, ask whether there
should be more features per-release, or if the package should be put
into 'alpha' or 'beta' status until it stabilizes.
 - Ask if anybody else is using the package you are maintaining.  If
not, ask yourself if it has be pre-maturely extract from an
application (remember 'You Aint Gonna Need It').
 - Ask if a quickly-rising package version indicates a family of
modules and functionality that should be re-arranged.  This could
happen if someone is developing a new application and dumping too much
application-specific functionality into an existing family.  Can the
family be split along functional lines?  Should the functionality be
kept in the application? (YAGNI again).


Well, that's about it.  You almost need to resurrect the role of
'Project Librarian' to keep track of it all! ;)

I would love to hear people's feedback on this idea, as I am sure that
I am not the first to tread this path.

Maris