[Python-Dev] an example of setuptools being used to good effect -- allmydata.org Tahoe
zooko at zooko.com
Sat Apr 12 06:02:04 CEST 2008
I'm sorry, but I am not caught up on the current conversation about
packaging. I'm very busy with exciting Python development -- http://
allmydata.com and http://allmydata.org . I understand from PJE's
message that he thinks I misunderstand some things about PEP 262;
this is entirely possible. I intend to catch up on reading the
emails of this conversation and to read carefully PJE's messages
about PEP 262 in the coming days.
Meanwhile, here is the last message that I wrote before I got swamped
with the aforementioned excitement:
On Apr 9, 2008, at 5:59 PM, Greg Ewing wrote:
> Paul Moore wrote:
>> I believe that Mac OS X goes for an even simpler structure -
>> applications store *everything* in the one directory, so that
>> install/uninstall is simply a directory copy/remove.
> Yep, and thereby cuts the whole gordian knot, throws the
> pieces on the fire and burns them. :-)
> Package managers have always seemed to me to be an
> excessively complex solution to a problem that needn't
> have existed in the first place.
Yes! I love the Zen of the Mac OS X packaging approach. The best
install is none at all! (Of course, I also love apt.)
> I keep hoping that someday Linux will support something
> like MacOSX application bundles and frameworks, but I
> haven't seen any sign of it yet.
We're slowly approaching this level of simplicity in packaging of the
*source code* of Allmydata.org "Tahoe", using setuptools.
The list of dependencies which are automatically resolved by
setuptools is visible here: . It currently includes zfec,
foolscap, simplejson, pycryptopp, nevow, zope.interface, twisted, and
This automatic resolution of dependencies works while fully
preserving the user's ability to use their own packages and their own
packaging tools. That is:
1. If any of these dependencies are already installed, such as in a
Debian package or if the user has installed them by hand, then
installing Tahoe will use the already-installed versions and not
install anything new, and
2. For any of these dependencies that are not already installed,
installing Tahoe will *not* write these dependencies into your
standard system directory (which is potentially a sacred place where
only your own packaging tool or your root account is allowed to
tread) but will instead write them into a local, newly-created
install directory from which you can then run Tahoe. (This part is
similar in spirit to the Mac OS packaging technique.)
Also, this install process never downloads anything from the Internet
at install time, per our policy [2, 3], which also happens to be a
policy some other people have, e.g. [4, 5].
This works on all of our supported platforms, which includes Linux,
Solaris, Windows, Cygwin, and Mac OS X.
Oh yes, we also have our buildbot  automatically produce Debian
packages for edgy, feisty, etch, and gutsy.
As far as I know, all of this is accomplished without breaking any of
the use cases traditionally associated with setuptools /
easy_install, for example "easy_install allmydata-tahoe" works, and
if you want "setup.py install" to install into your standard system
directory, it will.
The reason that I am posting this is to let other programmers know
that setuptools is actually a pretty useful tool, even if the use
cases that you want to support are incompatible with certain
easy_install traditions such as fetching new packages from the
internet at buildtime or installing into your system directory.
P.S. Two days ago I was able to remove twisted from the list of
"Manual Dependencies" that people have to be aware of in order to try
out Allmydata Tahoe from the source tarball.
I think I can safely remove pyOpenSSL now, but that remains to be
properly tested by our buildbot.
I will be able to remove Crypto++ soon, due to the pycryptopp 
If I can figure out a hack to work-around one of the major
frustrations of setuptools (that you can't simply run "./setup.py
install --prefix=$FOO"), and if I finish my setuptools plugin to run
Twisted trial instead pyunit when "./setup.py test", then I'll be
able to remove GNU make from the dependencies.
That will leave only g++, Python, and OpenSSL as packages that a
programmer has to manually deal with in order to try out Allmydata
Tahoe from source.
More information about the Python-Dev