[Python-Dev] an example of setuptools being used to good effect -- allmydata.org Tahoe

[zooko]

Folks:

I'm sorry, but I am not caught up on the current conversation about  
packaging.  I'm very busy with exciting Python development -- http:// 
allmydata.com and http://allmydata.org .  I understand from PJE's  
message that he thinks I misunderstand some things about PEP 262;  
this is entirely possible.  I intend to catch up on reading the  
emails of this conversation and to read carefully PJE's messages  
about PEP 262 in the coming days.

Meanwhile, here is the last message that I wrote before I got swamped  
with the aforementioned excitement:


On Apr 9, 2008, at 5:59 PM, Greg Ewing wrote:
> Paul Moore wrote:
>
>> I believe that Mac OS X goes for an even simpler structure -
>> applications store *everything* in the one directory, so that
>> install/uninstall is simply a directory copy/remove.
>
> Yep, and thereby cuts the whole gordian knot, throws the
> pieces on the fire and burns them. :-)
>
> Package managers have always seemed to me to be an
> excessively complex solution to a problem that needn't
> have existed in the first place.

Yes!  I love the Zen of the Mac OS X packaging approach.  The best  
install is none at all!  (Of course, I also love apt.)


> I keep hoping that someday Linux will support something
> like MacOSX application bundles and frameworks, but I
> haven't seen any sign of it yet.

We're slowly approaching this level of simplicity in packaging of the  
*source code* of Allmydata.org "Tahoe", using setuptools.

http://allmydata.org/source/tahoe/trunk/docs/install.html

The list of dependencies which are automatically resolved by  
setuptools is visible here: [1].  It currently includes zfec,  
foolscap, simplejson, pycryptopp, nevow, zope.interface, twisted, and  
pywin32.

This automatic resolution of dependencies works while fully  
preserving the user's ability to use their own packages and their own  
packaging tools.  That is:

1. If any of these dependencies are already installed, such as in a  
Debian package or if the user has installed them by hand, then  
installing Tahoe will use the already-installed versions and not  
install anything new, and

2. For any of these dependencies that are not already installed,  
installing Tahoe will *not* write these dependencies into your  
standard system directory (which is potentially a sacred place where  
only your own packaging tool or your root account is allowed to  
tread) but will instead write them into a local, newly-created  
install directory from which you can then run Tahoe.  (This part is  
similar in spirit to the Mac OS packaging technique.)

Also, this install process never downloads anything from the Internet  
at install time, per our policy [2, 3], which also happens to be a  
policy some other people have, e.g. [4, 5].

This works on all of our supported platforms, which includes Linux,  
Solaris, Windows, Cygwin, and Mac OS X.

Oh yes, we also have our buildbot [6] automatically produce Debian  
packages for edgy, feisty, etch, and gutsy.

As far as I know, all of this is accomplished without breaking any of  
the use cases traditionally associated with setuptools /  
easy_install, for example "easy_install allmydata-tahoe" works, and  
if you want "setup.py install" to install into your standard system  
directory, it will.

The reason that I am posting this is to let other programmers know  
that setuptools is actually a pretty useful tool, even if the use  
cases that you want to support are incompatible with certain  
easy_install traditions such as fetching new packages from the  
internet at buildtime or installing into your system directory.

Regards,

Zooko

P.S.  Two days ago I was able to remove twisted from the list of  
"Manual Dependencies" that people have to be aware of in order to try  
out Allmydata Tahoe from the source tarball.

I think I can safely remove pyOpenSSL now, but that remains to be  
properly tested by our buildbot.

I will be able to remove Crypto++ soon, due to the pycryptopp [7]  
library.

If I can figure out a hack to work-around one of the major  
frustrations of setuptools (that you can't simply run "./setup.py  
install --prefix=$FOO"), and if I finish my setuptools plugin to run  
Twisted trial instead pyunit when "./setup.py test", then I'll be  
able to remove GNU make from the dependencies.

That will leave only g++, Python, and OpenSSL as packages that a  
programmer has to manually deal with in order to try out Allmydata  
Tahoe from source.

[1] http://allmydata.org/trac/tahoe/browser/_auto_deps.py
[2] http://allmydata.org/trac/tahoe/wiki/Packaging
[3] http://allmydata.org/trac/tahoe/ticket/229
[4] http://bytes.com/forum/thread666455.html
[5] http://fedoraproject.org/wiki/Packaging/Python/Eggs
[6] http://allmydata.org/buildbot/waterfall?show_events=false
[7] http://allmydata.org/trac/pycryptopp