[Distutils] Newbie questions about setuptools

Phillip J. Eby pje at telecommunity.com
Thu Jun 21 15:09:55 CEST 2007

At 07:33 AM 6/21/2007 -0500, Edward Ream wrote:
>>It appears to me as though you do not have a project that can be 
>>meaningfully installed or distributed using the distutils or setuptools,
>Doesn't this say more about distutils and setuptools than about Leo?

Depends on what you mean by "say".  :)

Per http://www.python.org/doc/2.1/dist/ , the distutils were created 
"to make Python modules and extensions easily available to a wider 
audience with very little overhead for build/release/install mechanics."

They aren't for installing arbitrary applications.  It just happens 
that, if your "application" consists solely of scripts that tag along 
with some libraries and immutable data, then they also work.

It's not reasonable to expect the distutils to be a general purpose 
software installation toolkit; they are, after all, for sharing 
Python libraries.

>> > To put it another way, disutils and setuptools are designed to 
>> install > *immutable* collections of code and data.
>The distinction between mutable and immutable data makes some (but 
>not complete) sense on Linux. It is irrelevant on Windows.

Actually, Microsoft has been trying to get people to stop putting 
mutable data in the \Program Files tree for many years now.  That's 
why user-specific $APPDATA directories exist there.

>Here is what I particularly like about setuptools:
>setup.py sdist register upload
>easy_install leo cleverly uses a generic upload_url
>setuptools.set can (at least sometimes) find all files managed by cvs.
>Why do any of the above require any particular organization of files?

They don't -- but a LOT of other features of easy_install *do* depend 
on them.  Notably, easy_install is intended to support (relatively) 
clean uninstallation, and the simultaneous installation of multiple 
versions of a project along the same PYTHONPATH.

Aside from locating files in the first place, most of easy_install's 
complexity comes from either 1) supporting near-arbitrary 
distutils-based projects or 2) managing sys.path.  Your project 
doesn't use either of these features, but they're absolutely vital 
for setuptools' *real* goal, which is to make widespread library 
sharing and reuse practical.

The distutils do fine for distributing individual projects, but does 
badly for inter-project dependencies.  This makes people who want to 
depend on other libraries either bundle them inside their project, or 
write their own replacement just to avoid the hassle of updating 
their bundled versions of things.  Then, you either end up with 
multiple copies of some library installed anyway, or version skew/stomp.

Setuptools was intended to fix this by making it easy to say, "my 
library uses these other libraries" -- and have them automatically 
installed, with version skew problems handled.  This is a delicate 
enough problem to solve without throwing mutable files into the mix.

>We in the Leo community never argue over user preferences because my 
>guiding design principle is this: if there are two or more even 
>halfway reasonable ways of doing something, Leo provides an option 
>that allows users to choose what works for them, and that's is the 
>end of the discusssion.  It seems that allowing me to distribute Leo 
>as I and Leo's users prefer would be a useful addition to setuptools.

Projects that don't consist of packages, immutable data, and scripts, 
don't really support library reuse very well.  In fact, projects that 
include mutable files, would be in direct *opposition* to 
easy_install's goals for package management, so it really shouldn't 
be surprising that it doesn't support them very well.  :)

If you'd like to use the distutils or setuptools to distribute leo, I 
would suggest restructuring it as a set of Python packages rather 
than as a collection of modules and directories.  (E.g., "import 
leo.config" rather than "import leoConfig".)  For plugin support, I 
would suggest you investigate setuptools "entry points" system, which 
many packages and applications use to automatically discover 
separately-installed plugins:


This method has the advantage that people can distribute plugins as 
setuptools-based projects, and users can install them with 
"easy_install pluginprojectname".  (See, for example, all the various 
TurboGears widget plugin projects on the Cheeseshop, the TG/Buffet 
templating engines, Chandler plugins, etc., for examples of plugin 
systems built this way.)

I'm not saying this would be easy or simple; many existing projects 
with other plugin or extension systems or using other distribution 
mechanisms have taken some time to migrate or are still migrating to 
this approach, in order to take advantage of the benefits.  And 
truthfully, most projects using the full spectrum of setuptools' 
plugin abilities are relatively new, written after the features were available.

Certainly, you could also retain older plugin mechanisms, and scan 
configured directories for plugins in addition to any "built-in" 
plugins that you provide in your main distribution project, so that 
you keep continuity for your existing users.

Of course, you may also decide that the cost/benefit isn't there for 
you, and that's understandable.  In that case, you may wish to use 
some other distribution mechanism, or create your own; setuptools 
unfortunately can't both achieve its own goals, and support 
installing non-importable things (aside from scripts and immutable 
package data) or arbitrary installation locations for project contents.

More information about the Distutils-SIG mailing list