[Distutils] Splitting large packages into multiple eggs

Bob Ippolito etrepum at mac.com
Mon Dec 12 04:02:43 CET 2005

On Dec 11, 2005, at 2:49 PM, Bob Ippolito wrote:

> On Dec 11, 2005, at 2:11 PM, Phillip J. Eby wrote:
>> At 01:37 PM 12/11/2005 -0800, Bob Ippolito wrote:
>>> I'm currently refactoring PyObjC and py2app to be setuptools-
>>> friendly, but one issue I'm coming across is that PyObjC is a large
>>> package, and normally people wouldn't want to include all of it.   
>>> The
>>> way to do that would be to split it up into lots of eggs, so py2app
>>> can include the subset of eggs that are necessary.
>>> The issue at hand is how to structure the setup.py to support
>>> creation of multiple eggs, with an egg for installation purposes  
>>> that
>>> depends on everything.  PyObjC can be broken up into about 30 eggs,
>>> one for each package, one for the Xcode support (which depends on
>>> py2app and altgraph), one for all of the tests (or maybe separate
>>> eggs for each test suite).  Obviously I'm not looking to create 30+
>>> setup.py files, so what do I do?
>> Um, create a script to generate the setup.py files for you?  ;)
>> If you don't mind building everything together, you could just  
>> create a data structure and then call setup() in a "for" loop that  
>> loops through all the projects...  no, scratch that, it won't work  
>> unless you clean out the build directory every single time.
>> Seriously, distutils isn't made for this.  I originally intended  
>> to make setuptools do something like it for PEAK, and then gave up  
>> because it really just doesn't work.  Too many of the "install"  
>> commands depend on just copying everything that's in the build  
>> directories, which means you'll get inter-project crosstalk.
>> The One Obvious Way (30+ setup.py files) is in fact the only  
>> practical way without some pretty major work on the distutils or  
>> scaffolding to work around them.  I'd suggest, however, that maybe  
>> one egg per package is too fine-grained, and you break it into  
>> just a handful of eggs instead.
> Ok, so I'll start off with a handful of setup.py files to see how  
> it works out..
> For install_requires and setup_requires, how I can I let ez_setup  
> know that the subprojects are in the same tarball relative to the  
> main setup.py?

It looks like easy_install can't do this yet.  Here's the layout of  
what I have right now:

The main setup.py is at:

and each subproject lives in here:

Each subproject contains only a setup.py... The setup.py performs its  
job by changing back to the main source directory and then building  
some stuff as if it were just the main setup.py.  It's done this way  
because there are a lot of include files that these things share, and  
it makes the refactoring less painful.

It seems that the PackageIndex only wants to find eggs and source  
packages, but not source dirs... I can't make source packages out of  
them because they refer to source that exists elsewhere (in the  
parent tree).  I'm not very familiar with the sources yet, so it  
would take a while for me to write such a patch.

Also, I'm going to want a way to have "setup.py develop" ensure that  
all of the subprojects are up to date.


More information about the Distutils-SIG mailing list