[Distutils] Splitting large packages into multiple eggs

[Phillip J. Eby]

At 07:02 PM 12/11/2005 -0800, Bob Ippolito wrote:
>It looks like easy_install can't do this yet.

And it's not going to be able to.  The closest thing I can envision is that 
if you have a master project with a setup.py that runs each child project's 
setup.py (or just a setup() call), such that each gets passed a "bdist_egg" 
command by easy_install.  EasyInstall already detects when multiple eggs 
are built by a single setup.py, and processes all of them.

That way, you could have one PyObjC project that contains all the others, 
and builds multiple eggs, including one that just specifies dependencies on 
the others.

The only possible issue that might arise is if there are inter-egg 
dependencies and the eggs are built out-of-order.  In that case, 
easy_install might incorrectly conclude that it needs one of the built 
eggs, before it has processed it.  I could probably add some code to make 
this more robust.

Anyway, the only limitation of this approach is that you won't be able to 
build any individual packages from source, only the overall package as a 
whole.  However, it will be possible to build and even upload all the eggs 
using "setup.py bdist_egg upload", so easy_install will be able to find the 
binary packages.  Or you can just have your PyPI download URL point to a 
directory where you dump the latest eggs; if you're running your bdist 
commands on the web server (or rsync it), you can use the 'rotate' command 
to delete outdated snapshots.

But anybody who wants to build from source will have to use the master 
package, rather than any individual subpackages.


>Also, I'm going to want a way to have "setup.py develop" ensure that
>all of the subprojects are up to date.

This should work normally if your setup.py just calls setup() for each 
subproject and the master; again this should be in dependencies-first order.