[Distutils] Setuptools download cache
Phillip J. Eby
pje at telecommunity.com
Thu Aug 21 00:39:23 CEST 2008
At 04:40 PM 8/20/2008 -0400, David Turner wrote:
>Attached is a patch which adds download caching to setuptools.
>
>At TOPP (http://topp.openplans.org/), we use a system called fassembler
>to build our opencore stack. It creates approximately a dozen
>virtualenvs, each with their own lib/python, and then uses setuptools to
>install lots of libraries. Some of these libraries are common among
>multiple apps, but we install multiple copies for ease of development.
>And every time we rebuild, we start the whole process over again. The
>major slowdown in building is downloading a bunch of things which
>probably haven't changed since last time we downloaded them. This
>patch will let us maintain a cache of all downloads, and thus do builds
>much faster.
The process I'd suggest for this use case is to build the external
libraries using:
easy_install -f cache_dir -zmaxd cache_dir lib1 lib2 ...
This command will NOT go to the web for new versions of libraries,
unless you also use -U. But it will ensure that the specified
libraries have suitable eggs in cache_dir.
Then, to install a given set of libraries to a virtualenv, use:
easy_install -f cache_dir lib1 lib2 ...
Or, if you really insist on multiple copies of the eggs (instead of
just linking to them), use:
easy_install -af cache_dir lib1 lib2 ...
(which will copy the .egg files even if they could be used in place).
Unlike your caching proposal, this approach gives you finer control
over which libraries to update, when. You can also update the cache
without changing what's installed in a given virtualenv.
More information about the Distutils-SIG
mailing list