[Distutils] Packaging multiple wheels in the same archive

Nick Coghlan ncoghlan at gmail.com
Wed Nov 23 01:06:50 EST 2016


[Some folks are going to get this twice - unfortunately, Google's
mailing list mirrors are fundamentally broken, so replies to them
don't actually go to the original mailing list properly]

(Note for context: I stumbled across Wagon recently, and commented
that we don't currently have a good target-environment-independent way
of bundling up a set of wheels as a single transferable unit)

On 23 November 2016 at 03:44, Nir Cohen <nir36g at gmail.com> wrote:
> We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do just
> that and that's what we currently use to create and install our plugins.
> While wheel solves the problem of generating wheels, there is no single,
> standard method for taking an entire set of dependencies packaged in a
> single location and installing them in a different location.

Where I see this being potentially valuable is in terms of having a
common "multiwheel" transfer format that can be used for cases where
the goal is essentially wheelhouse caching and transfer. The two main
cases I'm aware of where this comes up:

- offline installation support (i.e. the Cloudify plugins use case,
where the installation environment doesn't have networked access to an
index server)
- saving and restoring the wheelhouse cache (e.g. this comes up in
container build pipelines)

The latter problem arises from an issue with the way some container
build environments (most notable Docker's) currently work: they always
run in a clean environment, which means they can't see the host's
wheel cache. One of the solutions to this is to let container builds
specify a "cache state" which is archived by the build management
service at the end of the build process, and then restored when
starting the next incremental image build.

This kind of cache transfer is already *possible* today, but having a
standardised way of doing it makes it easier for people to write
general purpose tooling around the concept, without requiring that the
tool used to create the archive be the same tool used to unpack it at
install time.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Distutils-SIG mailing list