[Distutils] PEP 426: proposed metadata caching convention

[Daniel Holth]

On Wed, Feb 27, 2013 at 6:45 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Wed, Feb 27, 2013 at 8:52 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> Just to clarify: when I say "hooks", what I mean is "setuptools-style entry
>> points that the installer looks for, which are used to customise the installation
>> process".
>
> The command to create a wheel from a source archive is currently still
> "./setup.py bdist_wheel". This may be executed on an appropriate build
> system rather than the target system, but aside from that everything
> in setup.py should still execute normally. This is the major
> difference between the current attempt and distutils2: du2 made moving
> from setup.py to setup.cfg a requirement to generate the new metadata
> format. By contrast, I want at least distribute, as well as the Python
> 3.4 distutils, to be able to generate wheels (including the new
> metadata) from current setup.py files.

Vinay's distlib has taken the wheel spec at its word, runs an
unmodified "install" command with all the various paths set to
wheel-compatible distname-1.0.data/scripts etc., and converts the
.egg-info directory to .dist-info the same as bdist_wheel's final
step.

All wheel does is it takes a basic assumption of distutils2 (avoid
running setup.py), rearranges it slightly (avoid running setup.py at
install time) and magically people seem to like it. I wanted lxml to
compile faster and wound up with a distutils escape hatch. Now I think
that avoiding running *distutils* at install time is much more
important than avoiding setup.py.

>> I believe it is possible to provide limited extensibility using hooks
>> without it leading to the complete ad-hocery that setup.py entails.
>
> For version 1.0, the only install-time modification that all wheel
> installers must implement is fanning files out to their target
> locations based on sysconfig directories and rewriting script shebang
> lines (they may also want to generate parallel Windows executables,
> but with the Windows launcher, that's less necessary).
>
> If a project needs more than that, they cannot ship wheels at this
> time, and will need to continue shipping source distributions that can
> execute arbitrary code at install time. Alternatively (and
> preferably), such a project could split out a support library that is
> wheel compatible, and have a separate component that must be installed
> from source and is able to make arbitrary changes to the target
> system.
>
> *Incremental* change, and explicitly leaving some use cases to source
> distribution and ./setup.py for the moment is the key to creating a
> distribution format that is as simple as we can make it while still
> supporting a wide variety of use cases. Will we eventually get
> pre-install and post-install hooks ala RPM and other platform specific
> systems? Quite possibly. But let's see how far we can get without them
> first - in particular, I want to focus people's initial efforts on
> putting the smarts into the wheel *creation* process rather than
> delaying decisions until install time.

It's just the 1.0 release. There's no hurry to write the document
entitled "PEP 376 is now the/a standard *interchange* format for
distribution metadata; here's how you can experiment with caching
runtime introspection." Other tasks such as "create the simplest
possible useful packaging system for the stdlib [by only including the
install feature]" and "create an ecosystem of interoperable
third-party products to do everything else" are higher up on the Grand
Python Packaging Plan or GP3 (tm) to-do list.

> The initial problem I believe we need to solve is the one of arcane
> build systems for key dependencies, and the simple fact that most
> Windows users aren't equipped to build software written in C in the
> first place. Eggs tried to tackle that problem years ago, but ignored
> things like the Filesystem Hierarchy Standard and the interests of OS
> distributions and system administrators, limiting its adoption to
> those developers that were happy with the idea of storing *everything*
> inside a single directory (the various legitimate concerns with the
> default behaviour of easy_install also didn't help). Wheel is designed
> to integrate more cleanly with platform specific conventions,
> hopefully overcoming some of those past objections to the egg format.

It's designed to make binary packaging generally interesting, even if
you don't have C extensions, or even if you do have a C compiler. This
will hopefully be a benefit to our Windows community as well.

> This preliminary approach also integrates well with centralised system
> management tools like Puppet, Chef and Salt - for those, the states
> and configurations of services and other components are handled
> through the management infrastructure, and the language specific
> package management tools are just a way to get the application code
> onto the target systems in a controlled fashion.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig