[Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes)

[Nick Coghlan]

Disclaimer: these are ideas that have been kicking around in my head
for a while, but they're still somewhat half-baked. By the time we get
around to turning them into a PEP, it's entirely possible I will have
changed my mind again about various aspects. There's also the fact
that I'll probably be trying to convince other people (like Vinay
Sajip) to actually implement my ideas rather than having the time to
implement them myself, so take most of the things you read below with
a rather large grain of salt :)

On Tue, Feb 12, 2013 at 9:25 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> However, the problem is that distutils currently only supports
> creating the old .egg-info format metadata. I believe it's possible to
> rename the egg-info file and munge the metadata directory structure to
> give a .dist-info format layout (IIRC, that's roughly what the
> existing setuptools bdist_wheel extension provided by the wheel
> project does) but I'm not sure if simply doing that meets the new
> standards. Retrofitting full support for the new PEPs into distutils
> may go beyond the limits of the "distutils freeze"...

Based on a recent python-dev discussion, the distutils freeze is
(mostly) dead. There are still greater limits on refactoring than is
normal for the rest of the stdlib (basically: treat even internal APIs
as public ones for maintenance releases, consult with the setuptools &
distribute developers if contemplating significant internal changes
for 3.4), but a bdist_wheel command would be a most welcome addition
for 3.4.

Besides, unlike the previous distutils2/packaging plan, in my
Archiver/Builder/Installer concept, distutils will still have a place
as the default Archiver/Builder (if all you want to do is a simple
pure Python distribution, with perhaps a simple C extension, distutils
works OK, and when you hit its limits distribute, distutils2 or bento
will only be a download away).

> Regardless, I'll do a proof of concept at least - it's not a big job
> (http://bugs.python.org/issue13189 was my last version, the
> bdist_simple format - here's hoping this one doesn't die of apathy the
> same way :-().

I've learned a lot about packaging in the last year and a half, mostly
through learning how to deal with the SRPM/RPM toolchain after
starting with RH. That's actually where I took the
Archiver->Builder->Installer model from, except that in RPM the spec
file is still the "one metadata file to rule them all".

In our case, I think the current existence of 4 distinct input formats
(classic setup.py, setuptools enhanced setup.py, distutils2 setup.cfg,
bento.info) means we need to do something slightly different, and
clearly define the interchange formats (sdist for the source form,
wheel for the binary form) where we only need to convince the tool
developers to agree, without trying to change the practices of the
entire Python community (which I think has been conclusively shown to
be an impossible task, as well as being a wonderful way to burn out
development leads).

The RPM spec also has lots of complexity to handle patching an
unmodified upstream tarball, but I'm happy to pass that task off to
the individual builders rather than trying to cover it in any meta-spec.

We could then take a decent run at creating pydist (aka "pysetup v2")
for Python 3.4 (probably under the distlib banner), which would aim to
handle the following tasks:

1. Given an sdist, sign it, upload it to PyPI and update the PyPI metadata
2. Given an sdist, unpack it into a specified directory
3. Given an unpacked directory, turn it into an sdist (and check validity)
4. Given an sdist, turn it into a wheel (using "./sdist2wheel.py" if
present, "./setup.py bdist_wheel" otherwise)
5. Given an unpacked directory, turn it into a wheel (and check validity)
6. Given a wheel, install it on the current system
7. Retrieve named sdists from PyPI (with dependencies and signature checking)
8. Retrieve named wheels from PyPI (with dependencies and signature checking)
9. Install a named distribution on the local system (retrieving from
PyPI as needed)
10. Remove an installed distribution from the local system

Note that pydist probably would *not* offer translation from sdist or
wheel to platform specific source or binary formats, instead leaving
that task to platform specific toolchains like deb_helper or pyp2rpm,
or third party templating projects like py2pack.

The big difference relative to pysetup in distutils2/packaging is that
the expected input format for pydist would be a complete sdist, rather
than the setup.cfg file expected by pysetup.

The responsibilities of the various components would then be:

Archiver: create an sdist with PKG-INFO, sdist2wheel.py, any other
metadata needed by the associated builder, and anything else you
choose to include (the builder will get a chance to filter things
further when creating the corresponding wheel files).

Builder: when "./sdist2wheel.py" is invoked with a destination
directory from an unpacked sdist, create an unpacked wheel file in the
specified directory (using a subdirectory that matches the desired
name of the wheel)
- pydist does *not* provide configuration options to the builder. If
you want to control the generated wheel, you have to know what build
system is being used and invoke it directly rather than going through
pydist or sdist2wheel.
- pydist takes care of unpacking the sdist and archiving the unpacked wheel

Installer: at least pydist features 2-11 listed above (these common
features are the ones distlib should cover), as well as any other
features the tool implementors consider useful to their users

As with the distutils2 design, pydist would be deliberately limited
relative to pip (e.g. only trusting the PSF cert, only allowing
downloads direct from PyPI) in order to keep it simple. For more
complex installation scenarios, "pydist install pip" would be a
necessary first step.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia