[Distutils] PEP 426 updated (with more than you ever wanted to know about version schemes)

[Nick Coghlan]

On Tue, Feb 12, 2013 at 4:10 PM, Marcus Smith <qwcode at gmail.com> wrote:
> On Mon, Feb 11, 2013 at 9:36 PM, Donald Stufft <donald.stufft at gmail.com>
> wrote:
>> I think the biggest point of contention here is how does a project
>> that doesn't _need_ any of those new fields opt into the stricter
>> version comparisons?
>
> I guess the answer is that the next wave of tool updates has to give the
> user a way to specify this choice.

Right. No current version of any tool is going to emit metadata 1.3,
so there's going to be a necessary update as the tools provide
supports for the new fields. Exactly how they do that is going to be
up to the individual tools.

The point of the compliance notes is that if a version is
non-compliant, then packaging tools *must NOT* emit v1.3 metadata.
Whether they allow it (and emit earlier metadata) or throw an error is
between them and their users.

As far as the sorting of non-compliant versions in older metadata
goes, I now think the PEP should just encourage tools to implement the
sorting scheme defined in this version. The decision of how to handle
non-compliant version numbers from old versions of the metadata is
then entirely up to the tool, with the recommended behaviour being to
filter them out.

Then there's no need for projects to explicitly request metadata v1.3,
and the compliance constraint will clearly be on the tools developers
rather than the projects themselves.

> and in talking with Nick, his idea is that it's up to each tool to gather
> this however they want from users.
> I.e. no PEP390 successor that standardizes how "users" put metadata into
> their projects.
>
> that's probably worth another thread... the lack of  a PEP390ish successor.

My overarching ambition is to encourage migration to a model where
Archivers create sdists from source control, Builders create wheels
from sdists, and Installers are able to retrieve wheels and deploy
them onto a system. That is:

source checkout -> Archiver -> sdist -> Builder -> wheel -> Installer
-> deployed

(Desirable variants include allowing the "sdist" and "wheel" to be
temporary directories rather than actual archives when performing
multiple steps at once, translating the wheel to a platform specific
binary format like RPM prior to installation, as well as translating
the sdist to a platform specific source format like SRPM prior to
building)

However, unlike RPM, I'm nowhere near convinced that we need to use a
single metadata file to cover every step, nor that the metadata
formats used for tool interoperability need to be the *same* formats
that the tools expose to their users. PEP 427 (wheel) is about
providing the info an installer needs in order to deploy most
pre-built binaries onto a system - creating the appropriate wheel,
including metadata, is the responsibility of the Builder. PEP 426 is
about providing the metadata needed for dependency retrieval, *and*
making it possible to configure a builder (more on that below). Making
sure the necessary metadata for the build step is provided as part of
the sdist is the responsibility of the Archiver.

We don't really *care* what format the Archiver uses to gather info
from the user, so long as it uses PEPs 426 and 427 to communicate
those details to later steps in the chain.

So, to my mind, the next PEP we're missing is actually one for the
*sdist* format itself, including a definition for how the
meta-packaging system should invoke the sdist->wheel build step,
rather than one for the Archiver/Builder configuration data (which is
what PEP 390 tries to be, and which I'm not convinced needs to be
standardised at all, so long as the Archiver takes care of translating
it to the standard formats).

For example, distutils needs to get out of the installer business
(because "./setup.py install" is a fundamentally bad idea), but I'm
fine with it remaining in the standard library as the default archiver
and builder (with support added for a bdist_wheel command). The input
format for distutils/setuptools/distribute would thus continue to be
setup.py. The difference is that the Builder and Installer steps in
the chain would no longer invoke the project's setup.py - instead,
distutils would gain the ability to say "given this sdist, create a
wheel" (or other format). If there is info that the builder step of
distutils needed from the archiver, then it would define
"Setup-Requires-Dist: distutils" and perhaps "Extension: distutils",
adding whatever it needs to the PEP 426 metadata (or include a
separate metadata file and reference that from the main metadata).

distutils2 and bento would then be alternative combined
archiver/builders, with the responsibility of ensuring that enough
information was captured in the sdist (again likely using a
combination of Setup-Requires-Dist, Extension and additional metadata
files) to allow the build step to be executed without access to the
original source repo.

My primary near(ish) term objectives are to ensure:
- a project's choice of Archiver or Builder should *NOT* affect a
user's choice of Installer (or vice-versa)
- given an sdist and no other information, it should be possible to
create any bdist_* format

PEPs 426 and 427 between them should achieve the first objective,
while the other parts of PEP 426 should get us a long way towards
achieving the second (with "./setup.py bdist_wheel" as the interim
build hook, pending the introduction of anything better in a PEP for a
clarified sdist format).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia