[Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0

Nick Coghlan ncoghlan at gmail.com
Tue Feb 19 12:42:20 CET 2013


On Tue, Feb 19, 2013 at 8:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Tue, 19 Feb 2013 10:24:15 +0000,
> Paul Moore <p.f.moore at gmail.com> a écrit :
>> On Tuesday, 19 February 2013, M.-A. Lemburg wrote:
>>
>> >
>> > The only tool in wide spread use that understands part of the 1.2
>> > data is setuptools/distribute, but it can only understand the
>> > Requires-Dist field of that version of the spec (only because the
>> > 1.1 Requires field was deprecated) and interprets a Provides-Extra
>> > field which isn't even standard. All other 1.2 fields are ignored.
>> > setuptools/distribute still writes 1.1 meta-data.
>> >
>> > I've never seen environment markers being used or supported
>> > in the wild.
>> >
>> > I'm not against modernizing the format, but given that version 1.2
>> > has been out for around 8 years now, without much following,
>> > I think we need to make the implementation bit a requirement
>> > before accepting the PEP.
>> >
>>
>> The wheel project uses metadata 2.0 and environment markers - indeed,
>> the PEP was written to formalise what wheel was implementing
>> (specifically so that pip was happy to incorporate support).
>
> I'm unsure what this means. Does the "wheel project" (I thought wheel
> was a format? is it also a project?) reimplement its own wheel of
> metadata parsing and generating?
>
>> I agree that standard library support would be good, either via
>> Distutils or by incorporating distlib, but I don't think it is
>> essential for acceptance.
>
> I think Marc-André is right that the acceptability of the standard
> should be judged on the availability of (preferably standard)
> implementations. If the standard isn't implemented, then perhaps it
> means it's too ambitious / too complex / victim of the second-system
> effect.

The most complex part of the PEP is still the version ordering, and
that's implemented in the analysis script (initially by Vinay, with
one small tweak by me to fix the regular expression for the new
validity rules). Everything else was either already implemented for
distutils2/distlib, or is a trivial check (like scanning for "a", "b",
"c" or "dev" to identify pre-releases and filter them out if none of
those characters are present in the corresponding version specifier).

Nothing in the PEP is particularly original - almost all of it is
either stolen from other build and packaging systems, or is designed
to provide a *discoverable* alternative to existing
setuptools/distribute/pip practices (specifically, the new extension
mechanism means things like entry points can be defined in the
standard metadata files without python-dev needing to get involved).

Probably the most novel concepts in the PEP are "Obsoleted-By" (so a
project can issue one last release to inform users directly that it is
dead, and there is an alternative available) and the "Private-Version"
field to handle the case where a projects' preferred version scheme
requires translation to match the more restricted format understood by
the installation tools.

Yes, the spec in aggregate is complex, but it's complex because
software distribution and dependency analysis are complex tasks. If
people want to simplify it, they can try to rebut the rationale for
the individual changes that is included at the end of the PEP, but
"it's complex" alone is not an argument.

> So I kind of agree with him the PEP shouldn't be accepted until there's
> a decent patch pending for the stdlib. That's not a judgement on the
> PEP's quality. Just an advice to remain cautious, especially if the
> previous metadata version already enjoys a very poor adoption rate.

There were at least a couple of very good reasons for not adopting metadata 1.2:
- it didn't eliminate the need for ad hoc metadata extension files
like entry_points.txt, so it didn't solve one of the problems it set
out to solve
- it's proposed version ordering scheme was incompatible with the
prevailing usage on PyPI in a couple of key ways (hence the systematic
statistical comparisons in the new PEP - the numbers that are already
there shows that the current version of the standard is highly
compatible even with *hidden* revisions included in the statistics.
I'm currently redoing the analysis to also show public versions only,
which should make the proportions look even better, as many of the
remaining glitches only affected old pre-releases).

The adoption of metadata 1.2 also got caught up in the "distutils must
die" rhetoric that meant the distutils2 project got bogged down in
trying to eliminate setup.py completely, when the only thing that
*really* needs to die is "./setup.py install" (and even that can be
acceptable on a development system, we just want to eliminate the need
to ever run it on a deployment target). That's the key change in
philosophy between this PEP and the previous one: I'm working
backwards from what *installers* need, and creating a specification
that allows build tools, including distutils, to give it to them.

I *don't really care* how those build tools get the info from their
users - setup.py, setup.cfg, bento.info, it doesn't matter to me.

> By the way, pip may be nice, but it's still a third-party tool, and
> perhaps not even as widespread as distribute / setuptools.

The main easy_install feature that pip couldn't previously match was
installing from pre-built binaries (since easy_install had eggs),
which is why the wheel format is so important.

My current plan for Python 3.4 is for pip to be formally endorsed as
the recommended tool that supports installing from an sdist, while the
stdlib will ship a "pydist" tool that can only install from prebuilt
wheel files. This will keep the stdlib tool simple (i.e. no need to
try to find installers on arbitrary target systems), while easily
allowing users to bootstrap a full fledged installer through "pydist
install pip". (This will all play it out in PEPs before 3.4 rolls
around - and there will likely be recruiting and documentation efforts
well in advance of that)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Python-Dev mailing list