[Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)

[Nick Coghlan]

On 21 Jun 2013 19:52, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 21 June 2013 00:24, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> > How close am I to reality here?
>>
>> Close, except that this is explicitly an "ecosystem first" plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification.
>>
>> The only "must have" on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month.
>
> Thanks. My remaining concern is the "chicken and egg" question of how this much richer metadata will get populated - things like dependency metadata is only as useful as the accuracy of the information. Some of it can be translated from existing legacy formats (notably setuptools) and hopefully once Appendix A is completed, we'll have a better understanding of how that will work, and how complete it will be.

The initial metadata will come from existing setuptools based requires
and install_requires definitions (thanks to easy_install and pip,
there's a lot of good dependency metadata available already). Having
PyPI extract and publish that info (rather than having to download
each distribution to read it) will already be a substantial step up
from the status quo.

Over the next few years, I expect professional developers and Linux
distribution packagers to help improve the metadata of their existing
dependencies. At the moment, we *can't* easily propose upstream
patches for dependency issues, but one of my goals with the PEP is to
make fully automated generation of distro policy compliant distro
specific packages feasible (at least for packages which don't have any
special requirements). At that point, the distro's preferred "upstream
first" approach becomes feasible for metadata fixes as well, rather
than being limited to source code changes.

> As regards the actual PEP contents, my thoughts are basically:
>
> - I'm +1 on the PEP as a whole. Most of it seems to be to be entirely reasonable and an important step forward
>
> - I don't have much personal use for the more complex items like semantic dependencies, extras, hooks, etc, but I see no particular problems with them

Yeah, they're definitely power tools. I see several of them as the
software distribution equivalent of metaclasses: if you're wondering
whether or not you need them, you don't need them. On the other hand,
when you *do* need them, you'd be completely stuck if they didn't
exist. I'm hopeful that install hooks, metadata extensions and direct
references will provide sufficient "escape valves" to handle some
seriously esoteric use cases without any further changes to the
specification.

> - Thinking about what I *do* use, the most glaring omission is a specific "supports Python 3" indicator. I know it's available in the classifiers, and it may have already been discussed, but is there realistically any benefit in formalising this? The environment markers section covers this somewhat, but is it within the remit of the PEP to recommend how (or even when) packages should declare what major versions of Python they support? (And should installers be warning about attempts to install packages that don't support the relevant Python major version?)

The quick compatibility check is actually part of the wheel file
naming specification (it's covered by the compatibility tags defined
in PEP 425). For PEP 426, I originally retained the separate
"Requires-Python" field, but eventually realised it was redundant
given the "supports_environments" field
(http://www.python.org/dev/peps/pep-0426/#supports-environments) and
the "python_version" and "python_full_version" variables in
environment markers. It gets a little clumsy if you want to express a
Python version constraint *and* a platform constraint (due to the
implicit or), but the current approach does have the virtue of
handling cases where version requirements differ between platforms.
For example, if you need atomic renaming capabilities, then they've
been available on POSIX systems since pretty much forever, but on
Windows you need Python 3.3 so you can have access to os.replace. In
the current PEP, that might look like:

    "supports_environments": ["sys_platform != 'win32' and
python_version >= 2.6", "sys_platform == 'win32' and python_version >=
3.3"]

If your package was source compatible between 2.6, 2.7 and 3.3+ you might say:

    "supports_environments": ["python_version == 2.6", "python_version
== 2.7", "python_version >= 3.3"]

However, looking at the PEP, I realise I didn't actually update the
remaining text to reflect that realisation - I just deleted the old
text that was no longer applicable. I'll add an example that shows
using "supports_environments" to specify a minimum Python version in a
way installation tools are expected to check.

I'll also make it clearer that "supports_environments" describes what
the *sdist* supports. Generating a wheel file may limit it further (as
indicated by the compatibility tags).

Cheers,
Nick.