On Thu, Jun 20, 2013 at 9:07 AM, Nick Coghlan wrote:

New versions of PEP 426 and PEP 440 are up on python.org:

PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/

(as before, not including them inline due to sheer length)

The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo)

There have been several significant changes to the main metadata spec in PEP 426:

* Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed)

The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them

Other less significant changes to PEP 426 include:

* Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments)

The other PEP 440 changes are also relatively minor:

- what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label

The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed.

Cheers, Nick.

FYI the tip revision of wheel (bdist_wheel, https://bitbucket.org/dholth/wheel/) produces pymeta.json that is almost up-to-date with this version of the spec, except that it still inlines the description rather than saving it to a separate file. bdist_wheel has always worked by converting setuptools metadata to PEP metadata.

I still think the testing part of "the interchange format between software publication and integration tools" is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? best and thanks for your good work, holger On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote:

New versions of PEP 426 and PEP 440 are up on python.org:

PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/

(as before, not including them inline due to sheer length)

The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo)

There have been several significant changes to the main metadata spec in PEP 426:

* Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed)

The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them

Other less significant changes to PEP 426 include:

* Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments)

The other PEP 440 changes are also relatively minor:

- what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label

The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote:

On 21 Jun 2013 00:36, "holger krekel" wrote:

...

I still think the testing part of "the interchange format between software publication and integration tools" is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something?

If "setup.py test" works for a distribution today, it will still work under PEP 426. Anything more is deliberately deferred (as noted in the PEP).

I am a little confused. Early in the PEP you say: The published metadata for distributions SHOULD allow integrators, with the aid of build and integration tools, to: ... * if supported by the distribution, run the distribution's automatic test suite on an installed instance of the distribution but a few sentences later: The current iteration of the metadata relies on the distutils commands system to support other necessary integration and deployment activities: ... python setup.py test: run the distribution's test suite on a built (but not yet installed) distribution And i am not sure what exactly PEP426 defines wrt to testing now. I saw the "test_requires" and "test_may_requires" fields and thought they were part of an idea already how to configure running of tests. But is there anything more in the current PEP426 that i missed? To help with my confusion, could you maybe describe how "python setup.py test" is to interoperate (if at all) with the new metadata format as is? thanks, holger

Cheers, Nick.

...

best and thanks for your good work,

holger

On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote:

...

New versions of PEP 426 and PEP 440 are up on python.org:

PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/

(as before, not including them inline due to sheer length)

The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo)

There have been several significant changes to the main metadata spec in PEP 426:

* Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed)

The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them

Other less significant changes to PEP 426 include:

* Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments)

The other PEP 440 changes are also relatively minor:

- what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label

The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 20 June 2013 21:54, Daniel Holth wrote:

These metadata PEPs are mostly about the installer, database of installed files, and package indexes.

In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) but my assumption is that it'll go something like this: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). 5. Distlib will follow suit. The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. How close am I to reality here? Paul

On 21 Jun 2013 06:19, "holger krekel" wrote:

On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote:

...

On 21 Jun 2013 00:36, "holger krekel" wrote:

...

I still think the testing part of "the interchange format between software publication and integration tools" is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something?

If "setup.py test" works for a distribution today, it will still work

under

...

PEP 426. Anything more is deliberately deferred (as noted in the PEP).

I am a little confused. Early in the PEP you say:

The published metadata for distributions SHOULD allow integrators, with the aid of build and integration tools, to:

... * if supported by the distribution, run the distribution's automatic test suite on an installed instance of the distribution

It's just an editing error - this was explicitly rejected (and is listed as a rejected feature), I just missed one of the references that was in the previous draft. Cheers, Nick.

but a few sentences later:

The current iteration of the metadata relies on the distutils commands system to support other necessary integration and deployment activities:

... python setup.py test: run the distribution's test suite on a built (but not yet installed) distribution

And i am not sure what exactly PEP426 defines wrt to testing now. I saw the "test_requires" and "test_may_requires" fields and thought they were part of an idea already how to configure running of tests. But is there anything more in the current PEP426 that i missed?

To help with my confusion, could you maybe describe how "python setup.py test" is to interoperate (if at all) with the new metadata format as is?

thanks, holger

...

Cheers, Nick.

...

best and thanks for your good work,

holger

On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote:

...

New versions of PEP 426 and PEP 440 are up on python.org:

PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/

(as before, not including them inline due to sheer length)

The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs

repo)

...

...

...

There have been several significant changes to the main metadata

spec

...

in PEP 426:

* Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed)

The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them

Other less significant changes to PEP 426 include:

* Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments)

The other PEP 440 changes are also relatively minor:

- what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label

The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 21 June 2013 00:24, Nick Coghlan wrote:

...

How close am I to reality here?

Close, except that this is explicitly an "ecosystem first" plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification.

The only "must have" on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month.

Thanks. My remaining concern is the "chicken and egg" question of how this much richer metadata will get populated - things like dependency metadata is only as useful as the accuracy of the information. Some of it can be translated from existing legacy formats (notably setuptools) and hopefully once Appendix A is completed, we'll have a better understanding of how that will work, and how complete it will be. As regards the actual PEP contents, my thoughts are basically: - I'm +1 on the PEP as a whole. Most of it seems to be to be entirely reasonable and an important step forward - I don't have much personal use for the more complex items like semantic dependencies, extras, hooks, etc, but I see no particular problems with them - Thinking about what I *do* use, the most glaring omission is a specific "supports Python 3" indicator. I know it's available in the classifiers, and it may have already been discussed, but is there realistically any benefit in formalising this? The environment markers section covers this somewhat, but is it within the remit of the PEP to recommend how (or even when) packages should declare what major versions of Python they support? (And should installers be warning about attempts to install packages that don't support the relevant Python major version?) Paul

On 21 June 2013 12:35, Nick Coghlan wrote:

The quick compatibility check is actually part of the wheel file naming specification (it's covered by the compatibility tags defined in PEP 425).

Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec?

For PEP 426, I originally retained the separate "Requires-Python" field, but eventually realised it was redundant given the "supports_environments" field (http://www.python.org/dev/peps/pep-0426/#supports-environments) and the "python_version" and "python_full_version" variables in environment markers. It gets a little clumsy if you want to express a Python version constraint *and* a platform constraint (due to the implicit or), but the current approach does have the virtue of handling cases where version requirements differ between platforms. For example, if you need atomic renaming capabilities, then they've been available on POSIX systems since pretty much forever, but on Windows you need Python 3.3 so you can have access to os.replace. In the current PEP, that might look like:

"supports_environments": ["sys_platform != 'win32' and python_version >= 2.6", "sys_platform == 'win32' and python_version >= 3.3"]

Yes, but that's very much the "power user" end of the spectrum, as you mentioned before.

If your package was source compatible between 2.6, 2.7 and 3.3+ you might say:

"supports_environments": ["python_version == 2.6", "python_version == 2.7", "python_version >= 3.3"]

More often than not, my thinking is more at the level of "did I take the time to make the code dual-version compatible" - things like putting brackets round print arguments. Or simply "will I accept bug reports for both Python 2 and Python 3, even though I only really test on Python3". The supports_environments tag seems too precise for that sort of very broad brush approach. However, looking at the PEP, I realise I didn't actually update the

remaining text to reflect that realisation - I just deleted the old text that was no longer applicable. I'll add an example that shows using "supports_environments" to specify a minimum Python version in a way installation tools are expected to check.

I'll also make it clearer that "supports_environments" describes what the *sdist* supports. Generating a wheel file may limit it further (as indicated by the compatibility tags).

That sounds good. I'd suggest examples like twisted (doesn't support Python 3 yet), virtualenv (supports Python 2.6 or greater, including 3.x) and maybe something that is purely for Python 3. I'd also like to see a paragraph explaining that installers SHOULD at least respect python version specs (even if they don't handle the full environment marker spec). I'm not sure how practical it would be to require that sort of partial support, though, which is my main reservation about using the environment spec for this. (I think respecting language version specs is important, but the full environment spec is no more than a nice to have). Also, it seems to me that wheel could use the Python version spec to generate appropriate compatibility tags. Again, that's not possible for tags in their full generality, but is possible for "Python version only" tags. And again, I'd like to see something saying that wheel generators should do this (otherwise, there's a duplication somewhere as this information would need to be specified elsewhere in a generator-specific manner, like the setup.cfg extension I mentioned above). Paul

On 21 Jun, 2013, at 1:24, Nick Coghlan wrote:

...

1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar.

The initial vehicles for migration will be Daniel's bdist_wheel command and upgrades to setuptools.

Standard library support may not land for 3.4, but we will at least have the pip bootstrapping mechanism in place (which in turn will allow retrieval of setuptools).

What would be needed to land support for 3.4? Is it "just" a matter of pushing distlib to the stdlib and updating distutils (add a way to configure 2.0 metadata, add bdist_wheel and update sdist), or are there other other issues? Well, other than finalizing the PEP and its implementation in distlib? Ronald

On 22 June 2013 00:10, Paul Moore wrote:

On 21 June 2013 14:30, Ronald Oussoren wrote:

...

...

Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec?

No, but it should be in the spec/documentation for the tool that builds sdists or wheels. That is, when metadata 2.0 support is added to distutils/setuptools/... it should document how the metadata is configured.

So a package author creates a package that only works with Python 3, builds a sdist using a tool-specific option that states this, and that data isn't recorded in the sdist so that whoever uses the sdist to build a wheel has to find that out elsewhere and re-specify it to build a wheel? That sounds wrong, unless I'm misunderstanding you...

This is a build problem. The metadata 2.0 spec *deliberately* avoids trying to solve build problems, it merely leaves the door open to attempting to solve them in the future. It *may* make sense for a build tool to check the "supports_environment" flags for implied options when building a wheel, but that's up to the build tool developer. Designing a metabuild system that is actually better than the setup.py based command line API is an incredibly hard problem, and I'm not yet convinced it's a problem we really need to solve. Killing "setup.py install"? Sure, that definitely needs to happen and decoupling the intergrator's choice of install tool from the developer's choice of build tool is the main aim of metadata 2.0. Killing "setup.py dist_info/sdist/bdist_wheel/test", though? I'm *not* convinced that needs to happen, since there's no way to decouple of project's build system from itself and projects like d2to1 show that creating a setup.py based shim for an alternate build system doesn't need to be *that* painful, particularly once there are well documented standard archive formats to target. Even if a setup.py replacement on the build side does happen eventually, it's a long way down the priority list. In the meantime, once sdist 2.0 is officially defined in a PEP, build tools will be able to define their own metadata extensions to record whatever they want in the sdist files. They'll also be free to add arbitrary files to the dist-info directory and have them propagate through the wheel files and all the way to the installation database. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Tip revision of bdist_wheel ( https://bitbucket.org/dholth/wheel/ ) supports this revision of the spec, with the description in a separate file. It also has better support for representing conditional dependencies in the metadata. On Fri, Jun 21, 2013 at 10:57 AM, Paul Moore wrote:

On 21 June 2013 15:52, Nick Coghlan wrote:

...

In the meantime, once sdist 2.0 is officially defined in a PEP, build tools will be able to define their own metadata extensions to record whatever they want in the sdist files. They'll also be free to add arbitrary files to the dist-info directory and have them propagate through the wheel files and all the way to the installation database.

Ah, OK. I see how that would work now. Thanks for the clarification. Paul

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 23 June 2013 08:05, Nick Coghlan wrote:

PEP 426 has now been updated based on the feedback on this thread (and to handle some "todo" items that were pending anyway): http://hg.python.org/peps/rev/3f733fe7c06c

Sorry I didn't think of this sooner, but I don't see in the PEP a brief summary of precisely what metadata is required. From the "Core Metadata" section I infer that a project with *only* Name and Version (and Metadata-Version, which will be supplied by the build tool) is valid. That's fine, but notably more permissive than earlier versions of the spec. One thing I see is "Index servers MAY require that this field be provided" for some metadata (e.g., Summary") which is fine, I guess, but implies that somewhere, the rules for PyPI should be documented. Also, is it true that build tools MAY require certain fields to be provided? I'm pretty sure setup.py sdist does at present. My logic here is that previously I would go to the metadata PEP to check precisely what fields I need when building my project. Now, I'm not sure where I'd go. (Note that most of my projects are internal only, so it's actually the extra distutils/setuptools requirements that interest me more than those of PyPI...) I can imagine the answer would be "the documentation of the individual tool" - but in reality said documentation often isn't present (possibly because earlier metadata PEPs documented this information centrally) Paul

On 23 June 2013 13:53, Nick Coghlan wrote:

I think I'll just move the summary back up into the core metadata. You can't make a sane index server without it, and it used to be mandatory, so "maybe mandatory, maybe not" just makes things more complicated than they need to be rather than simplifying anything.

Yes, it's the "maybe mandatory, depending on your tools" state that's annoying. If summary was the only one of them, then this change makes sense. Thanks. Older metadata versions made things like Download-URL mandatory, which was a pain for internal projects that didn't actually *have* a download URL, just a file on the software server or something (and no, file URLs don't count :-)) I guess they all become optional under Metadata 2.0 (as per the "Changes from Previous Versions" appendix).

I expect tools to come into line with PEP 426 on that front over time. Those that still generate legacy metadata will likely follow the legacy requirements, and for tools with stronger opinions about what should be provided, it will be up to the authors to document that properly.

OK, that's cool. As tools move towards *validating* to the 2.0 spec, the problem goes away. It's just that there's no way to make older tools validate to the new spec, so there's a transition here which we have to live with (as opposed to *generating* metadata 2.0, which can be handled by converters).

Contributions to the Python packaging user guide (a shared introduction to setuptools and pip) are also more than welcome (see http://python-packaging-user-guide.readthedocs.org). Marcus has made a good start based on the distribute guide (with less opinionated hyperbole), and I expect it to settle down to something quite usable once pip 1.4 is released and it can simply direct people towards setuptools 0.7+ as the default build tool (plus "wheel" for bdist_wheel support) and pip 1.4+ as the default installer (including support for installation from wheels, and automatic use of cached local builds for installation from source).

Yes, I should have a look at that.

We really don't want end users needing to read PEPs just to get started.

Agreed, but again there's not much else to work with at the moment :-) Paul

Nick Coghlan writes:

New versions of PEP 426 and PEP 440 are up on python.org:

Thanks for the update; I'm in the process of updating distlib to provide support for the updated spec. The tip version of distlib provides good coverage for all except the very latest updates to the spec, and they should be included soon. It's a very minor point, but I noticed that all the keys of the main metadata dictionary and all sub-dictionaries are lower case with underscore, except for the project_urls dictionary which has keys like "Home", "Documentation" etc. Is there any reason for this stylistic departure in just this part of the spec? Regards, Vinay Sajip

On Sun, Jun 23, 2013 at 1:52 PM, Vinay Sajip wrote:

Nick Coghlan writes:

...

* References wheel and warehouse from legacy metadata appendix

It may also be worth mentioning that:

* distlib supports converting legacy metadata to the new format; it's as simple as:

from distlib.metadata import Metadata metadata = Metadata(path='PKG-INFO') metadata.write(path='pymeta.json')

* Metadata extracted from setup.py, which includes full dependency information, is available through distlib's locate function. This includes setuptools' newest format for extras, which has support for for environment markers. For example:

from distlib.locators import locate dist = locate('setuptools') dist.metadata.write('pymeta.json')

Then pymeta.json contains (some fields omitted for brevity, but the dependency stuff is there):

{ "version": "0.7.4", "name": "setuptools", "metadata_version": "2.0", "run_may_require": [ { "environment": "sys_platform=='win32'", "dependencies": [ "wincertstore (== 0.1)" ], "extra": "ssl" }, { "environment": "sys_platform=='win32' and python_version=='2.4'", "dependencies": [ "ctypes (== 1.0.2)" ], "extra": "ssl" }, { "dependencies": [ "certifi (== 0.0.8)" ], "extra": "certs" }, { "environment": "python_version in '2.4, 2.5'", "dependencies": [ "ssl (== 1.16)" ], "extra": "ssl" } ], "extras": [ "certs", "ssl" ], }

I would welcome feedback from anyone who cares to clone/sync the distlib repository. A release of distlib with this functionality will occur once incorporation of the latest spec changes occurs, and I'm comfortable that distlib is reasonably stable following these changes. Additional testing by others can speed up getting to that point :-)

Regards,

Vinay Sajip

Here is the setuptools extras+conditionals format. It just separates them with a colon: extras_require = { "ssl:sys_platform=='win32'": "wincertstore==0.1", "ssl:sys_platform=='win32' and python_version=='2.4'": "ctypes==1.0.2", "ssl:python_version in '2.4, 2.5'":"ssl==1.16", "certs": "certifi==0.0.8", },

No I just forgot the syntax so I looked it up. On Jun 23, 2013 7:58 PM, "Vinay Sajip" wrote:

Daniel Holth writes:

...

Here is the setuptools extras+conditionals format. It just separates them with a colon:

Thanks, but I've already seen that. The example I gave showed how that was migrated to the PEP 426 format as pymeta.json :-)

Did I miss anything?

Regards,

Vinay Sajip

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig