On 27 Feb 2014 04:00, "Donald Stufft" wrote:

...

I will accordingly be updating the defined metadata version in PEP 426 to 3.0, and including an explicit admonition to *never* include experimental metadata (whether in the base format or as part of an experimental extension) in the main pydist.json file. Experimental metadata should only ever appear in tool-specific files (which don't need to guarantee any kind of interoperability with other tools).

In the meantime: please don't publish pydist.json files, use some other filename if you want to experiment with JSON based metadata in advance of the acceptance of PEP 426.

Regards, Nick.

This shouldn't matter. Tools shouldn't be trusting a pydist.json just because one exists and the Wheel format should have a version bump before you can start trusting them anyways.

It's post-install and *post PEP 426 acceptance* that concerns me. Once PEP 426 is accepted, we'll need a reliable way to distinguish pydist.json files in the standard format from these experimental prototypes. Now, we could add some extra validity rules (such as "must come from a 1.1+ wheel file or a 2.0+ sdist"), but that seems fragile to me, since those markers likely won't be available once the package is installed. By contrast, skipping straight to 3.0 will make it easy for tools to distinguish between files that are intended to be PEP compliant, and those that may contain data in formats based on an earlier draft. That said, I guess another alternative is to handle it through parsing error fallbacks, and include a recommendation in PEP 426 that if pydist.json is missing or fails to parse correctly, they should fall back to checking for setuptools style metadata. If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour that would mean we could leave the metadata version alone, and just add the guidelines regarding publishing draft metadata and handling malformed or missing metadata. Cheers, Nick.

On 26 February 2014 21:51, Donald Stufft wrote:

Well I don't really care if we do 3.0 or 2.0 it's just a number. I just mean that you shouldn't parse a pydist.json inside of a Wheel unless you know it's inside of a Wheel with Wheel-Version: Whatever-We-Formally-Add-Pydistjson-To-Wheel in.

pydist.json files appear in dist-info directories currently, as well. Probably from installs of wheels (after all, installing a wheel essentially just dumps everything into site-packages, including the pydist.json). And there's no marker that I know of for the version of a dist-info directory. (Actually, the WHEEL file is also in there, so you could ignore pydist.json if there's also a WHEEL file saying version 1.0, but that's more in the realm of heuristics than actual standards-based rules...) Paul

 If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour  that would mean we could leave the metadata version alone

I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file. IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time. I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out, (b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and (c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering). Regards, Vinay Sajip

On 26 February 2014 22:07, Donald Stufft wrote:

We should probably start versioning dist-info directories.

Probably. I'm not actually sure there's a formal spec for what's in a dist-info directory TBH. There's definitely a "plus anything else people put there" clause in practice, but I don't know if there's a list of the actual mandated files anywhere. PEP 426 says you put pydist.json in there, but that's the only file I know of that's documented. Paul

On 27 Feb 2014 08:41, "Paul Moore" wrote:

On 26 February 2014 22:07, Donald Stufft wrote:

...

We should probably start versioning dist-info directories.

Probably. I'm not actually sure there's a formal spec for what's in a dist-info directory TBH. There's definitely a "plus anything else people put there" clause in practice, but I don't know if there's a list of the actual mandated files anywhere. PEP 426 says you put pydist.json in there, but that's the only file I know of that's documented.

http://www.python.org/dev/peps/pep-0376/ The installation database is one of the things listed at the top of PEP 426 as requiring an update to document the inclusion of pydist.json. Cheers, Nick.

Paul

On Feb 26, 2014, at 6:17 PM, Vinay Sajip wrote:

...

It makes perfect sense to version the dist-info directory. You don’t know how to interpret the files inside that directory without it. You have to rely on heuristics and guessing.

Not if you specify up front how it will work, which is doable.

It's not clear to me if you mean putting the metadata version in the directory name itself - if so, that's a -1 from me. It doesn't actually make things better as far as the PEP 376 database logic goes, given that a site-packages could have any number of dist-info dirs which could have been installed with different metadata versions.

No I don’t mean the directory name.

If you mean putting some marker inside the directory, then pydist.json is as good a place as any, because as packaging evolves, the chances are that pydist.json will need to contain metadata version information telling how to process it, and there's little point in providing the information in two places. All of the legacy stuff is in separate files purely for speed of processing or through historical accident, but those are implementation details which should be under the hood as far as possible.

There is always going to be multiple files, it’s kind of silly to tie the definition of the dist-info directory to the pydist.json when that’s perhaps not the file you care about how to interpret. How do you interpret the RECORD file? The INSTALLER file? The versioned definition of the dist-info directory would say, “RECORD file is an XYZ file” “pydist.json is an ABC file”. It’s the only sane way to handle the case where you can have an unbounded number of unknown files in the directory.

Regards,

Vinay Sajip

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Feb 26, 2014, at 6:25 PM, Nick Coghlan wrote:

On 27 Feb 2014 08:37, "Vinay Sajip" wrote:

...

...

If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour that would mean we could leave the metadata version alone

I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file.

IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time.

I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out,

Wheels work almost as well with setuptools metadata as they will with the initial release of the new metadata spec (because I will be removing the metadata hook extension for the time being - I'm still not convinced that mechanism is a good idea, and I don't think it should delay the other benefits, like providing a format for PyPI to publish dependency metadata directly).

...

(b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and

The wheel spec needs revision to cover more use cases. That's not a reason to avoid encouraging it's use for the wide range of cases that it already handles.

However, people do know that I'm not planning to write all these spec updates myself, right? I've claimed 426/440/459, but if you wait for me to write the others as well, we're going to be waiting a long time.

I still need to review the changes to those :(

I know Donald already has a very early draft of wheel 2.0 in the works, but I'm not aware of anyone that has even started thinking seriously about an sdist 2.0 spec, or an update to the installation database spec - if the latter could include an SQLite based caching mechanism, that would be great, since it lays the foundation for allowing metaimporters to provide installation database entries. And PEP 425 does need an update to cover at least Linux distros (likely based on /etc/os-release).

I have a Wheel 2.0 as well as a new one (or maybe just extending 425) that splits the filename convention out of Wheel. Could possibly get a dist-info 2.0 in there but I also have two PyPI PEPS and a Python PEP on my TODO list too :/

Would it help if I created tasks for all the things that still need to be done in the metadata formats repo, and assigned the ones I'm actually working on to myself? That way people could see clearly where we already know work is needed, but nobody has volunteered to do it yet.

...

(c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering).

PEP 425 explicitly covers that. Report bugs against tools that are non-compliant because they expect distutils to automatically do the right thing (when we know it doesn't, especially on Windows and in 2.x).

Cheers, Nick.

...

Regards,

Vinay Sajip

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 27 Feb 2014 10:16, "Vinay Sajip" wrote:

...

PEP 425 explicitly covers that.

It says "The version is py_version_nodot. CPython gets away with no dot,

but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18, pp19".

The "probably" leaves some room for doubt as to what exactly is meant:

both wheel and distlib use py_version_nodot, and the last sentence I quoted looks provisional rather than definitive. That's because API/ABI versioning is technically up to the individual implementations. The PEP only mandates particular behaviour for CPython and the implementation independent tags. If if you want a definitive answer on that, ask the PyPy devs to write one up. I would have preferred to mandate behaviour based on sys.implementation, but that still wouldn't have covered 2.x. Cheers, Nick.

Regards,

Vinay Sajip

However, people do know that I'm not planning to write all these spec updates myself, right? I've claimed 426/440/459, but if you wait for me to write the others as well, we're going to be waiting a long time.

I know Donald already has a very early draft of wheel 2.0 in the works, but I'm not aware of anyone that has even started thinking seriously about an sdist 2.0 spec, or an update to the installation database spec - if the latter could include an SQLite based caching mechanism, that would be great, since it lays the foundation for allowing metaimporters to provide installation database entries. And PEP 425 does need an update to cover at least Linux distros (likely based on /etc/os-release).

Would it help if I created tasks for all the things that still need to be done in the metadata formats repo, and assigned the ones I'm actually working on to myself? That way people could see clearly where we already know work is needed, but nobody has volunteered to do it yet.

that would be good. If you did, I would link to the tasks from the PUG future page.

On Wed, Feb 26, 2014 at 8:22 PM, Marcus Smith wrote:

...

Calling it version 3 is a fine solution, and the spec has changed enough from key/value 2.0 to json 2.0, and grown a year older, that it's also appropriate to bump the version again.

appropriate to bump? "key/value 2.0" was never a thing. I don't like switching to 3.0, unless we really have to. It looks odd from the outside. like we're switching gears again in some major way

btw, what is in wheel's "2.0" METADATA file that is 2.0? and what's using it?

can't bdist_wheel at least stop setting metadata to 2.0 from here on?

METADATA is the key/value file that setuptools uses when it finds a .dist-info directory. It resembles pre-JSON PEP 426. It can be classified as setuptools metadata now. The next bdist_wheel will remove pydist.json.

but my question was what are you adding, if anything, that warranted it having "Metadata-Version: 2.0"?

I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON. Regards, Vinay Sajip _______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 27 February 2014 11:22, Marcus Smith wrote:

...

Calling it version 3 is a fine solution, and the spec has changed enough from key/value 2.0 to json 2.0, and grown a year older, that it's also appropriate to bump the version again.

appropriate to bump? "key/value 2.0" was never a thing. I don't like switching to 3.0, unless we really have to. It looks odd from the outside. like we're switching gears again in some major way

Yeah, after chatting with Donald a bit, I think we can deal with it without bumping the metadata version again (I think we could sell that if we really had to, but I've been persuaded that isn't necessary at this point). 1. Before reading a pydist.json file from a wheel file, check the wheel version to be sure it's one that includes a valid pydist.json file as part of the spec. 2/ Before reading it from a dist-info directory, check the layout version there as well While PEP 376 doesn't include a way to version dist-info directories, we can add one in the same update that adds pydist.json to the expected files list - the simplest way would be something like a "LAYOUT" file with an incrementing number as its only contents (so PEP 376 would be considered layout 1, it's successor that adds pydist.json layout 2, etc) Coming up with an efficient sqlite backed install-time metadata caching scheme or new metapath and path hook friendly metadata APIs would be postponed for the time being, since those are both a long way down our priority list at this point. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I don't think it matters which number we use in the METADATA key/value metadata... it's not even checked by anything. On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan wrote:

On 27 February 2014 18:16, Vinay Sajip wrote:

...

...

but my question was what are you adding, if anything, that warranted it having "Metadata-Version: 2.0"?

I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON.

For the record, http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last version prior to the switch to JSON, and the summary of differences is at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263

At that time, I think wheel still had a dependency on PEP 426 - changing wheels to work with the setuptools metadata was a relatively late change after we realised it made sense to decouple the two activities.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yes, but the METADATA version of the metadata is not a standard. On Thu, Feb 27, 2014 at 9:14 AM, Donald Stufft wrote:

On Feb 27, 2014, at 9:13 AM, Daniel Holth wrote:

...

I don't think it matters which number we use in the METADATA key/value metadata... it's not even checked by anything.

On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan wrote:

...

On 27 February 2014 18:16, Vinay Sajip wrote:

...

...

but my question was what are you adding, if anything, that warranted it having "Metadata-Version: 2.0"?

I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON.

For the record, http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last version prior to the switch to JSON, and the summary of differences is at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263

At that time, I think wheel still had a dependency on PEP 426 - changing wheels to work with the setuptools metadata was a relatively late change after we realised it made sense to decouple the two activities.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

---

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Compliance to standards matter. It's how you get reasonable interoperability.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

No PEP describes how METADATA supports setuptools' extras. On Thu, Feb 27, 2014 at 9:17 AM, Donald Stufft wrote:

On Feb 27, 2014, at 9:16 AM, Daniel Holth wrote:

...

Yes, but the METADATA version of the metadata is not a standard.

On Thu, Feb 27, 2014 at 9:14 AM, Donald Stufft wrote:

...

On Feb 27, 2014, at 9:13 AM, Daniel Holth wrote:

...

I don't think it matters which number we use in the METADATA key/value metadata... it's not even checked by anything.

On Thu, Feb 27, 2014 at 7:16 AM, Nick Coghlan wrote:

...

On 27 February 2014 18:16, Vinay Sajip wrote:

...

> but my question was what are you adding, if anything, > that warranted it having "Metadata-Version: 2.0"?

I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON.

For the record, http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt is the last version prior to the switch to JSON, and the summary of differences is at http://hg.python.org/peps/file/3b67372b39ba/pep-0426.txt#l1263

At that time, I think wheel still had a dependency on PEP 426 - changing wheels to work with the setuptools metadata was a relatively late change after we realised it made sense to decouple the two activities.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

---

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Compliance to standards matter. It's how you get reasonable interoperability.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Uh, it's defined in the PEP standards.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Thu, Feb 27, 2014 at 12:16 AM, Vinay Sajip wrote:

...

but my question was what are you adding, if anything, that warranted it having "Metadata-Version: 2.0"?

I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON.

from a quick scan of the wheel code, I only see "Provides-Extra" as something additional to 1.2 Setuptools additionally acquired logic for "Provides-Extra" in "DistInfoDistribution" afaik, "Provides-Extra" is not literally in PEP426, so it has no PEP home? but it's implemented? it sounds like we need some version to capture this change?, or just consider it a mess, and leave at that.

On Thu, Feb 27, 2014 at 9:41 PM, Marcus Smith wrote:

On Thu, Feb 27, 2014 at 6:24 AM, Daniel Holth wrote:

...

No PEP describes how METADATA supports setuptools' extras.

i.e. "Provides-Extras". maybe we should at least add a footnote to PEP-345, or *something*, so this is not a mystery down the road where this came from, and how it ended being in "production" use for wheels

https://bitbucket.org/pypa/pypi-metadata-formats/issue/29/document-provides-...