Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
I've never seen environment markers being used or supported in the wild.
I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP.
Elephant in the room? Somethin's been around for so long and hasn't got any sort of mainstream support is *not* a good candidate for a requirement but maybe it's me being naive.
On Feb 19, 2013 6:57 AM,
I've never seen environment markers being used or supported in the wild.
I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP.
Elephant in the room? Somethin's been around for so long and hasn't got any sort of mainstream support is *not* a good candidate for a requirement but maybe it's me being naive.
I think 1.2 was started and then amended over a long period of time to become what it is today. I wanted to edit it again just to add Provides-Extra. It turned out to be more practical to make a new PEP. Nick explains that for complicated reasons the implementation of Metadata 1.2 was not successful. For Metadata 2.0 we had a different strategy: put it into distribute first. You will find that this aids adoption. distlib will provide another implementation that is in many ways interchangeable. We also have three implementations of an installer for the related wheel format, three implementations of the wheel builder, and two surprisingly short implementations of environment markers. Yes, the wheel project has the same name as the wheel format. Metadata 2.0's Provides-Extra field is there to represent the setuptools "extras" feature which is necessary for a very large number of distributions on pypi. For a while the most common environment markers will look like just "extra == 'foo'". So it should not be surprising that distribute is the first tool to recognize Provides-Extra. Previously distribute did not use METADATA / PKG-INFO for dependencies, instead reading from requires.txt If you can get around the fact that we've incorporated most of another PEP 386 into v2.0, rewritten and explained everything to death, I would argue that 2.0 is actually simpler than 1.2. Only four headers are actually required in a valid Metadata 2.0 file and a number of the fields are informational and do not have to be understood by the installer.
On Tue, Feb 19, 2013 at 9:57 PM,
I've never seen environment markers being used or supported in the wild.
I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP.
PEP 345 has NOT been an approved specification for 8 years. It was first drafted in 2005, but not Accepted until 2010: http://hg.python.org/peps/rev/7d78653a0e79 (as Daniel noted, what became PEP 426 actually started as *another* round of edits to PEP 345) PEP 345 was a first attempt at defining a formal standard that adequately described an *existing* implementation defined software distribution ecosystem built around the behaviour of setuptools. It failed, because it did *not* achieve that goal, and because there were associated declarations along the lines of "to migrate to this standard, you must completely change the way you build and distribute your Python code" (by conflating the setup.py -> setup.cfg migration of distutils2 with the use of the new metadata standard). The response was an overwhelming meh, as people willing migrated to distribute and pip (based on the de facto setuptools standards), while the distutils2 project never made it to production readiness because it was trying to solve too many problems at once. By contrast, the PEP 376 metadata format created at the same time has been adopted quite well. PEPs 345 and 386 were also accepted at a time when python-dev had neither interest nor extensive experience in packaging systems (or if anyone did have such experience, they weren't volunteering that information). So what's changed since 2010 that makes me think the time is right for a new metadata version? 1. We've recognised that without setuptools and/or distribute on board to generate it, and pip to consume it, any new metadata version is dead in the water. That groundwork has been done (mostly by Daniel as part of his work on the wheel format) 2. We've systematically gone through and resolved the significant discrepancies between the formal versioning scheme and the implementation defined behaviour of setuptools and distribute. This means that standards compliant versions will sort the same way in those tools, even if they are not updated. 3. We've addressed other objections PJ Eby (the author of setuptools) had to the previous version of the format (notably the handling of project obsolescence). 4. Various other missing features from setuptools are now supported directly (such as extras) 5. The defined interaction of version specifiers and pre- and post- releases was not useful. The new version fixes that to be something developers are more likely to want (i.e. they won't get pre-releases unless they explicitly reference one). 6. We've enhanced the format so it can handle extensions in a systematic fashion with clearly defined lines of authority based on PyPI distribution names, rather than adding arbitrarily named files to a distribution's info directory without any clear indication of where to find that specification for the file's meaning. 7. I've updated the metadata specification itself, including the addition of the Private-Version field, to more clearly explain to developers that the public Version in their metadata is primarily a tool for communicating the relative ordering of versions to installation tools, but they're still free to use their own preferred version labels internally, and certainly are not obliged to actually use the full complexity of the standard versioning scheme that the installers understand. It's OK if people don't want to read the detailed rationale provided for each of the major changes as part of the PEP, or if they want to dispute a particular piece of that rationale. But merely going "it's too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" is not a reasonable response. Software distribution is complicated - trying to oversimplify it is one of the reasons setuptools became necessary. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 February 2013 13:59, Nick Coghlan
It's OK if people don't want to read the detailed rationale provided for each of the major changes as part of the PEP, or if they want to dispute a particular piece of that rationale. But merely going "it's too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" is not a reasonable response. Software distribution is complicated - trying to oversimplify it is one of the reasons setuptools became necessary.
Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, 390, 376 and 386 are all relevant to one extent or another, and only PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - setup.cfg) are still in Draft format. The others are all Accepted or Final. And yet PEPs 345, 390 and 386 are unused and likely to remain so. I believe that the only ones that are *actually* of use are 241, 314 and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of Installed Python Distributions) although 376 still has a number of flaws. I'd suggest that these should be marked as Final, and the others as Rejected, so that we have a clear statement of what is actually supported. Making sure that users have the means to write code that *uses* these standards using functionality available in the stdlib is then the next step as you say. It is critical that this is done, because packaging tools are unique in that the barrier to using external dependencies is particularly high for them - for example, pip's reliance on distribute/setuptools is necessary, but has been problematic at times. Paul PS Apologies for using a load of PEP numbers without explanation. Here's a glossary: Metadata 1.0 - PEP 241 Metadata 1.1 - PEP 314 Metadata 1.2 - PEP 345 Metadata 1.3 - PEP 426 Static Metadata for Distutils - PEP 390 Database of Installed Python Distributions - PEP 376 Changing the version comparison module in Distutils - PEP 386
On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore
On 19 February 2013 13:59, Nick Coghlan
wrote: It's OK if people don't want to read the detailed rationale provided for each of the major changes as part of the PEP, or if they want to dispute a particular piece of that rationale. But merely going "it's too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" is not a reasonable response. Software distribution is complicated - trying to oversimplify it is one of the reasons setuptools became necessary.
Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, 390, 376 and 386 are all relevant to one extent or another, and only PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - setup.cfg) are still in Draft format. The others are all Accepted or Final. And yet PEPs 345, 390 and 386 are unused and likely to remain so.
I believe that the only ones that are *actually* of use are 241, 314 and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of Installed Python Distributions) although 376 still has a number of flaws. I'd suggest that these should be marked as Final, and the others as Rejected, so that we have a clear statement of what is actually supported.
Making sure that users have the means to write code that *uses* these standards using functionality available in the stdlib is then the next step as you say. It is critical that this is done, because packaging tools are unique in that the barrier to using external dependencies is particularly high for them - for example, pip's reliance on distribute/setuptools is necessary, but has been problematic at times.
Paul
PS Apologies for using a load of PEP numbers without explanation. Here's a glossary:
Metadata 1.0 - PEP 241 Metadata 1.1 - PEP 314 Metadata 1.2 - PEP 345 Metadata 1.3 - PEP 426 Static Metadata for Distutils - PEP 390 Database of Installed Python Distributions - PEP 376 Changing the version comparison module in Distutils - PEP 386
The documentation is bad. The standard library will almost certainly grow code that can interpret these PEPs. Packaging tool authors can handle these specs but no sane person reads PEPs to figure out how to submit a simple library to pypi. No end user cares about metadata versions either or even notices that sdists contain PKG-INFO at their root. You get what setup() produces and that's it. Wheel installers can work without being installed into the target environment at all. This is very hard to understand if you have been using the distutils model (the installed package imports the installer and packages are always rebuilt from source) for more than a decade. This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-) Once the installer doesn't need to be installed, repeatedly, in every virtualenv, hundreds of times a day, users could just have a single up-to-date standalone copy of pip invoked as "python pip.zip install x". Additional packages needed only for builds (even setuptools) may be installed automatically by a build system (like pip) when building rather than using packages.
On 2/19/2013 11:06 AM, Daniel Holth wrote:
This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-)
This is the most artfully crafted comment I've seen on topic on this list for some months! Thanks!
participants (5)
-
a.cavallo@cavallinux.eu
-
Daniel Holth
-
Glenn Linderman
-
Nick Coghlan
-
Paul Moore