[Distutils] PEP 439 and pip bootstrap updated

[Donald Stufft]

On Jul 12, 2013, at 3:25 PM, Brett Cannon <brett at python.org> wrote:

> 
> 
> 
> On Fri, Jul 12, 2013 at 2:16 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> On Jul 12, 2013, at 2:00 PM, Brett Cannon <brett at python.org> wrote:
> 
>> Speaking with my python-dev hat on which has a badge from when I led the stdlib cleanup for Python 3, I would say anything that has a PEP should probably have a module in the stdlib for it. That way standard management of whatever is specified in the PEP will be uniform and expected to be maintained and work. Beyond that code will exist outside the stdlib.
> 
> This is basically the exact opposite of what Nick has said the intent has been (Ecosystem first).
> 
> 
> Not at all as no module will go in immediately until after a PEP has landed and been vetted as needed.
>  
> Adding packaging tools beyond bootstrapping pip at this point in the game is IMO a huge mistake. If what Nick has said and PEPs are not appropriate for things that don't have a module in the standard lib well that's fine I guess.
> 
> You misunderstand what I mean. I'm just saying that *if* anything were to go into the stdlib it would only be to have code which implements a PEP in the stdlib to prevent everyone from re-implementing a standard.
>  
> I just won't worry about trying to write PEPs :)
> 
> No, the PEPs are important to prevent version skew and make sure everyone is on the same page. And that's also what a module in the stdlib would do; make sure everyone is on the same page in terms of semantics by using a single code base.
> 
> I mean I wouldn't expect anything more than maybe code parsing the JSON metadata that does some validation and parsing version numbers that can support comparisons and verifying platform requirements; that's it. Stuff that every installation tool will need to do in order to follow the PEPs properly. And it wouldn't go in until everyone was very happy with the PEPs and ready to commit to code enshrining it in the stdlib. Otherwise I hope distlib moves into PyPA and everyone who develops installation tools, etc. uses that library.

I could probably be convinced about something that makes handling versions easier going into the standard lib, but that's about it.

There's a few reasons that I don't want these things added to the stdlib themselves.

One of the major ones is that of "agility". We've seen with distutils how impossible it can be to make improvements to the system. Now some of this is made better with the way the new system is being designed  with versioned metadata but it doesn't completely go away. We can look at Python's past to see just how long any individual version sticks around and we can assume that if something gets added now that particular version will be around for a long time.

Another is because of how long it can take a new version of Python to become "standard", especially in the 3.x series since the entire 3.x series itself isn't standard, any changes made to the standard lib won't be usable for years and years. This can be mitigated by releasing a backport on PyPI, but if every version of Python but the latest one is going to require installing these libs from PyPI in order to usefully interact with the "world", then you might as well just require all versions of Python to install bits from PyPI.

Yet another is by blessing a particular implementation, that implementations behaviors become the standard (indeed the way the PEP system generally works for this is once it's been added to the standard lib the PEP is a historical document and the documentation becomes the standard). However packaging is not like Enums or urllibs, or smtp. We are essentially defining a protocol, one that non Python tools will be expected to use (for Debian and RPMs for example). We are using these PEPs more like a RFC than a proposal to include something in the stdlib.

There's also the case of usefulness. You mention some code that can parse the JSON metadata and validate it. Weel assumingly we'll have the metadata for 2.0 set down by the time 3.4 comes around. So sure 3.4 could have that, but then maybe we release metadata 2.1 and now 3.4 can only parse _some_ of the metadata. Maybe we release a metadata 3.0 and now it can't parse any metadata. But even if it can parse the metadata what does it do with it? The major places you'd be validating the metadata (other than merely consuming it) is either on the tools that create packages or in PyPI performing checks on a valid file upload. In the build tool case they are going to either need to write their own code for actually creating the package or, more likely, they'll reuse something like distlib. If those tools are already going to be using a distlib-like library then we might as just keep the validation code in there.

Now the version parsing stuff which I said I could be convinced is slightly different. It is really sort of it's own thing. It's not dependent on the other pieces of packaging to be useful, and it's not versioned. It's also the only bit that's really useful on it's own. People consuming the (future) PyPI API could use it to fully depict the actual metadata so it's kind of like JSON itself in that regard.

The installer side of things the purist side of me doesn't like adding it to the standard library for all the same reasons but the pragmatic side of me wants it there because it enables fetching the other bits that are needed for "pip install X" to be a reasonable official response to these kind of questions. But I pushed for and still believe that if a prerequisite for doing that involves "locking" in pip or any of it's dependencies by adding them to the standard library then I am vehemently against doing it.

Wow that was a lot of words...

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130712/ec583077/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130712/ec583077/attachment.pgp>