[Distutils] Expectations on how pip needs to change for Python 3.4

[Noah Kantrowitz]

On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote:

> On 14 July 2013 17:13, Donald Stufft <donald at stufft.io> wrote:
> I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize).
> 
> The main elements of CPython's backwards compatibility policy that I consider relevant are:
> 
> * Use leading underscores to denote private APIs with no backwards compatibility guarantees
> * Be conservative with deprecating public APIs that aren't fundamentally broken
> * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change
> 
> We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely.
> 
> That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy:
> 
> * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip
> * otherwise, bundle the same version of pip as the previous release
> 
> This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default.
> 
> On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode).

If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will support that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is "reasonable" for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer.

--Noah