[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Richard Jones]

Thanks for writing that detailed explanation, Paul (and all your other hard
work!)


     Richard


On 21 October 2017 at 21:03, Paul Moore <p.f.moore at gmail.com> wrote:

> On 20 October 2017 at 23:53, Richard Jones <r1chardj0n3s at gmail.com> wrote:
> > Hiya Paul,
> >
> > There's a bunch of tooling out there using pip's internals to extending
> > pip's functionality. Could you please provide a some reasoning as to why
> > they're all going to be broken at pip 10, and possibly some guidance on
> how
> > to get that functionality back?
>
> Hi Richard,
> There was a change to the pip docs that clarified the status of pip's
> internal code. The PR for that is at
> https://github.com/pypa/pip/pull/4743 but unfortunately it appears
> that the docs build has been failing so it hasn't yet made it to the
> formal pip docs site.
>
> To summarise, pip has *never* supported the use of its internal APIs
> by external code. Over time, we've had a steady trickle of people
> raising issues when their code broke because of doing so, and it
> usually turned out to be because they violated assumptions made by the
> pip code - such as that it's running in a single-threaded application,
> or it has sole control over the logging subsystem, or even just that
> you can run your own code after calling a pip API. We've explained
> this position regularly on those issues, but as is typical, people
> don't manage to find similar issues when raising new ones, so we spent
> a lot of time repeating ourselves.
>
> Coming up to pip 10, there's been a *lot* of internal work going on,
> and it's fairly likely that a decent chunk of code using pip's
> internal APIs will break anyway. We don't document internals changes,
> so we faced the possibility of an extended period of people raising
> issues saying "you broke my code" and us having no better response
> than "you shouldn't do that", which would likely hinder adoption of
> pip 10, and cause problems for the ecosystem as a whole. Rather than
> do this, we decided to make a clean compatibility break, where we
> could send out a clear message - "everything's moved, if that matters
> to you, then you were using unsupported functionality and you should
> find a better way". The breakage is still there (and certainly we made
> it affect more people, as there are no doubt some people who would
> have survived the pip 10 release unscathed if we hadn't done this) but
> at least it's clearly defined and contained.
>
> As to alternatives, we don't have all the answers here but I can offer
> the following suggestions:
>
> 1. There are a number of external packages that provide functionality
> equivalent to what pip does - packaging, wheel, distlib, pkg_resources
> are the ones I'm aware of. These are designed as libraries and so *do*
> provide supported APIs.
> 2. We're making a strong push to standardise *all* of the external
> interfaces that pip uses, so you should be far more able to write your
> own code if that's necessary, without worrying that it'll work
> differently than pip does, or that things will suddenly change and
> break your code.
> 3. You can call pip as a subprocess - that's always been supported and
> will remain so. We've added some automation-friendly features there
> (such as json output format for "pip list") and we'd be happy to add
> more if people want to submit PRs.
>
> Likely the biggest problems will be for people who call into the pip
> resolver and build APIs, as I don't know of any alternatives out
> there. But they were *definitely* breaking anyway, as we've made major
> changes to that code (and will be making more).
>
> Also, I should note that we didn't take this decision lightly. We
> don't have any particular objection in principle to having a supported
> stable pip API, it's just that we don't have anything even close to
> the resources needed to define a supported API, maintain it with
> acceptable backward compatibility guarantees, and support users who
> will inevitably be using it in unexpected and creative ways (we don't
> even have the resources to support the limited use of pip's internals
> that we see today). Also, pip was never designed originally as a
> library, so we *would* have to design that API from scratch. As I
> alluded to above, the existing internals code makes some strong
> assumptions about how it's called - assumptions that would be
> unacceptable in library code, but are fine in an application's
> internal code.
>
> Paul
>
> PS People who want to, can of course hunt out the new equivalents of
> the code they were using, and just switch. It's not like we can stop
> them. But the new names make it clear that they shouldn't be doing
> this, so there's an obvious warning there.
> PPS Please disregard xoviat's response. This is something we've been
> considering for a while, and most definitely not a spur of the moment
> decision. It's unfortunate that he was the one most immediately
> affected by the change when we made it, but that was just bad timing
> (we didn't suddenly do this just because "someone complained").
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/ccc15032/attachment-0001.html>