On 20 October 2017 at 23:53, Richard Jones <r1chardj0n3s@gmail.com> wrote:
> Hiya Paul,
>
> There's a bunch of tooling out there using pip's internals to extending
> pip's functionality. Could you please provide a some reasoning as to why
> they're all going to be broken at pip 10, and possibly some guidance on how
> to get that functionality back?

Hi Richard,
There was a change to the pip docs that clarified the status of pip's
internal code. The PR for that is at
https://github.com/pypa/pip/pull/4743 but unfortunately it appears
that the docs build has been failing so it hasn't yet made it to the
formal pip docs site.

To summarise, pip has *never* supported the use of its internal APIs
by external code. Over time, we've had a steady trickle of people
raising issues when their code broke because of doing so, and it
usually turned out to be because they violated assumptions made by the
pip code - such as that it's running in a single-threaded application,
or it has sole control over the logging subsystem, or even just that
you can run your own code after calling a pip API. We've explained
this position regularly on those issues, but as is typical, people
don't manage to find similar issues when raising new ones, so we spent
a lot of time repeating ourselves.

Coming up to pip 10, there's been a *lot* of internal work going on,
and it's fairly likely that a decent chunk of code using pip's
internal APIs will break anyway. We don't document internals changes,
so we faced the possibility of an extended period of people raising
issues saying "you broke my code" and us having no better response
than "you shouldn't do that", which would likely hinder adoption of
pip 10, and cause problems for the ecosystem as a whole. Rather than
do this, we decided to make a clean compatibility break, where we
could send out a clear message - "everything's moved, if that matters
to you, then you were using unsupported functionality and you should
find a better way". The breakage is still there (and certainly we made
it affect more people, as there are no doubt some people who would
have survived the pip 10 release unscathed if we hadn't done this) but
at least it's clearly defined and contained.

As to alternatives, we don't have all the answers here but I can offer
the following suggestions:

1. There are a number of external packages that provide functionality
equivalent to what pip does - packaging, wheel, distlib, pkg_resources
are the ones I'm aware of. These are designed as libraries and so *do*
provide supported APIs.
2. We're making a strong push to standardise *all* of the external
interfaces that pip uses, so you should be far more able to write your
own code if that's necessary, without worrying that it'll work
differently than pip does, or that things will suddenly change and
break your code.
3. You can call pip as a subprocess - that's always been supported and
will remain so. We've added some automation-friendly features there
(such as json output format for "pip list") and we'd be happy to add
more if people want to submit PRs.

Likely the biggest problems will be for people who call into the pip
resolver and build APIs, as I don't know of any alternatives out
there. But they were *definitely* breaking anyway, as we've made major
changes to that code (and will be making more).

Also, I should note that we didn't take this decision lightly. We
don't have any particular objection in principle to having a supported
stable pip API, it's just that we don't have anything even close to
the resources needed to define a supported API, maintain it with
acceptable backward compatibility guarantees, and support users who
will inevitably be using it in unexpected and creative ways (we don't
even have the resources to support the limited use of pip's internals
that we see today). Also, pip was never designed originally as a
library, so we *would* have to design that API from scratch. As I
alluded to above, the existing internals code makes some strong
assumptions about how it's called - assumptions that would be
unacceptable in library code, but are fine in an application's
internal code.

Paul

PS People who want to, can of course hunt out the new equivalents of
the code they were using, and just switch. It's not like we can stop
them. But the new names make it clear that they shouldn't be doing
this, so there's an obvious warning there.
PPS Please disregard xoviat's response. This is something we've been
considering for a while, and most definitely not a spur of the moment
decision. It's unfortunate that he was the one most immediately
affected by the change when we made it, but that was just bad timing
(we didn't suddenly do this just because "someone complained").