[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Elvis Stansvik]

2017-10-21 14:34 GMT+02:00 Paul Moore <p.f.moore at gmail.com>:
> On 21 October 2017 at 12:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> (Note: this is entirely speculative, and I have no idea how hard it would
>> be, so please read it as the question it's intended to be)
>
> No problem - I don't know myself how hard some of this would be, either ;-)
>
>> Do you know if there any key APIs (like installation) that could be turned
>> into wrappers around pip CLI calls in order to mitigate some of the impact?
>
> The obvious one is pip.main(). That's the one a lot of people use, but
> it's easily replaceable by a simple subprocess call. That's actually
> one of the reasons this was so frustrating to us - the bug reports we
> got were often from people doing things they didn't need to, that they
> could handle trivially via a supported approach.
>
> Otherwise, no. We've had little or no feedback on the tracker from
> people using more complex internals, so our working assumption has
> been there's very little that can't be handled via the CLI or existing
> packages. Feedback so far from this mail hints that maybe we were
> wrong, but it's still hard to know if it's one or two key projects, or
> a whole range of people that we've yet to hear from. I'm pretty sure,
> for example, that pipenv uses internals, either directly or via one of
> their dependencies, but we've not seen any feedback from them yet.

Another one that immediately comes to mind is pip-tools [1], which I
think is quite widely used.

But I just checked, and they filed a "check out how to deal with pip
10" issue two days ago [2] (I'm guessing in response to this thread).

Elvis

[1] https://github.com/jazzband/pip-tools
[2] https://github.com/jazzband/pip-tools/issues/580

>
>> The reason I ask is because it's unlikely anyone else is going to understand
>> how to emulate the previous functionality better than the pip devs would,
>> and if there's an API for those particular invocations, than they can be
>> covered directly by pip's test suite.
>
> Understood. We understand *how*, but don't know what is needed. One of
> the points of all this is to tease out such requirements. We'd hope to
> get them addressed by including them into *other* packages like
> distlib or packaging, but that's mostly just a matter of where the API
> goes, not what is needed. It does help us avoid tying fixes to pip's
> release cycle, though.
>
> It's also worth noting that the pip devs are possibly way too deep
> into how pip does things, rather than what the standards say. Getting
> others to implement libraries based on the published standards would
> help immensely to ensure that we're not falling into the
> "implementation defined" trap of the community being stuck with having
> to use pip because no-one else knows how to do what pip does.
>
>> Plus if there are previous API capabilities that *can't* currently be
>> emulated via the CLI, then the pip devs are the folks in the best position
>> to add the necessary CLI enhancements (such as the ones Noah asked about for
>> doing a more selective `pip list`).
>
> Oh, sure - apart from the aforementioned "pip.main()", most
> capabilities of the internal API are *not* easily emulated by the CLI.
>
> But 3rd party libraries are just as much an option. Remember, the
> issue here isn't so much about designing an API as about exposing (and
> therefore locking in stone) internal implementation details of how pip
> does things. And the pip devs are arguably in the *worst* position to
> handle that option, precisely because they know so much about how pip
> does things.
>
>> If that's an approach you might be amenable to, then a 10.0 pre-release
>> could be a good time to solicit PRs from folks that were using particular
>> APIs and would be prepared to invest time in defining comparable CLI
>> wrappers for them.
>
> Well, I get your point here, but the implication of this is that we
> have to design and build an API before we can release pip 10. Calling
> it CLI wrappers (and implementing it that way) doesn't do anything to
> ease the work needed on design, or the maintenance burden of providing
> stability. We're already getting a lot of pressure to release pip 10,
> and trying to do that would push it way further off.
>
> To be blunt, no-one on the pip team is really interested in trying to
> provide or support a stable API at this point in time. We have our
> hands full, as everyone is aware, and this is way down all our
> priorities. Community-submitted PRs would help, but there's still work
> in agreeing a design, reviewing those PRs, and maintaining them
> (exposing details from the internal APIs limits how much we can change
> those internals later, which is something we have to consider).
>
> What would be ideal would be for the community to build
> standards-based libraries that satisfied the needs that people
> currently handle by using pip internals. Heck, we could even consider
> vendoring such libraries inside of pip and saving ourselves a bunch of
> complexity ;-) The packaging, pkg_resources and distlib libraries are
> examples of this happening. But it doesn't seem to be a popular route,
> unfortunately. And those particular libraries are all maintained by
> PyPA members, so don't really count as examples of the community
> getting involved...
>
> The most likely alternative solution would be to revert the internal
> API moves. If that happened, I'd still prefer to lose pip.main, as
> that's the major pain point for us and the easiest for users to solve,
> but we could put the rest back. But it would be naive to assume that
> as a result people like Noah would be unaffected - they'd actually be
> worse off, as the internal APIs they use would probably break anyway
> (the installer/resolver stuff has been heavily modified for pip 10)
> but not all would do so in obvious ways.
>
> Beyond this, I don't know.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig