2017-10-21 14:34 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 21 October 2017 at 12:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
No problem - I don't know myself how hard some of this would be, either ;-)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The obvious one is pip.main(). That's the one a lot of people use, but it's easily replaceable by a simple subprocess call. That's actually one of the reasons this was so frustrating to us - the bug reports we got were often from people doing things they didn't need to, that they could handle trivially via a supported approach.
Otherwise, no. We've had little or no feedback on the tracker from people using more complex internals, so our working assumption has been there's very little that can't be handled via the CLI or existing packages. Feedback so far from this mail hints that maybe we were wrong, but it's still hard to know if it's one or two key projects, or a whole range of people that we've yet to hear from. I'm pretty sure, for example, that pipenv uses internals, either directly or via one of their dependencies, but we've not seen any feedback from them yet.
Another one that immediately comes to mind is pip-tools [1], which I think is quite widely used. But I just checked, and they filed a "check out how to deal with pip 10" issue two days ago [2] (I'm guessing in response to this thread). Elvis [1] https://github.com/jazzband/pip-tools [2] https://github.com/jazzband/pip-tools/issues/580
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Understood. We understand *how*, but don't know what is needed. One of the points of all this is to tease out such requirements. We'd hope to get them addressed by including them into *other* packages like distlib or packaging, but that's mostly just a matter of where the API goes, not what is needed. It does help us avoid tying fixes to pip's release cycle, though.
It's also worth noting that the pip devs are possibly way too deep into how pip does things, rather than what the standards say. Getting others to implement libraries based on the published standards would help immensely to ensure that we're not falling into the "implementation defined" trap of the community being stuck with having to use pip because no-one else knows how to do what pip does.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
Oh, sure - apart from the aforementioned "pip.main()", most capabilities of the internal API are *not* easily emulated by the CLI.
But 3rd party libraries are just as much an option. Remember, the issue here isn't so much about designing an API as about exposing (and therefore locking in stone) internal implementation details of how pip does things. And the pip devs are arguably in the *worst* position to handle that option, precisely because they know so much about how pip does things.
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Well, I get your point here, but the implication of this is that we have to design and build an API before we can release pip 10. Calling it CLI wrappers (and implementing it that way) doesn't do anything to ease the work needed on design, or the maintenance burden of providing stability. We're already getting a lot of pressure to release pip 10, and trying to do that would push it way further off.
To be blunt, no-one on the pip team is really interested in trying to provide or support a stable API at this point in time. We have our hands full, as everyone is aware, and this is way down all our priorities. Community-submitted PRs would help, but there's still work in agreeing a design, reviewing those PRs, and maintaining them (exposing details from the internal APIs limits how much we can change those internals later, which is something we have to consider).
What would be ideal would be for the community to build standards-based libraries that satisfied the needs that people currently handle by using pip internals. Heck, we could even consider vendoring such libraries inside of pip and saving ourselves a bunch of complexity ;-) The packaging, pkg_resources and distlib libraries are examples of this happening. But it doesn't seem to be a popular route, unfortunately. And those particular libraries are all maintained by PyPA members, so don't really count as examples of the community getting involved...
The most likely alternative solution would be to revert the internal API moves. If that happened, I'd still prefer to lose pip.main, as that's the major pain point for us and the easiest for users to solve, but we could put the rest back. But it would be naive to assume that as a result people like Noah would be unaffected - they'd actually be worse off, as the internal APIs they use would probably break anyway (the installer/resolver stuff has been heavily modified for pip 10) but not all would do so in obvious ways.
Beyond this, I don't know.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig