Announcement: Pip 10 is coming, and will move all internal APIs
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul
Hi,
On Fri, Oct 20, 2017 at 2:22 PM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks for the heads-up.
Will y'all be doing a PyPI pre-release so we can test with `pip install --pre -U pip`?
Cheers,
Matthew
On 20 October 2017 at 14:26, Matthew Brett matthew.brett@gmail.com wrote:
Thanks for the heads-up.
Will y'all be doing a PyPI pre-release so we can test with `pip install --pre -U pip`?
We've not yet decided on that. Traditionally I don't think we have done so, but I'm inclined to think it's a good idea. It might not be until noticeably closer to the release, though...
Paul
+1 on pre-release wheels. I've seen the process in action with SciPy, and it helped to catch at least one significant bug.
2017-10-20 8:30 GMT-05:00 Paul Moore p.f.moore@gmail.com:
On 20 October 2017 at 14:26, Matthew Brett matthew.brett@gmail.com wrote:
Thanks for the heads-up.
Will y'all be doing a PyPI pre-release so we can test with `pip install --pre -U pip`?
We've not yet decided on that. Traditionally I don't think we have done so, but I'm inclined to think it's a good idea. It might not be until noticeably closer to the release, though...
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Yeah, +1 from me too. Pip is one project where people will highly likely try out the pre-release versions.
xoviat kirjoitti 20.10.2017 klo 20:34:
+1 on pre-release wheels. I've seen the process in action with SciPy, and it helped to catch at least one significant bug.
2017-10-20 8:30 GMT-05:00 Paul Moore <p.f.moore@gmail.com mailto:p.f.moore@gmail.com>:
On 20 October 2017 at 14:26, Matthew Brett <matthew.brett@gmail.com <mailto:matthew.brett@gmail.com>> wrote: > Thanks for the heads-up. > > Will y'all be doing a PyPI pre-release so we can test with `pip > install --pre -U pip`? We've not yet decided on that. Traditionally I don't think we have done so, but I'm inclined to think it's a good idea. It might not be until noticeably closer to the release, though... Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Oct 20, 2017, at 9:30 AM, Paul Moore p.f.moore@gmail.com wrote:
On 20 October 2017 at 14:26, Matthew Brett matthew.brett@gmail.com wrote:
Thanks for the heads-up.
Will y'all be doing a PyPI pre-release so we can test with `pip install --pre -U pip`?
We've not yet decided on that. Traditionally I don't think we have done so, but I'm inclined to think it's a good idea. It might not be until noticeably closer to the release, though…
I used to cut pre-releases for pip, and after awhile I gave up on doing them because it felt like nobody ever actually reported any issues with them anyways, and it wasn’t until we cut the final release that we started finding bugs with them. I don’t have any problem with us starting to issue them again though and seeing if we start catching issues earlier this time.
Thanks for the heads-up, Paul.
I’m currently using `pip.get_installed_distributions` and as far as I can see that has moved into `_internal`, too: https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333 https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
Any recommendations?
Thanks,
Jannis
On 20. Oct 2017, at 15:22, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 20 October 2017 at 14:55, Jannis Gebauer ja.geb@me.com wrote:
Thanks for the heads-up, Paul.
I’m currently using `pip.get_installed_distributions` and as far as I can see that has moved into `_internal`, too: https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
Any recommendations?
See https://github.com/pypa/pip/pull/4743
Unfortunately, the "latest" docs build doesn't seem to reflect this (I don't know why).
I guess you probably want something from pkg_resources? Depends precisely what you're trying to do, I guess. Paul
Perhaps pkg_resources.find_distributions()?
http://setuptools.readthedocs.io/en/latest/pkg_resources.html#getting-or-cre...
Jannis Gebauer kirjoitti 20.10.2017 klo 16:55:
Thanks for the heads-up, Paul.
I’m currently using `pip.get_installed_distributions` and as far as I can see that has moved into `_internal`, too: https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
Any recommendations?
Thanks,
Jannis
On 20. Oct 2017, at 15:22, Paul Moore <p.f.moore@gmail.com mailto:p.f.moore@gmail.com> wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources... which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
There's no dry-run functionality that I know of so far. However, you could use the following:
pip install --prefix=tmpdir
This command is actually about the same speed as a proper implementation, because we can't actually know what we're installing until we build the requirements.
2017-10-20 12:42 GMT-05:00 Noah Kantrowitz noah@coderanger.net:
So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/ poise_python/resources/python_package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Installing to a temp dir is really not an option for automated tooling (if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.
--Noah
On Oct 20, 2017, at 11:35 AM, xoviat xoviat@gmail.com wrote:
There's no dry-run functionality that I know of so far. However, you could use the following:
pip install --prefix=tmpdir
This command is actually about the same speed as a proper implementation, because we can't actually know what we're installing until we build the requirements.
2017-10-20 12:42 GMT-05:00 Noah Kantrowitz noah@coderanger.net: So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources... which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
A correct dry-run implementation will do about the same amount of work as installing to a temporary directory right now. In the future, that could be optimized, but any patch to the finder doesn't actually detect the requirements correctly (as they're not necessarily known until after the wheels are built).
2017-10-20 13:41 GMT-05:00 Noah Kantrowitz noah@coderanger.net:
Installing to a temp dir is really not an option for automated tooling (if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.
--Noah
On Oct 20, 2017, at 11:35 AM, xoviat xoviat@gmail.com wrote:
There's no dry-run functionality that I know of so far. However, you
could use the following:
pip install --prefix=tmpdir
This command is actually about the same speed as a proper
implementation, because we can't actually know what we're installing until we build the requirements.
2017-10-20 12:42 GMT-05:00 Noah Kantrowitz noah@coderanger.net: So as someone on the tooling side, is there any kind of install dry-run
yet? I've got https://github.com/poise/poise-python/blob/master/lib/ poise_python/resources/python_package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
While I understand that pip itself has to be very careful about edge cases and all the pathological things you can do in setup.py, as a higher-level tooling author my priorities are on the happy path UX and speed is a big factor there. So yes, using PackageFinder is potentially inaccurate, but it's also _usually_ accurate :) Anyways, if there is true concern that finder-based approaches are too risky, probably don't offer it in the pip list output.
--Noah
On Oct 20, 2017, at 11:43 AM, xoviat xoviat@gmail.com wrote:
A correct dry-run implementation will do about the same amount of work as installing to a temporary directory right now. In the future, that could be optimized, but any patch to the finder doesn't actually detect the requirements correctly (as they're not necessarily known until after the wheels are built).
2017-10-20 13:41 GMT-05:00 Noah Kantrowitz noah@coderanger.net: Installing to a temp dir is really not an option for automated tooling (if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.
--Noah
On Oct 20, 2017, at 11:35 AM, xoviat xoviat@gmail.com wrote:
There's no dry-run functionality that I know of so far. However, you could use the following:
pip install --prefix=tmpdir
This command is actually about the same speed as a proper implementation, because we can't actually know what we're installing until we build the requirements.
2017-10-20 12:42 GMT-05:00 Noah Kantrowitz noah@coderanger.net: So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources... which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
The "installing collected packages" output is not always correct. The "successfully installed [package list]" is always correct (see pypa/pip/issues/4724 for an example).
I understand your requirements, but is the practical implication that pip should offer a command that outputs packages that are potentially incorrect? What happens when someone relies on that and then files a bug report?
2017-10-20 13:48 GMT-05:00 Noah Kantrowitz noah@coderanger.net:
While I understand that pip itself has to be very careful about edge cases and all the pathological things you can do in setup.py, as a higher-level tooling author my priorities are on the happy path UX and speed is a big factor there. So yes, using PackageFinder is potentially inaccurate, but it's also _usually_ accurate :) Anyways, if there is true concern that finder-based approaches are too risky, probably don't offer it in the pip list output.
--Noah
On Oct 20, 2017, at 11:43 AM, xoviat xoviat@gmail.com wrote:
A correct dry-run implementation will do about the same amount of work
as installing to a temporary directory right now. In the future, that could be optimized, but any patch to the finder doesn't actually detect the requirements correctly (as they're not necessarily known until after the wheels are built).
2017-10-20 13:41 GMT-05:00 Noah Kantrowitz noah@coderanger.net: Installing to a temp dir is really not an option for automated tooling
(if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.
--Noah
On Oct 20, 2017, at 11:35 AM, xoviat xoviat@gmail.com wrote:
There's no dry-run functionality that I know of so far. However, you
could use the following:
pip install --prefix=tmpdir
This command is actually about the same speed as a proper
implementation, because we can't actually know what we're installing until we build the requirements.
2017-10-20 12:42 GMT-05:00 Noah Kantrowitz noah@coderanger.net: So as someone on the tooling side, is there any kind of install
dry-run yet? I've got https://github.com/poise/ poise-python/blob/master/lib/poise_python/resources/python_ package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
--Noah
On Oct 20, 2017, at 6:22 AM, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not
always
that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the
incentive
to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hiya Paul,
There's a bunch of tooling out there using pip's internals to extending pip's functionality. Could you please provide a some reasoning as to why they're all going to be broken at pip 10, and possibly some guidance on how to get that functionality back?
Cheers,
Richard
On 21 October 2017 at 00:22, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul
The short answer is that someone complained about the pip API not doing what they wanted it to do, so we broke the API to make it clear to everyone that the API is not something that should be depended on.
2017-10-20 17:53 GMT-05:00 Richard Jones r1chardj0n3s@gmail.com:
Hiya Paul,
There's a bunch of tooling out there using pip's internals to extending pip's functionality. Could you please provide a some reasoning as to why they're all going to be broken at pip 10, and possibly some guidance on how to get that functionality back?
Cheers,
RichardOn 21 October 2017 at 00:22, Paul Moore p.f.moore@gmail.com wrote:
We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications.
Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-)
So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released.
And many thanks to anyone else who helps by testing out the new version, as well :-)
Thanks, Paul
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 20 October 2017 at 23:53, Richard Jones r1chardj0n3s@gmail.com wrote:
Hiya Paul,
There's a bunch of tooling out there using pip's internals to extending pip's functionality. Could you please provide a some reasoning as to why they're all going to be broken at pip 10, and possibly some guidance on how to get that functionality back?
Hi Richard, There was a change to the pip docs that clarified the status of pip's internal code. The PR for that is at https://github.com/pypa/pip/pull/4743 but unfortunately it appears that the docs build has been failing so it hasn't yet made it to the formal pip docs site.
To summarise, pip has *never* supported the use of its internal APIs by external code. Over time, we've had a steady trickle of people raising issues when their code broke because of doing so, and it usually turned out to be because they violated assumptions made by the pip code - such as that it's running in a single-threaded application, or it has sole control over the logging subsystem, or even just that you can run your own code after calling a pip API. We've explained this position regularly on those issues, but as is typical, people don't manage to find similar issues when raising new ones, so we spent a lot of time repeating ourselves.
Coming up to pip 10, there's been a *lot* of internal work going on, and it's fairly likely that a decent chunk of code using pip's internal APIs will break anyway. We don't document internals changes, so we faced the possibility of an extended period of people raising issues saying "you broke my code" and us having no better response than "you shouldn't do that", which would likely hinder adoption of pip 10, and cause problems for the ecosystem as a whole. Rather than do this, we decided to make a clean compatibility break, where we could send out a clear message - "everything's moved, if that matters to you, then you were using unsupported functionality and you should find a better way". The breakage is still there (and certainly we made it affect more people, as there are no doubt some people who would have survived the pip 10 release unscathed if we hadn't done this) but at least it's clearly defined and contained.
As to alternatives, we don't have all the answers here but I can offer the following suggestions:
1. There are a number of external packages that provide functionality equivalent to what pip does - packaging, wheel, distlib, pkg_resources are the ones I'm aware of. These are designed as libraries and so *do* provide supported APIs. 2. We're making a strong push to standardise *all* of the external interfaces that pip uses, so you should be far more able to write your own code if that's necessary, without worrying that it'll work differently than pip does, or that things will suddenly change and break your code. 3. You can call pip as a subprocess - that's always been supported and will remain so. We've added some automation-friendly features there (such as json output format for "pip list") and we'd be happy to add more if people want to submit PRs.
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
Paul
PS People who want to, can of course hunt out the new equivalents of the code they were using, and just switch. It's not like we can stop them. But the new names make it clear that they shouldn't be doing this, so there's an obvious warning there. PPS Please disregard xoviat's response. This is something we've been considering for a while, and most definitely not a spur of the moment decision. It's unfortunate that he was the one most immediately affected by the change when we made it, but that was just bad timing (we didn't suddenly do this just because "someone complained").
On 21 October 2017 at 20:03, Paul Moore p.f.moore@gmail.com wrote:
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Cheers, Nick.
On 21 October 2017 at 12:15, Nick Coghlan ncoghlan@gmail.com wrote:
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
No problem - I don't know myself how hard some of this would be, either ;-)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The obvious one is pip.main(). That's the one a lot of people use, but it's easily replaceable by a simple subprocess call. That's actually one of the reasons this was so frustrating to us - the bug reports we got were often from people doing things they didn't need to, that they could handle trivially via a supported approach.
Otherwise, no. We've had little or no feedback on the tracker from people using more complex internals, so our working assumption has been there's very little that can't be handled via the CLI or existing packages. Feedback so far from this mail hints that maybe we were wrong, but it's still hard to know if it's one or two key projects, or a whole range of people that we've yet to hear from. I'm pretty sure, for example, that pipenv uses internals, either directly or via one of their dependencies, but we've not seen any feedback from them yet.
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Understood. We understand *how*, but don't know what is needed. One of the points of all this is to tease out such requirements. We'd hope to get them addressed by including them into *other* packages like distlib or packaging, but that's mostly just a matter of where the API goes, not what is needed. It does help us avoid tying fixes to pip's release cycle, though.
It's also worth noting that the pip devs are possibly way too deep into how pip does things, rather than what the standards say. Getting others to implement libraries based on the published standards would help immensely to ensure that we're not falling into the "implementation defined" trap of the community being stuck with having to use pip because no-one else knows how to do what pip does.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
Oh, sure - apart from the aforementioned "pip.main()", most capabilities of the internal API are *not* easily emulated by the CLI.
But 3rd party libraries are just as much an option. Remember, the issue here isn't so much about designing an API as about exposing (and therefore locking in stone) internal implementation details of how pip does things. And the pip devs are arguably in the *worst* position to handle that option, precisely because they know so much about how pip does things.
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Well, I get your point here, but the implication of this is that we have to design and build an API before we can release pip 10. Calling it CLI wrappers (and implementing it that way) doesn't do anything to ease the work needed on design, or the maintenance burden of providing stability. We're already getting a lot of pressure to release pip 10, and trying to do that would push it way further off.
To be blunt, no-one on the pip team is really interested in trying to provide or support a stable API at this point in time. We have our hands full, as everyone is aware, and this is way down all our priorities. Community-submitted PRs would help, but there's still work in agreeing a design, reviewing those PRs, and maintaining them (exposing details from the internal APIs limits how much we can change those internals later, which is something we have to consider).
What would be ideal would be for the community to build standards-based libraries that satisfied the needs that people currently handle by using pip internals. Heck, we could even consider vendoring such libraries inside of pip and saving ourselves a bunch of complexity ;-) The packaging, pkg_resources and distlib libraries are examples of this happening. But it doesn't seem to be a popular route, unfortunately. And those particular libraries are all maintained by PyPA members, so don't really count as examples of the community getting involved...
The most likely alternative solution would be to revert the internal API moves. If that happened, I'd still prefer to lose pip.main, as that's the major pain point for us and the easiest for users to solve, but we could put the rest back. But it would be naive to assume that as a result people like Noah would be unaffected - they'd actually be worse off, as the internal APIs they use would probably break anyway (the installer/resolver stuff has been heavily modified for pip 10) but not all would do so in obvious ways.
Beyond this, I don't know.
Paul
2017-10-21 14:34 GMT+02:00 Paul Moore p.f.moore@gmail.com:
On 21 October 2017 at 12:15, Nick Coghlan ncoghlan@gmail.com wrote:
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
No problem - I don't know myself how hard some of this would be, either ;-)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The obvious one is pip.main(). That's the one a lot of people use, but it's easily replaceable by a simple subprocess call. That's actually one of the reasons this was so frustrating to us - the bug reports we got were often from people doing things they didn't need to, that they could handle trivially via a supported approach.
Otherwise, no. We've had little or no feedback on the tracker from people using more complex internals, so our working assumption has been there's very little that can't be handled via the CLI or existing packages. Feedback so far from this mail hints that maybe we were wrong, but it's still hard to know if it's one or two key projects, or a whole range of people that we've yet to hear from. I'm pretty sure, for example, that pipenv uses internals, either directly or via one of their dependencies, but we've not seen any feedback from them yet.
Another one that immediately comes to mind is pip-tools [1], which I think is quite widely used.
But I just checked, and they filed a "check out how to deal with pip 10" issue two days ago [2] (I'm guessing in response to this thread).
Elvis
[1] https://github.com/jazzband/pip-tools [2] https://github.com/jazzband/pip-tools/issues/580
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Understood. We understand *how*, but don't know what is needed. One of the points of all this is to tease out such requirements. We'd hope to get them addressed by including them into *other* packages like distlib or packaging, but that's mostly just a matter of where the API goes, not what is needed. It does help us avoid tying fixes to pip's release cycle, though.
It's also worth noting that the pip devs are possibly way too deep into how pip does things, rather than what the standards say. Getting others to implement libraries based on the published standards would help immensely to ensure that we're not falling into the "implementation defined" trap of the community being stuck with having to use pip because no-one else knows how to do what pip does.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
Oh, sure - apart from the aforementioned "pip.main()", most capabilities of the internal API are *not* easily emulated by the CLI.
But 3rd party libraries are just as much an option. Remember, the issue here isn't so much about designing an API as about exposing (and therefore locking in stone) internal implementation details of how pip does things. And the pip devs are arguably in the *worst* position to handle that option, precisely because they know so much about how pip does things.
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Well, I get your point here, but the implication of this is that we have to design and build an API before we can release pip 10. Calling it CLI wrappers (and implementing it that way) doesn't do anything to ease the work needed on design, or the maintenance burden of providing stability. We're already getting a lot of pressure to release pip 10, and trying to do that would push it way further off.
To be blunt, no-one on the pip team is really interested in trying to provide or support a stable API at this point in time. We have our hands full, as everyone is aware, and this is way down all our priorities. Community-submitted PRs would help, but there's still work in agreeing a design, reviewing those PRs, and maintaining them (exposing details from the internal APIs limits how much we can change those internals later, which is something we have to consider).
What would be ideal would be for the community to build standards-based libraries that satisfied the needs that people currently handle by using pip internals. Heck, we could even consider vendoring such libraries inside of pip and saving ourselves a bunch of complexity ;-) The packaging, pkg_resources and distlib libraries are examples of this happening. But it doesn't seem to be a popular route, unfortunately. And those particular libraries are all maintained by PyPA members, so don't really count as examples of the community getting involved...
The most likely alternative solution would be to revert the internal API moves. If that happened, I'd still prefer to lose pip.main, as that's the major pain point for us and the easiest for users to solve, but we could put the rest back. But it would be naive to assume that as a result people like Noah would be unaffected - they'd actually be worse off, as the internal APIs they use would probably break anyway (the installer/resolver stuff has been heavily modified for pip 10) but not all would do so in obvious ways.
Beyond this, I don't know.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi,
On Sun, Oct 22, 2017 at 8:52 AM, Elvis Stansvik elvis.stansvik@orexplore.com wrote:
2017-10-21 14:34 GMT+02:00 Paul Moore p.f.moore@gmail.com:
On 21 October 2017 at 12:15, Nick Coghlan ncoghlan@gmail.com wrote:
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
No problem - I don't know myself how hard some of this would be, either ;-)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The obvious one is pip.main(). That's the one a lot of people use, but it's easily replaceable by a simple subprocess call. That's actually one of the reasons this was so frustrating to us - the bug reports we got were often from people doing things they didn't need to, that they could handle trivially via a supported approach.
Otherwise, no. We've had little or no feedback on the tracker from people using more complex internals, so our working assumption has been there's very little that can't be handled via the CLI or existing packages. Feedback so far from this mail hints that maybe we were wrong, but it's still hard to know if it's one or two key projects, or a whole range of people that we've yet to hear from. I'm pretty sure, for example, that pipenv uses internals, either directly or via one of their dependencies, but we've not seen any feedback from them yet.
Another one that immediately comes to mind is pip-tools [1], which I think is quite widely used.
But I just checked, and they filed a "check out how to deal with pip 10" issue two days ago [2] (I'm guessing in response to this thread).
Now pip 10 is out, of course, I discover that I've lost the implementation of `get_supported` in pip.pep425tags. It's my fault for not remembering I had used it.
Is the suggestion to use the `_internal` import, or carry a copy of the pep425tags code myself, that I have to keep up to date with the internal pip copy? That would also involve me copying the `glibc` part of the code. I see that the `wheel` package has an old copy of that code too, which doesn't deal with manylinux wheels. You probably saw that `pip-tools` ended up vendoring the whole of pip9 [1].
Cheers,
Matthew
[1] https://github.com/jazzband/pip-tools/tree/master/piptools/_vendored/pip
On Apr 14, 2018, at 4:57 PM, Matthew Brett matthew.brett@gmail.com wrote:
Is the suggestion to use the `_internal` import, or carry a copy of the pep425tags code myself, that I have to keep up to date with the internal pip copy? That would also involve me copying the `glibc` part of the code. I see that the `wheel` package has an old copy of that code too, which doesn't deal with manylinux wheels. You probably saw that `pip-tools` ended up vendoring the whole of pip9 [1].
The best solution is to figure out what APIs people need, and either add them to packaging and have pip consume that as well as anyone else, or make a new library for the same.
If that’s unacceptable, vendoring or version pinning is probably the best option.
On 15 April 2018 at 07:31, Donald Stufft donald@stufft.io wrote:
On Apr 14, 2018, at 4:57 PM, Matthew Brett matthew.brett@gmail.com wrote:
Is the suggestion to use the `_internal` import, or carry a copy of the pep425tags code myself, that I have to keep up to date with the internal pip copy? That would also involve me copying the `glibc` part of the code. I see that the `wheel` package has an old copy of that code too, which doesn't deal with manylinux wheels. You probably saw that `pip-tools` ended up vendoring the whole of pip9 [1].
The best solution is to figure out what APIs people need, and either add them to packaging and have pip consume that as well as anyone else, or make a new library for the same.
If that’s unacceptable, vendoring or version pinning is probably the best option.
I think there are going to be at least two steps involved for most projects affected by the API change:
1. The quick fix to add pip 10 compatibility (which is likely a matter of "copy the code you need into the project that needs it") 2. The technical debt reduction to reduce code duplication (which is likely a matter of "add the required APIs to the 'packaging' project")
Step 2 is the step that the pip internals refactoring is designed to encourage, as we believe a lot of tool developers were just using pip's internal APIs rather than filing RFEs and submitting PRs to help guide the evolution of the stable APIs in packaging in a use-case driven manner.
FWIW, `pipenv`'s currently still on "Step 1" at the moment (and has a lot of internal refactoring of its own ahead of it before it will really move on to step 2).
Cheers, Nick.
FWIW we are using quite a bit of the internal api. My plan was to go over this once we cut over to the new warehouse uris. Of note might be the fact that pip-tools is a core dependency we bundle in pipenv and the current maintainer is a pipenv maintainer as well. For our specific case we have made sizeable changes to the dependency resolution stack and bundling allows us to patch freely.
I don’t know that we are a good example though, we are doing significantly more with pip internals than the average project
-dan
Dan Ryan // pipenv maintainer gh: @techalchemy
On Apr 14, 2018, at 11:24 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 15 April 2018 at 07:31, Donald Stufft donald@stufft.io wrote:
On Apr 14, 2018, at 4:57 PM, Matthew Brett matthew.brett@gmail.com wrote:
Is the suggestion to use the `_internal` import, or carry a copy of the pep425tags code myself, that I have to keep up to date with the internal pip copy? That would also involve me copying the `glibc` part of the code. I see that the `wheel` package has an old copy of that code too, which doesn't deal with manylinux wheels. You probably saw that `pip-tools` ended up vendoring the whole of pip9 [1].
The best solution is to figure out what APIs people need, and either add them to packaging and have pip consume that as well as anyone else, or make a new library for the same.
If that’s unacceptable, vendoring or version pinning is probably the best option.
I think there are going to be at least two steps involved for most projects affected by the API change:
- The quick fix to add pip 10 compatibility (which is likely a matter
of "copy the code you need into the project that needs it") 2. The technical debt reduction to reduce code duplication (which is likely a matter of "add the required APIs to the 'packaging' project")
Step 2 is the step that the pip internals refactoring is designed to encourage, as we believe a lot of tool developers were just using pip's internal APIs rather than filing RFEs and submitting PRs to help guide the evolution of the stable APIs in packaging in a use-case driven manner.
FWIW, `pipenv`'s currently still on "Step 1" at the moment (and has a lot of internal refactoring of its own ahead of it before it will really move on to step 2).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick:
That's generally a good idea, but one significant problem that can occur is that the Python import system will cache certain libraries, people will run "pip install," and then they will expect such libraries to be available. I don't even know exactly how the caching for the import system works, so I don't want to go and make claims about it that may be incorrect (maybe you do?). However, it is important to keep that in mind when considering an API.
2017-10-21 6:15 GMT-05:00 Nick Coghlan ncoghlan@gmail.com:
On 21 October 2017 at 20:03, Paul Moore p.f.moore@gmail.com wrote:
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sat, 21 Oct 2017 at 11:03 xoviat xoviat@gmail.com wrote:
Nick:
That's generally a good idea, but one significant problem that can occur is that the Python import system will cache certain libraries, people will run "pip install," and then they will expect such libraries to be available. I don't even know exactly how the caching for the import system works, so I don't want to go and make claims about it that may be incorrect (maybe you do?). However, it is important to keep that in mind when considering an API.
Nick does know how the import system works, but to specifically address this point, as long as the module isn't already imported it's fine. If it has, though, then you will need to do a reload instead of an import, and even then that doesn't necessarily work the way some people want it to.
-Brett
2017-10-21 6:15 GMT-05:00 Nick Coghlan ncoghlan@gmail.com:
On 21 October 2017 at 20:03, Paul Moore p.f.moore@gmail.com wrote:
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
(Note: this is entirely speculative, and I have no idea how hard it would be, so please read it as the question it's intended to be)
Do you know if there any key APIs (like installation) that could be turned into wrappers around pip CLI calls in order to mitigate some of the impact?
The reason I ask is because it's unlikely anyone else is going to understand how to emulate the previous functionality better than the pip devs would, and if there's an API for those particular invocations, than they can be covered directly by pip's test suite.
Plus if there are previous API capabilities that *can't* currently be emulated via the CLI, then the pip devs are the folks in the best position to add the necessary CLI enhancements (such as the ones Noah asked about for doing a more selective `pip list`).
If that's an approach you might be amenable to, then a 10.0 pre-release could be a good time to solicit PRs from folks that were using particular APIs and would be prepared to invest time in defining comparable CLI wrappers for them.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Oct 21, 2017, at 2:15 PM, Brett Cannon brett@python.org wrote:
as long as the module isn't already imported it's fine.
Negative imports get cached too don’t they?
On Sat, 21 Oct 2017 at 11:26 Donald Stufft donald@stufft.io wrote:
On Oct 21, 2017, at 2:15 PM, Brett Cannon brett@python.org wrote:
as long as the module isn't already imported it's fine.
Negative imports get cached too don’t they?
Yes, that too. :) (aside: don't reply to technical emails while you're waiting to head out the door for a weekend away, you will forget details ;) .
-Brett
On 22 October 2017 at 04:03, xoviat xoviat@gmail.com wrote:
Nick:
That's generally a good idea, but one significant problem that can occur is that the Python import system will cache certain libraries, people will run "pip install," and then they will expect such libraries to be available. I don't even know exactly how the caching for the import system works, so I don't want to go and make claims about it that may be incorrect (maybe you do?). However, it is important to keep that in mind when considering an API.
Yep, since we switched to the implementation that uses fewer stat calls, you need to call `importlib.invalidate_caches()` to be sure your current process will see anything you just installed. However, for modules you haven't previously imported, that's independent of how you installed them - the problem is that the granularity on the import system's automated cache invalidation depends on the granularity of the filesystem's mtime records for directories, so you either have to sleep for a few seconds (since the mtime resolution on filesystems like FAT & FAT32 is only a couple of seconds), or else force the cache invalidation.
The problem with the sys.modules cache and already imported libraries is different: for those, you either need to use `importlib.reload(existing_module)` to force an in-place update of the existing namespace, or else `del sys.modules[existing_module.__name__]` to force creation of a new module without affecting old references to it.
It's those subtleties that keep "Restart all Python processes if you expect them to see components you just installed" in place as our default guidance: it's the only option that's guaranteed to work in all cases. While hot reloading can be made to work, it has assorted caveats and restrictions that you need to learn in order to do it correctly and reliably (many of them are actually pretty similar to the self-imposed restrictions needed to make lazy loading work reliably).
However, none of that impacts the question of whether `pip.main()` runs code in the current process or implicitly runs it in a subprocess - `pip` doesn't import the modules it installs either way, so it will all look the same as far as the import system is concerned.
Cheers, Nick.
On Oct 21, 2017, at 10:30 PM, Nick Coghlan ncoghlan@gmail.com wrote:
However, none of that impacts the question of whether `pip.main()` runs code in the current process or implicitly runs it in a subprocess - `pip` doesn't import the modules it installs either way, so it will all look the same as far as the import system is concerned.
That of course also ignores things like “foo.py optionally imports bar.py if it is available”, with something like:
try: import bar except ImportError: bar = None
If you then did ``import foo``, noticed the optional features powered by bar weren’t added, you would also have to reload ``foo`` (and track down any references to it!). Reloading modules in Python is hard :/
The additional thing here is that the import system isn’t the only cache at play either— pkg_resources builds a cache of installed packages at import time too and that has to get invalidated somehow as well. This took pip like 3 or 4 versions to get right because we tried to detect what version of pip was installed _after_ we installed all the versions in order to stop doing the outdated warning with ``pip install -U pip`` and I’m still not entirely convinced we’re not erroneously spitting out the warning in some cases.
The end result of all of this is unless you’re really really careful and design things just right and know exactly how not only yourself, but all your dependencies are written (and track how they change in different versions!) and also how things that use _you_ are going to import you and your dependencies… it’s basically playing whack a mole and is almost never worth the effort.
Thanks for writing that detailed explanation, Paul (and all your other hard work!)
Richard
On 21 October 2017 at 21:03, Paul Moore p.f.moore@gmail.com wrote:
On 20 October 2017 at 23:53, Richard Jones r1chardj0n3s@gmail.com wrote:
Hiya Paul,
There's a bunch of tooling out there using pip's internals to extending pip's functionality. Could you please provide a some reasoning as to why they're all going to be broken at pip 10, and possibly some guidance on
how
to get that functionality back?
Hi Richard, There was a change to the pip docs that clarified the status of pip's internal code. The PR for that is at https://github.com/pypa/pip/pull/4743 but unfortunately it appears that the docs build has been failing so it hasn't yet made it to the formal pip docs site.
To summarise, pip has *never* supported the use of its internal APIs by external code. Over time, we've had a steady trickle of people raising issues when their code broke because of doing so, and it usually turned out to be because they violated assumptions made by the pip code - such as that it's running in a single-threaded application, or it has sole control over the logging subsystem, or even just that you can run your own code after calling a pip API. We've explained this position regularly on those issues, but as is typical, people don't manage to find similar issues when raising new ones, so we spent a lot of time repeating ourselves.
Coming up to pip 10, there's been a *lot* of internal work going on, and it's fairly likely that a decent chunk of code using pip's internal APIs will break anyway. We don't document internals changes, so we faced the possibility of an extended period of people raising issues saying "you broke my code" and us having no better response than "you shouldn't do that", which would likely hinder adoption of pip 10, and cause problems for the ecosystem as a whole. Rather than do this, we decided to make a clean compatibility break, where we could send out a clear message - "everything's moved, if that matters to you, then you were using unsupported functionality and you should find a better way". The breakage is still there (and certainly we made it affect more people, as there are no doubt some people who would have survived the pip 10 release unscathed if we hadn't done this) but at least it's clearly defined and contained.
As to alternatives, we don't have all the answers here but I can offer the following suggestions:
- There are a number of external packages that provide functionality
equivalent to what pip does - packaging, wheel, distlib, pkg_resources are the ones I'm aware of. These are designed as libraries and so *do* provide supported APIs. 2. We're making a strong push to standardise *all* of the external interfaces that pip uses, so you should be far more able to write your own code if that's necessary, without worrying that it'll work differently than pip does, or that things will suddenly change and break your code. 3. You can call pip as a subprocess - that's always been supported and will remain so. We've added some automation-friendly features there (such as json output format for "pip list") and we'd be happy to add more if people want to submit PRs.
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
Paul
PS People who want to, can of course hunt out the new equivalents of the code they were using, and just switch. It's not like we can stop them. But the new names make it clear that they shouldn't be doing this, so there's an obvious warning there. PPS Please disregard xoviat's response. This is something we've been considering for a while, and most definitely not a spur of the moment decision. It's unfortunate that he was the one most immediately affected by the change when we made it, but that was just bad timing (we didn't suddenly do this just because "someone complained").
Paul Moore kirjoitti 21.10.2017 klo 13:03:
On 20 October 2017 at 23:53, Richard Jones r1chardj0n3s@gmail.com wrote:
Hiya Paul,
There's a bunch of tooling out there using pip's internals to extending pip's functionality. Could you please provide a some reasoning as to why they're all going to be broken at pip 10, and possibly some guidance on how to get that functionality back?
Hi Richard, There was a change to the pip docs that clarified the status of pip's internal code. The PR for that is at https://github.com/pypa/pip/pull/4743 but unfortunately it appears that the docs build has been failing so it hasn't yet made it to the formal pip docs site.
To summarise, pip has *never* supported the use of its internal APIs by external code. Over time, we've had a steady trickle of people raising issues when their code broke because of doing so, and it usually turned out to be because they violated assumptions made by the pip code - such as that it's running in a single-threaded application, or it has sole control over the logging subsystem, or even just that you can run your own code after calling a pip API. We've explained this position regularly on those issues, but as is typical, people don't manage to find similar issues when raising new ones, so we spent a lot of time repeating ourselves.
Coming up to pip 10, there's been a *lot* of internal work going on, and it's fairly likely that a decent chunk of code using pip's internal APIs will break anyway. We don't document internals changes, so we faced the possibility of an extended period of people raising issues saying "you broke my code" and us having no better response than "you shouldn't do that", which would likely hinder adoption of pip 10, and cause problems for the ecosystem as a whole. Rather than do this, we decided to make a clean compatibility break, where we could send out a clear message - "everything's moved, if that matters to you, then you were using unsupported functionality and you should find a better way". The breakage is still there (and certainly we made it affect more people, as there are no doubt some people who would have survived the pip 10 release unscathed if we hadn't done this) but at least it's clearly defined and contained.
As to alternatives, we don't have all the answers here but I can offer the following suggestions:
- There are a number of external packages that provide functionality
equivalent to what pip does - packaging, wheel, distlib, pkg_resources are the ones I'm aware of. These are designed as libraries and so *do* provide supported APIs.
I need to correct you here: wheel does *NOT* have a public API!
It did previously have some documented functions but it was not really well thought out and we recently decided to remove all traces of API documentation so as to not cause any confusion about it.
- We're making a strong push to standardise *all* of the external
interfaces that pip uses, so you should be far more able to write your own code if that's necessary, without worrying that it'll work differently than pip does, or that things will suddenly change and break your code. 3. You can call pip as a subprocess - that's always been supported and will remain so. We've added some automation-friendly features there (such as json output format for "pip list") and we'd be happy to add more if people want to submit PRs.
Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more).
Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code.
Paul
PS People who want to, can of course hunt out the new equivalents of the code they were using, and just switch. It's not like we can stop them. But the new names make it clear that they shouldn't be doing this, so there's an obvious warning there. PPS Please disregard xoviat's response. This is something we've been considering for a while, and most definitely not a spur of the moment decision. It's unfortunate that he was the one most immediately affected by the change when we made it, but that was just bad timing (we didn't suddenly do this just because "someone complained"). _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 21 October 2017 at 19:11, Alex Grönholm alex.gronholm@nextday.fi wrote:
I need to correct you here: wheel does *NOT* have a public API!
Ah, sorry. I hadn't realised that.
It did previously have some documented functions but it was not really well thought out and we recently decided to remove all traces of API documentation so as to not cause any confusion about it.
That's probably what I was remembering.
Paul
participants (12)
-
Alex Grönholm -
Brett Cannon -
Dan Ryan -
Donald Stufft -
Elvis Stansvik -
Jannis Gebauer -
Matthew Brett -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Richard Jones -
xoviat