So, pip 7.0 depends on the wheel module for its automatic wheel building, and installing pip from get-pip.py, or the bundled copy in virtualenvs will automatically install wheel. But ensurepip doesn't bundle wheel, so we're actually installing a slightly crippled pip 7.1, which will lead to folk having a poorer experience. Is this a simple bug, or do we need to update the PEP? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On August 2, 2015 at 8:47:46 PM, Robert Collins (robertc@robertcollins.net) wrote:
Personally, I think it's not going to be worth the pain to add wheel to ensurepip. We (pip) already have a somewhat rocky relationship with some downstream vendors because of the bundling of pip and setuptools that I'm not sure that wheel makes sense. Especially given that I want the optional dependency on Wheel to be a temporary measure until we can just implicitly install wheel as a build time dependency within pip and no longer need to install it implicitly in get-pip.py or virtualenv. In the future I expect setuptools to be removed as well at a similar time when we can implicitly install setuptools as a build time dependency of an sdist and do not require end users to install it explicitly. That being said, I think the PEP would need to be updated (and possibly a new PEP?) since we explicitly called out the fact that setuptools would currently be included until pip no longer needed it to be installed seperately. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 3 August 2015 at 11:06, Donald Stufft <donald@stufft.io> wrote:
I'm going to contradict what I said to Robert at the PyCon AU sprints earlier this week, and agree with Donald here. setuptools is in the situation where because it also includes pkg_resources, it blurs the line between "build time" and "run time" dependency. While it would be nice to split that and have a "just pkg_resources" runtime dependency distinct from the build time dependency, that isn't likely to happen any time soon. wheel, by contrast, is already a pure build time dependency for bdist_wheel, and thus should be getting brought in as an implied "build requires" by pip itself when building from source. This does pose an interesting challenge from the perspective of the "offline installation" use case for ensurepip, where wheels are used as a local build caching mechanism, but we don't assume PyPI access, but it isn't one we really considered in the original ensurepip PEP. So actually doing this would probably require a PEP to update ensurepip with some additional options related to whether the build dependencies should be installed or not, and give downstream vendors a free pass to exclude the build dependencies from the default installation set. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On August 5, 2015 at 10:01:50 AM, Nick Coghlan (ncoghlan@gmail.com) wrote:
Just a small correction, in general setuptools does blur that line, but for pip itself setuptools is completely a build time dependency which isn’t *technically* any different than our dependency on wheel. We work perfectly fine without it installed you just don’t get certain features available to you if you don’t have it installed. However we left setuptools installing because the feature you lose if you don’t have it pre-installed is the ability to install from sdists entirely. It was determined that not being able to install from sdists was a large enough “breakage” that considering setuptools a dependency of pip in the terms of ensurepip was considered better than minimizing the things we bundled. On the flip side, the thing you lose if you don’t have wheel installed is more like a “nice to have” than something that breaks functionality that most people would consider mandatory in the current landscape. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 6 August 2015 at 00:10, Donald Stufft <donald@stufft.io> wrote:
Sorry, I omitted a downstream-related step in my thought process there. We currently have a hard dependency from python to pip and setuptools in Fedora, so we can ensure ensurepip works properly inside virtual environments. Enough things require setuptools at runtime for pkg_resources that that falls into the category of "annoying runtime dependency we'd like to see go away, but we can live with it since a lot of production systems are still going to end up with it installed regardless of what's in the base image". A hard dependency on wheel wouldn't fit into the same category - when folks are using a build pipeline to minimise the installation footprint on production systems, the wheel package itself has no business being installed anywhere other than developer systems and build servers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 5 août 2015 17:12, "Nick Coghlan" <ncoghlan@gmail.com> a écrit :
I'm quite sure that virtualenv is used to deploy python on production. Pip 7 automatically creates wheel packages when no build wheel package is available on PyPI. Examples numpy and any pure python package only providing a tarball. For me it makes sense to embed wheel in ensurepip and to install wheel on production systems (to install pacakes, not to build them). Victor
On 6 August 2015 at 09:29, Victor Stinner <victor.stinner@gmail.com> wrote:
pip can install from wheels just fine without the wheel package being present - that's how ensurepip already works. The wheel package itself is only needed in order to support the setuptools "bdist_wheel" command, which then allows pip to implicitly cache wheel files when installing from an sdist. Installing from sdist in production is a *fundamentally bad idea*, because it means you have to have a build toolchain on your production servers. One of the benefits of the wheel format and projects like devpi is that it makes it easier to discourage people from doing that. Even without getting into Linux containers and tools like pyp2rpm, it's also possible to create an entire virtualenv on a build server, bundle that up as an RPM or DEB file, and use the system package manager to do the production deployment. However, production Linux servers aren't the only case we need to care about, and there's a strong user experience argument to be made for providing wheel by default upstream, and telling downstream redistributors that care about the distinction to do the necessary disentangling to make it easy to have "build dependency free" production images. We've learned from experience that things go far more smoothly if we thrash out those kinds of platform dependent behavioural differences *before* we inflict them on end users, rather than having downstream redistributors tackle foreseeable problems independently of both each other and upstream :) Hence my request for a PEP - I can see why adding wheel to the ensurepip bundle would be a good idea for upstream, but I can also see why it's a near certainty downstream Linux distros (including Fedora) would take it out again in at least some situations to better meet the needs of *our* user base. (Since RPM has weak dependency support now, we'd likely make python-wheel a "Recommends:" dependency, rather than a "Requires:" dependency - still installed by default, but easy to omit if not wanted or needed) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 August 2015 at 15:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
pip can also do this without setuptools being installed; yet we bundle setuptools with pip in ensurepip. I am thus confused :). When I consider the harm to a production pipeline that using setuptools can cause (in that it triggers easy_install, and easy_install has AFAIK none of the security improvements pip has added over the last couple years....), I find the acceptance of setuptools, but non-acceptance of wheel flummoxing.
Yes: but the logic chain from 'its a bad idea' to 'we don't include wheel but we do include setuptools' is the bit I'm having a hard time with.
Does Fedora also take out setuptools? If not, why not?
So, a new PEP? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On 6 August 2015 at 19:04, Robert Collins <robertc@robertcollins.net> wrote:
When ensurepip was implemented, pip couldn't install from wheel files without setuptools yet, and the level of adoption of wheel files in general was lower than it is today.
Just an accident of history due to the relative timing of ensurepip's introduction, pip gaining the ability to install wheel files without setuptools, and high levels of adoption of the wheel format on PyPI. If PEP 453 was redone today, it's entirely possible setuptools wouldn't have been bundled, but it wasn't a viable option at the time. Accepting the bundling was a nice piece of technical debt that bought several additional months of feature availability :)
Not at the moment - while I'd like to see the dependency go away eventually, there are plenty of other things in the world that bother me more, especially since it comes back the moment someone has an "import pkg_resources" anywhere in their application.
Yeah. I don't think it needs to be too fancy, just provide a way to indicate whether or not ensurepip should install the wheel package, and make it clear that if folks want to ensure pip can build wheels, they should install it explicitly (at the command line or as a dependency), rather than assuming it will always be there by default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
In my opinion, it’s the severity of how crippled their experience is without that particular thing installed. In the case of wheel not being installed they lose the ability to have an implicit wheel cache and to run ``pip wheel``. This makes pip less good but, unless they are running ``pip wheel`` everything is still fully functioning. In the case of setuptools they lose the ability to ``pip install`` when there isn’t a wheel available and the ability to run ``pip wheel``. This is making pip completely unusable for a lot of people, and if we did not pre-install setup tools the number one thing people would do is to ``pip install setuptools``, most likely while bitching under their breath about the command that just failed because they tried to install from sdist. So it’s really just “how bad are we going to break people’s expectations”.
On 7 August 2015 at 03:28, Donald Stufft <donald@stufft.io> wrote:
So - I was in a talk at PyCon AU about conda[*], and the author believed they were using the latest pip with all the latest caching features, but their experience (16 minute installs) wasn't that. I dug into that with them after the talk, and it was due to Conda not installing wheel by default. Certainly the framing of ensurepip as 'this installs pip' is going to be confusing and misleading if it doesn't install pip the way get-pip.py (or virtualenv) install pip, leading to confusion such as that. Given the inconsequential impact of installing wheel, I see only harm in holding it back, and only benefits in adding it. All the harm from having source builds comes in with setuptools ;). -Rob *) https://www.youtube.com/watch?v=Fqknoni5aX0 -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On 7 August 2015 at 08:50, Robert Collins <robertc@robertcollins.net> wrote:
Right, this is the main reason I'm actually *in favour* of adding wheel to the ensurepip bundle upstream - it significantly improves the "out of the box" experience of pyvenv by implicitly caching builds. (I'm also in favour because it will lead to redistributors providing "pip wheel" support by default, and having to make an explicit design decision *not* to provide it if we want to do something different). The only reason I'm asking for a PEP is because I'm confident we're going to want a "support prebuilt wheels only" installation option downstream in the Linux distro world - shipping setuptools by default is a pragmatic concession to practical reality rather than something we *want* to be doing. As such, I do think Robert raises a good point that any new ensurepip option should probably prevent installation of both wheel *and* setuptools, since pip can install from wheel files without setuptools these days. The CLI option name might be something like "--no-build-tools", and could also be added to the public pyvenv and virtualenv interfaces. Downstream in Fedora, now that we have weak dependency support, I'd advocate for switching the python->setuptools dependency over to Recommends, and adding wheel as a Recommends dependency from the start. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I’m not sure if —no-build-tools make sense, since I plan on removing setuptools from ensurepip completely once pip can implicitly install it. PEP 453 explicitly called out the fact that setuptools was installed as an implementation detail with an eye to remove it in the future. Adding flags that deal with it specifically doesn’t seem like the right path to go down.
On 7 August 2015 at 17:20, Donald Stufft <donald@stufft.io> wrote:
I’m not sure if —no-build-tools make sense, since I plan on removing setuptools from ensurepip completely once pip can implicitly install it. PEP 453 explicitly called out the fact that setuptools was installed as an implementation detail with an eye to remove it in the future. Adding flags that deal with it specifically doesn’t seem like the right path to go down.
I'd be happy for the flag to go the other way, and have to supply "--build-tools" in order to opt in to having ensurepip install setuptools and wheel in addition to pip itself. If we did that, the downstream setup would likely be the even weaker "Suggests" dependency. My use case here is the "offline Python installation" one - having the build tools bundled with CPython and readily available is still useful for cases where you have your own code you want to build, but can't go to the internet to get a build toolchain. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Interesting-- so move to a Python specific binary distribution option -- rather than using rm or deb packages? Doesn't lead to a dependency heck? I.e no way to express non-python dependencies? And while we are moving forward, can we please deprecate dependency management and installation from setuptools? Is there a philosophy of intended separation of concerns articulated somewhere? -Chris
Le 7 août 2015 00:51, "Robert Collins" <robertc@robertcollins.net> a écrit :
If an expert user is unaware of having to explicitly install wheel, what about other users? Packaging is the most hated feature of Python. Please don't add extra pain for purity and make sure that ensurepip installs "pip" and not "slow pip until you install wheel in the venv". To develop on OpenStack, I have more than 20 virtual environment on my PC. I recreate them regulary because I like downgarding or upgrading a package, or edit the code of a package directly in the venv (usually to debug). Since pip7, the creation of venv is much faster. Please don't make pip slower. Victor
This is a really good point -- other than purity, what is the downside? Arguably, the only reason setuptools, pip, and wheel are not in the standard library are because the need a more rapid development/release cycle. Ensurepip is the way to get the best of both worlds -- why not make it complete? -Chris
So my opinion is basically that in a vacuum I would absolutely add wheel to ensurepip (and I did add wheel to get-pip.py and to virtualenv). However, this does not exist in a vacuum and there is still animosity about PEP 453 and downstream’s are still trying to figure out how they are going to handle it for real. During the 3.4 release there were downstream redisttibutors who completely removed ensurepip and were talking about possibly removing pip entirely from their archives. So my hesitation is basically that I consider is a short (or medium) term need until pip can implicitly install wheel and setuptools as part of the build process for a particular project and I worry that it will reopen some wounds and cause more strife. I do however think it would make ensurepip itself better, so I’m not dead set against it, mostly just worried about ramifications.
On 8 August 2015 at 02:12, Donald Stufft <donald@stufft.io> wrote:
[I'm wearing my professional Fedora Environments & Stack WG and RHEL Developer Experience hats in this post, moreso than my CPython core developer one] It seems to me that most modern open source developers (especially those using dynamic languages) perceive Linux distros more as impediments to be worked around, rather than as allies to collaborate with, and that's *our* UX issue to figure out downstream (hence design concepts like https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManag... and https://fedoraproject.org/wiki/Env_and_Stacks/Projects/PackageReviewProcessR... in the Fedora space) It's not CPython's problem to resolve, and it's only CPython's responsibility to work around to the extent that it makes things easier for *end users* developing in Python. If a distro is being unreasonably intransigent about developer experience concerns, then that's the distro's problem, and we can advise people to download and use a cross-platform distro like ActivePython, EPD/Canopy or Anaconda instead of the system Python.
So my hesitation is basically that I consider is a short (or medium) term need until pip can implicitly install wheel and setuptools as part of the build process for a particular project and I worry that it will reopen some wounds and cause more strife.
I don't believe it's a good idea to avoid strife for the sake of avoiding strife - many Linux distros are in the wrong here, and we need to get with the program in suitably meeting the needs of open source developers, not just folks running Linux on production servers. Fedora started that process with the launch of the Fedora.next initiatives a couple of years ago, but there's still a lot of work to be done in retooling our online and desktop experience to make it more developer friendly.
I do however think it would make ensurepip itself better, so I’m not dead set against it, mostly just worried about ramifications.
I'd advise against letting concerns about Linux distro politics hold you back from making ensurepip as good as you can make it - if nothing else, the developer experience folks at commercial Linux vendors are specifically paid to advocate for the interests of software developers when it comes to the Linux user experience (that's part of my own day job in the Fedora/RHEL/CentOS case - I moved over to the software management team in RHEL Developer Experience at the start of June). That means that while I will have some *requests* to make certain things easier downstream (like going through the PEP process to figure out an upstream supported way to omit the build-only dependencies when running ensurepip), I also wholeheartedly endorse the idea of having the default upstream behaviour focus on making the initial experience for folks downloading Windows or Mac OS X binaries from python.org as compelling as we can make it. python-dev needs to put the needs of Python first, and those of Linux second. This does mean that any Linux distro that can't figure out how to provide a better open source developer experience for Pythonistas than Windows or Mac OS X is at risk of falling by the wayside in the Python community, but if those of us that care specifically about the viability of desktop Linux as a platform for open source development stand by and let that happen, then we'll *deserve* the consequences. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8 August 2015 at 04:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sorry I'm late to this, but I would very much like to see wheel installed with ensurepip on at least Windows. (I don't see any reason why OSX would not be similar, but as I'm not an OSX user I can't say for certain). If Linux distros have issue with this, then maybe we need to do something different there (I like Nick's comments, and would rather we didn't make the Linux situation worse due to distro politics, but that's not my call) but that shouldn't affect Windows/OSX. There's a certain irony to me in the fact that we're reaching a point where the Windows experience is the benchmark Linux users need to aim for, but I'll avoid saying anything more on that one ;-) Paul
On Sun, Aug 16, 2015 at 02:17:09PM +0100, Paul Moore wrote:
Sorry I'm late to this, but I would very much like to see wheel installed with ensurepip on at least Windows.
I seem to be missing something critical to this entire discussion. As I understand it, ensurepip is *only* intended to bootstrap pip itself. So the idea is, you install Python, including ensurepip, use that to install the latest pip *including wheel*, and Bob's your uncle. At worst, you install pip, then install wheel. So what is the benefit of including wheel with ensurepip? -- Steve
On August 16, 2015 at 10:41:42 AM, Steven D'Aprano (steve@pearwood.info) wrote:
pip has an optional dependency on wheel, if you install that optional dependency than you’ll get the implicit wheel cache enabled by default which can drastically improve installation speeds by caching built artifacts (i.e. ``pip instal lxml`` multiple times only compiles it once). The goal is to get more people getting the benefits of that by default instead of requiring them to know they need to ``pip install wheel`` after the fact. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On August 16, 2015 at 11:26:08 AM, Steven D'Aprano (steve@pearwood.info) wrote:
That’s the proposal here, ensurepip only installs things it has bundled inside of it, so we’d add a .whl file for wheel and slightly tweak ensurepip so it also installs wheel. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On August 2, 2015 at 8:47:46 PM, Robert Collins (robertc@robertcollins.net) wrote:
Personally, I think it's not going to be worth the pain to add wheel to ensurepip. We (pip) already have a somewhat rocky relationship with some downstream vendors because of the bundling of pip and setuptools that I'm not sure that wheel makes sense. Especially given that I want the optional dependency on Wheel to be a temporary measure until we can just implicitly install wheel as a build time dependency within pip and no longer need to install it implicitly in get-pip.py or virtualenv. In the future I expect setuptools to be removed as well at a similar time when we can implicitly install setuptools as a build time dependency of an sdist and do not require end users to install it explicitly. That being said, I think the PEP would need to be updated (and possibly a new PEP?) since we explicitly called out the fact that setuptools would currently be included until pip no longer needed it to be installed seperately. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 3 August 2015 at 11:06, Donald Stufft <donald@stufft.io> wrote:
I'm going to contradict what I said to Robert at the PyCon AU sprints earlier this week, and agree with Donald here. setuptools is in the situation where because it also includes pkg_resources, it blurs the line between "build time" and "run time" dependency. While it would be nice to split that and have a "just pkg_resources" runtime dependency distinct from the build time dependency, that isn't likely to happen any time soon. wheel, by contrast, is already a pure build time dependency for bdist_wheel, and thus should be getting brought in as an implied "build requires" by pip itself when building from source. This does pose an interesting challenge from the perspective of the "offline installation" use case for ensurepip, where wheels are used as a local build caching mechanism, but we don't assume PyPI access, but it isn't one we really considered in the original ensurepip PEP. So actually doing this would probably require a PEP to update ensurepip with some additional options related to whether the build dependencies should be installed or not, and give downstream vendors a free pass to exclude the build dependencies from the default installation set. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On August 5, 2015 at 10:01:50 AM, Nick Coghlan (ncoghlan@gmail.com) wrote:
Just a small correction, in general setuptools does blur that line, but for pip itself setuptools is completely a build time dependency which isn’t *technically* any different than our dependency on wheel. We work perfectly fine without it installed you just don’t get certain features available to you if you don’t have it installed. However we left setuptools installing because the feature you lose if you don’t have it pre-installed is the ability to install from sdists entirely. It was determined that not being able to install from sdists was a large enough “breakage” that considering setuptools a dependency of pip in the terms of ensurepip was considered better than minimizing the things we bundled. On the flip side, the thing you lose if you don’t have wheel installed is more like a “nice to have” than something that breaks functionality that most people would consider mandatory in the current landscape. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 6 August 2015 at 00:10, Donald Stufft <donald@stufft.io> wrote:
Sorry, I omitted a downstream-related step in my thought process there. We currently have a hard dependency from python to pip and setuptools in Fedora, so we can ensure ensurepip works properly inside virtual environments. Enough things require setuptools at runtime for pkg_resources that that falls into the category of "annoying runtime dependency we'd like to see go away, but we can live with it since a lot of production systems are still going to end up with it installed regardless of what's in the base image". A hard dependency on wheel wouldn't fit into the same category - when folks are using a build pipeline to minimise the installation footprint on production systems, the wheel package itself has no business being installed anywhere other than developer systems and build servers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 5 août 2015 17:12, "Nick Coghlan" <ncoghlan@gmail.com> a écrit :
I'm quite sure that virtualenv is used to deploy python on production. Pip 7 automatically creates wheel packages when no build wheel package is available on PyPI. Examples numpy and any pure python package only providing a tarball. For me it makes sense to embed wheel in ensurepip and to install wheel on production systems (to install pacakes, not to build them). Victor
On 6 August 2015 at 09:29, Victor Stinner <victor.stinner@gmail.com> wrote:
pip can install from wheels just fine without the wheel package being present - that's how ensurepip already works. The wheel package itself is only needed in order to support the setuptools "bdist_wheel" command, which then allows pip to implicitly cache wheel files when installing from an sdist. Installing from sdist in production is a *fundamentally bad idea*, because it means you have to have a build toolchain on your production servers. One of the benefits of the wheel format and projects like devpi is that it makes it easier to discourage people from doing that. Even without getting into Linux containers and tools like pyp2rpm, it's also possible to create an entire virtualenv on a build server, bundle that up as an RPM or DEB file, and use the system package manager to do the production deployment. However, production Linux servers aren't the only case we need to care about, and there's a strong user experience argument to be made for providing wheel by default upstream, and telling downstream redistributors that care about the distinction to do the necessary disentangling to make it easy to have "build dependency free" production images. We've learned from experience that things go far more smoothly if we thrash out those kinds of platform dependent behavioural differences *before* we inflict them on end users, rather than having downstream redistributors tackle foreseeable problems independently of both each other and upstream :) Hence my request for a PEP - I can see why adding wheel to the ensurepip bundle would be a good idea for upstream, but I can also see why it's a near certainty downstream Linux distros (including Fedora) would take it out again in at least some situations to better meet the needs of *our* user base. (Since RPM has weak dependency support now, we'd likely make python-wheel a "Recommends:" dependency, rather than a "Requires:" dependency - still installed by default, but easy to omit if not wanted or needed) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 August 2015 at 15:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
pip can also do this without setuptools being installed; yet we bundle setuptools with pip in ensurepip. I am thus confused :). When I consider the harm to a production pipeline that using setuptools can cause (in that it triggers easy_install, and easy_install has AFAIK none of the security improvements pip has added over the last couple years....), I find the acceptance of setuptools, but non-acceptance of wheel flummoxing.
Yes: but the logic chain from 'its a bad idea' to 'we don't include wheel but we do include setuptools' is the bit I'm having a hard time with.
Does Fedora also take out setuptools? If not, why not?
So, a new PEP? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On 6 August 2015 at 19:04, Robert Collins <robertc@robertcollins.net> wrote:
When ensurepip was implemented, pip couldn't install from wheel files without setuptools yet, and the level of adoption of wheel files in general was lower than it is today.
Just an accident of history due to the relative timing of ensurepip's introduction, pip gaining the ability to install wheel files without setuptools, and high levels of adoption of the wheel format on PyPI. If PEP 453 was redone today, it's entirely possible setuptools wouldn't have been bundled, but it wasn't a viable option at the time. Accepting the bundling was a nice piece of technical debt that bought several additional months of feature availability :)
Not at the moment - while I'd like to see the dependency go away eventually, there are plenty of other things in the world that bother me more, especially since it comes back the moment someone has an "import pkg_resources" anywhere in their application.
Yeah. I don't think it needs to be too fancy, just provide a way to indicate whether or not ensurepip should install the wheel package, and make it clear that if folks want to ensure pip can build wheels, they should install it explicitly (at the command line or as a dependency), rather than assuming it will always be there by default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
In my opinion, it’s the severity of how crippled their experience is without that particular thing installed. In the case of wheel not being installed they lose the ability to have an implicit wheel cache and to run ``pip wheel``. This makes pip less good but, unless they are running ``pip wheel`` everything is still fully functioning. In the case of setuptools they lose the ability to ``pip install`` when there isn’t a wheel available and the ability to run ``pip wheel``. This is making pip completely unusable for a lot of people, and if we did not pre-install setup tools the number one thing people would do is to ``pip install setuptools``, most likely while bitching under their breath about the command that just failed because they tried to install from sdist. So it’s really just “how bad are we going to break people’s expectations”.
On 7 August 2015 at 03:28, Donald Stufft <donald@stufft.io> wrote:
So - I was in a talk at PyCon AU about conda[*], and the author believed they were using the latest pip with all the latest caching features, but their experience (16 minute installs) wasn't that. I dug into that with them after the talk, and it was due to Conda not installing wheel by default. Certainly the framing of ensurepip as 'this installs pip' is going to be confusing and misleading if it doesn't install pip the way get-pip.py (or virtualenv) install pip, leading to confusion such as that. Given the inconsequential impact of installing wheel, I see only harm in holding it back, and only benefits in adding it. All the harm from having source builds comes in with setuptools ;). -Rob *) https://www.youtube.com/watch?v=Fqknoni5aX0 -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On 7 August 2015 at 08:50, Robert Collins <robertc@robertcollins.net> wrote:
Right, this is the main reason I'm actually *in favour* of adding wheel to the ensurepip bundle upstream - it significantly improves the "out of the box" experience of pyvenv by implicitly caching builds. (I'm also in favour because it will lead to redistributors providing "pip wheel" support by default, and having to make an explicit design decision *not* to provide it if we want to do something different). The only reason I'm asking for a PEP is because I'm confident we're going to want a "support prebuilt wheels only" installation option downstream in the Linux distro world - shipping setuptools by default is a pragmatic concession to practical reality rather than something we *want* to be doing. As such, I do think Robert raises a good point that any new ensurepip option should probably prevent installation of both wheel *and* setuptools, since pip can install from wheel files without setuptools these days. The CLI option name might be something like "--no-build-tools", and could also be added to the public pyvenv and virtualenv interfaces. Downstream in Fedora, now that we have weak dependency support, I'd advocate for switching the python->setuptools dependency over to Recommends, and adding wheel as a Recommends dependency from the start. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I’m not sure if —no-build-tools make sense, since I plan on removing setuptools from ensurepip completely once pip can implicitly install it. PEP 453 explicitly called out the fact that setuptools was installed as an implementation detail with an eye to remove it in the future. Adding flags that deal with it specifically doesn’t seem like the right path to go down.
On 7 August 2015 at 17:20, Donald Stufft <donald@stufft.io> wrote:
I’m not sure if —no-build-tools make sense, since I plan on removing setuptools from ensurepip completely once pip can implicitly install it. PEP 453 explicitly called out the fact that setuptools was installed as an implementation detail with an eye to remove it in the future. Adding flags that deal with it specifically doesn’t seem like the right path to go down.
I'd be happy for the flag to go the other way, and have to supply "--build-tools" in order to opt in to having ensurepip install setuptools and wheel in addition to pip itself. If we did that, the downstream setup would likely be the even weaker "Suggests" dependency. My use case here is the "offline Python installation" one - having the build tools bundled with CPython and readily available is still useful for cases where you have your own code you want to build, but can't go to the internet to get a build toolchain. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Interesting-- so move to a Python specific binary distribution option -- rather than using rm or deb packages? Doesn't lead to a dependency heck? I.e no way to express non-python dependencies? And while we are moving forward, can we please deprecate dependency management and installation from setuptools? Is there a philosophy of intended separation of concerns articulated somewhere? -Chris
Le 7 août 2015 00:51, "Robert Collins" <robertc@robertcollins.net> a écrit :
If an expert user is unaware of having to explicitly install wheel, what about other users? Packaging is the most hated feature of Python. Please don't add extra pain for purity and make sure that ensurepip installs "pip" and not "slow pip until you install wheel in the venv". To develop on OpenStack, I have more than 20 virtual environment on my PC. I recreate them regulary because I like downgarding or upgrading a package, or edit the code of a package directly in the venv (usually to debug). Since pip7, the creation of venv is much faster. Please don't make pip slower. Victor
This is a really good point -- other than purity, what is the downside? Arguably, the only reason setuptools, pip, and wheel are not in the standard library are because the need a more rapid development/release cycle. Ensurepip is the way to get the best of both worlds -- why not make it complete? -Chris
So my opinion is basically that in a vacuum I would absolutely add wheel to ensurepip (and I did add wheel to get-pip.py and to virtualenv). However, this does not exist in a vacuum and there is still animosity about PEP 453 and downstream’s are still trying to figure out how they are going to handle it for real. During the 3.4 release there were downstream redisttibutors who completely removed ensurepip and were talking about possibly removing pip entirely from their archives. So my hesitation is basically that I consider is a short (or medium) term need until pip can implicitly install wheel and setuptools as part of the build process for a particular project and I worry that it will reopen some wounds and cause more strife. I do however think it would make ensurepip itself better, so I’m not dead set against it, mostly just worried about ramifications.
On 8 August 2015 at 02:12, Donald Stufft <donald@stufft.io> wrote:
[I'm wearing my professional Fedora Environments & Stack WG and RHEL Developer Experience hats in this post, moreso than my CPython core developer one] It seems to me that most modern open source developers (especially those using dynamic languages) perceive Linux distros more as impediments to be worked around, rather than as allies to collaborate with, and that's *our* UX issue to figure out downstream (hence design concepts like https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManag... and https://fedoraproject.org/wiki/Env_and_Stacks/Projects/PackageReviewProcessR... in the Fedora space) It's not CPython's problem to resolve, and it's only CPython's responsibility to work around to the extent that it makes things easier for *end users* developing in Python. If a distro is being unreasonably intransigent about developer experience concerns, then that's the distro's problem, and we can advise people to download and use a cross-platform distro like ActivePython, EPD/Canopy or Anaconda instead of the system Python.
So my hesitation is basically that I consider is a short (or medium) term need until pip can implicitly install wheel and setuptools as part of the build process for a particular project and I worry that it will reopen some wounds and cause more strife.
I don't believe it's a good idea to avoid strife for the sake of avoiding strife - many Linux distros are in the wrong here, and we need to get with the program in suitably meeting the needs of open source developers, not just folks running Linux on production servers. Fedora started that process with the launch of the Fedora.next initiatives a couple of years ago, but there's still a lot of work to be done in retooling our online and desktop experience to make it more developer friendly.
I do however think it would make ensurepip itself better, so I’m not dead set against it, mostly just worried about ramifications.
I'd advise against letting concerns about Linux distro politics hold you back from making ensurepip as good as you can make it - if nothing else, the developer experience folks at commercial Linux vendors are specifically paid to advocate for the interests of software developers when it comes to the Linux user experience (that's part of my own day job in the Fedora/RHEL/CentOS case - I moved over to the software management team in RHEL Developer Experience at the start of June). That means that while I will have some *requests* to make certain things easier downstream (like going through the PEP process to figure out an upstream supported way to omit the build-only dependencies when running ensurepip), I also wholeheartedly endorse the idea of having the default upstream behaviour focus on making the initial experience for folks downloading Windows or Mac OS X binaries from python.org as compelling as we can make it. python-dev needs to put the needs of Python first, and those of Linux second. This does mean that any Linux distro that can't figure out how to provide a better open source developer experience for Pythonistas than Windows or Mac OS X is at risk of falling by the wayside in the Python community, but if those of us that care specifically about the viability of desktop Linux as a platform for open source development stand by and let that happen, then we'll *deserve* the consequences. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8 August 2015 at 04:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sorry I'm late to this, but I would very much like to see wheel installed with ensurepip on at least Windows. (I don't see any reason why OSX would not be similar, but as I'm not an OSX user I can't say for certain). If Linux distros have issue with this, then maybe we need to do something different there (I like Nick's comments, and would rather we didn't make the Linux situation worse due to distro politics, but that's not my call) but that shouldn't affect Windows/OSX. There's a certain irony to me in the fact that we're reaching a point where the Windows experience is the benchmark Linux users need to aim for, but I'll avoid saying anything more on that one ;-) Paul
On Sun, Aug 16, 2015 at 02:17:09PM +0100, Paul Moore wrote:
Sorry I'm late to this, but I would very much like to see wheel installed with ensurepip on at least Windows.
I seem to be missing something critical to this entire discussion. As I understand it, ensurepip is *only* intended to bootstrap pip itself. So the idea is, you install Python, including ensurepip, use that to install the latest pip *including wheel*, and Bob's your uncle. At worst, you install pip, then install wheel. So what is the benefit of including wheel with ensurepip? -- Steve
On August 16, 2015 at 10:41:42 AM, Steven D'Aprano (steve@pearwood.info) wrote:
pip has an optional dependency on wheel, if you install that optional dependency than you’ll get the implicit wheel cache enabled by default which can drastically improve installation speeds by caching built artifacts (i.e. ``pip instal lxml`` multiple times only compiles it once). The goal is to get more people getting the benefits of that by default instead of requiring them to know they need to ``pip install wheel`` after the fact. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On August 16, 2015 at 11:26:08 AM, Steven D'Aprano (steve@pearwood.info) wrote:
That’s the proposal here, ensurepip only installs things it has bundled inside of it, so we’d add a .whl file for wheel and slightly tweak ensurepip so it also installs wheel. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (7)
-
Chris Barker - NOAA Federal -
Donald Stufft -
Nick Coghlan -
Paul Moore -
Robert Collins -
Steven D'Aprano -
Victor Stinner