PEP439 and backward compat / easy_install / distlib
Hi Richard, all, two first notes on PEP439. backward compat with present-day release files: the PEP should state it as a goal or at least discuss it in some depth. In that context, the choice of providing a bootstrap for pip rather than easy_install needs reasoning. One problem with pip, compared to easy_install, is that it doesn't support eggs which is a problem particularly on Windows machines where often no fitting C compiler is available. If the remedy here is to support wheels and recommend it's use, it is still a backward compatibility problem: many users will not be able to use the builtin-supported installer to install todays existing egg release files. setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package. I am highlighting these two backward-compat aspects because otherwise we might run into this problem: http://xkcd.com/927/ and i understood that most people involved in improving the packaging ecology want to avoid that :) best, holger
Did you know that the "wheel convert" command masks wheels from eggs? On Mar 24, 2013 5:48 AM, "holger krekel" <holger@merlinux.eu> wrote:
Hi Richard, all,
two first notes on PEP439.
backward compat with present-day release files: the PEP should state it as a goal or at least discuss it in some depth. In that context, the choice of providing a bootstrap for pip rather than easy_install needs reasoning. One problem with pip, compared to easy_install, is that it doesn't support eggs which is a problem particularly on Windows machines where often no fitting C compiler is available. If the remedy here is to support wheels and recommend it's use, it is still a backward compatibility problem: many users will not be able to use the builtin-supported installer to install todays existing egg release files.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
I am highlighting these two backward-compat aspects because otherwise we might run into this problem: http://xkcd.com/927/ and i understood that most people involved in improving the packaging ecology want to avoid that :)
best, holger _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Mar 24, 2013 at 09:49 -0400, Daniel Holth wrote:
Did you know that the "wheel convert" command masks wheels from eggs?
Wasn't aware, thanks for the pointer. That should certainly be part of the PEP439 discussion i ask for in the first paragraph. holger
On Mar 24, 2013 5:48 AM, "holger krekel" <holger@merlinux.eu> wrote:
Hi Richard, all,
two first notes on PEP439.
backward compat with present-day release files: the PEP should state it as a goal or at least discuss it in some depth. In that context, the choice of providing a bootstrap for pip rather than easy_install needs reasoning. One problem with pip, compared to easy_install, is that it doesn't support eggs which is a problem particularly on Windows machines where often no fitting C compiler is available. If the remedy here is to support wheels and recommend it's use, it is still a backward compatibility problem: many users will not be able to use the builtin-supported installer to install todays existing egg release files.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
I am highlighting these two backward-compat aspects because otherwise we might run into this problem: http://xkcd.com/927/ and i understood that most people involved in improving the packaging ecology want to avoid that :)
best, holger _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Mar 24, 2013 at 2:48 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Richard, all,
two first notes on PEP439.
PEP 439 is just one small piece of a much larger puzzle, and the entire puzzle won't be explained in this PEP. I realise this makes it hard to evaluate in isolation, but the beta freeze for 3.4 is still several months away and all the other pieces will be in place well before then. Please be patient with us as we get all the pieces documented and published over the coming weeks. However, I do have some comments on your specific questions: - while easy_install has provided good service for many years, it is not a viable choice as the officially supported installer. It's default behaviour is completely broken from a system administrator's point of view, as using it to install things has side effects on every Python application run on that system, and the lack of uninstall support is not acceptable either. By contrast, pip has tried from the beginning to accommodate the interests of system administrators *as well as* developers, making it much easier to justify its being blessed as the official installer. The approach taken to this over the coming months will be to identify the reasons that people still use easy_install for some tasks and add support for them to pip. - eggs are too closely associated with easy_install to easily rehabilitate their image with system administrators and platform packagers, and also lack some of the necessary metadata to interoperate correctly with platform packaging tools. The new wheel format builds on a combination of eggs and the sysconfig installation path concept to create a format that can be more readily mapped to FHS compliant platform specific packages. Wheel also introduces the enhanced "compatibility tag" format for filenames, which covers more details of the Python and platform dependencies of the built distribution. - We could potentially provide server side support for automatically generating wheels from eggs uploaded to PyPI, but that would be a question for catalog-sig (since it is purely about PyPI's feature set and behaviour, and independent of the packaging and distribution standards themselves) - metadata 2.0 specifically includes the "Setup-Requires-Dist" field so that projects that require additional dependencies when building from source will be correctly supported by pip and other metadata 2.0 compliant installers. This will be supported transparently for users, so long as they update to metadata 2.0 compliant versions of their build tools (setuptools/distribute, distutils2, d2to1, hashdist, etc). Python 3.4 will also continue to provide distutils, so that metadata 1.0 and 1.1 packages generated with older versions of distutils will continue to work correctly. - once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools" Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Mar 24, 2013 at 09:04 -0700, Nick Coghlan wrote:
On Sun, Mar 24, 2013 at 2:48 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Richard, all,
two first notes on PEP439.
PEP 439 is just one small piece of a much larger puzzle, and the entire puzzle won't be explained in this PEP. I realise this makes it hard to evaluate in isolation, but the beta freeze for 3.4 is still several months away and all the other pieces will be in place well before then. Please be patient with us as we get all the pieces documented and published over the coming weeks.
Looking forward to more details. It's indeed a bit hard to get a grasp on what's going on currently. Even though i've even just written the related PEP438 myself. In any case, each PEP should make enough sense on its own or refer to other PEPs or documents for details.
However, I do have some comments on your specific questions:
- while easy_install has provided good service for many years, it is not a viable choice as the officially supported installer. It's default behaviour is completely broken from a system administrator's point of view, as using it to install things has side effects on every Python application run on that system, and the lack of uninstall support is not acceptable either. By contrast, pip has tried from the beginning to accommodate the interests of system administrators *as well as* developers, making it much easier to justify its being blessed as the official installer. The approach taken to this over the coming months will be to identify the reasons that people still use easy_install for some tasks and add support for them to pip.
I understand your high level motivation and views here. Can you point to a more technical detailed comparison of easy_install and pip? FWIW I've heart from many people that they have to use easy_install because of egg support and the many packages in that format out there. This is why i asked about a "backward compatibility" strategy specifically.
- eggs are too closely associated with easy_install to easily rehabilitate their image with system administrators and platform packagers, and also lack some of the necessary metadata to interoperate correctly with platform packaging tools. The new wheel format builds on a combination of eggs and the sysconfig installation path concept to create a format that can be more readily mapped to FHS compliant platform specific packages. Wheel also introduces the enhanced "compatibility tag" format for filenames, which covers more details of the Python and platform dependencies of the built distribution.
I also have the impression that wheels are a very good development. My last mail didn't question the merits of wheels over eggs, though.
- We could potentially provide server side support for automatically generating wheels from eggs uploaded to PyPI, but that would be a question for catalog-sig (since it is purely about PyPI's feature set and behaviour, and independent of the packaging and distribution standards themselves)
Maybe. In any case, I see this the issue of automated egg->wheel conversion is on topic for PEP439. It should be part of a focus on an evolutionary strategy (rather than a pure replacement one) which helps to get as many people on board and benefiting from the PEP as possible.
- metadata 2.0 specifically includes the "Setup-Requires-Dist" field so that projects that require additional dependencies when building from source will be correctly supported by pip and other metadata 2.0 compliant installers. This will be supported transparently for users, so long as they update to metadata 2.0 compliant versions of their build tools (setuptools/distribute, distutils2, d2to1, hashdist, etc).
Sounds good. The present-day packages which will not be updated (soon or ever), may be a problem, though.
Python 3.4 will also continue to provide distutils, so that metadata 1.0 and 1.1 packages generated with older versions of distutils will continue to work correctly.
- once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools"
Well, one of the stated goals is to make it easier for package maintainers to explain things. If you have to explain pip-bootstrapping, then install setuptools, then the actual package, it's not much of an improvement anymore. Especially if you need to explain things depending on Python version. To avoid the latter the PEP439 should include offering installer-bootstraps from python.org for older Python versions, particularly Python2. This way we could see the benefits of PEP439 and related developments much much earlier than if we all need to wait until everyone uses Python3.4 :) FWIW I am open to do a hangout over all this some time. best, holger
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 March 2013 20:48, holger krekel <holger@merlinux.eu> wrote:
backward compat with present-day release files: the PEP should state it as a goal or at least discuss it in some depth. In that context, the choice of providing a bootstrap for pip rather than easy_install needs reasoning. One problem with pip, compared to easy_install, is that it doesn't support eggs which is a problem particularly on Windows machines where often no fitting C compiler is available. If the remedy here is to support wheels and recommend it's use, it is still a backward compatibility problem: many users will not be able to use the builtin-supported installer to install todays existing egg release files.
This is a valid concern. Obviously "pip install easy_install" is not a solution - especially since the general intention is to deprecate easy_install eventually (as explained in Nick's response). I did not discuss eggs with the pip developers while at PyCon which is quite unfortunate. I would appreciate any insights from those devs on the matter. It may be that wheel convert can solve this issue for some eggs. Unless it can be fully automated it's not going to solve it for all.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
This is not the bootstrap's problem (and hence not the PEP's) since the bootstrap exists *solely* to install the pip implementation. If that's not clear enough in the PEP then I can attempt to make it clearer. On 25 March 2013 03:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
- once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools"
I'm -0 on the idea of also including an easy_install bootstrap in the Python install, since I personally would prefer not to require users to have to deal with two install tools which behave slightly differently. On 25 March 2013 07:21, holger krekel <holger@merlinux.eu> wrote:
If you have to explain pip-bootstrapping, then install setuptools, then the actual package, it's not much of an improvement anymore.
The point of this PEP is to remove the first "explain pip bootstrapping" step from this equation. I had thrown around the idea of the pip bootstrap installing both pip implementation *and* setuptools. At the time my justification was that pip depended on it. The pip devs have indicated that they could remove the setuptools dependency when distlib and wheel support are in the Python standard library. ISTM however that there is still quite a good justification for installing setuptools during the bootstrapping, for the reasons you state. Richard
On 3/24/13, Richard Jones <richardjones@optushome.com.au> wrote:
This is a valid concern. Obviously "pip install easy_install" is not a solution - especially since the general intention is to deprecate easy_install eventually (as explained in Nick's response). I did not discuss eggs with the pip developers while at PyCon which is quite unfortunate. I would appreciate any insights from those devs on the matter.
Why is "pip install setuptools" not a solution? It's easier than getting setuptools installed is today.
It may be that wheel convert can solve this issue for some eggs. Unless it can be fully automated it's not going to solve it for all.
The simplest solution is likely for pip to gain support for installing from eggs, despite their known issues.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
This is not the bootstrap's problem (and hence not the PEP's) since the bootstrap exists *solely* to install the pip implementation. If that's not clear enough in the PEP then I can attempt to make it clearer.
Right, in every PEP we should probably make the builder vs installer distinction clear, and be explicit that the PEP only covers the installer side. "install from sdist" unfortunately blurs that boundary, and we may need an egregious hack like doing a substring search for "import setuptools" in setup.py when installing from an sdist with metadata 1.x in PKG-INFO. One of my key long term goals is to eventually allow system administrators to control whether or not a build system ends up on their production servers. At the moment that's not possible (and it likely won't be for 3.4 either), but that's where I would like us to get to somewhere along the line.
On 25 March 2013 03:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
- once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools"
I'm -0 on the idea of also including an easy_install bootstrap in the Python install, since I personally would prefer not to require users to have to deal with two install tools which behave slightly differently.
I only meant "pip install setuptools && easy_install other_project", not a separate bootstrap command.
On 25 March 2013 07:21, holger krekel <holger@merlinux.eu> wrote:
If you have to explain pip-bootstrapping, then install setuptools, then the actual package, it's not much of an improvement anymore.
The point of this PEP is to remove the first "explain pip bootstrapping" step from this equation.
I had thrown around the idea of the pip bootstrap installing both pip implementation *and* setuptools. At the time my justification was that pip depended on it. The pip devs have indicated that they could remove the setuptools dependency when distlib and wheel support are in the Python standard library.
ISTM however that there is still quite a good justification for installing setuptools during the bootstrapping, for the reasons you state.
I can make this part simple: I won't accept a PEP that proposes automatically installing setuptools even if you never install a package from source, and never install anything that needs pkg_resources :) We will make it easy for people to install setuptools if they need it. Projects built with newer versions of setuptools will have "Setup-Requires-Dist: setuptools" and "Requires-Dist: setuptools" configured appropriately, while pip will also correctly pick up a runtime dependency identifies in a requires.txt file. But the idea is to eventually deprecate setuptools/pkg_resources/easy_install as components that get deployed to production systems, and leave setuptools as a build system only. It's going to take us a while to get there (especially since we still need a path hook to replace pkg_resource.requires), but even when we do, they will always remain only a "pip install" away for projects that still need them. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 25 March 2013 14:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 3/24/13, Richard Jones <richardjones@optushome.com.au> wrote:
This is a valid concern. Obviously "pip install easy_install" is not a solution - especially since the general intention is to deprecate easy_install eventually (as explained in Nick's response). I did not discuss eggs with the pip developers while at PyCon which is quite unfortunate. I would appreciate any insights from those devs on the matter.
Why is "pip install setuptools" not a solution? It's easier than getting setuptools installed is today.
Because of the reason I stated later; it's a second hurdle that users have to get over before installing the actual thing they wish to install. All packages that depend on setuptools must include the instructions "but first install setuptools."
It may be that wheel convert can solve this issue for some eggs. Unless it can be fully automated it's not going to solve it for all.
The simplest solution is likely for pip to gain support for installing from eggs, despite their known issues.
This would indeed be simpler.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
This is not the bootstrap's problem (and hence not the PEP's) since the bootstrap exists *solely* to install the pip implementation. If that's not clear enough in the PEP then I can attempt to make it clearer.
Right, in every PEP we should probably make the builder vs installer distinction clear, and be explicit that the PEP only covers the installer side.
Yep.
"install from sdist" unfortunately blurs that boundary, and we may need an egregious hack like doing a substring search for "import setuptools" in setup.py when installing from an sdist with metadata 1.x in PKG-INFO.
Hm. I'm not sure I get this point. The intention in the PEP is to install the pip implementation from a wheel, not sdist. So maybe I'm missing the relevance of sdist being mentioned here.
On 25 March 2013 03:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
- once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools"
I'm -0 on the idea of also including an easy_install bootstrap in the Python install, since I personally would prefer not to require users to have to deal with two install tools which behave slightly differently.
I only meant "pip install setuptools && easy_install other_project", not a separate bootstrap command.
OK. So I'm -1 on that ;-)
On 25 March 2013 07:21, holger krekel <holger@merlinux.eu> wrote:
If you have to explain pip-bootstrapping, then install setuptools, then the actual package, it's not much of an improvement anymore.
The point of this PEP is to remove the first "explain pip bootstrapping" step from this equation.
I had thrown around the idea of the pip bootstrap installing both pip implementation *and* setuptools. At the time my justification was that pip depended on it. The pip devs have indicated that they could remove the setuptools dependency when distlib and wheel support are in the Python standard library.
ISTM however that there is still quite a good justification for installing setuptools during the bootstrapping, for the reasons you state.
I can make this part simple: I won't accept a PEP that proposes automatically installing setuptools even if you never install a package from source, and never install anything that needs pkg_resources :)
We will make it easy for people to install setuptools if they need it. Projects built with newer versions of setuptools will have "Setup-Requires-Dist: setuptools" and "Requires-Dist: setuptools" configured appropriately, while pip will also correctly pick up a runtime dependency identifies in a requires.txt file.
But the idea is to eventually deprecate setuptools/pkg_resources/easy_install as components that get deployed to production systems, and leave setuptools as a build system only. It's going to take us a while to get there (especially since we still need a path hook to replace pkg_resource.requires), but even when we do, they will always remain only a "pip install" away for projects that still need them.
I think we have too much legacy to support here. Sure it'd be nice if everyone just switched over to PEP 426 style overnight, but it ain't gonna happen. The intent of the automatic setuptools installation is to mirror the *current* situation which people rely on: when pip's installed you also have setuptools (/distribute) installed. Packages may then depend on setuptools in their setup.py with fair confidence that it'll be there. Having this PEP support pip without setuptools will make packaging more complex which is antithetical my goal with the PEP. I can't support a PEP that will make things more complex :-) Richard
Wheels are very similar to eggs. The initial wheel implementation converts *every* newly built wheel from .egg-info to .dist-info since setuptools doesn't know how to generate the new metadata. I think you will find the conversion to be reliable, though it can't download the eggs for you right now. The hypothetical no-setuptools pip will probably add an implicit setuptools dependency to every sdist, installing it automatically as an additional dependency, until we invent a way for sdists to say that they don't need setuptools. "pip install" always forces its setup.py subprocesses to use setuptools. On Sun, Mar 24, 2013 at 11:49 PM, Richard Jones <richardjones@optushome.com.au> wrote:
On 25 March 2013 14:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 3/24/13, Richard Jones <richardjones@optushome.com.au> wrote:
This is a valid concern. Obviously "pip install easy_install" is not a solution - especially since the general intention is to deprecate easy_install eventually (as explained in Nick's response). I did not discuss eggs with the pip developers while at PyCon which is quite unfortunate. I would appreciate any insights from those devs on the matter.
Why is "pip install setuptools" not a solution? It's easier than getting setuptools installed is today.
Because of the reason I stated later; it's a second hurdle that users have to get over before installing the actual thing they wish to install. All packages that depend on setuptools must include the instructions "but first install setuptools."
It may be that wheel convert can solve this issue for some eggs. Unless it can be fully automated it's not going to solve it for all.
The simplest solution is likely for pip to gain support for installing from eggs, despite their known issues.
This would indeed be simpler.
setuptools and distlib: Even if Python3.4+ had a mature distlib providing minimal setuptools functionality, how would it work for the typical "python setup.py install" which is invoked by pip? Often those setup.py scripts depend on a setuptools package.
This is not the bootstrap's problem (and hence not the PEP's) since the bootstrap exists *solely* to install the pip implementation. If that's not clear enough in the PEP then I can attempt to make it clearer.
Right, in every PEP we should probably make the builder vs installer distinction clear, and be explicit that the PEP only covers the installer side.
Yep.
"install from sdist" unfortunately blurs that boundary, and we may need an egregious hack like doing a substring search for "import setuptools" in setup.py when installing from an sdist with metadata 1.x in PKG-INFO.
Hm. I'm not sure I get this point. The intention in the PEP is to install the pip implementation from a wheel, not sdist. So maybe I'm missing the relevance of sdist being mentioned here.
On 25 March 2013 03:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
- once we can bootstrap pip, then bootstrapping easy_install if it still needed for some edge cases will be as easy as installing anything else that is either pure Python or publishes an appropriate wheel for the platform: "pip install setuptools"
I'm -0 on the idea of also including an easy_install bootstrap in the Python install, since I personally would prefer not to require users to have to deal with two install tools which behave slightly differently.
I only meant "pip install setuptools && easy_install other_project", not a separate bootstrap command.
OK. So I'm -1 on that ;-)
On 25 March 2013 07:21, holger krekel <holger@merlinux.eu> wrote:
If you have to explain pip-bootstrapping, then install setuptools, then the actual package, it's not much of an improvement anymore.
The point of this PEP is to remove the first "explain pip bootstrapping" step from this equation.
I had thrown around the idea of the pip bootstrap installing both pip implementation *and* setuptools. At the time my justification was that pip depended on it. The pip devs have indicated that they could remove the setuptools dependency when distlib and wheel support are in the Python standard library.
ISTM however that there is still quite a good justification for installing setuptools during the bootstrapping, for the reasons you state.
I can make this part simple: I won't accept a PEP that proposes automatically installing setuptools even if you never install a package from source, and never install anything that needs pkg_resources :)
We will make it easy for people to install setuptools if they need it. Projects built with newer versions of setuptools will have "Setup-Requires-Dist: setuptools" and "Requires-Dist: setuptools" configured appropriately, while pip will also correctly pick up a runtime dependency identifies in a requires.txt file.
But the idea is to eventually deprecate setuptools/pkg_resources/easy_install as components that get deployed to production systems, and leave setuptools as a build system only. It's going to take us a while to get there (especially since we still need a path hook to replace pkg_resource.requires), but even when we do, they will always remain only a "pip install" away for projects that still need them.
I think we have too much legacy to support here. Sure it'd be nice if everyone just switched over to PEP 426 style overnight, but it ain't gonna happen. The intent of the automatic setuptools installation is to mirror the *current* situation which people rely on: when pip's installed you also have setuptools (/distribute) installed. Packages may then depend on setuptools in their setup.py with fair confidence that it'll be there. Having this PEP support pip without setuptools will make packaging more complex which is antithetical my goal with the PEP. I can't support a PEP that will make things more complex :-)
Richard _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Mar 24, 2013 at 8:49 PM, Richard Jones <richardjones@optushome.com.au> wrote:
I think we have too much legacy to support here. Sure it'd be nice if everyone just switched over to PEP 426 style overnight, but it ain't gonna happen. The intent of the automatic setuptools installation is to mirror the *current* situation which people rely on: when pip's installed you also have setuptools (/distribute) installed. Packages may then depend on setuptools in their setup.py with fair confidence that it'll be there. Having this PEP support pip without setuptools will make packaging more complex which is antithetical my goal with the PEP. I can't support a PEP that will make things more complex :-)
I am not adding setuptools to the standard library, which is effectively what we're doing for anything installed automatically via the pip bootstrap script. Things have moved on since Guido approved setuptools for inclusion in 2.5, and I'm not adding an entire legacy module only to immediately deprecate it in favour of the updated toolchains. That means anyone assuming setuptools will be present as part of all Python installations is just plain wrong, both now and in the future. Therefore, any installation of setuptools on a system *must* be requested explicitly, either directly (via "pip install setuptools") or indirectly (by installing something that explicitly depends on it through a setuptools requirements file, Setup-Requires-Dist or Requires-Dist). The *only* case this approach doesn't immediately cover is a project that: 1. Doesn't publish a pre-built wheel for the current platform (or egg, assuming pip gains support for those, perhaps by implicitly converting them to wheels) 2. Doesn't publish 2.0 metadata with "Setup-Requires-Dist: setuptools" 3. imports setuptools in its setup.py file This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it). Yes, it's a hack, but I am *not* going to approve a PEP that further entrenches something even its creator would like to see waved off into the sunset, giving thanks for its good service :) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
The *only* case this approach doesn't immediately cover is a project that: 1. Doesn't publish a pre-built wheel for the current platform (or egg, assuming pip gains support for those, perhaps by implicitly converting them to wheels) 2. Doesn't publish 2.0 metadata with "Setup-Requires-Dist: setuptools" 3. imports setuptools in its setup.py file
so that's most everything on pypi right now in the short and medium term.
This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it).
so you're asking pip to get this working soon, right? like before python3.4, so this PEP can go in? Marcus
Yes, it's a hack, but I am *not* going to approve a PEP that further entrenches something even its creator would like to see waved off into the sunset, giving thanks for its good service :)
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Mon, Mar 25, 2013 at 9:42 AM, Marcus Smith <qwcode@gmail.com> wrote:
The *only* case this approach doesn't immediately cover is a project that: 1. Doesn't publish a pre-built wheel for the current platform (or egg, assuming pip gains support for those, perhaps by implicitly converting them to wheels) 2. Doesn't publish 2.0 metadata with "Setup-Requires-Dist: setuptools" 3. imports setuptools in its setup.py file
so that's most everything on pypi right now in the short and medium term.
This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it).
so you're asking pip to get this working soon, right? like before python3.4, so this PEP can go in?
Our messages crossed in flight - I realised I'm fine with assuming a setuptools dependency for eggs and all sdist's without 2.0+ metadata, it's only the idea of installing setuptools as a dependency of bootstrapping pip itself that I'm not happy with. That means pip only needs to support two configurations: 1. No setuptools, can only install from wheel files and sdists with 2.0+ metadata 2. Has setuptools, can also install from sdists with legacy metadata and eggs By default, installing setuptools when necessary should be automatic, but advanced users should be able to ask that it instead be treated as an error if no wheel is available to satisfy an installation request or dependency (so they don't inadvertently install setuptools on their production systems if they don't want to). To make this work, we'll need to get wheels published on PyPI for setuptools before 3.4, as well as ensuring pip doesn't require setuptools to install from wheel files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Mar 25, 2013 at 12:50 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Mar 25, 2013 at 9:42 AM, Marcus Smith <qwcode@gmail.com> wrote:
The *only* case this approach doesn't immediately cover is a project that: 1. Doesn't publish a pre-built wheel for the current platform (or egg, assuming pip gains support for those, perhaps by implicitly converting them to wheels) 2. Doesn't publish 2.0 metadata with "Setup-Requires-Dist: setuptools" 3. imports setuptools in its setup.py file
so that's most everything on pypi right now in the short and medium term.
This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it).
so you're asking pip to get this working soon, right? like before python3.4, so this PEP can go in?
Our messages crossed in flight - I realised I'm fine with assuming a setuptools dependency for eggs and all sdist's without 2.0+ metadata, it's only the idea of installing setuptools as a dependency of bootstrapping pip itself that I'm not happy with. That means pip only needs to support two configurations:
1. No setuptools, can only install from wheel files and sdists with 2.0+ metadata 2. Has setuptools, can also install from sdists with legacy metadata and eggs
By default, installing setuptools when necessary should be automatic, but advanced users should be able to ask that it instead be treated as an error if no wheel is available to satisfy an installation request or dependency (so they don't inadvertently install setuptools on their production systems if they don't want to).
To make this work, we'll need to get wheels published on PyPI for setuptools before 3.4, as well as ensuring pip doesn't require setuptools to install from wheel files.
Cheers, Nick.
My vision for the setuptools deprecation process is that distutils rides into the sunset with it. In this future eventually bugs in setuptools will be solved by porting setup.py to one of (X, Y, Z) which haven't necessarily been invented yet. Like Marcus said, pip itself only uses pkg_resources which we can bundle and eventually replace with a shim backed by distlib (Vinay has tried this already). Only the individual distributions' "setup.py" subprocesses (are forced to) import setuptools. The --only-wheel option should be easy. It is going to be problematic to try to find and download legacy eggs correctly and automatically since they don't for example tell you whether they are wide Unicode. Those problems and more would apply to automatically downloading and converting bdist_wininst .exe installers which aren't even named consistently.
On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth@gmail.com> wrote:
It is going to be problematic to try to find and download legacy eggs correctly and automatically since they don't for example tell you whether they are wide Unicode. Those problems and more would apply to automatically downloading and converting bdist_wininst .exe installers which aren't even named consistently.
However, given that this problem exists now, and that this is strictly a transitional feature, all you really need to do is make reasonably similar choices to easy_install. It might be worth looking at the actual stats, but my impression is that most eggs on PyPI are either for Windows or aren't platform-specific at all. Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools. If for political reasons you want to not provide setuptools even in that case (since there are packages that don't use it), you could just throw in an import hook that notices when you're trying to import something from setuptools and handles the install. But ISTM the simplest case is just to look for (the absence of) 2.0 metadata. (Of course, that kind of assumes that the 2.0 standard includes build metadata and procedures, and that we know what a new-style sdist looks like in this scheme.)
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway. Ronald
On Mar 26, 2013 5:28 AM, "Ronald Oussoren" <ronaldoussoren@mac.com> wrote:
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using
package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway.
Ronald
We really really really want to get rid of ez_setup. It is considered by many to be the example of something that should not happen as a side effect of running a build script. When packages no longer have to install themselves, they can just mention setup-requires and the installer grabs the necessary setuptools.
On 26 Mar, 2013, at 12:06, Daniel Holth <dholth@gmail.com> wrote:
On Mar 26, 2013 5:28 AM, "Ronald Oussoren" <ronaldoussoren@mac.com> wrote:
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway.
Ronald
We really really really want to get rid of ez_setup. It is considered by many to be the example of something that should not happen as a side effect of running a build script.
That can't be helped with the current tool versions, distutils in current release of python doesn't support setup-requires and hence the only way to use setuptools is by using ez_setup.py. Ez_setup.py will still have to be present for python 2.7 users that want to use "python setup.py ...", and hence can be assumed to be present for now. I'm all for adding support for metadata 2.0 to the stdlib for Python 3.4, that way ez_setup.py can be phased out in the long run.
When packages no longer have to install themselves, they can just mention setup-requires and the installer grabs the necessary setuptools.
That can only be done for sdists with 2.0 metadata, sdists for older versions don't have a setup-requires in their metadata. This is not just for installing, if you want to use setuptools in your setup.py you'll have to make sure it is installed in your setup.py, and with the current version of the packaging tools this means you have to use something like ez_setup.py or tell users to install setuptools themselves. And with some luck a large subset of packages will ship wheels in the future, that way the installer doesn't even have to look at sdists. Ronald
On Mar 26, 2013 7:26 AM, "Ronald Oussoren" <ronaldoussoren@mac.com> wrote:
On 26 Mar, 2013, at 12:06, Daniel Holth <dholth@gmail.com> wrote:
On Mar 26, 2013 5:28 AM, "Ronald Oussoren" <ronaldoussoren@mac.com>
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can
be
done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway.
Ronald
We really really really want to get rid of ez_setup. It is considered by many to be the example of something that should not happen as a side effect of running a build script.
That can't be helped with the current tool versions, distutils in current release of python doesn't support setup-requires and hence the only way to use setuptools is by using ez_setup.py. Ez_setup.py will still have to be
wrote: present for python 2.7 users that want to use "python setup.py ...", and hence can be assumed to be present for now. I'm all for adding support for metadata 2.0 to the stdlib for Python 3.4, that way ez_setup.py can be phased out in the long run.
When packages no longer have to install themselves, they can just
mention setup-requires and the installer grabs the necessary setuptools.
That can only be done for sdists with 2.0 metadata, sdists for older versions don't have a setup-requires in their metadata. This is not just for installing, if you want to use setuptools in your setup.py you'll have to make sure it is installed in your setup.py, and with the current version of the packaging tools this means you have to use something like ez_setup.py or tell users to install setuptools themselves.
Yes, which is why we propose to assume Setup-Requires-Dist: setuptools if Metadata-Version < 2.0. Then a no-op ez_setup.py can be added to sys.modules before setup.py runs and the installer will have a lot more control over that side effect. These improved installers will target both 2.7 and 3.4. I do understand that some people feel it is harder to say "manually download the installer and then install what you want" rather than "manually download and install the package you want".
And with some luck a large subset of packages will ship wheels in the future, that way the installer doesn't even have to look at sdists.
On 26 Mar, 2013, at 13:27, Daniel Holth <dholth@gmail.com> wrote:
stall themselves, they can just mention setup-requires and the installer grabs the necessary setuptools.
That can only be done for sdists with 2.0 metadata, sdists for older versions don't have a setup-requires in their metadata. This is not just for installing, if you want to use setuptools in your setup.py you'll have to make sure it is installed in your setup.py, and with the current version of the packaging tools this means you have to use something like ez_setup.py or tell users to install setuptools themselves.
Yes, which is why we propose to assume Setup-Requires-Dist: setuptools if Metadata-Version < 2.0. Then a no-op ez_setup.py can be added to sys.modules before setup.py runs and the installer will have a lot more control over that side effect.
Just because I'm curious, is that control needed to make sure that a new enough version of setuptools gets used (e.g. one that supports modern features, instead of the 2 year old version that is mentioned in ez_setup.py for $SOME_OLD_PACKAGE)? Just assuming that every sdist with old metadata requires setuptools would work, although it will be strange to see that some packages @work that use plain disutils suddenly seem to require setuptool :-)
These improved installers will target both 2.7 and 3.4. I do understand that some people feel it is harder to say "manually download the installer and then install what you want" rather than "manually download and install the package you want".
No me. I'm glad to see that the hard work by everyone working in the packaging space is coming to fruition. Infrastructure work is almost never glamorous, and work on Python's packaging system appears to be more stressful than average. Ronald
Just assuming that every sdist with old metadata requires setuptools would work, although it will be strange to see that some packages @work that use plain disutils suddenly seem to require setuptool :-)
pip does this already, importing setuptools before running any setup.py
These improved installers will target both 2.7 and 3.4. I do understand that some people feel it is harder to say "manually download the installer and then install what you want" rather than "manually download and install the package you want".
No me. I'm glad to see that the hard work by everyone working in the packaging space is coming to fruition. Infrastructure work is almost never glamorous, and work on Python's packaging system appears to be more stressful than average.
You said it.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/26/2013 05:28 AM, Ronald Oussoren wrote:
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup.
No, they are not. That usage was for bootstrapping in an era when setuptools was not widely presetn. Most packages have *removed* those files today. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFRvPAACgkQ+gerLs4ltQ6XhgCgknMlM9drnL5KJKSvoEcuoKqw 60gAn1QyyUersaUdKXbJrpnJuu3AXkzz =i63/ -----END PGP SIGNATURE-----
On 26 Mar, 2013, at 16:21, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/26/2013 05:28 AM, Ronald Oussoren wrote:
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup.
No, they are not. That usage was for bootstrapping in an era when setuptools was not widely presetn. Most packages have *removed* those files today.
I didn't know that, all my project still include the bootstrap code to make it easier to install them in a fresh build of python. The distribute docs still mention that you should use distribute_setup.py (their version of ez_setup.py) in your project. Ronald
On Tue, Mar 26, 2013 at 11:21 AM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/26/2013 05:28 AM, Ronald Oussoren wrote:
On 25 Mar, 2013, at 19:16, PJ Eby <pje@telecommunity.com> wrote:
Also, as far as detecting the need for setuptools, I think that can be done just by noticing whether the PKG-INFO included in an sdist is metadata 2.0 or not. If it is, then setuptools should be explicitly declared as a build-time dependency, otherwise it's not needed. If it's an older metadata version, then you probably need setuptools.
Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup.
No, they are not. That usage was for bootstrapping in an era when setuptools was not widely presetn. Most packages have *removed* those files today.
I still use distribute_setup.py very regularly. I'm dealing with scientific users, mostly on Macs or Windows who barely even know what version of Python they have installed (or even what distribution of Python--python.org/macports/homebrew/etc.) much less that they need some variant of setuptools to install a large percentage of packages out there. Sometimes they do have setuptools installed but it's an outdated version, or they didn't install it properly, or something to that effect. I don't think "downloading the installer" should be a side effect of running the installation either, but until this mess is cleaned up it's a necessary evil. Yes, making things easier for users who don't know what they're doing is a legitimate use case. Erik
On Tue, Mar 26, 2013 at 11:54 AM, Erik Bray <erik.m.bray@gmail.com> wrote:
I don't think "downloading the installer" should be a side effect of running the installation either, but until this mess is cleaned up it's a necessary evil. Yes, making things easier for users who don't know what they're doing is a legitimate use case.
I should clarify--when I write "until this mess is cleaned up" what I really mean is, "as soon as most packages have wheels built for them for a wide range of platforms". Then I don't really see it as an issue :) Erik
On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth@gmail.com> wrote:
My vision for the setuptools deprecation process is that distutils rides into the sunset with it. In this future eventually bugs in setuptools will be solved by porting setup.py to one of (X, Y, Z) which haven't necessarily been invented yet.
That would be nice (really!) but what are you proposing replace it for building packages with heavy reliance on C extensions? Because for that one use case (and perhaps that alone) it works "pretty okay" for most cases. I don't want to start seeing an infinite number of ways to configure and build extension modules. The great thing about using distutils (or some variant) for this is that if I had the source for a package I could just `./setup.py build` and it would "just work" for all but the most complex cases (SciPy for example). I don't want to have a situation where some projects are using bento and others are using scons and some are using waf and others are using autoconf, etc, etc. It's fine if a few projects have their own special needs for build toolchains and I've been saying all along that building should be separate from installing, and it should be easier to drop in one's own build system. Another thing that setuptools provides that currently works "pretty well" with extension modules is `./setup.py develop`. It calls `setup.py build_ext --inplace` to make extension modules importable. Any build system for extension modules needs to be able to do something similar to support in-place install functionality like `setup.py develop`, `pip install -e`, etc. Erik
On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray <erik.m.bray@gmail.com> wrote:
On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth@gmail.com> wrote:
My vision for the setuptools deprecation process is that distutils rides into the sunset with it. In this future eventually bugs in setuptools will be solved by porting setup.py to one of (X, Y, Z) which haven't necessarily been invented yet.
That would be nice (really!) but what are you proposing replace it for building packages with heavy reliance on C extensions? Because for that one use case (and perhaps that alone) it works "pretty okay" for most cases. I don't want to start seeing an infinite number of ways to configure and build extension modules. The great thing about using distutils (or some variant) for this is that if I had the source for a package I could just `./setup.py build` and it would "just work" for all but the most complex cases (SciPy for example).
I don't want to have a situation where some projects are using bento and others are using scons and some are using waf and others are using autoconf, etc, etc. It's fine if a few projects have their own special needs for build toolchains and I've been saying all along that building should be separate from installing, and it should be easier to drop in one's own build system.
Another thing that setuptools provides that currently works "pretty well" with extension modules is `./setup.py develop`. It calls `setup.py build_ext --inplace` to make extension modules importable. Any build system for extension modules needs to be able to do something similar to support in-place install functionality like `setup.py develop`, `pip install -e`, etc.
It's true that de-standardization of the build process has its own problems. As a consolation you don't have to do it as often because we have binary packages. It's hard to say how much fragmentation will happen, but distutils Extension() is an awful way to compile things! It has very little to recommend it apart from that it's there. For example, no parallel builds and no partial re-compiles based on what's changed. The trouble is that we know the packages on the cheeseshop mostly work but it's harder to count the stuff that avoids the cheeseshop because setuptools and setup.py wasn't a good solution. The example I've had to deal with recently is pycairo. They already use waf to compile their Python extension and don't use setup.py at all, so I argue that de-standardization has already happened. Whatever very easy (for pycairo) thing we can do make them "pip install"-able again is a plus. Even my own simple bcrypt wrapper "cryptacular" would compile an assembler file if I knew how to make that work with distutils. It doesn't and it's a little slower than it would have been if I had a decent build tool. I suspect at least 80% of packages will use some simple thing that comes with Python, two third party build tools will dominate, and we will discover interesting things that just weren't possible before. At least if someone wants to improve packaging we can make it easy for them to try without having to ask distutils-sig.
On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth <dholth@gmail.com> wrote:
I suspect at least 80% of packages will use some simple thing that comes with Python, two third party build tools will dominate, and we will discover interesting things that just weren't possible before. At least if someone wants to improve packaging we can make it easy for them to try without having to ask distutils-sig.
...and for that matter, without them having to monkeypatch distutils, scrape HTML from the PyPI UI, and sandbox-execute other people's setup.py files. ;-)
On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth <dholth@gmail.com> wrote:
On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray <erik.m.bray@gmail.com> wrote:
On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth@gmail.com> wrote:
My vision for the setuptools deprecation process is that distutils rides into the sunset with it. In this future eventually bugs in setuptools will be solved by porting setup.py to one of (X, Y, Z) which haven't necessarily been invented yet.
That would be nice (really!) but what are you proposing replace it for building packages with heavy reliance on C extensions? Because for that one use case (and perhaps that alone) it works "pretty okay" for most cases. I don't want to start seeing an infinite number of ways to configure and build extension modules. The great thing about using distutils (or some variant) for this is that if I had the source for a package I could just `./setup.py build` and it would "just work" for all but the most complex cases (SciPy for example).
I don't want to have a situation where some projects are using bento and others are using scons and some are using waf and others are using autoconf, etc, etc. It's fine if a few projects have their own special needs for build toolchains and I've been saying all along that building should be separate from installing, and it should be easier to drop in one's own build system.
Another thing that setuptools provides that currently works "pretty well" with extension modules is `./setup.py develop`. It calls `setup.py build_ext --inplace` to make extension modules importable. Any build system for extension modules needs to be able to do something similar to support in-place install functionality like `setup.py develop`, `pip install -e`, etc.
It's true that de-standardization of the build process has its own problems. As a consolation you don't have to do it as often because we have binary packages. It's hard to say how much fragmentation will happen, but distutils Extension() is an awful way to compile things! It has very little to recommend it apart from that it's there. For example, no parallel builds and no partial re-compiles based on what's changed. The trouble is that we know the packages on the cheeseshop mostly work but it's harder to count the stuff that avoids the cheeseshop because setuptools and setup.py wasn't a good solution.
The example I've had to deal with recently is pycairo. They already use waf to compile their Python extension and don't use setup.py at all, so I argue that de-standardization has already happened. Whatever very easy (for pycairo) thing we can do make them "pip install"-able again is a plus.
Even my own simple bcrypt wrapper "cryptacular" would compile an assembler file if I knew how to make that work with distutils. It doesn't and it's a little slower than it would have been if I had a decent build tool.
I suspect at least 80% of packages will use some simple thing that comes with Python, two third party build tools will dominate, and we will discover interesting things that just weren't possible before. At least if someone wants to improve packaging we can make it easy for them to try without having to ask distutils-sig.
I pretty much agree with you on all of this, but I don't think the question should be ignored either--avoiding this question is one of the things that got previous packaging reform efforts into trouble. Though the agreement to treat "build" and "installation" as two different stories mitigates the issue this time around. In any case it's sort of off topic for this thread so I'll bring it up again elsewhen. One thing I see as a possible short-term solution is to still rely on some version of distutils as a build tool *only*. But it would still be nice to have some easy way to standardize "in-place" installation regardless of how extension modules get built. Erik P.S. pycairo does have a setup.py which worked for me, but the installation instructions say it's "unsupported", though I don't see the waf script doing anything enormously different from it.
On Wed, Mar 27, 2013 at 6:42 AM, Erik Bray <erik.m.bray@gmail.com> wrote:
I pretty much agree with you on all of this, but I don't think the question should be ignored either--avoiding this question is one of the things that got previous packaging reform efforts into trouble. Though the agreement to treat "build" and "installation" as two different stories mitigates the issue this time around. In any case it's sort of off topic for this thread so I'll bring it up again elsewhen. One thing I see as a possible short-term solution is to still rely on some version of distutils as a build tool *only*. But it would still be nice to have some easy way to standardize "in-place" installation regardless of how extension modules get built.
That's exactly the interim solution I have in mind: for the moment, the "archive system" will be "python setup.py sdist" in an appropriate location and the "build system" will be "python setup.py bdist_wheel". Both will be modelled on pip's current behaviour when installing from sdists - the difference will be in the explicit invocation of the separate steps, rather than handling the whole chain with "setup.py install". Longer term I want to make setup.py optional even for source installs, but that requires further enhancements to the metadata. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Mar 25, 2013 at 9:23 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it). Yes, it's a hack, but I am *not* going to approve a PEP that further entrenches something even its creator would like to see waved off into the sunset, giving thanks for its good service :)
I would also be fine with a simpler version of this approach, which works the way pip does now: if pip encounters a metadata 1.0 or 1.1 sdist, then it *assumes* "Setup-Requires-Dist: setuptools". That way, if you give it an index with no setuptools and fully populated with pre-built wheels, you can avoid deploying setuptools to the target environment. The bootstrap script itself should not install setuptools though - it's up to pip to do that before running a setup.py file without the explicit Setup-Requires-Dist support in metadata 2.0+ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I created a pip issue for this https://github.com/pypa/pip/issues/863 On Mon, Mar 25, 2013 at 9:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Mar 25, 2013 at 9:23 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This can be handled in pip, by using the AST module to scan for setuptools imports in setup.py (or else by checking for a setuptools related ImportError after trying to run it). Yes, it's a hack, but I am *not* going to approve a PEP that further entrenches something even its creator would like to see waved off into the sunset, giving thanks for its good service :)
I would also be fine with a simpler version of this approach, which works the way pip does now: if pip encounters a metadata 1.0 or 1.1 sdist, then it *assumes* "Setup-Requires-Dist: setuptools".
That way, if you give it an index with no setuptools and fully populated with pre-built wheels, you can avoid deploying setuptools to the target environment.
The bootstrap script itself should not install setuptools though - it's up to pip to do that before running a setup.py file without the explicit Setup-Requires-Dist support in metadata 2.0+
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
25 March 2013 03:49, Richard Jones <richardjones@optushome.com.au> wrote:
On 25 March 2013 14:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 3/24/13, Richard Jones <richardjones@optushome.com.au> wrote:
This is a valid concern. Obviously "pip install easy_install" is not a solution - especially since the general intention is to deprecate easy_install eventually (as explained in Nick's response). I did not discuss eggs with the pip developers while at PyCon which is quite unfortunate. I would appreciate any insights from those devs on the matter.
Why is "pip install setuptools" not a solution? It's easier than getting setuptools installed is today.
Because of the reason I stated later; it's a second hurdle that users have to get over before installing the actual thing they wish to install. All packages that depend on setuptools must include the instructions "but first install setuptools."
People have suggested pip offer egg (or wininst) installation for a long time now. But nobody has ever come up with actual code to do this. I don't think anyone should assume that such support will "just happen". Paul.
I had thrown around the idea of the pip bootstrap installing both pip implementation *and* setuptools. At the time my justification was that pip depended on it. The pip devs have indicated that they could remove the setuptools dependency when distlib and wheel support are in the Python standard library.
pip's wheel install support is native to pip right now, but it does use pkg_resources. (pip's wheel build support is done via the external wheel package as a setuptools extension) if pip "vendorized" pkg_resources, then it could install a Setuptools wheel. (pip will ultimately refactor its pkg_resources use over to distlib, but that seems less likely in the short term) I have a hard time imagining implementing the "MEBs" idea (i.e. removing the "setuptools dependency") by python 3.4, if that's what people are considering? (possibly only in the simplest way, where pip just automatically installs Setuptools from wheel if it's not installed when it encounters an sdist, because it knows it's the only build option right now) but that really reinforces that we need to get these plans posted up to the packaging developer hub so can really grok all this and talk phases and timelines. https://python-packaging-developer-hub.readthedocs.org/en/latest/ Marcus
participants (10)
-
Daniel Holth -
Erik Bray -
holger krekel -
Marcus Smith -
Nick Coghlan -
Paul Moore -
PJ Eby -
Richard Jones -
Ronald Oussoren -
Tres Seaver