flip the pip dependencies (was Current status of PEP 439)
I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead.
not sure about the flip, but let me break some things down a bit for those who don't know: what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that) what pip has "bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic) what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources) Marcus
On Jul 13, 2013, at 2:30 PM, Marcus Smith <qwcode@gmail.com> wrote:
I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead.
not sure about the flip, but let me break some things down a bit for those who don't know:
what pip has internally already (i.e. literally in it's package namespace): - pypi crawling/downloading - wheel installing (does not require the pypi wheel project; only building wheels requires that)
what pip has "bundled' already: - distlib (in 'pip.vendor'; currently only used for some --pre version logic)
what pip still needs to be self-sufficient to do wheel installs: - something bundled or internal that does what pkg_resources does
theoretical options: 1) bundle setuptools/pkg_resources 2) use the bundled distlib to replace our use of pkg_resources 3) internalize pkg_resources as pip.pkg_resources (i.e. fork off pkg_resources)
Marcus
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip.
I'm a fan of bundling too (if it works), but the "dynamic install of setuptools" idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves. The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing. Marcus
On Jul 13, 2013, at 3:35 PM, Marcus Smith <qwcode@gmail.com> wrote:
As you're aware I think it makes the most sense to just bundle setuptools wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip.
I'm a fan of bundling too (if it works), but the "dynamic install of setuptools" idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves.
The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing.
Marcus
1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have weird bugs/side effects due to people using different combinations of pip/setuptools with pip than we've tested. 2) This much is true, the question then becomes how important is that? If there's a major regression in setuptools that needs fixed I'd think we'd release an updated pip. If there's new functionality I would guess we'd need to expose that in pip anyways. 3) I think this isn't as big of a deal as it sounds. Especially given we can write tooling to make it simpler :) 4) Even if MEBs were here *right now* we'd still have nearly 150k source dists that required setuptools. So either in the MEB system we'd be grabbing setuptools *a lot* or we could just bundle it to provide a better UX for people using the large corpus of existing software. I think it will be a long time once the MEBs exist before they gain enough traction that even the bulk of installs are using that system. MEBs depend on sdist 2.0 which hasn't even been started yet ;) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 13 July 2013 20:35, Marcus Smith <qwcode@gmail.com> wrote:
The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD)
One possible issue is with the "install from sdist" code, which runs Python in a subprocess with the "import setuptools, etc etc" incantation to force always using setuptools. That may break (or at least need changing) for a bundled setuptools, which won't be visible from the top level by default. Worth checking, anyway. Of course, this code path becomes less important as we move towards installing from wheels, but it's going to be a while before that's the norm. Paul PS Apologies if I already said this. I'm losing track of what I've replied to and what I've just thought about on this thread :-(
yea, all those comebacks make sense to me. we should try the bundle and see if it works. we already do some fancy footwork when working with setup.py https://github.com/pypa/pip/blob/develop/pip/req.py#L602 https://github.com/pypa/pip/blob/develop/pip/req.py#L687 https://github.com/pypa/pip/blob/develop/pip/req.py#L269 https://github.com/pypa/pip/blob/develop/pip/wheel.py#L291 I guess we'd be doing some additional override work in sys.modules. Marcus On Sat, Jul 13, 2013 at 12:50 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 13, 2013, at 3:35 PM, Marcus Smith <qwcode@gmail.com> wrote:
As you're aware I think it makes the most sense to just bundle setuptools
wholesale. This makes it impossible to "break" pip by something going wrong in setuptools causing it to be uninstalled and means that for users who are only doing installs, they don't need setuptools installed just pip.
I'm a fan of bundling too (if it works), but the "dynamic install of setuptools" idea also offers what you mention, although admittedly with more fragility. If a user uninstalled setuptools, it would be installed again when needed, and users only need pip to get started, and don't have to think about the setuptools dependency themselves.
The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD) 2) users can't upgrade themselves (for use in pip) 3) more tedium in our release process. 4) feels odd to bundle it knowing we'd likely drop it later, if we do the MEBs thing.
Marcus
1) That's kinda FUD-y yea ;) But I'd say it's equally as likely to have weird bugs/side effects due to people using different combinations of pip/setuptools with pip than we've tested.
2) This much is true, the question then becomes how important is that? If there's a major regression in setuptools that needs fixed I'd think we'd release an updated pip. If there's new functionality I would guess we'd need to expose that in pip anyways.
3) I think this isn't as big of a deal as it sounds. Especially given we can write tooling to make it simpler :)
4) Even if MEBs were here *right now* we'd still have nearly 150k source dists that required setuptools. So either in the MEB system we'd be grabbing setuptools *a lot* or we could just bundle it to provide a better UX for people using the large corpus of existing software. I think it will be a long time once the MEBs exist before they gain enough traction that even the bulk of installs are using that system. MEBs depend on sdist 2.0 which hasn't even been started yet ;)
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 13, 2013, at 4:08 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 13 July 2013 20:35, Marcus Smith <qwcode@gmail.com> wrote: The drawbacks of bundling setuptools: 1) maybe some weird bug/side-effect shows up after we do it (ok, maybe that's FUD)
One possible issue is with the "install from sdist" code, which runs Python in a subprocess with the "import setuptools, etc etc" incantation to force always using setuptools. That may break (or at least need changing) for a bundled setuptools, which won't be visible from the top level by default. Worth checking, anyway.
Of course, this code path becomes less important as we move towards installing from wheels, but it's going to be a while before that's the norm.
Paul
PS Apologies if I already said this. I'm losing track of what I've replied to and what I've just thought about on this thread :-(
Pip already wraps that code to force things to use setuptools even if they use distutils. So in that case pip would just need to modify it's own code to add setuptools to sys.modules (or extend sys.path in that sub process). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (3)
-
Donald Stufft -
Marcus Smith -
Paul Moore