I would like to remove dependency_links from pip, and ideally also setuptools. In implementing the ensurepip module from PEP453 I realized that even with the ``--no-index`` flag pip was still attempting to reach the internet. After a little bit of investigation I realized that the reason for this was setuptools use of dependency links. From my investigation it appears that setuptools uses these in order to enable secure automatic installation of the ssl dependencies on Python < 2.6. Overall this feature is a security concern, a malicous package could "pin" any package they want by depending on it and adding a dependency link a version 100000. This would be more or less transparent to the end user. I was looking to see what sort of impact this would have. There are currently 167,796 source files hosted on PyPI and of those files 4,005 of them have any dependency links at all. Looking at it a different way, there are 36,070 total projects on PyPI and 411 of them use this feature. So this is ~2% of the files or ~1% of the projects. So it appears that this isn't a particularly popular feature, I believe that it is a *bad* idea that inverts the expected control and should be removed from both pip and setuptools. In setuptools case it does use it in the only reasonable way I can imagine, however I think setuptools should just stop trying to automatically install those dependencies for Pythons < 2.6 and similarly to pip just print an error and expect users to get and install them on their own. As a reminder there are very few downloads from PyPI that are from Pythons < 2.6 [1] [1] https://caremad.io/blog/a-look-at-pypi-downloads/ [2] https://gist.github.com/dstufft/7173539 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Oct 26, 2013, at 10:14 PM, Donald Stufft <donald@stufft.io> wrote:
I would like to remove dependency_links from pip, and ideally also setuptools.
In implementing the ensurepip module from PEP453 I realized that even with the ``--no-index`` flag pip was still attempting to reach the internet. After a little bit of investigation I realized that the reason for this was setuptools use of dependency links. From my investigation it appears that setuptools uses these in order to enable secure automatic installation of the ssl dependencies on Python < 2.6.
Overall this feature is a security concern, a malicous package could "pin" any package they want by depending on it and adding a dependency link a version 100000. This would be more or less transparent to the end user.
I was looking to see what sort of impact this would have. There are currently 167,796 source files hosted on PyPI and of those files 4,005 of them have any dependency links at all. Looking at it a different way, there are 36,070 total projects on PyPI and 411 of them use this feature. So this is ~2% of the files or ~1% of the projects.
So it appears that this isn't a particularly popular feature, I believe that it is a *bad* idea that inverts the expected control and should be removed from both pip and setuptools. In setuptools case it does use it in the only reasonable way I can imagine, however I think setuptools should just stop trying to automatically install those dependencies for Pythons < 2.6 and similarly to pip just print an error and expect users to get and install them on their own. As a reminder there are very few downloads from PyPI that are from Pythons < 2.6 [1]
[1] https://caremad.io/blog/a-look-at-pypi-downloads/ [2] https://gist.github.com/dstufft/7173539
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
A list of projects that use dependency links: https://gist.github.com/dstufft/7177500 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Bleh scratch that, it was adding everything :( On Oct 26, 2013, at 10:59 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 10:14 PM, Donald Stufft <donald@stufft.io> wrote:
I would like to remove dependency_links from pip, and ideally also setuptools.
In implementing the ensurepip module from PEP453 I realized that even with the ``--no-index`` flag pip was still attempting to reach the internet. After a little bit of investigation I realized that the reason for this was setuptools use of dependency links. From my investigation it appears that setuptools uses these in order to enable secure automatic installation of the ssl dependencies on Python < 2.6.
Overall this feature is a security concern, a malicous package could "pin" any package they want by depending on it and adding a dependency link a version 100000. This would be more or less transparent to the end user.
I was looking to see what sort of impact this would have. There are currently 167,796 source files hosted on PyPI and of those files 4,005 of them have any dependency links at all. Looking at it a different way, there are 36,070 total projects on PyPI and 411 of them use this feature. So this is ~2% of the files or ~1% of the projects.
So it appears that this isn't a particularly popular feature, I believe that it is a *bad* idea that inverts the expected control and should be removed from both pip and setuptools. In setuptools case it does use it in the only reasonable way I can imagine, however I think setuptools should just stop trying to automatically install those dependencies for Pythons < 2.6 and similarly to pip just print an error and expect users to get and install them on their own. As a reminder there are very few downloads from PyPI that are from Pythons < 2.6 [1]
[1] https://caremad.io/blog/a-look-at-pypi-downloads/ [2] https://gist.github.com/dstufft/7173539
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
A list of projects that use dependency links: https://gist.github.com/dstufft/7177500
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Ok here’s the real list: https://gist.github.com/dstufft/7177500 On Oct 26, 2013, at 11:00 PM, Donald Stufft <donald@stufft.io> wrote:
Bleh scratch that, it was adding everything :(
On Oct 26, 2013, at 10:59 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 10:14 PM, Donald Stufft <donald@stufft.io> wrote:
I would like to remove dependency_links from pip, and ideally also setuptools.
In implementing the ensurepip module from PEP453 I realized that even with the ``--no-index`` flag pip was still attempting to reach the internet. After a little bit of investigation I realized that the reason for this was setuptools use of dependency links. From my investigation it appears that setuptools uses these in order to enable secure automatic installation of the ssl dependencies on Python < 2.6.
Overall this feature is a security concern, a malicous package could "pin" any package they want by depending on it and adding a dependency link a version 100000. This would be more or less transparent to the end user.
I was looking to see what sort of impact this would have. There are currently 167,796 source files hosted on PyPI and of those files 4,005 of them have any dependency links at all. Looking at it a different way, there are 36,070 total projects on PyPI and 411 of them use this feature. So this is ~2% of the files or ~1% of the projects.
So it appears that this isn't a particularly popular feature, I believe that it is a *bad* idea that inverts the expected control and should be removed from both pip and setuptools. In setuptools case it does use it in the only reasonable way I can imagine, however I think setuptools should just stop trying to automatically install those dependencies for Pythons < 2.6 and similarly to pip just print an error and expect users to get and install them on their own. As a reminder there are very few downloads from PyPI that are from Pythons < 2.6 [1]
[1] https://caremad.io/blog/a-look-at-pypi-downloads/ [2] https://gist.github.com/dstufft/7173539
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
A list of projects that use dependency links: https://gist.github.com/dstufft/7177500
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 27 October 2013 14:13, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them.
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself? If so, then the bare minimum is to provide such a flag in the bundled versions of pip and setuptools and have ensurepip use it. I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440). That would make the migration look something like: pip 1.5 (and associated minimum required version of setuptools): - add a disable switch for dependency link handling - add at least a per-project opt-in for dependency link handling (and perhaps a global opt-in) - deprecate implicit handling of dependency links pip 1.6: - dependency links are disabled by default, must opt-in to process them Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Oct 27, 2013, at 12:30 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 27 October 2013 14:13, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them.
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself?
If so, then the bare minimum is to provide such a flag in the bundled versions of pip and setuptools and have ensurepip use it.
Yes, it only needs to exist in pip as well, it does not need to exist in setuptools for ensurepip’s purposes.
I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440).
That would make the migration look something like:
pip 1.5 (and associated minimum required version of setuptools): - add a disable switch for dependency link handling - add at least a per-project opt-in for dependency link handling (and perhaps a global opt-in) - deprecate implicit handling of dependency links
pip 1.6: - dependency links are disabled by default, must opt-in to process them
Cheers, Nick.
What if pip 1.5 added a —no-dependency-links flag, and then pip 1.6 ignored them by default but if a package cannot be installed it would print something like… The package {foo} was unable to be found which was depended on by {bar}, {bar} has suggested some additional links for locating dependencies, you can use any of them by using the —find-links flag such as pip install —find-links <url>. The suggested urls are: https://…./ https://…./ This allows users to opt in on a per url basis (and under the covers the implementation would be the same, dependency links just get added to find-links) without adding yet another flag. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 27 October 2013 14:35, Donald Stufft <donald@stufft.io> wrote:
On Oct 27, 2013, at 12:30 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 27 October 2013 14:13, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them.
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself?
If so, then the bare minimum is to provide such a flag in the bundled versions of pip and setuptools and have ensurepip use it.
Yes, it only needs to exist in pip as well, it does not need to exist in setuptools for ensurepip’s purposes.
Excellent, so that's the only mandatory-due-to-PEP-453 part for pip 1.5.
I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440).
That would make the migration look something like:
pip 1.5 (and associated minimum required version of setuptools): - add a disable switch for dependency link handling - add at least a per-project opt-in for dependency link handling (and perhaps a global opt-in) - deprecate implicit handling of dependency links
pip 1.6: - dependency links are disabled by default, must opt-in to process them
Cheers, Nick.
What if pip 1.5 added a —no-dependency-links flag, and then pip 1.6 ignored them by default but if a package cannot be installed it would print something like…
The package {foo} was unable to be found which was depended on by {bar}, {bar} has suggested some additional links for locating dependencies, you can use any of them by using the —find-links flag such as pip install —find-links <url>.
The suggested urls are: https://…./ https://…./
This allows users to opt in on a per url basis (and under the covers the implementation would be the same, dependency links just get added to find-links) without adding yet another flag.
With that error message, I think it's reasonable to do as Holger suggests and opt out of processing them by default even in 1.5. To me, the best part of the more aggressive timeline is it means CPython would never ship a version of pip that allows that particular attack vector by default. In that approach, I'd still suggest offering a "--process-dependency-links" flag and there wouldn't be a flag to turn the processing off (since they'd be off by default). This suggestion is born out of the "we don't know what happens inside corporate firewalls" perspective, so I think it's beneficial to have a "make it work the way it used to" fallback for at least one release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
"we don't know what happens inside corporate firewalls"
non-published use of dependency links could turn out to be the use-cases that we'd get complaints about
To me, the best part of the more aggressive timeline is it means CPython would never ship a version of pip that allows that particular attack vector by default.
over IRC and on pypa-dev, I brought up the deprecate first point of view in the context that we would be *removing the feature*. It's less drastic to flip defaults (and add a turn on) it's probably right that nobody will complain, but my thinking was this: - donald can add a hidden option for now for the sake of ensurepip (it wouldn't clutter the cli, and can be removed later care-free) - separate from that, pip and setuptools deprecates together, then completely removes dep-links support. if its bad, it's bad. get rid of it. let's reduce the options and clutter.
On 27 Oct 2013 18:38, "Marcus Smith" <qwcode@gmail.com> wrote:
"we don't know what happens inside corporate firewalls"
non-published use of dependency links could turn out to be the use-cases
that we'd get complaints about
To me, the best part of the more aggressive timeline is it means CPython would never ship a version of pip that allows that particular attack vector by default.
over IRC and on pypa-dev, I brought up the deprecate first point of view
in the context that we would be *removing the feature*.
It's less drastic to flip defaults (and add a turn on)
it's probably right that nobody will complain, but my thinking was this: - donald can add a hidden option for now for the sake of ensurepip (it wouldn't clutter the cli, and can be removed later care-free)
Yeah, we at least need to do that much to meet the "ensurepip doesn't talk to the internet" guarantee.
- separate from that, pip and setuptools deprecates together, then completely removes dep-links support. if its bad, it's bad. get rid of it. let's reduce the options and clutter.
I'm happy to go with whatever you folks (as in pip & setuptools devs) decide on that front. I prefer "flip the default & deprecate, then remove later if nobody campaigns to keep it", but I'm also OK with the more conservative "deprecate, then remove later". Cheers, Nick.
More numbers, of the 411 projects who have ever used dependency links, only 311 of them use them in their latest release. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Here’s the list of dependency links for the projects that still use them in their latest releases: https://gist.github.com/dstufft/7185162 A good number of them are either bogus, are pointing directly to PyPI, or are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception). So honestly I think this could just go away completely. I don’t see any use for it anymore and apparently neither does most of PyPI. On Oct 27, 2013, at 1:00 PM, Donald Stufft <donald@stufft.io> wrote:
More numbers, of the 411 projects who have ever used dependency links, only 311 of them use them in their latest release.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sun, Oct 27, 2013 at 10:44 AM, Donald Stufft <donald@stufft.io> wrote:
Here’s the list of dependency links for the projects that still use them in their latest releases:
https://gist.github.com/dstufft/7185162
A good number of them are either bogus, are pointing directly to PyPI, or are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception).
I actually know a couple people on this list. I can ask them and see if the list can be reduced further. :) --Chris
So honestly I think this could just go away completely. I don’t see any use for it anymore and apparently neither does most of PyPI.
On Oct 27, 2013, at 1:00 PM, Donald Stufft <donald@stufft.io> wrote:
More numbers, of the 411 projects who have ever used dependency links, only 311 of them use them in their latest release.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Oct 27, 2013 at 12:04 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Oct 27, 2013 at 10:44 AM, Donald Stufft <donald@stufft.io> wrote:
Here’s the list of dependency links for the projects that still use them in their latest releases:
https://gist.github.com/dstufft/7185162
A good number of them are either bogus, are pointing directly to PyPI, or are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception).
I actually know a couple people on this list. I can ask them and see if the list can be reduced further. :)
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library." Is there another approach or work-around he can be using? What is the "right" way for him to do it? --Chris
On Oct 27, 2013, at 5:49 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Oct 27, 2013 at 12:04 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Oct 27, 2013 at 10:44 AM, Donald Stufft <donald@stufft.io> wrote:
Here’s the list of dependency links for the projects that still use them in their latest releases:
https://gist.github.com/dstufft/7185162
A good number of them are either bogus, are pointing directly to PyPI, or are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception).
I actually know a couple people on this list. I can ask them and see if the list can be reduced further. :)
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
--Chris
Upload the package to PyPI under a different name? Vendor the package inside the source? Maybe his fork is incompatible, the way he’s doing it it’ll install, pretending to be the unforked library, and then if something *else* depends on it, it’ll get a fundamentally incompatible version of that library (in the theoretical situation). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in "/my/forks" and then "pip install --find-links=/my/forks/ my_app" as for fork versioning in the long run, that is intended to be covered in PEP440, with a conversation happening here: https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-local-numbering...
--Chris _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Chris: to be clear, after talking to Donald, we interpreted your question differently. - If you're distributing library Y, and it depends on X, but it now needs a custom/fixed fork of X, then what Donald said, rename and re-publish (or vendor it). - If you just need to override a buggy pypi package locally in your app Y install, then what I said (you can also use a vcs requirement form, and not have to package you fork) On Sun, Oct 27, 2013 at 3:52 PM, Marcus Smith <qwcode@gmail.com> wrote:
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in "/my/forks" and then "pip install --find-links=/my/forks/ my_app"
as for fork versioning in the long run, that is intended to be covered in PEP440, with a conversation happening here:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-local-numbering...
--Chris _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 Oct 2013 09:11, "Marcus Smith" <qwcode@gmail.com> wrote:
Chris: to be clear, after talking to Donald, we interpreted your question
differently.
- If you're distributing library Y, and it depends on X, but it now needs
a custom/fixed fork of X, then what Donald said, rename and re-publish (or vendor it).
- If you just need to override a buggy pypi package locally in your app Y install, then what I said (you can also use a vcs requirement form, and not have to package you fork)
And to clarify the rationale for these as the official solutions: installing modified software under the *same* name as the original is not something we wish to support through PyPI, as it makes even cursory security audits far more difficult than they should be, and can cause additional confusion when debugging problems. If a package maintainer for a dependency doesn't release regular updates or is otherwise not responsive to bug reports, then the appropriate solutions are: - offering a function for runtime monkey patching of the dependency - bundling a patched copy directly - forking the dependency - using application level dependencies to force installation of the patched version Implicit monkey patching and silently replacing the upstream dependency with a patched version is quite user hostile when publishing software for use by others in a Python installation that may be used to run multiple independent applications. Thinking about this further, perhaps we should institute a PyPI side lockout for usage of dependency links in *new* projects? (similar to what happened for the external hosting support and is proposed for direct references in PEP 426) (Which would put this in YAPP territory - *sigh*) Cheers, Nick. P.S. Yet Another Packaging PEP. Feel free to infer the implied presence of additional adjectives ;)
On Sun, Oct 27, 2013 at 3:52 PM, Marcus Smith <qwcode@gmail.com> wrote:
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in
fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then
your app install) package it, and place it in "/my/forks"
and then "pip install --find-links=/my/forks/ my_app"
as for fork versioning in the long run, that is intended to be covered in PEP440, with a conversation happening here:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-local-numbering...
--Chris _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
https://github.com/pypa/pip/pull/1264 On Oct 27, 2013, at 7:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 Oct 2013 09:11, "Marcus Smith" <qwcode@gmail.com> wrote:
Chris: to be clear, after talking to Donald, we interpreted your question differently.
- If you're distributing library Y, and it depends on X, but it now needs a custom/fixed fork of X, then what Donald said, rename and re-publish (or vendor it). - If you just need to override a buggy pypi package locally in your app Y install, then what I said (you can also use a vcs requirement form, and not have to package you fork)
And to clarify the rationale for these as the official solutions: installing modified software under the *same* name as the original is not something we wish to support through PyPI, as it makes even cursory security audits far more difficult than they should be, and can cause additional confusion when debugging problems.
If a package maintainer for a dependency doesn't release regular updates or is otherwise not responsive to bug reports, then the appropriate solutions are: - offering a function for runtime monkey patching of the dependency - bundling a patched copy directly - forking the dependency - using application level dependencies to force installation of the patched version
Implicit monkey patching and silently replacing the upstream dependency with a patched version is quite user hostile when publishing software for use by others in a Python installation that may be used to run multiple independent applications.
Thinking about this further, perhaps we should institute a PyPI side lockout for usage of dependency links in *new* projects? (similar to what happened for the external hosting support and is proposed for direct references in PEP 426)
(Which would put this in YAPP territory - *sigh*)
Cheers, Nick.
P.S. Yet Another Packaging PEP. Feel free to infer the implied presence of additional adjectives ;)
On Sun, Oct 27, 2013 at 3:52 PM, Marcus Smith <qwcode@gmail.com> wrote:
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in "/my/forks" and then "pip install --find-links=/my/forks/ my_app"
as for fork versioning in the long run, that is intended to be covered in PEP440, with a conversation happening here: https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-local-numbering...
--Chris _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sun, Oct 27, 2013 at 03:52:04PM -0700, Marcus Smith wrote:
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in "/my/forks" and then "pip install --find-links=/my/forks/ my_app"
This is a useful feature and losing it would cause some measure of pain. I tend to use this via find-links in buildout.cfg files. Not just for forks, but also for private packages not released to PyPI. Specifying dependency_links in random packages' setup.py's is a nuisance and I would rather it went away. I always turn it off by specifying allow-hosts = *.python.org in buildout.cfg So, two different things here. One is specified by the user who runs pip (or some other installation tool). The other is specified by the package builder. One is fine, the other is not. Marius Gedminas -- Never assume the reader has read the subject line.
On Tue, Oct 29, 2013 at 5:45 AM, Marius Gedminas <marius@pov.lt> wrote:
Specifying dependency_links in random packages' setup.py's is a nuisance and I would rather it went away. I always turn it off by specifying allow-hosts = *.python.org in buildout.cfg
Some packages indexed on PyPI have downloads elsewhere for various reasons. You can specify use-dependency-links = false to get buildout to ignore dependency_links without limiting the collection of hosts it is willing to download from (useful if you use any private repositories). -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Oct 29, 2013, at 5:45 AM, Marius Gedminas <marius@pov.lt> wrote:
On Sun, Oct 27, 2013 at 03:52:04PM -0700, Marcus Smith wrote:
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
e.g. you find a bug in SomeProject-1.4 on pypi (which is a dependency in your app install) fork it, fix it, and re-version it to SomeProject-1.4-myfork1, then package it, and place it in "/my/forks" and then "pip install --find-links=/my/forks/ my_app"
This is a useful feature and losing it would cause some measure of pain. I tend to use this via find-links in buildout.cfg files. Not just for forks, but also for private packages not released to PyPI.
Specifying dependency_links in random packages' setup.py's is a nuisance and I would rather it went away. I always turn it off by specifying allow-hosts = *.python.org in buildout.cfg
So, two different things here. One is specified by the user who runs pip (or some other installation tool). The other is specified by the package builder. One is fine, the other is not.
Marius Gedminas -- Never assume the reader has read the subject line. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
—find-links aren’t going anywhere, it’s strictly about the package builder case. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sun, Oct 27, 2013 at 10:49 PM, Chris Jerdonek <chris.jerdonek@gmail.com>wrote:
On Sun, Oct 27, 2013 at 12:04 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Oct 27, 2013 at 10:44 AM, Donald Stufft <donald@stufft.io>
Here’s the list of dependency links for the projects that still use
https://gist.github.com/dstufft/7185162
A good number of them are either bogus, are pointing directly to PyPI,
or are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception).
I actually know a couple people on this list. I can ask them and see if
wrote: them in their latest releases: the list can be reduced further. :)
So I asked the person I know, and this is what he said, "Yes, we have to use it! It's the only way to allow a package to install other packages that aren't on PyPI-- for instance, a custom fork of a library."
Is there another approach or work-around he can be using? What is the "right" way for him to do it?
--Chris
Note that some projects also use the dependency_links to point to development packages. See for example https://github.com/tangentlabs/django-oscar/blob/master/setup.py#L63 Although this approach doesn't always work reliably in my experience
On 28 Oct 2013 03:44, "Donald Stufft" <donald@stufft.io> wrote:
Here’s the list of dependency links for the projects that still use them
in their latest releases:
https://gist.github.com/dstufft/7185162
A good number of them are either bogus, are pointing directly to PyPI, or
are file:// urls that are highly unlikely to exist on anyones computer but the author’s. All in all there are 307 total unique links in this set of packages, and 99 of them are not reachable from my computer (requests.get(…) raises an exception).
So honestly I think this could just go away completely. I don’t see any
use for it anymore and apparently neither does most of PyPI. When making compatibility decisions, it's worth remembering that pre-packaged software (let alone the open source subset of that) is only the tip of a very large software iceberg that, as far as I am aware, still consists mostly of custom purpose specific code written for particular organisations. In this case, I think the vulnerability argument is strong enough and good use cases rare enough to justify turning dependency link support off by default, but it should be easy to turn back on in at least pip 1.5 as a risk mitigation strategy. Cheers, Nick.
On Oct 27, 2013, at 1:00 PM, Donald Stufft <donald@stufft.io> wrote:
More numbers, of the 411 projects who have ever used dependency links,
only 311 of them use them in their latest release.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sun, Oct 27, 2013 at 12:30 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself?
There's been a setting for this in buildout for some time, and I don't build without it. Your deprecation path sounds reasonable to me. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Sun, Oct 27, 2013 at 14:30 +1000, Nick Coghlan wrote:
On 27 October 2013 14:13, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them.
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself?
If so, then the bare minimum is to provide such a flag in the bundled versions of pip and setuptools and have ensurepip use it.
I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440).
That would make the migration look something like:
pip 1.5 (and associated minimum required version of setuptools): - add a disable switch for dependency link handling - add at least a per-project opt-in for dependency link handling (and perhaps a global opt-in) - deprecate implicit handling of dependency links
pip 1.6: - dependency links are disabled by default, must opt-in to process them
So 400 projects out of 35000 ever used dependency links. I checked three random ones: - flask-mongorest: does not use it anymore - Pylons: deplink goes to 502 page, and has the latest release on pypi. - OpenCoreRedirect: one of out three deplinks work but goes to a page that doesn't appear to be one. Latest release is 0.5.1, available on pypi Project, four years old. Judging from this little sample: if a questionable feature is used by <1% of projects and even they likely to not work/don't rely on it anymore, i don't think we should spend or make Donald spend much efforts on it. Rather do the supposed 1.6 change for 1.5 already. Note that I was the guy publically pressing for backward-compat but that was for the introduction of "--pre" which broke many usages. This does not start to compare to this change here. Also pip-1.5 would cleanly bail out and tell what to do whereas the need for "--pre" was more implicit as people could get the wrong version suddenly without noticing/understanding. best, holger
On Oct 27, 2013, at 1:07 AM, holger krekel <holger@merlinux.eu> wrote:
On Sun, Oct 27, 2013 at 14:30 +1000, Nick Coghlan wrote:
On 27 October 2013 14:13, Donald Stufft <donald@stufft.io> wrote:
On Oct 26, 2013, at 11:59 PM, Donald Stufft <donald@stufft.io> wrote:
Ok here’s the real list: https://gist.github.com/dstufft/7177500
Quick note that this list is a list of projects that have *ever* used dependency links on PyPI. Some of these projects are no longer using them.
Am I correct in thinking that providing a flag to disable them completely will be enough to get ensurepip to behave itself?
If so, then the bare minimum is to provide such a flag in the bundled versions of pip and setuptools and have ensurepip use it.
I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440).
That would make the migration look something like:
pip 1.5 (and associated minimum required version of setuptools): - add a disable switch for dependency link handling - add at least a per-project opt-in for dependency link handling (and perhaps a global opt-in) - deprecate implicit handling of dependency links
pip 1.6: - dependency links are disabled by default, must opt-in to process them
So 400 projects out of 35000 ever used dependency links. I checked three random ones:
- flask-mongorest: does not use it anymore - Pylons: deplink goes to 502 page, and has the latest release on pypi. - OpenCoreRedirect: one of out three deplinks work but goes to a page that doesn't appear to be one. Latest release is 0.5.1, available on pypi Project, four years old.
Heh, Webtest and Flask-Security were two I checked who no longer use them.
Judging from this little sample: if a questionable feature is used by <1% of projects and even they likely to not work/don't rely on it anymore, i don't think we should spend or make Donald spend much efforts on it. Rather do the supposed 1.6 change for 1.5 already.
I’m definitely +1 on doing the change in 1.5 instead. I really don’t think it’s going to affect hardly anyone.
Note that I was the guy publically pressing for backward-compat but that was for the introduction of "--pre" which broke many usages. This does not start to compare to this change here. Also pip-1.5 would cleanly bail out and tell what to do whereas the need for "--pre" was more implicit as people could get the wrong version suddenly without noticing/understanding.
best, holger
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I also think it is reasonable to continue offering a feature like dependency_links on an opt-in basis for controlled environments (I see it as analagous to the direct references feature in PEP 440).
with pip already, the idea** is that you can enforce a concrete location for a dependency that will override the default pypi location. e.g. I can do pip install http://myserver/peppercorn-0.3.tar.gz#egg=peppercorn deform (where deform requires peppercorn) and my peppercorn will be installed the point being, I don't think we need setup.py dependency links to be able enforce concrete url fulfillment. ** I say "idea", because I know there are bugs related to archive urls and egg fragments and overriding other dependencies, but the point still stands.
participants (8)
-
Chris Jerdonek
-
Donald Stufft
-
Fred Drake
-
holger krekel
-
Marcus Smith
-
Marius Gedminas
-
Michael van Tellingen
-
Nick Coghlan