Re: [Distutils] Please do not remove dependency_links

Thank you for your suggestions, Donald. They are both feasible but they feel like workarounds to me. Manually listing transitive dependencies seems like a step backwards to me. Isn't the whole point of setuptools/distutils that each component manages its own list of dependencies? How will this scale to dozens of transitive dependencies and busy development teams where dependencies are revved frequently. I really liked what Maven did for Java and therefore liked what pip is trying to do for Python. I haven't used --find-links yet so I may be wrong but tarring up packages and serving them by a web server is additional work that I'd rather avoid. It just creates yet another copy of the package and requires maintaining an additional server component. I loved the fact that I could point pip directly at my source repos for both my top-level projects as well as their dependencies. That seemed like a perfect fit for an interpreted language like Python where packages are distributed in source form. Removing dependency_links makes that feature useless to me because it doesn't extend to dependencies, only top level components. On Fri, Jan 17, 2014 at 4:44 AM, Donald Stufft wrote:

On Jan 16, 2014, at 9:52 PM, Hannes Schmidt wrote:

I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach.

The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature.

Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there.

I understand the motivation for this change – security – but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two?

[1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html

Pip 1.5 has already been released which turns dependency links off by default and adds a flag —process-dependency-links and they are scheduled to be removed in 1.6. It’s possible we can “undeprecate” them and leave them off by default however my question to you would be why you can’t use a requirements.txt file to handle installing things from bitbucket?

Fundamentally a setup.py should not contain “concrete” locations but instead should contain abstract names/version specifiers whereas a requirements.txt (or the pip invocation itself) takes those abstract names and makes them concrete by pairing them with an index.

So to be more explicit, is there any reason why one of:

1) Create a requirements.txt and point to bitbucket in that and install your environment using ``pip install -r requirements.txt`` 2) Create tar balls of your packages and host them in a directory somewhere behind nginx and use —find-links https://example.com/that-directory/as your install location.

Won’t work just as fine for you?

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes@ucsc.edu