[Distutils] Please do not remove dependency_links

[Nick Coghlan]

On 18 Jan 2014 04:45, "Hannes Schmidt" <hannes at ucsc.edu> wrote:
>
> Thank you for your suggestions, Donald. They are both feasible but they
feel like workarounds to me.
>
> Manually listing transitive dependencies seems like a step backwards to
me. Isn't the whole point of setuptools/distutils that each component
manages its own list of dependencies? How will this scale to dozens of
transitive dependencies and busy development teams where dependencies are
revved frequently. I really liked what Maven did for Java and therefore
liked what pip is trying to do for Python.

Hannes, this is the role we consider to be best served by running a private
index server and using the full dependency resolution mechanism. You may
want to review PEP 426 to see the intended mode of operation.

However, the question at this point is whether there is a legitimate model
of use where dependency links (or an equivalent) are used to create a more
concrete dependency tree in a closed environment where the software
developer/publisher and integrator/user are the *same* organisation.

PEP 426 focuses on the case where those are different organisations (since
that case poses the greatest security risk, allowing publishers to
compromise the installations of integrators and users, beyond just running
the software they provide). While it does include the "meta requirements"
field to handle cases of (effective) dependency bundling rather than normal
versioning, that model is deliberately more restrictive than dependency
links.

It seems to me that overall, it's much, much safer to simply avoid
supporting this degree of transitive trust directly in the core
installation toolchain. A more appropriate solution to handling the trusted
internal distribution use case may be a dependency scanner that processes
the dependency links metadata to generate a suitable requirements.txt file.
>From pip's perspective, trusting that generated file is the same as
trusting any other explicitly specified set of packages, while it also
provides an opportunity for the resolved dependency links to be audited
before trusting them.

Unfortunately, we can't actually do that easily in a secure fashion,
because obtaining the metadata currently requires executing an untrusted
setup.py. So trusting dependency links "just" to generate a
requirements.txt file is almost as bad as trusting them for installing
software. The main advantage is that you can always generate the dependency
list as an unprivileged user or in a sandbox, even if operating in an
untrusted environment. Installation, by contrast, is often going to involve
elevated privileges.

Regardless, perhaps there's a case to be made that such a requirements.txt
generator, with opt-in dependency links support, should remain available,
with the feature being removed entirely only for actual package
installation.

That approach is still far more secure by default than the status quo,
strongly encourages the use of a private index to better align with the
upstream distribution model, but restores support for users like Hannes
that are currently relying on dependency links as a dependency resolution
solution that doesn't require additional server components in a trusted
environment.

> I haven't used --find-links yet so I may be wrong but tarring up packages
and serving them by a web server is additional work that I'd rather avoid.
It just creates yet another copy of the package and requires maintaining an
additional server component.
>
> I loved the fact that I could point pip directly at my source repos for
both my top-level projects as well as their dependencies. That seemed like
a perfect fit for an interpreted language like Python where packages are
distributed in source form.

In a trusted environment, it can be, but on the internet, it is intolerably
insecure.

However, making it easy to generate a transitive requirements.txt for a
trusted environment may be a usage model we can support without leaving
upstream distribution and usage open to compromise.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140118/674afe76/attachment.html>