[Distutils] setup_requires for dev environments

[Robert Collins]

On 17 March 2015 at 09:42, Donald Stufft <donald at stufft.io> wrote:
>
..
> Ahhhh, wait a minute. I think something might have just clicked here.
>
> You’re expecting/wanting the results of setup_requres to be installed
> into the environment itself and not just made available to the setup.py?
> That’s not going to work and I’d be against making that work.

One of the common things I do is 'python setup.py sdist bdist_wheel
upload -s'. That requires setup.py to work, and is a wonky world of
pain right now. Anything that depends on pip intercepting setup.py
isn't going to work there, is it? Debian build environments for these
things generally run without the internet, without access to a pip
mirror - distributors are used to having the setup requirements be
safely installable, and I'd argue that setup_requires that violate
that rule have been getting pushback and selection pressure for years.
So I very much doubt there is a case where installing the deps is bad.
I can well imagine the lack of resolver in pip issue being a concern,
but its a concern regardless.

> For something like that I’d say it would more cleanly map to something
> like tests_requires (in setup.py and PEP 426) and dev_requires (in PEP 426).
> I think that it would be reasonable for pip to install both of those
> types of requirements into the environment when you’re installing as
> an editable installation.

I can grok that, though I suspect it runs the risk of being
over-modelled. Do we know any uses of setup_requires where installing
the requirements into the build environment would do the wrong thing?
E.g. is it a theoretical concern, or a omg X would do Y
we've-been-bitten-before issue?
..

> So it appears there’s actually two problems here, one is the one above,
> that you want some sort of “these are required to do development”
> requirements, and that setup_requires has some problems (it’s inside
> of setup.py, and it’s installed by easy_install instead of pip).

Sure. Note too that folk need to be able to run setup.py without
triggering easy_install and without running pip. Thats a requirement
for e.g. Debian. The way thats handled today is to have the build
fail, look at it, and add build-depends lines to debian/control - the
explicit metadata in Debian. If we had explicit metadata for Python
sources (git, not dists), then folk could use that to reflect
dependencies across semi-automatically (e.g. flagging new ones more
clearly).

> Ignoring the “development requirement” problem (even though I think that’s
> a more interesting problem!) for a moment, I think that yea it’d be great
> to specify setup_requires statically, but that right now defining requirements
> as Python inside of setup.py is the standard we have. I’m aware that pbr
> routes around this standard, but that’s pbr and it’s not hardly the norm. I
> think that it’s worse to have a weird one off place to specify a particular
> type of dependency than to continue to use the normal mechanism and add
> things in pip to work around the deficiencies in that.

A concern I haven't expressed so far is that the route you're
proposing is very clever. Clever tends to break, be hard to diagnose
and hard to understand. I understand the benefit to folk for all the
stale-won't-update packages out there, and perhaps we can do multiple
things that all independently solve the issue. That is, what if we do
explicit metadata *and* have pip do wonky magic.

> The other benefit to my proposal is that every existing use of setup_requires
> starts to get installed by pip instead of by easy_install, which solves a
> whole class of problems like not supporting Wheels, proxy settings, SSL settings,
> etc.

Yep, which is beneficial on its own. But not a reason not to do
explicit metadata :).

> Going back to the development requirement problem, I think it would be reasonable
> for setuptools to start to gain some of the concepts from PEP 426, it already
> has tests_requires and I think that an official dev_requires wouldn’t be a bad
> idea either. If it then exposed those things as something pip could inspect we
> could start doing things like automatically installing them inside of a development
> installation. This would probably even allow backwards compat by having a setup.py
> dynamically add things to the setup_requires based upon what version of setuptools
> is executing the setup.py. If it’s an older one, add a shim that’ll implement the
> new functionality as a plugin instead of part of core.

Ok, so what I need to know is what:
 - can I do
 - that solves my problem (not the other 1000 problems that PEP-426 solves)
 - that the setuptools // pip maintainers would be willing to merge.

I'm happy to put tuits into this, but I don't want to boil the ocean -
I want to solve the specific thing that makes me curse my laptop
screen on a regular basis.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud