[Distutils] setup_requires for dev environments

[Donald Stufft]

> On Mar 16, 2015, at 6:35 PM, Robert Collins <robertc at robertcollins.net> wrote:
> 
> On 17 March 2015 at 09:42, Donald Stufft <donald at stufft.io> wrote:
>> 
> ..
>> Ahhhh, wait a minute. I think something might have just clicked here.
>> 
>> You’re expecting/wanting the results of setup_requres to be installed
>> into the environment itself and not just made available to the setup.py?
>> That’s not going to work and I’d be against making that work.
> 
> One of the common things I do is 'python setup.py sdist bdist_wheel
> upload -s'. That requires setup.py to work, and is a wonky world of
> pain right now. Anything that depends on pip intercepting setup.py
> isn't going to work there, is it? Debian build environments for these
> things generally run without the internet, without access to a pip
> mirror - distributors are used to having the setup requirements be
> safely installable, and I'd argue that setup_requires that violate
> that rule have been getting pushback and selection pressure for years.
> So I very much doubt there is a case where installing the deps is bad.
> I can well imagine the lack of resolver in pip issue being a concern,
> but its a concern regardless.

I’m not sure what solution is actually going to work here besides something
like setup_requires. If you’re executing ``python setup.py …`` then the only
"hooks" that exist (and as far as I can tell, can exist) are ones that exist
now. The only real improvement I can think of is setuptools offering a seperate
API call to install the setup_requires so you can do something like:

    import setuptools

    setuptools.setup_requires("pbr", "otherthing")

    import pbr

    setuptools.setup(**pbr.get_metadata())

Anything that relies on parsing a setup.cfg is going to be roughly equivilant
to that (or the current solution) because there's no other place to call that
will make ``python setup.py`` work. Unless I'm missing something, how does
having a setup_requires inside of setup.cfg help with this sequence of
commands:

    $ git clone ....
    $ cd ....
    $ virtualenv .env
    $ .env/bin/python setup.py sdist


> 
>> For something like that I’d say it would more cleanly map to something
>> like tests_requires (in setup.py and PEP 426) and dev_requires (in PEP 426).
>> I think that it would be reasonable for pip to install both of those
>> types of requirements into the environment when you’re installing as
>> an editable installation.
> 
> I can grok that, though I suspect it runs the risk of being
> over-modelled. Do we know any uses of setup_requires where installing
> the requirements into the build environment would do the wrong thing?
> E.g. is it a theoretical concern, or a omg X would do Y
> we've-been-bitten-before issue?
> ..

setup_requires don’t just run in “build” environments, they also run anytime
you install from sdist in a final environment. To answer the question though,
yes there are real concerns, some of the current setuptools extensions don’t
play well if they are installed and available when you’re installing something
that *doesn’t* use them. You could argue that this is a bug with those particular
things, but currently some things do assume that if they are installed they are
expected to be doing things to setup.py.

Further more, I’m not sure that’s actually possible for this to generically work,
if two different setuptools extensions both want to override the same thing, only
one of them can work. So if you had two things that relied on overriding the install
command (or the build command, or whatever), one would “win” and the other would
act as if it were not installed.

> 
>> So it appears there’s actually two problems here, one is the one above,
>> that you want some sort of “these are required to do development”
>> requirements, and that setup_requires has some problems (it’s inside
>> of setup.py, and it’s installed by easy_install instead of pip).
> 
> Sure. Note too that folk need to be able to run setup.py without
> triggering easy_install and without running pip. Thats a requirement
> for e.g. Debian. The way thats handled today is to have the build
> fail, look at it, and add build-depends lines to debian/control - the
> explicit metadata in Debian. If we had explicit metadata for Python
> sources (git, not dists), then folk could use that to reflect
> dependencies across semi-automatically (e.g. flagging new ones more
> clearly).

I don’t think most Debian (or Linux in General) Distributions are consuming
packages directly from VCS. As I understand it from following debian-python the
folks packaging Openstack are the outliers in that.

That being said, most of the work thus far have ignored the part of the
process where what we have is a VCS checkout. That's because it's somewhat
dependent on breaking the reference cycle where the entire toolchain only
really works because we assume setuptools everywhere.


> 
>> Ignoring the “development requirement” problem (even though I think that’s
>> a more interesting problem!) for a moment, I think that yea it’d be great
>> to specify setup_requires statically, but that right now defining requirements
>> as Python inside of setup.py is the standard we have. I’m aware that pbr
>> routes around this standard, but that’s pbr and it’s not hardly the norm. I
>> think that it’s worse to have a weird one off place to specify a particular
>> type of dependency than to continue to use the normal mechanism and add
>> things in pip to work around the deficiencies in that.
> 
> A concern I haven't expressed so far is that the route you're
> proposing is very clever. Clever tends to break, be hard to diagnose
> and hard to understand. I understand the benefit to folk for all the
> stale-won't-update packages out there, and perhaps we can do multiple
> things that all independently solve the issue. That is, what if we do
> explicit metadata *and* have pip do wonky magic.

Yea, clever is often times broken, I can agree with that.

> 
>> The other benefit to my proposal is that every existing use of setup_requires
>> starts to get installed by pip instead of by easy_install, which solves a
>> whole class of problems like not supporting Wheels, proxy settings, SSL settings,
>> etc.
> 
> Yep, which is beneficial on its own. But not a reason not to do
> explicit metadata :).

Right, explicit metadata is absolutely the end goal. My concern is about adding more
random cruft that we’ll have to support forever on the path to that.

> 
>> Going back to the development requirement problem, I think it would be reasonable
>> for setuptools to start to gain some of the concepts from PEP 426, it already
>> has tests_requires and I think that an official dev_requires wouldn’t be a bad
>> idea either. If it then exposed those things as something pip could inspect we
>> could start doing things like automatically installing them inside of a development
>> installation. This would probably even allow backwards compat by having a setup.py
>> dynamically add things to the setup_requires based upon what version of setuptools
>> is executing the setup.py. If it’s an older one, add a shim that’ll implement the
>> new functionality as a plugin instead of part of core.
> 
> Ok, so what I need to know is what:
> - can I do
> - that solves my problem (not the other 1000 problems that PEP-426 solves)
> - that the setuptools // pip maintainers would be willing to merge.
> 
> I'm happy to put tuits into this, but I don't want to boil the ocean -
> I want to solve the specific thing that makes me curse my laptop
> screen on a regular basis.

I’ll come back to this if you can answer the above about how a setup.cfg solves the
``python setup.py …`` situation from my first paragraph, because that’s going to
influence my thoughts.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150316/74b25c93/attachment.sig>