On Mar 16, 2015, at 6:35 PM, Robert Collins
wrote: On 17 March 2015 at 09:42, Donald Stufft
wrote: ..
Ahhhh, wait a minute. I think something might have just clicked here.
You’re expecting/wanting the results of setup_requres to be installed into the environment itself and not just made available to the setup.py? That’s not going to work and I’d be against making that work.
One of the common things I do is 'python setup.py sdist bdist_wheel upload -s'. That requires setup.py to work, and is a wonky world of pain right now. Anything that depends on pip intercepting setup.py isn't going to work there, is it? Debian build environments for these things generally run without the internet, without access to a pip mirror - distributors are used to having the setup requirements be safely installable, and I'd argue that setup_requires that violate that rule have been getting pushback and selection pressure for years. So I very much doubt there is a case where installing the deps is bad. I can well imagine the lack of resolver in pip issue being a concern, but its a concern regardless.
I’m not sure what solution is actually going to work here besides something like setup_requires. If you’re executing ``python setup.py …`` then the only "hooks" that exist (and as far as I can tell, can exist) are ones that exist now. The only real improvement I can think of is setuptools offering a seperate API call to install the setup_requires so you can do something like: import setuptools setuptools.setup_requires("pbr", "otherthing") import pbr setuptools.setup(**pbr.get_metadata()) Anything that relies on parsing a setup.cfg is going to be roughly equivilant to that (or the current solution) because there's no other place to call that will make ``python setup.py`` work. Unless I'm missing something, how does having a setup_requires inside of setup.cfg help with this sequence of commands: $ git clone .... $ cd .... $ virtualenv .env $ .env/bin/python setup.py sdist
For something like that I’d say it would more cleanly map to something like tests_requires (in setup.py and PEP 426) and dev_requires (in PEP 426). I think that it would be reasonable for pip to install both of those types of requirements into the environment when you’re installing as an editable installation.
I can grok that, though I suspect it runs the risk of being over-modelled. Do we know any uses of setup_requires where installing the requirements into the build environment would do the wrong thing? E.g. is it a theoretical concern, or a omg X would do Y we've-been-bitten-before issue? ..
setup_requires don’t just run in “build” environments, they also run anytime you install from sdist in a final environment. To answer the question though, yes there are real concerns, some of the current setuptools extensions don’t play well if they are installed and available when you’re installing something that *doesn’t* use them. You could argue that this is a bug with those particular things, but currently some things do assume that if they are installed they are expected to be doing things to setup.py. Further more, I’m not sure that’s actually possible for this to generically work, if two different setuptools extensions both want to override the same thing, only one of them can work. So if you had two things that relied on overriding the install command (or the build command, or whatever), one would “win” and the other would act as if it were not installed.
So it appears there’s actually two problems here, one is the one above, that you want some sort of “these are required to do development” requirements, and that setup_requires has some problems (it’s inside of setup.py, and it’s installed by easy_install instead of pip).
Sure. Note too that folk need to be able to run setup.py without triggering easy_install and without running pip. Thats a requirement for e.g. Debian. The way thats handled today is to have the build fail, look at it, and add build-depends lines to debian/control - the explicit metadata in Debian. If we had explicit metadata for Python sources (git, not dists), then folk could use that to reflect dependencies across semi-automatically (e.g. flagging new ones more clearly).
I don’t think most Debian (or Linux in General) Distributions are consuming packages directly from VCS. As I understand it from following debian-python the folks packaging Openstack are the outliers in that. That being said, most of the work thus far have ignored the part of the process where what we have is a VCS checkout. That's because it's somewhat dependent on breaking the reference cycle where the entire toolchain only really works because we assume setuptools everywhere.
Ignoring the “development requirement” problem (even though I think that’s a more interesting problem!) for a moment, I think that yea it’d be great to specify setup_requires statically, but that right now defining requirements as Python inside of setup.py is the standard we have. I’m aware that pbr routes around this standard, but that’s pbr and it’s not hardly the norm. I think that it’s worse to have a weird one off place to specify a particular type of dependency than to continue to use the normal mechanism and add things in pip to work around the deficiencies in that.
A concern I haven't expressed so far is that the route you're proposing is very clever. Clever tends to break, be hard to diagnose and hard to understand. I understand the benefit to folk for all the stale-won't-update packages out there, and perhaps we can do multiple things that all independently solve the issue. That is, what if we do explicit metadata *and* have pip do wonky magic.
Yea, clever is often times broken, I can agree with that.
The other benefit to my proposal is that every existing use of setup_requires starts to get installed by pip instead of by easy_install, which solves a whole class of problems like not supporting Wheels, proxy settings, SSL settings, etc.
Yep, which is beneficial on its own. But not a reason not to do explicit metadata :).
Right, explicit metadata is absolutely the end goal. My concern is about adding more random cruft that we’ll have to support forever on the path to that.
Going back to the development requirement problem, I think it would be reasonable for setuptools to start to gain some of the concepts from PEP 426, it already has tests_requires and I think that an official dev_requires wouldn’t be a bad idea either. If it then exposed those things as something pip could inspect we could start doing things like automatically installing them inside of a development installation. This would probably even allow backwards compat by having a setup.py dynamically add things to the setup_requires based upon what version of setuptools is executing the setup.py. If it’s an older one, add a shim that’ll implement the new functionality as a plugin instead of part of core.
Ok, so what I need to know is what: - can I do - that solves my problem (not the other 1000 problems that PEP-426 solves) - that the setuptools // pip maintainers would be willing to merge.
I'm happy to put tuits into this, but I don't want to boil the ocean - I want to solve the specific thing that makes me curse my laptop screen on a regular basis.
I’ll come back to this if you can answer the above about how a setup.cfg solves the ``python setup.py …`` situation from my first paragraph, because that’s going to influence my thoughts. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA