[Distutils] setup_requires for dev environments

[Chris Barker - NOAA Federal]

Folks,

I'm having a hard time catching up with this, but maybe a few comments
from someone well outside the trenches will be helpful.

And note that one of use cases I'm now wrestling with is using another
package manager (conda), and setuptools is currently a PITA in that
context.

>> you that "python setup.py ..." should
>> install setup_requires? For me I'd be quite happy if installing the
>> requirements was my own problem in the absence of an installer.

Yes, yes, yes! Separation of concerns is key here -- the tool that
builds and/or installs one package shouldn't do ANYTHING to get me
dependencies, except maybe warn that they are not there. And raise a
sensible error if a build dependency is not there.

Particularly for developers, they really are capable of installing dependencies.

> I've no particular thoughts on that. It would certainly avoid the pain
> of easy_install being triggered.

Ahh! Is that why this is so painful? Not only is setuptools trying to
install stuff for me, but it's using easy_install to do so? Aargh!

> Success criteria for my immediate personal needs:
> - pip install -e . works on a clean checkout of my projects

Sure.

> - easy_install doesn't go and download stuff

easy_install doesn't do anything, ever!

> - my setup.py can refer to things (usually the version) inside the
> project itself, safely

Yeah, that would be nice.

A few other notes:

If I have this right, this thread, and a number of other issues are
triggered by the fact that setup() is not declarative -- i.e. You
don't have access to the metadata until it's been run.

But maybe we can kludge a declarative interface I top of the existing
system. Something like requiring:

Setup_params = a_big_dict_of_stuff_to_pass_to_setup

setup(**a_big_dict_of_stuff_to_pass_to_setup)

 Code could look for that big dict before running setup. If it's not
there, you don't get any new functionality.

Note that I'm wary of a completely declarative system, there ends up
being a lot if stuff you don't want to hard-code, so you have to start
building up a whole macro-expansion system, etc. I'd much rather
simply let the user build up a python data structure however they want
-- the default, simple cases would still be basic declarative
hard-coding.

I suppose it's too late now, but the really painful parts of all this
seem to be due to overly aggressive backward compatibility. We now
have wheels, but also eggs, we now have pip, but also easy_install,
etc.

Perhaps it's time to restore "distribute" -- setuptools without the
cruft. Folks could choose to use distribute (or maybe setuptools2)
instead of setuptools, and not get the cruft.

pip would, of course, still need to work with setuptools, and
setuptools would have to be maintained, but it would give us a path
forward out of the mess.

Another issue I don't see a way out of is that the package name that
you use to ask for a package, say on pypi, is not necessarily the name
of the python package you can import. So it's really tricky to check
if a package is installed independently of the package manager at
hand. This is the source of my conda issues -- conda installs the
dependencies, but setuptools doesn't know that, so it tries to do it
again -- ouch.

Final note: setuptools has always bugged me, even though it provides
some great features. I think all my struggles with it come down to a
key issue: it does not make clear distinctions between what should
happen at build-time vs install-time vs run-time.

For example:

I don't want it downloading and installing dependencies when I go to
build. That's an install- time task.

I don't want it selecting versions at run time--also an install time task.

There are others I can't recall -- but a couple years back I was
bundling up an application with py2exe and py2app, and found I had to
put an enormous amount of cruft in to satisfy setuptools at run time
(including setuptools itself) -- it was pretty painful.

And of course, using it within another package manager, like conda --
I really want it to build, and only build, I'm taking care of
dependencies another way.

OK, I've had my rant!

-Chris