[Distutils] setup_requires for dev environments

[Daniel Holth]

On Mon, Mar 16, 2015 at 3:19 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> On Mar 16, 2015, at 2:53 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> No one should be asked to learn how to extend distutils, and in
>> practice no one knows how.
>
> Some people know how, pytest figured it out, pbr figured it out. There’s
> some documentation at https://pythonhosted.org/setuptools/setuptools.html#extending-and-reusing-setuptools
>
> It is true though that it’s not particularly great documentation and
> actually doing it is somewhat of an annoyance.
>
>>
>> People have been begging for years for working setup_requires, far
>> longer than I've been interested in it, and all they want to do is
>>
>> import fetch_version
>> setup(version=fetch_version(), ...)
>>
>> Then they will eventually notice setup_requires has never worked the
>> way most people expect. As a result there are too few setup.py
>> abstractions.
>>
>> The other proposal is a little bit interesting. Parse setup.py without
>> running it, extract setup_requires, and pip install locally before
>> running the file? It would be easy as long as the setup_requires were
>> defined as a literal list in the setup() call, but you would have to
>> tell people they were not allowed to write normal clever Python code.
>> I think the gotchas would be severe…
>
> I wasn’t going to try and parse the setup.py, I was going to execute it.
>
> Here’s a proof of concept I started on awhile back to try and validate
> the overall idea: https://github.com/pypa/pip/compare/develop...dstufft:eldritch-horror
>
> Since then I’ve thought about it more and decided it’d probably be better
> to instead of trying to shuffle arguments into the subprocess, have the
> subprocess write out into a file or stdout or something what all of the
> setup_requires are. This would require executing the setup.py 3x instead
> of 2x like pip is currently doing.
>
> This would also enable people do something like:
>
> try:
>     import fetch_version
> except ImportError:
>     def fetch_version():
>         return “UNKNOWN”
>
> setup(version=fetch_version(), …)
>
> If they are happy with mandating that their thing can only be installed from
> sdist with pip newer than X, because given a three pass installation (once
> to discover setup_requires, once to write egg_info, once to actually install)
> as long as the setup_requires list doesn’t rely on anything installed then
> the first pass can have no real information except the setup_requires.
>
> It actually wouldn’t even really be completely broken, it’d just have a nonsense
> version number (or whatever piece of metadata can’t be located).

But it would still work in older pip if the setup requirements were
installed already? Users would have to try/catch every import? Explain
why this is better than reading out of a different file in the sdist,
no matter the format? Would it let you change your setup_requires with
Python code before the initial .egg-info is written out?

We've already talked too many times about my 34-line setup.py prefix
that does exactly what I want by parsing and installing requirements
from a config file, but people tend to complain about it not being a
pip feature. If it would help, I could have it accept a different
format, then it would be possible to publish
backwards-compatible-with-pip sdists that parsed some preferable
requirements format. If they were already installed the bw compat code
would do nothing.
https://bitbucket.org/dholth/setup-requires/src/03eda33c7681bc4102164c976e5a8cec3c4ffa9c/setup.py