[Distutils] moving things forward (was: wheel including files it shouldn't)

[Nick Coghlan]

On 5 May 2016 at 14:22, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, May 4, 2016 at 6:28 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 4 May 2016 at 23:00, Daniel Holth <dholth at gmail.com> wrote:
>>> +1 It would be great to start with a real setup_requires and probably would
>>> not interfere with later build system abstractions at all.
>>
>> If we're going to go down that path, perhaps it might make sense to
>> just define a standard [setup_requires] section in setup.cfg?
>>
>> Quite a few projects already have one of those thanks to distutiils2,
>> d2to1 and pbr, which means the pragmatic approach here might be to ask
>> what needs to change so the qualifier can be removed from this current
>> observation in the PBR docs: "The setup.cfg file is an ini-like file
>> that can mostly replace the setup.py file."
>>
>> The build system abstraction config could then also just be another
>> setup.cfg section.
>
> I'm sympathetic to the general approach, but on net I think I prefer a
> slightly different proposal.
>
> Downsides to just standardizing [setup_requires]:
>
> - if projects have existing ones, that's actually kinda good / kinda
> bad -- pip has never been respecting these before, so if we suddenly
> start doing that then existing stuff could break. I don't know how
> likely this is, but experience suggests that *something* will break
> and make someone angry.

(Robert & Nathaniel worked through this while I was writing it, but I
figured it was worth clarifying my expectations anyway)

I'm not aware of any current widely adopted systems that use a
"setup_requires" or comparable bootstrapping section in setup.cfg -
they pass setup_requires to the setup() call in setup.py. Examples:

pbr: http://docs.openstack.org/developer/pbr/#setup-py
d2to1: https://pypi.python.org/pypi/d2to1

I didn't find any clear examples like those for numpy.distutils, but
it seems it works the same way.

That's the whole problem: as long as people have to do their build
dependency bootstrapping *from* setup.py, module level imports in
setup.py don't work, and any replaced packages (such as a newer or
older setuptools) need to be injected into an already running Python
process.

The closest thing that we have today to a bootstrapping system that's
independent of easy_install that I'm aware of is Daniel Holth's
concept demonstrator that uses setup.py to read a multi-valued key
from setup.cfg, uses pip to install the extracted modules, and then
goes on to execute a "real-setup.py" in a subprocess:
https://bitbucket.org/dholth/setup-requires/src

The fact that the "Python-style ini format" used for setup.cfg is an
implementation defined format defined by the standard library's
configparser module rather than a properly specified language
independent format is certainly a downside, but we recently decided
we're OK with assuming "Python related build tools will be implemented
in Python (or at least have a Python shim)" when opting for a Python
API rather than a CLI as the preferred build system abstraction.

Changing that one assumption (i.e. that we're actually OK with using
Python-centric interoperability formats where they make pragmatic
sense) means that a number of other previously rejected ideas (like
"let's just use a new section or multi-valued key in setup.cfg") can
also be reconsidered.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia