[Distutils] A smaller step towards de-specializing setuptools/distutils

[Nick Coghlan]

(Note: I'm still traveling, so I'm not fully caught up on all the
threads, this one just caught my eye)

On 30 October 2015 at 00:05, Donald Stufft <donald at stufft.io> wrote:
> On October 29, 2015 at 6:59:00 PM, Robert Collins (robertc at robertcollins.net) wrote:
>> On 30 October 2015 at 11:50, Robert Collins wrote:
>> ...
>> > However, the big step you're proposing that I think is fundamentally
>> > unsound is that of requiring a wheel be built before dependencies can
>> > be queried. Wheels are *expensive* to build in enough cases that I
>> > really believe we have a choice between being able to fix issue 988 or
>> > utilising wheels as the interface to get at install metadata. Remember
>> > too that the numpy ABI discussion is going to cause install
>> > dependencies to depend on the version of numpy present at build-time,
>> > so the resolver is going to have to re-evaluate the dependencies for
>> > those distributions when it selects a new numpy version: so its not
>> > even a one-time-per-package cost, its a
>> > one-time-per-arc-of-the-N!-graph-paths cost.
>>
>> On further thought there is one mitigating factor here - we'd have to
>> be building numpy fully anyway for the cases where numpy is a
>> build-dependency for things further up the stack, so even in my most
>> paranoid world we will *anyway* have to pay that cost - but we can
>> avoid the cost of multiple full rebuilds of C extensions built on top
>> of numpy. I think avoiding that cost is still worth it, its just a
>> little more into the grey area rather than being absolutely obviously
>> worth it.
>>
>
> We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel.
>
> I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it.

I think this is the key. If the *core* build-and-dependency-resolution
is defined in terms of:

* building and caching wheels (so they get built at most once per
venv, and potentially per machine)
* inspecting their metadata for dependencies

then "dist-info" can be introduced later as an optimisation that
*just* generates the wheel metadata, without actually doing the full
binary build. For wheels we're downloading, we won't need to build
them, and for wheels we're installing, we'll need to build them
anyway.

That means we can move defining the interface for a dist-info
subcommand off the critical path for decoupling the build system, and
instead use the existing directory based metadata format defined for
wheel files.

However, I also think there's one refinement we can make that lets us
drop the need for a copy-and-paste "setup.py", *without* needing to
define a programmatic build system API: let setup.cfg define a module
name to invoke with "python -m <module>" instead of running
"setup.py".

That way, instead of the static setup.py needing to be copied and
pasted into each project, there's just another line in the config file
like:

    [bootstrap]
    setup_requires = my_build_system
    setup_cli_module = my_build_system.setup_cli

If setup_cli_module isn't specified, then pip will expect to find a
setup.py in the project directory.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia