[Distutils] A smaller step towards de-specializing setuptools/distutils

[Robert Collins]

On 2 November 2015 at 12:45, Nick Coghlan <ncoghlan at gmail.com> wrote:
> (Note: I'm still traveling, so I'm not fully caught up on all the
> threads, this one just caught my eye)
...
>> We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel.
>>
>> I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it.
>
> I think this is the key. If the *core* build-and-dependency-resolution
> is defined in terms of:
>
> * building and caching wheels (so they get built at most once per
> venv, and potentially per machine)
> * inspecting their metadata for dependencies
>
> then "dist-info" can be introduced later as an optimisation that
> *just* generates the wheel metadata, without actually doing the full
> binary build. For wheels we're downloading, we won't need to build
> them, and for wheels we're installing, we'll need to build them
> anyway.

... and for wheels we *might* install we'll have to build them unnecessarily.

I'm very worried that folk are underestimating the cost of complex
operations in a resolver context.

> That means we can move defining the interface for a dist-info
> subcommand off the critical path for decoupling the build system, and
> instead use the existing directory based metadata format defined for
> wheel files.

We can do that anyway, if we choose to encode the existing interface
rather than aim at a clean one. See
https://mail.python.org/pipermail/distutils-sig/2015-October/027464.html
for more discussion.

> However, I also think there's one refinement we can make that lets us
> drop the need for a copy-and-paste "setup.py", *without* needing to
> define a programmatic build system API: let setup.cfg define a module
> name to invoke with "python -m <module>" instead of running
> "setup.py".

This is now back to a bootstrap API which is what my PEP describes,
just with different syntax wrapped around it.

You can equally take my PEP and rather than piping metadata on stdout,
have it write a directory and consume that.

But - my PEP is stalled on the decision between what flit folk have
asked for, and what Donald is concerned about (see the mail in the
link above).

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud