[Distutils] A smaller step towards de-specializing setuptools/distutils

[Donald Stufft]

On October 29, 2015 at 6:59:00 PM, Robert Collins (robertc at robertcollins.net) wrote:
> On 30 October 2015 at 11:50, Robert Collins wrote:
> ...
> > However, the big step you're proposing that I think is fundamentally
> > unsound is that of requiring a wheel be built before dependencies can
> > be queried. Wheels are *expensive* to build in enough cases that I
> > really believe we have a choice between being able to fix issue 988 or
> > utilising wheels as the interface to get at install metadata. Remember
> > too that the numpy ABI discussion is going to cause install
> > dependencies to depend on the version of numpy present at build-time,
> > so the resolver is going to have to re-evaluate the dependencies for
> > those distributions when it selects a new numpy version: so its not
> > even a one-time-per-package cost, its a
> > one-time-per-arc-of-the-N!-graph-paths cost.
>  
> On further thought there is one mitigating factor here - we'd have to
> be building numpy fully anyway for the cases where numpy is a
> build-dependency for things further up the stack, so even in my most
> paranoid world we will *anyway* have to pay that cost - but we can
> avoid the cost of multiple full rebuilds of C extensions built on top
> of numpy. I think avoiding that cost is still worth it, its just a
> little more into the grey area rather than being absolutely obviously
> worth it.
>  

We’d only need to pay that cost on the assumption that we don’t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel. 

I mentioned it on IRC, but just for the folks who aren’t on IRC, I’m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn’t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA