[Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

On Tue, Jan 26, 2016 at 9:39 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On 27 January 2016 at 16:37, Nathaniel Smith <njs at pobox.com> wrote:
>> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft <donald at stufft.io> wrote:
>>> As many know there has been an effort to get a generalized interface that
>>> a build system can implement so that we can break the hard dependency on
>>> setuptools that Python packaging currently has. After a lot of deliberation
>>> and threads back and forth (as well as video meetings!) I think that the
>>> version of the PEP that is currently on the PR in
>>> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s
>>> generally the right thing to move forward with. I made a few little comments
>>> but overall I think it’s there and we’re ready to move forward on trying to
>>> get a reference implementation done that can validate the decisions made in
>>> that PEP (and then, hopefully finalize the PEP and merge those
>>> implementations).
>>>
>>> So many thanks to everyone involved in hammering this out thus far :) I’m
>>> nervous but excited about the possibility of making setuptools just another
>>> build system.
>>
>> AFAICR at the time the set of public threads petered out (I guess because
>> Robert and I both got busy with other things), there were several
>> fundamental disagreements still unresolved:
>
> There were two pep drafts in flight, but I would hesitate to call it a
> fundamental disagreement :).

Yeah, that may be overstating it :-). I guess I mean something like
"basic design issues that people were still arguing about". (As
compared to stuff like "what file format should we use for the config
file" where it's worth talking about but fundamentally it just doesn't
really matter.)

>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm pretty sure there was another big subthread about this just before
things petered out, but yeah, fair enough. And as far as I'm concerned
there's only one possible answer here since if pip insists on
copy-before-build then all the projects I'm involved in will simply
refuse to use pip for building, so I certainly don't object to
anything in your draft in this regard :-).

>> - Robert was in favor of a command-line based interface; I favored a hook
>> based interface. I won't try to summarize the disagreement here b/c I'm
>> unable to articulate any advantages of the command-line based interface :-)
>
> it has the fairly large advantage that the developers of pip are in
> favour of it - as the folk that will have to deal with and debug what
> goes on, thats more than sufficient for me.

Can you link to where they said this?

>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>> as-is, and in particular keeping the current behavior where it's the build
>> system's job to pick which directory the .egg-link file should be installed
>> into and then do the actual work; I favored an interface where the build
>> system says "here's the path to a directory with the package and metadata",
>> and then the installation tool (pip) picks the location and installs the
>> .egg-link file (or whatever) pointing to that directory. The argument in
>> favor of the current semantics is that hey, we need to get this show on the
>> road, and this will require less code added to pip. The information is
>> against is that the current design gets the separation of concerns wrong,
>> impossible to implement without relying on unstandardized features like
>> egg-link, and also I don't think it actually works right currently (e.g. if
>> the user supplies --user or --root to the pip install -e command, then what
>> happens? But I could be wrong -- haven't had a chance to fully reverse
>> engineer pip/setuptools interaction here).
>
> That feels like an exaggeration of my position. I'd love to evolve
> those semantics. But since such an evolution is a hard API break
> across everything, I don't want to make fixing the much more
> significant problems of setup_requires and having to use setuptools
> conditional on that other thing: its a massive adoption hurdle I
> suspect.

Are you concerned that this would make it impossible to do a simple
shim to run old setuptools projects under the new interface? I think
this part is fine: basically for the hook that I propose, a
PEPXX->setup.py shim would implement it by running

  setup.py build_ext -i && setup.py egg_info

which sets up the current directory to be (a) importable, and (b) have
metadata, and then pip would be responsible for installing a .egg-link
pointing to this directory. IIUC this is exactly what 'develop'
currently does, except that it creates the .egg-link itself.

Doh, and of course that was the other substantive disagreement:
whether to mandate egg_info.

Well, at least now it looks like we can forget about that whole debate
about the need to quickly get metadata from scipy sdists, since
apparently scipy will have wheels on all supported platforms before
pip implements either the new build system interface or the resolver
:-). Of course there will still be an issue for more obscure platforms
and for projects whose developers can't be arsed to provide wheels.

[...]
> No; there was the one video call, which was advertised here, and then
> the subsequent drafts. The PR hasn't been changed in a couple of
> months: in that time we've had Xmas, various work things, hacking
> older pip version compat stuff into pbr, and me nagging Donald a lot
> to get a review of the PR :).

Plus all your great work on getting PEP 508 sorted out so it wouldn't
block this build system stuff :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org