[Distutils] A possible refactor/streamlining of PEP 517

[Nathaniel Smith]

On Tue, Jul 4, 2017 at 11:03 AM, Donald Stufft <donald at stufft.io> wrote:
>
> I don’t think it’s (entirely) rehashing. The discussion has made me realize
> that the purported cases covered by the hook aren’t actually going to be
> covered except in a narrow set of circumstances, which suggests that it’s
> not actually a good hook.

The thought pattern that led to my counter-proposal is basically:

This prepare_build_files hooks feels half-baked – the pip devs would
just as rather leave it out, and if you look at Thomas's rationale for
why it's important, it's to let him circumvent pip in the cases pip
doesn't want him to circumvent, while he also points out that it adds
overhead for no reason. I really honestly don't mind if we add
something like this, but what we have now does *not* feel like an
obvious one-way-to-do-it design that's ready for standardization. It
feels like the sort of thing that we'll look back on in a few years
and be like "why is this the way it is?" and the answer will be "well,
you kinda had to be there".

Okay, so is there any way to get the half-baked part out of the PEP
517 critical path? The idea came up of adding an extension namespace,
and it seems to me that this is obviously a good idea totally
independent of the specifics of which hooks make the cut for PEP 517.
Nick has characterized it as being like the PEP 225 approach to matrix
multiplication (PEP 225 proposed to add 6 new infix operators to
Python with semantics to be specified later). But the inspiration is
really more like - you know how people pop up on python-ideas saying
"hey X should be added to the stdlib", and the response is "well, put
it on PyPI first to work it out, and then we can talk"? You can't do
that for infix operators, which is part of why it took *14 years* to
sort out matrix multiplication. If we ever want to extend the build
backend interface again, having some way to get that "try it first,
standardize second" workflow is obviously helpful. And all it requires
is reserving the 'extensions' attribute on backends, which has *zero*
cost in terms of spec complexity or added code.

Really that's enough on its own; when pip gets around to switching
from copytree->sdist, they can hash out some prototype with Thomas,
ship the new feature without breaking the world, and standardize
whenever makes sense.

Alternatively, there's the NotImplemented proposal, which is also
sufficient to fix the immediate issue on its own. TBH I'm pretty
confused about the reaction its gotten. The reason I like it is that
we all seem to agree that "this backend has determined dynamically
that it can't build an sdist" is an important case that we want to
support. So a straightforward way of representing that in our spec
seems like a nice future-proof thing. It's simple, and it makes sense
totally independently of the details of pip's build pipeline. No-one's
going to look back later and be like "why did you add a
straightforward way for the frontend and backend to communicate about
this thing that can happen?". But... for some reason everyone seems to
think that it's very important that PEP 517 *handle* this case, but
that also that it *pretend that this case doesn't exist*. I don't get
it.

Is the issue the slightly-weird use of a sentinel value
(NotImplemented) instead of raising an error (like declaring that
NotImplementedError is the standard way to indicate this issue?). I
chose a sentinel because it avoids the chance of a bug in the hook
causing an internal exception to leak out and be misinterpreted by the
frontend (which I guess is why dunder methods use NotImplemented for
exactly the analogous situation), but it doesn't really matter that
much.

Anyway, both of these proposals seem *obvious* in a way that
prepare_build_files just isn't, and either one is sufficient to make
PEP 517 workable.

(Re: prepare_build_metadata: I basically agree with Donald that this
is probably something we will add eventually, and it doesn't matter a
lot whether we do it now or later. I went ahead and dropped it from my
version of the proposal because I was trying to simplify to just the
core proposal and it's very easy to add later, but really it's the
issues around prepare_build_files where there's disagreement.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org