[Distutils] Update to my skeletal PEP for a new build system interface

[Nathaniel Smith]

On Mon, Nov 9, 2015 at 7:34 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 November 2015 at 05:20, Nathaniel Smith <njs at pobox.com> wrote:
>> A *source tree* is something like a VCS checkout. We need a standard
>> interface for installing from this format, to support usages like
>> ``pip install some-directory/``.
>
> I still find these two definitions unhelpful, sorry.
>
> We don't *need* an interface to install from a source tree. It's
> entirely feasible to have a standard interface to build a sdist from a
> source tree and go source tree -> sdist -> wheel -> install. That
> doesn't cater for editable installs, nor does it cater for reusing
> things like object files from previous builds, so there may be
> *benefits* to having a richer interface than this, but it's wrong to
> say it's needed.

I am confuse. All that sentence is saying is that (a) it is useful to
have the phrase "source tree" as distinct from "sdist" so we can talk
about them (which I assume you agree about because you use that phrase
in your response :-)), and (b) there must be *some* interface that
allows people to type "pip install some-directory/" and have it work
because that's a feature we have to support (which I assume you agree
about because you immediately propose an interface for supporting that
feature).

It sounds like we do disagree about the details of what this interface
should look like and thus how "pip install some-directory/" should
work internally, but that's not a problem with the definition (or
indeed something that this PEP's text currently takes any stance on at
all :-)).

> I suspect you're reluctant to require a "source tree -> sdist"
> interface, because the author of flit isn't comfortable with having
> such a thing. That's OK - if you want to note that a benefit of going
> direct to install (or wheel) is that tools that don't allow you to
> create a sdist are supported, then let's make that explicit. Expect
> plenty of pushback on the idea of tools that don't supply sdists
> though...

I actually haven't talked to Thomas about this particular point at
all, and actually part of what started all this was my looking at flit
and going "this is cool, but c'mon, you can't just throw away sdists"
:-).

The reason I'm reluctant to require a "source tree -> sdist" interface
is described here:
    https://mail.python.org/pipermail/distutils-sig/2015-November/027636.html

and also at the very top of this long email (which for some reason I
can't seem to find in the mail.python.org archives?):
    https://www.mail-archive.com/distutils-sig@python.org/msg23144.html

The TL;DR is: obviously we need source tree -> sdist operations
somewhere, and obviously we need mechanisms to increase the
reliability of builds -- we all agree that there's some irreducible
complexity there, those issues need to be addressed, the question is
just where to put that complexity. I think putting it into the PEP for
the build frontend <-> build backend interface is the wrong place,
because it increases spec complexity (the worst kind of complexity)
and it rules out the useful feature of incremental rebuilds. (And by
"useful feature" there I mean "if we regress from distutils by failing
to support this, then there's a good chance downstream devs will
simply refuse to use our new design".)

>> A *source distribution* is a static snapshot representing a particular
>> release of some source code, like ``lxml-3.4.4.zip``. Source
>> distributions serve many purposes: they form an archival record of
>> releases, they provide a stupid-simple de facto standard for tools
>> that want to ingest and process large corpora of code, possibly
>> written in many languages (e.g. code search), they act as the input to
>> downstream packaging systems like Debian/Fedora/Conda/..., and so
>> forth. In the Python ecosystem they additionally have a particularly
>> important role to play, because packaging tools like ``pip`` are able
>> to use source distributions to fulfill binary dependencies, e.g. if
>> there is a distribution ``foo.whl`` which declares a dependency on
>> ``bar``, then we need to support the case where ``pip install bar`` or
>> ``pip install foo`` automatically locates the sdist for ``bar``,
>> downloads it, builds it, and installs the resulting package.
>>
>> Source distributions are also known as *sdists* for short.
>
> One key feature of the current sdists that you are either overlooking
> or ignoring is that they can, and do, contain *built* files. The best
> example is projects using Cython. The sdist contains generated C
> files, so that users building wheels from the sdist don't need cython
> installed.
>
> Certainly your definition of a sdist is general enough that it doesn't
> preclude such things. But on the other hand, it doesn't offer any
> suggestion that this is an important feature of a sdist (and it is - I
> say that as someone who has needed to build wheels from a sdist and
> doesn't have Cython installed). From your definition, people will
> infer that zipping up a development directory makes a sdist, and so
> that's what they'll do. Because after all, making Cython a build
> requirement and generating the C at build time is *also* an option,
> it's just not as friendly to the average user.

Hmm, I certainly agree that it doesn't preclude such things, because I
am very aware of this use case (I maintain projects that handle Cython
in exactly the way you describe), and it never occurred to me that
this could *not* be supported :-). I'm not sure what you're worried
about exactly? Right now, zipping up a development directory actually
is a valid way of making an sdist, and nonetheless projects actually
do go to elaborate lengths to trick distutils into including generated
.c files. So I don't think it's likely they'll stop because of some
PEP that neglected to explicitly point out that this was possible :-).
But if you think the wording could be improved I'm certainly open to
that.

(I guess I do have some generic preference that we not insist on PEPs
serving as end-user documentation -- the intended audience here is
experts, the definitions are written to mean exactly what they say,
etc., and there are real trade-offs between being precise and being
easily comprehensible by non-experts. But I also would like you to be
happy :-).)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org