[Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 10 February 2016 at 01:19, Robert Collins <robertc at robertcollins.net> wrote:
> On 10 February 2016 at 13:09, Paul Moore <p.f.moore at gmail.com> wrote:
>> [I need to read and digest the rest of this, but it's late here, so
>> that will be tomorrow]
>
> OK, cool.

Right, I've been thinking about this a bit more, and I've re-read the
PEP. I wasn't at all clear in my concern - in my own mind, and as a
consequence in how I explained the issue. My apologies.

I don't have any problem with the basic proposed interface, and I see
what you mean that it doesn't need to require a "sdist" command. I was
wrong on that. But the proposal is written in terms of how pip "works
from a Python source tree". The example used throughout the PEP is a
local source directory. While that's an easy and understandable
example to use, it glosses over the detail of how pip will *get* such
a source tree.

What the PEP omits, it seems to me, is an explanation of how pip will
get a "source tree" in the form defined by the PEP. In essence, there
is no transition plan in the PEP. At present, pip sets up a source
tree by downloading a sdist from PyPI and unpacking it. The PEP
explains that sources without a pypa.json should be treated as current
setuptools-style sdists for backward compatibility, but offers no
guidance on how projects (or build tools, on behalf of projects that
use them) can transition to a pypa.json-based approach.

That's really what I'm trying to get at with my focus on "source
distributions". As a project author, how do I switch to this new
format? What do I publish? Publishing binaries in the form of wheels
is irrelevant - projects using flit or other build tools can do that
right now. This PEP is entirely about enabling them to publish
*source* using new build tools, and yet it doesn't cover the
publishing side at all.

So while the PEP is fine, it's incomplete from the user's point of
view, and my concern is that if we finalise it in its current form,
people will have to roll their own publishing solutions, and we'll
then be playing "catch up" trying to define a standard that takes into
account the approaches people have developed in the meantime. And as a
user, I'll then be left with projects I can't easily build wheels for.
It's all very well to say "the project should build wheels", and I
wish they would, but not all do or will - I care because one of my
ongoing side projects is an automated wheel build farm (for simple
cases only, don't anyone get their hopes up! ;-)) and being able to
build wheels in a standard way is key to that.

We don't have to solve the whole "sdist 2.0" issue right now. Simply
saying that in order to publish pypa.json-based source trees you need
to zip up the source directory, name the file "project-version.zip"
and upload to PyPI, would be sufficient as a short-term answer
(assuming that this *would* be a viable "source file" that pip could
use - and I must be clear that I *haven't checked this*!!!) until
something like Nathaniel's source distribution proposal, or a
full-blown sdist-2.0 spec, is available. We'd need to support whatever
stopgap proposal we recommend for backward compatibility in those new
proposals, but that's a necessary cost of not wanting to delay the
current PEP on those other ones.

Hope this is clearer.
Paul