[Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 9 February 2016 at 21:38, Robert Collins <robertc at robertcollins.net> wrote:
>> In the absence of a "pip sdist" command I can see that you're saying
>> that we can implement pip's functionality without caring about this.
>> But fundamentally, people upload sdists and wheels to PyPI. A build
>> system that doesn't support sdists (which this PEP allows) will
>> therefore have to publish wheel-only builds to PyPI, and I am very
>> much against the idea of PyPI hosting "binary only" distributions.
>
> As you know - https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
> - flit /already/ does this. If we want to require sdists as a thing,
> PyPI needs to change to do that. I don't care whether it does or
> doesn't myself: folk that want to not ship sources will always find a
> way. I predict empty sdists, for instance.

I know flit does this. We're obviously never going to be able to
*stop* people generating wheels any way they like. But there is value
in having a standard invocation that builds the wheel - anyone who has
ever used make is aware that a standardised build command has value.

At the moment, that standard is "pip wheel foo". Or maybe it's
"setup.py bdist_wheel", but I prefer the layer of build system
abstraction offered by pip.

>
>> If project foo has no sdist, "pip wheel foo" for a platform where
>> there's no compatible wheel available will fail, as there's no way for
>> pip to locate the source.
>
> Thats true; but pip isn't a build system itself - and so far pip has
> no role in the uploading of wheels to PyPI. Nor does - last I checked
> - twine build things for you. The pattern is that developers prepare
> there artifacts, then ask twine to upload them.

As someone who frequently builds wheels for multiple projects, where
those projects don't have complex build requirements but the projects
themselves do not upload wheels, I find significant value in being
able to just do "pip wheel foo". Maybe it's not pip's core
functionality, maybe in the long term it'll be replaced by a separate
utility, but the key point is that one command, given a project name,
locates it on PyPI, downloads it and builds the source into a wheel.

Having that capability is a *huge* benefit (I remember the days before
distutils and PyPI, when every single package build was a process of
locating files on obscure websites, and struggling with custom build
processes - I'm admittedly over-sensitive about the risks of going
back to that).

I don't care in the slightest how build systems implement
"<buildsystem> sdist". They can write a custom "setup.py" script that
downloads the real sources from github and bulids them using flit, for
all I care.

What I do care about is ensuring that if a projects hasn't published a
wheel for some particular environment, and the project is *not*
actively trying to avoid publishing sources, then we provide them with
a means to publish those sources in such a way that "pip wheel
theproject" (or whatever command replaces it as the canonical "build a
wheel" command) can automate the process of generating a wheel for the
user's platform.

Of course projects who want to make their sources unavailable can, and
will. I'm not trying to force them not to.
Of course users can manually locate the sources and build from them. I
believe that is a major step back in usability.
Of course projects can upload wheels. They will never cover all platforms.

Longer term, do we want a standard "source distribution" format? I
thought we did (Nathaniel's proposals generated a lot of debate, but
no-one was saying "let's just leave projects to do whatever they want,
there's no point standardising"). If we do, then a standard process
for building a wheel from that format seems like an obvious thing to
expect. On the way to that goal, let's not start off by going
backwards and abandoning the standard we currently have in "pip wheel
project-with-source-on-pypi".

>> So can we please revisit the question of whether build systems will be
>> permitted to refuse to generate sdists? Note that I don't care whether
>> we formally define a new sdist format, or go with something adhoc, or
>> whatever. All I care about is that the PEP states that build systems
>> must support generating a file that can be uploaded to PyPI and used
>> by pip to build a wheel as described above (not "git clone this
>> location and do 'pip wheel .'"). I think that making build systems
>> expose how you make such a file by requiring a "sdist" subcommand is a
>> reasonable approach, but I'm OK with naming the subcommand
>> differently.
>
> I truely have no opinion here. I don't think it harms the abstract
> build system to have it, but I do know that Donald very much does not
> want pip to have to know or care about building sdists per se, and he
> may see this as an attractive nuisance.

As I said above, all I really care about is that "pip wheel foo"
either continue to work for any project with sources published on
PyPI, or that we have a replacement available before we
deprecate/remove it.

IMO, requiring build systems to support generating "something that pip
can treat as a sdist" is a less onerous solution in the short term
than requiring pip to work out how to support "pip wheel" to create
wheels from source in the absence of a sdist command. Longer term,
we'll have to work out how to move away from the current sdist format,
but I can't conceive of a way that we can have a standard tool
(whether pip or something else) that builds wheels from PyPI-hosted
sources unless we have *some* commonly-agreed format - and the "build
system generate-a-source-file command" can remain as an interface for
creating that format.

> Who / what tool do we expect to use the sdist command in the abstract interface?

In the short term, "pip wheel foo".
Longer term, if we want to remove that functionality from pip, then
whatever command we create to replace it.

I do not accept any proposal that removes "pip wheel <source>" without
providing *any* replacement.

>> PS I agree with Nathaniel, insofar as there were definitely a number
>> of unresolved points remaining after the various public discussions,
>> and I don't think it's reasonable to say that any proposal is "ready
>> for implementation" without a public confirmation that those issues
>> have been resolved.
>
> I suspect it had been quescient too long and folk had paged out the
> state. Certainly I don't think Donald intended to bypass any community
> debate, and neither did I.

No problem - I didn't expect that either of you had any such
intention. I just wanted to raise a red flag that (at least in my
view) there were unanswered concerns about the proposal that needed to
be addressed before it could be signed off. Donald's note certainly
gave the impression that he had forgotten about those issues.

Paul