[Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

On Tue, Feb 9, 2016 at 2:40 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 February 2016 at 21:38, Robert Collins <robertc at robertcollins.net> wrote:
>>> In the absence of a "pip sdist" command I can see that you're saying
>>> that we can implement pip's functionality without caring about this.
>>> But fundamentally, people upload sdists and wheels to PyPI. A build
>>> system that doesn't support sdists (which this PEP allows) will
>>> therefore have to publish wheel-only builds to PyPI, and I am very
>>> much against the idea of PyPI hosting "binary only" distributions.
>>
>> As you know - https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
>> - flit /already/ does this. If we want to require sdists as a thing,
>> PyPI needs to change to do that. I don't care whether it does or
>> doesn't myself: folk that want to not ship sources will always find a
>> way. I predict empty sdists, for instance.
>
> I know flit does this. We're obviously never going to be able to
> *stop* people generating wheels any way they like. But there is value
> in having a standard invocation that builds the wheel - anyone who has
> ever used make is aware that a standardised build command has value.
>
> At the moment, that standard is "pip wheel foo". Or maybe it's
> "setup.py bdist_wheel", but I prefer the layer of build system
> abstraction offered by pip.

Everyone agrees that we absolutely need to support "pip wheel
packagename" and and "pip wheel sdist.zip" and "pip wheel some-path/".
The (potential) disagreement is about whether the most primitive input
to "pip wheel" is an sdist or a directory tree.

I.e., option A is that "pip wheel some-path/" is implemented as "pip
sdist some-path/ && pip wheel some-path-sdist.zip", and option B is
"pip wheel sdist.zip" implemented as "unzip sdist.zip && pip wheel
sdist-tree/". Robert's draft PEP doesn't explicitly come down on
either side, but in practice I think that at least to start with it
would require the "option B" approach, since it doesn't provide any
"pip sdist" command.

>>> If project foo has no sdist, "pip wheel foo" for a platform where
>>> there's no compatible wheel available will fail, as there's no way for
>>> pip to locate the source.
>>
>> Thats true; but pip isn't a build system itself - and so far pip has
>> no role in the uploading of wheels to PyPI. Nor does - last I checked
>> - twine build things for you. The pattern is that developers prepare
>> there artifacts, then ask twine to upload them.
>
> As someone who frequently builds wheels for multiple projects, where
> those projects don't have complex build requirements but the projects
> themselves do not upload wheels, I find significant value in being
> able to just do "pip wheel foo". Maybe it's not pip's core
> functionality, maybe in the long term it'll be replaced by a separate
> utility, but the key point is that one command, given a project name,
> locates it on PyPI, downloads it and builds the source into a wheel.
>
> Having that capability is a *huge* benefit (I remember the days before
> distutils and PyPI, when every single package build was a process of
> locating files on obscure websites, and struggling with custom build
> processes - I'm admittedly over-sensitive about the risks of going
> back to that).
>
> I don't care in the slightest how build systems implement
> "<buildsystem> sdist". They can write a custom "setup.py" script that
> downloads the real sources from github and bulids them using flit, for
> all I care.
>
> What I do care about is ensuring that if a projects hasn't published a
> wheel for some particular environment, and the project is *not*
> actively trying to avoid publishing sources, then we provide them with
> a means to publish those sources in such a way that "pip wheel
> theproject" (or whatever command replaces it as the canonical "build a
> wheel" command) can automate the process of generating a wheel for the
> user's platform.

Both of the options under discussion give us all of these properties.
You can absolutely implement a "pip wheel" command that handles all
these cases without needing any "pip sdist" command.

> Of course projects who want to make their sources unavailable can, and
> will. I'm not trying to force them not to.
> Of course users can manually locate the sources and build from them. I
> believe that is a major step back in usability.
> Of course projects can upload wheels. They will never cover all platforms.
>
> Longer term, do we want a standard "source distribution" format? I
> thought we did (Nathaniel's proposals generated a lot of debate, but
> no-one was saying "let's just leave projects to do whatever they want,
> there's no point standardising"). If we do, then a standard process
> for building a wheel from that format seems like an obvious thing to
> expect. On the way to that goal, let's not start off by going
> backwards and abandoning the standard we currently have in "pip wheel
> project-with-source-on-pypi".
>
>>> So can we please revisit the question of whether build systems will be
>>> permitted to refuse to generate sdists? Note that I don't care whether
>>> we formally define a new sdist format, or go with something adhoc, or
>>> whatever. All I care about is that the PEP states that build systems
>>> must support generating a file that can be uploaded to PyPI and used
>>> by pip to build a wheel as described above (not "git clone this
>>> location and do 'pip wheel .'"). I think that making build systems
>>> expose how you make such a file by requiring a "sdist" subcommand is a
>>> reasonable approach, but I'm OK with naming the subcommand
>>> differently.
>>
>> I truely have no opinion here. I don't think it harms the abstract
>> build system to have it, but I do know that Donald very much does not
>> want pip to have to know or care about building sdists per se, and he
>> may see this as an attractive nuisance.
>
> As I said above, all I really care about is that "pip wheel foo"
> either continue to work for any project with sources published on
> PyPI, or that we have a replacement available before we
> deprecate/remove it.
>
> IMO, requiring build systems to support generating "something that pip
> can treat as a sdist" is a less onerous solution in the short term
> than requiring pip to work out how to support "pip wheel" to create
> wheels from source in the absence of a sdist command.

What's onerous about it? Right now sdists are just zipped up source
trees; the way you generate a wheel from an sdist is to unpack the
zipfile and then build from the resulting tree. If you *start* with a
tree, then the only reason you would need an sdist command is if you
want to zip up the tree and then unzip it again. There are reasons you
might want to do that, but skipping it would hardly be an extra burden
:-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org