[Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 10 February 2016 at 11:40, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 February 2016 at 21:38, Robert Collins <robertc at robertcollins.net> wrote:
>>> In the absence of a "pip sdist" command I can see that you're saying
>>> that we can implement pip's functionality without caring about this.
>>> But fundamentally, people upload sdists and wheels to PyPI. A build
>>> system that doesn't support sdists (which this PEP allows) will
>>> therefore have to publish wheel-only builds to PyPI, and I am very
>>> much against the idea of PyPI hosting "binary only" distributions.
>>
>> As you know - https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
>> - flit /already/ does this. If we want to require sdists as a thing,
>> PyPI needs to change to do that. I don't care whether it does or
>> doesn't myself: folk that want to not ship sources will always find a
>> way. I predict empty sdists, for instance.
>
> I know flit does this. We're obviously never going to be able to
> *stop* people generating wheels any way they like. But there is value
> in having a standard invocation that builds the wheel - anyone who has
> ever used make is aware that a standardised build command has value.
>
> At the moment, that standard is "pip wheel foo". Or maybe it's
> "setup.py bdist_wheel", but I prefer the layer of build system
> abstraction offered by pip.

But 'pip wheel foo' already reuses existing wheels: things like flit
that upload universal wheels work with 'pip wheel foo' just fine (as
long as --no-binary has not been passed). Things that upload an sdist
will also still work, unchanged, with this proposal.

>>> If project foo has no sdist, "pip wheel foo" for a platform where
>>> there's no compatible wheel available will fail, as there's no way for
>>> pip to locate the source.
>>
>> Thats true; but pip isn't a build system itself - and so far pip has
>> no role in the uploading of wheels to PyPI. Nor does - last I checked
>> - twine build things for you. The pattern is that developers prepare
>> there artifacts, then ask twine to upload them.
>
> As someone who frequently builds wheels for multiple projects, where
> those projects don't have complex build requirements but the projects
> themselves do not upload wheels, I find significant value in being
> able to just do "pip wheel foo". Maybe it's not pip's core
> functionality, maybe in the long term it'll be replaced by a separate
> utility, but the key point is that one command, given a project name,
> locates it on PyPI, downloads it and builds the source into a wheel.
>
> Having that capability is a *huge* benefit (I remember the days before
> distutils and PyPI, when every single package build was a process of
> locating files on obscure websites, and struggling with custom build
> processes - I'm admittedly over-sensitive about the risks of going
> back to that).
>
> I don't care in the slightest how build systems implement
> "<buildsystem> sdist". They can write a custom "setup.py" script that
> downloads the real sources from github and bulids them using flit, for
> all I care.

I'm struggling to connect 'pip wheel foo' to 'they can do an sdist'.
'pip wheel foo' does not invoke sdist. There is a bug proposing that
it should ( https://github.com/pypa/pip/issues/2195 and
https://github.com/pypa/pip/pull/3219 ) - but that proposal is
contentious - see the ongoing related debate about editable mode /
incremental builds and also the discussion on PR 3219.

> What I do care about is ensuring that if a projects hasn't published a
> wheel for some particular environment, and the project is *not*
> actively trying to avoid publishing sources, then we provide them with
> a means to publish those sources in such a way that "pip wheel
> theproject" (or whatever command replaces it as the canonical "build a
> wheel" command) can automate the process of generating a wheel for the
> user's platform.

Sure, they should have that. I don't understand why pip's
*abstraction* over project *building* should know about that. Issue
2195 aside, which seems to be entirely about avoiding one particular
developer failure mode, and other than that makes no sense to me.


> Of course projects who want to make their sources unavailable can, and
> will. I'm not trying to force them not to.
> Of course users can manually locate the sources and build from them. I
> believe that is a major step back in usability.
> Of course projects can upload wheels. They will never cover all platforms.
>
> Longer term, do we want a standard "source distribution" format? I
> thought we did (Nathaniel's proposals generated a lot of debate, but
> no-one was saying "let's just leave projects to do whatever they want,
> there's no point standardising"). If we do, then a standard process
> for building a wheel from that format seems like an obvious thing to
> expect. On the way to that goal, let's not start off by going
> backwards and abandoning the standard we currently have in "pip wheel
> project-with-source-on-pypi".

We certainly want to have any metadata published in an sdist be
trustable, which isn't a format change thing per se - but in a sense,
my proposal is a new sdist format: rather than setup.py, it has
pypa.json, the rest is the same.

There's never been any guarantee that an sdist can produce a new sdist
:- and there are lots of ways that that can break, already.

>>> So can we please revisit the question of whether build systems will be
>>> permitted to refuse to generate sdists? Note that I don't care whether
>>> we formally define a new sdist format, or go with something adhoc, or
>>> whatever. All I care about is that the PEP states that build systems
>>> must support generating a file that can be uploaded to PyPI and used
>>> by pip to build a wheel as described above (not "git clone this
>>> location and do 'pip wheel .'"). I think that making build systems
>>> expose how you make such a file by requiring a "sdist" subcommand is a
>>> reasonable approach, but I'm OK with naming the subcommand
>>> differently.
>>
>> I truely have no opinion here. I don't think it harms the abstract
>> build system to have it, but I do know that Donald very much does not
>> want pip to have to know or care about building sdists per se, and he
>> may see this as an attractive nuisance.
>
> As I said above, all I really care about is that "pip wheel foo"
> either continue to work for any project with sources published on
> PyPI, or that we have a replacement available before we
> deprecate/remove it.

The current proposal doesn't stop it working,and doesn't make it any
easier for it to stop working. It is, AFAICT, unrelated / orthogonal.

> IMO, requiring build systems to support generating "something that pip
> can treat as a sdist" is a less onerous solution in the short term

Which we already do/have. Remember the abstraction is not the build
system. Its not intended to - and it would be counterproductive for it
to be - an end user interface to build systems. They are too varied,
with too many different styles, for consolidating all that as an end
user tool to make sense.

> than requiring pip to work out how to support "pip wheel" to create
> wheels from source in the absence of a sdist command. Longer term,
> we'll have to work out how to move away from the current sdist format,
> but I can't conceive of a way that we can have a standard tool
> (whether pip or something else) that builds wheels from PyPI-hosted
> sources unless we have *some* commonly-agreed format - and the "build
> system generate-a-source-file command" can remain as an interface for
> creating that format.
>
>> Who / what tool do we expect to use the sdist command in the abstract interface?
>
> In the short term, "pip wheel foo".
> Longer term, if we want to remove that functionality from pip, then
> whatever command we create to replace it.

So - to be clear - you're proposing that PR 3219 be merged?

> I do not accept any proposal that removes "pip wheel <source>" without
> providing *any* replacement.

But the current proposal *DOES NOT REMOVE IT*.

We're clearly miscommunicating about something :).

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud