[Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 9 February 2016 at 22:59, Nathaniel Smith <njs at pobox.com> wrote:
>> At the moment, that standard is "pip wheel foo". Or maybe it's
>> "setup.py bdist_wheel", but I prefer the layer of build system
>> abstraction offered by pip.
>
> Everyone agrees that we absolutely need to support "pip wheel
> packagename" and and "pip wheel sdist.zip" and "pip wheel some-path/".
> The (potential) disagreement is about whether the most primitive input
> to "pip wheel" is an sdist or a directory tree.
>
> I.e., option A is that "pip wheel some-path/" is implemented as "pip
> sdist some-path/ && pip wheel some-path-sdist.zip", and option B is
> "pip wheel sdist.zip" implemented as "unzip sdist.zip && pip wheel
> sdist-tree/". Robert's draft PEP doesn't explicitly come down on
> either side, but in practice I think that at least to start with it
> would require the "option B" approach, since it doesn't provide any
> "pip sdist" command.

Hmm, I see what you're getting at.

But if project X uses flit, what do they need to do to upload
something to PyPI so that "pip wheel X" builds a wheel? Let's assume
they aren't actively trying to hide their sources, but equally they
aren't looking to do a load of work beyond their normal workflow just
to support that usage.

And if a project currently using setuptools wants to switch to flit,
how do they upload what's needed to allow pip to install on platforms
where they don't supply wheels?

Maybe that's not functionality pip *needs* to do its job, but it seems
far too likely to me that projects will build and upload wheels for
some platforms, and then say "to get the source go to <our github
site>". With luck they'll include build instructions, although they
may just be "git clone <url>; pip wheel ."

I have set up a number of systems that automate wheel builds - these
days, that's as simple as putting "pip wheel -r mypackages.txt" in a
scheduled job. I do *not* want to have to go back to maintaining a
mapping from project to source URL, or working out how to get things
to work on systems without git installed and how to detect the correct
way of getting the latest release version from a source repo.

Documenting that build systems are required to define how they support
the operation of "build a release source file that can be uploaded to
PyPI" makes it clear that we consider this to be an important
operation. Even if it's not technically needed for pip to operate.

>> What I do care about is ensuring that if a projects hasn't published a
>> wheel for some particular environment, and the project is *not*
>> actively trying to avoid publishing sources, then we provide them with
>> a means to publish those sources in such a way that "pip wheel
>> theproject" (or whatever command replaces it as the canonical "build a
>> wheel" command) can automate the process of generating a wheel for the
>> user's platform.
>
> Both of the options under discussion give us all of these properties.
> You can absolutely implement a "pip wheel" command that handles all
> these cases without needing any "pip sdist" command.

I don't think (bit it is some time ago, and I haven't reread the
thread yet) I ever said that I want "pip sdist" to be mandated. (I
have no real opinion on that). What I want is that we make it clear
that build systems need to provide a means to create "a source bundle
that can be uploaded to PyPI that pip can use to build wheels" (I'll
buy a beer for anyone who can come up with an acceptable name for this
that doesn't make people thing "but we want to get away from the sdist
format" :-))

I also want people to agree that it's a good thing to expect build
systems to support creating such a source artifact.

I'm aware that this "source artifact" may well be what you were
proposing be a "directory tree". That's another debate that as far as
I know petered out without conclusion. I'm trying to keep the two
discussions separate - for the purposes of *this* discussion, whether
it's a (zipped) source tree or a sdist (current or 2.0 format) or
something else is beside the point (except insofar as how we name the
command we expect the build system to provide :-))

>> IMO, requiring build systems to support generating "something that pip
>> can treat as a sdist" is a less onerous solution in the short term
>> than requiring pip to work out how to support "pip wheel" to create
>> wheels from source in the absence of a sdist command.
>
> What's onerous about it? Right now sdists are just zipped up source
> trees; the way you generate a wheel from an sdist is to unpack the
> zipfile and then build from the resulting tree. If you *start* with a
> tree, then the only reason you would need an sdist command is if you
> want to zip up the tree and then unzip it again. There are reasons you
> might want to do that, but skipping it would hardly be an extra burden

What I meant was that right now, sdists contain a setup.py that pip
can use to build a wheel. By "less onerous" I meant that pip doesn't
change and build tool X writes out a setup.py that does

    subprocess.call(['build-tool-x', '.'])

(massively oversimplified, but you get the point).

But on reflection, pip has to change anyway to detect what build tool
to use, and the whole point of this proposal is that pip can then ask
the build tool how to create a wheel, so the minimal set of changes
needed in pip make the remaining steps trivial. So I'm wrong on this
point - sorry for the confusion.

OK, I'm part convinced. What's in the proposal is fine. But I strongly
want to avoid giving the impression that we consider providing a means
to build a "source thingy that can go on PyPI" in any way optional.
That may be nothing more than a filename convention that must be
followed, but even that is not something I think we should expect
people to get right by hand. And I don't think it's wise to open up
support of other build systems without resolving that question at
least.

Let me think on this some more. You've convinced me that the risk
isn't as high as I'd originally thought, but I suspect the PEP might
need a bit more explanatory context.

Paul