[Distutils] PEP: Build system abstraction for pip/conda etc

[Nathaniel Smith]

The state of the art right now is that sdist generation is already a
quirky build-system-specific interface that isn't used by any other
tools -- 'python setup.py sdist --whatever=whatever' -- and that the
de facto sdist format is 'a zip/tarball named NAME-VERSION.EXT which
contains a setup.py'. So nothing in the current build interface
proposals makes this any better, or any worse.

I understand your concern that we want to encourage people to upload
sdists, but if it reassures you any, recall that part of what started
this whole discussion is that flit already *wants* to upload sdists --
the problem is just that they currently can't :-). See also
   https://github.com/takluyver/flit/issues/74

So there are definitely things we can do to improve sdists in general,
but I think we can treat that as an independent discussion, and I'm
not terribly worried that distutils-sig supervision is necessary to
prevent build system authors from running free ignoring the concept of
source releases. And if they do then we can shake a finger at them
when it happens :-)

-n

On Tue, Feb 9, 2016 at 3:56 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 February 2016 at 22:59, Nathaniel Smith <njs at pobox.com> wrote:
>>> At the moment, that standard is "pip wheel foo". Or maybe it's
>>> "setup.py bdist_wheel", but I prefer the layer of build system
>>> abstraction offered by pip.
>>
>> Everyone agrees that we absolutely need to support "pip wheel
>> packagename" and and "pip wheel sdist.zip" and "pip wheel some-path/".
>> The (potential) disagreement is about whether the most primitive input
>> to "pip wheel" is an sdist or a directory tree.
>>
>> I.e., option A is that "pip wheel some-path/" is implemented as "pip
>> sdist some-path/ && pip wheel some-path-sdist.zip", and option B is
>> "pip wheel sdist.zip" implemented as "unzip sdist.zip && pip wheel
>> sdist-tree/". Robert's draft PEP doesn't explicitly come down on
>> either side, but in practice I think that at least to start with it
>> would require the "option B" approach, since it doesn't provide any
>> "pip sdist" command.
>
> Hmm, I see what you're getting at.
>
> But if project X uses flit, what do they need to do to upload
> something to PyPI so that "pip wheel X" builds a wheel? Let's assume
> they aren't actively trying to hide their sources, but equally they
> aren't looking to do a load of work beyond their normal workflow just
> to support that usage.
>
> And if a project currently using setuptools wants to switch to flit,
> how do they upload what's needed to allow pip to install on platforms
> where they don't supply wheels?
>
> Maybe that's not functionality pip *needs* to do its job, but it seems
> far too likely to me that projects will build and upload wheels for
> some platforms, and then say "to get the source go to <our github
> site>". With luck they'll include build instructions, although they
> may just be "git clone <url>; pip wheel ."
>
> I have set up a number of systems that automate wheel builds - these
> days, that's as simple as putting "pip wheel -r mypackages.txt" in a
> scheduled job. I do *not* want to have to go back to maintaining a
> mapping from project to source URL, or working out how to get things
> to work on systems without git installed and how to detect the correct
> way of getting the latest release version from a source repo.
>
> Documenting that build systems are required to define how they support
> the operation of "build a release source file that can be uploaded to
> PyPI" makes it clear that we consider this to be an important
> operation. Even if it's not technically needed for pip to operate.
>
>>> What I do care about is ensuring that if a projects hasn't published a
>>> wheel for some particular environment, and the project is *not*
>>> actively trying to avoid publishing sources, then we provide them with
>>> a means to publish those sources in such a way that "pip wheel
>>> theproject" (or whatever command replaces it as the canonical "build a
>>> wheel" command) can automate the process of generating a wheel for the
>>> user's platform.
>>
>> Both of the options under discussion give us all of these properties.
>> You can absolutely implement a "pip wheel" command that handles all
>> these cases without needing any "pip sdist" command.
>
> I don't think (bit it is some time ago, and I haven't reread the
> thread yet) I ever said that I want "pip sdist" to be mandated. (I
> have no real opinion on that). What I want is that we make it clear
> that build systems need to provide a means to create "a source bundle
> that can be uploaded to PyPI that pip can use to build wheels" (I'll
> buy a beer for anyone who can come up with an acceptable name for this
> that doesn't make people thing "but we want to get away from the sdist
> format" :-))
>
> I also want people to agree that it's a good thing to expect build
> systems to support creating such a source artifact.
>
> I'm aware that this "source artifact" may well be what you were
> proposing be a "directory tree". That's another debate that as far as
> I know petered out without conclusion. I'm trying to keep the two
> discussions separate - for the purposes of *this* discussion, whether
> it's a (zipped) source tree or a sdist (current or 2.0 format) or
> something else is beside the point (except insofar as how we name the
> command we expect the build system to provide :-))
>
>>> IMO, requiring build systems to support generating "something that pip
>>> can treat as a sdist" is a less onerous solution in the short term
>>> than requiring pip to work out how to support "pip wheel" to create
>>> wheels from source in the absence of a sdist command.
>>
>> What's onerous about it? Right now sdists are just zipped up source
>> trees; the way you generate a wheel from an sdist is to unpack the
>> zipfile and then build from the resulting tree. If you *start* with a
>> tree, then the only reason you would need an sdist command is if you
>> want to zip up the tree and then unzip it again. There are reasons you
>> might want to do that, but skipping it would hardly be an extra burden
>
> What I meant was that right now, sdists contain a setup.py that pip
> can use to build a wheel. By "less onerous" I meant that pip doesn't
> change and build tool X writes out a setup.py that does
>
>     subprocess.call(['build-tool-x', '.'])
>
> (massively oversimplified, but you get the point).
>
> But on reflection, pip has to change anyway to detect what build tool
> to use, and the whole point of this proposal is that pip can then ask
> the build tool how to create a wheel, so the minimal set of changes
> needed in pip make the remaining steps trivial. So I'm wrong on this
> point - sorry for the confusion.
>
> OK, I'm part convinced. What's in the proposal is fine. But I strongly
> want to avoid giving the impression that we consider providing a means
> to build a "source thingy that can go on PyPI" in any way optional.
> That may be nothing more than a filename convention that must be
> followed, but even that is not something I think we should expect
> people to get right by hand. And I don't think it's wise to open up
> support of other build systems without resolving that question at
> least.
>
> Let me think on this some more. You've convinced me that the risk
> isn't as high as I'd originally thought, but I suspect the PEP might
> need a bit more explanatory context.
>
> Paul



-- 
Nathaniel J. Smith -- https://vorpus.org