[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Brett Cannon]

On Sat, 3 Oct 2015 at 04:51 Paul Moore <p.f.moore at gmail.com> wrote:

> On 3 October 2015 at 02:03, Nathaniel Smith <njs at pobox.com> wrote:
> > In particular I hesitate a little bit to just drop in everything from
> > PEP 426 and friends, because previous specs haven't really thought
> > through the distinction between sdists and wheels -- e.g. if an sdist
> > generates two wheels, they probably won't have the same name,
> > description, trove classifiers, etc. They may not even have the same
> > version (e.g. if two different tools with existing numbering schemes
> > get merged into a single distribution -- esp. if one of them needs an
> > epoch marker). So it may well make sense to have an "sdist description
> > field", but it's not immediately obvious that it's identical to a
> > wheel's description field.
>
> I can only assume you're using the term sdist differently from the way
> it is normally used (as in the PUG glossary), because for me the idea
> that a sdist generates multiple wheels makes no sense.
>
> If I do "pip wheel <some_sdist>" I expect to get a single wheel that
> can be installed to give the same results as "pip install
> <some_sdist>" would, The wheel install will work on my build machine,
> and on any machine where the wheel's compatibility metadata (the
> compatibility tags, currently) says it's valid.
>
> The above behaviour is key to pip's mechanism for auto-generating and
> caching wheels when doing an install, so I don't see how it could
> easily be discarded.
>
> If what you're calling a "sdist" doesn't work like this, maybe you
> should invent a new term, so that people don't get confused? If it
> *does* work like this, I don't see what you mean by a sdist generating
> two wheels.
>

I think sdist is getting a bit overloaded in this discussion. From my
understanding of what Paul and Donald are after, the process should be:

VCS -> sdist/source wheel -> binary wheel.

Here, "VCS" is literally a git/hg clone of some source tree. A
"sdist/source wheel" is carefully constructed zip file that contains all
the source code from the VCS necessary to build a binary wheel for a
project as long as the proper dependencies exist (e.g., proper version of
NumPy, BLAS in one of its various forms, etc.). The binary wheel is
obviously the final artifact that can just be flat-out loaded by Python
without any work.

So Paul doesn't see sdist/source wheels producing more than one binary
wheel because in its own way an sdist/source wheel is a "compiled" artifact
of select source code whose only purpose is to generate a binary wheel for
a single project. So while a VCS clone might have multiple subprojects,
each project should generate a single sdist/source wheel.

Now this isn't to say that an sdist/source wheel won't generate different
*versions* of a binary wheel based on whether e.g. BLAS is system-linked,
statically linked, or dynamically loaded from a Linux binary wheel. But the
key point is that the sdist/source wheel still **only** makes one kind of
project.

>From this  perspective I don't see Nathaniel's desire for installing from a
VCS as something other than requiring a step to bundle up the source code
into an sdist/source wheel that pip knows how to handle. But I think what
Paul and Donald are saying is pip doesn't want to have anything to do with
the VCS -> sdist/source wheel step and that is entirely up to the project
to manage through whatever tooling they choose. I also view the
sdist//source wheel as almost a mini-VCS checkout since it is just a
controlled view of the source code with a bunch of helpful, static metadata
for pip to use to execute whatever  build  steps are needed to get  to a
binary wheel.

Or I'm totally wrong. =) But for me I actually like the idea of an
sdist/source wheel being explained as "a blob of source in a structured way
that can produce a binary wheel for the package" and a binary wheel as "a
blob of bytes for a package that Python can import" and I'm totally fine
having C extensions not just shuttle around a blind zip but a structured
zip in the form of an sdist/source wheel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151003/2a814c94/attachment.html>