[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Donald Stufft]

On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote:
> On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote:
>  
> > On 3 October 2015 at 02:03, Nathaniel Smith wrote:
> > > In particular I hesitate a little bit to just drop in everything from
> > > PEP 426 and friends, because previous specs haven't really thought
> > > through the distinction between sdists and wheels -- e.g. if an sdist
> > > generates two wheels, they probably won't have the same name,
> > > description, trove classifiers, etc. They may not even have the same
> > > version (e.g. if two different tools with existing numbering schemes
> > > get merged into a single distribution -- esp. if one of them needs an
> > > epoch marker). So it may well make sense to have an "sdist description
> > > field", but it's not immediately obvious that it's identical to a
> > > wheel's description field.
> >
> > I can only assume you're using the term sdist differently from the way
> > it is normally used (as in the PUG glossary), because for me the idea
> > that a sdist generates multiple wheels makes no sense.
> >
> > If I do "pip wheel " I expect to get a single wheel that
> > can be installed to give the same results as "pip install
> > " would, The wheel install will work on my build machine,
> > and on any machine where the wheel's compatibility metadata (the
> > compatibility tags, currently) says it's valid.
> >
> > The above behaviour is key to pip's mechanism for auto-generating and
> > caching wheels when doing an install, so I don't see how it could
> > easily be discarded.
> >
> > If what you're calling a "sdist" doesn't work like this, maybe you
> > should invent a new term, so that people don't get confused? If it
> > *does* work like this, I don't see what you mean by a sdist generating
> > two wheels.
> >
>  
> I think sdist is getting a bit overloaded in this discussion. From my
> understanding of what Paul and Donald are after, the process should be:
>  
> VCS -> sdist/source wheel -> binary wheel.
>  
> Here, "VCS" is literally a git/hg clone of some source tree. A
> "sdist/source wheel" is carefully constructed zip file that contains all
> the source code from the VCS necessary to build a binary wheel for a
> project as long as the proper dependencies exist (e.g., proper version of
> NumPy, BLAS in one of its various forms, etc.). The binary wheel is
> obviously the final artifact that can just be flat-out loaded by Python
> without any work.
>  
> So Paul doesn't see sdist/source wheels producing more than one binary
> wheel because in its own way an sdist/source wheel is a "compiled" artifact
> of select source code whose only purpose is to generate a binary wheel for
> a single project. So while a VCS clone might have multiple subprojects,
> each project should generate a single sdist/source wheel.
>  
> Now this isn't to say that an sdist/source wheel won't generate different
> *versions* of a binary wheel based on whether e.g. BLAS is system-linked,
> statically linked, or dynamically loaded from a Linux binary wheel. But the
> key point is that the sdist/source wheel still **only** makes one kind of
> project.
>  
> From this perspective I don't see Nathaniel's desire for installing from a
> VCS as something other than requiring a step to bundle up the source code
> into an sdist/source wheel that pip knows how to handle. But I think what
> Paul and Donald are saying is pip doesn't want to have anything to do with
> the VCS -> sdist/source wheel step and that is entirely up to the project
> to manage through whatever tooling they choose. I also view the
> sdist//source wheel as almost a mini-VCS checkout since it is just a
> controlled view of the source code with a bunch of helpful, static metadata
> for pip to use to execute whatever build steps are needed to get to a
> binary wheel.
>  
> Or I'm totally wrong. =) But for me I actually like the idea of an
> sdist/source wheel being explained as "a blob of source in a structured way
> that can produce a binary wheel for the package" and a binary wheel as "a
> blob of bytes for a package that Python can import" and I'm totally fine
> having C extensions not just shuttle around a blind zip but a structured
> zip in the form of an sdist/source wheel.

This is basically accurate, the only thing is that I think that we do need an
answer for handling the VCS side of things, but I think that we should defer
it until after this PEP. Right now I think the PEP is trying to tackle two
different problems at once because on the surface they look similar but IMO
a VCS/arbitrary directory and a sdist are two entirely different things that
are only similar on the surface.

So Yes the path would be:

VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install
\
 \-> "In Place" install (aka Development Install)

Right now, we have the Wheel -> Install part, and I'd like for this PEP to
focus on the sdist -> Wheel part, and then a future PEP focus on the VCS part.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA