[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Wes Turner]

On Oct 3, 2015 12:36 PM, "Donald Stufft" <donald at stufft.io> wrote:
>
>
>
> On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote:
> > On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote:
> >
> > > On 3 October 2015 at 02:03, Nathaniel Smith wrote:
> > > > In particular I hesitate a little bit to just drop in everything
from
> > > > PEP 426 and friends, because previous specs haven't really thought
> > > > through the distinction between sdists and wheels -- e.g. if an
sdist
> > > > generates two wheels, they probably won't have the same name,
> > > > description, trove classifiers, etc. They may not even have the same
> > > > version (e.g. if two different tools with existing numbering schemes
> > > > get merged into a single distribution -- esp. if one of them needs
an
> > > > epoch marker). So it may well make sense to have an "sdist
description
> > > > field", but it's not immediately obvious that it's identical to a
> > > > wheel's description field.
> > >
> > > I can only assume you're using the term sdist differently from the way
> > > it is normally used (as in the PUG glossary), because for me the idea
> > > that a sdist generates multiple wheels makes no sense.
> > >
> > > If I do "pip wheel " I expect to get a single wheel that
> > > can be installed to give the same results as "pip install
> > > " would, The wheel install will work on my build machine,
> > > and on any machine where the wheel's compatibility metadata (the
> > > compatibility tags, currently) says it's valid.
> > >
> > > The above behaviour is key to pip's mechanism for auto-generating and
> > > caching wheels when doing an install, so I don't see how it could
> > > easily be discarded.
> > >
> > > If what you're calling a "sdist" doesn't work like this, maybe you
> > > should invent a new term, so that people don't get confused? If it
> > > *does* work like this, I don't see what you mean by a sdist generating
> > > two wheels.
> > >
> >
> > I think sdist is getting a bit overloaded in this discussion. From my
> > understanding of what Paul and Donald are after, the process should be:
> >
> > VCS -> sdist/source wheel -> binary wheel.
> >
> > Here, "VCS" is literally a git/hg clone of some source tree. A
> > "sdist/source wheel" is carefully constructed zip file that contains all
> > the source code from the VCS necessary to build a binary wheel for a
> > project as long as the proper dependencies exist (e.g., proper version
of
> > NumPy, BLAS in one of its various forms, etc.). The binary wheel is
> > obviously the final artifact that can just be flat-out loaded by Python
> > without any work.
> >
> > So Paul doesn't see sdist/source wheels producing more than one binary
> > wheel because in its own way an sdist/source wheel is a "compiled"
artifact
> > of select source code whose only purpose is to generate a binary wheel
for
> > a single project. So while a VCS clone might have multiple subprojects,
> > each project should generate a single sdist/source wheel.
> >
> > Now this isn't to say that an sdist/source wheel won't generate
different
> > *versions* of a binary wheel based on whether e.g. BLAS is
system-linked,
> > statically linked, or dynamically loaded from a Linux binary wheel. But
the
> > key point is that the sdist/source wheel still **only** makes one kind
of
> > project.
> >
> > From this perspective I don't see Nathaniel's desire for installing
from a
> > VCS as something other than requiring a step to bundle up the source
code
> > into an sdist/source wheel that pip knows how to handle. But I think
what
> > Paul and Donald are saying is pip doesn't want to have anything to do
with
> > the VCS -> sdist/source wheel step and that is entirely up to the
project
> > to manage through whatever tooling they choose. I also view the
> > sdist//source wheel as almost a mini-VCS checkout since it is just a
> > controlled view of the source code with a bunch of helpful, static
metadata
> > for pip to use to execute whatever build steps are needed to get to a
> > binary wheel.
> >
> > Or I'm totally wrong. =) But for me I actually like the idea of an
> > sdist/source wheel being explained as "a blob of source in a structured
way
> > that can produce a binary wheel for the package" and a binary wheel as
"a
> > blob of bytes for a package that Python can import" and I'm totally fine
> > having C extensions not just shuttle around a blind zip but a structured
> > zip in the form of an sdist/source wheel.
>
> This is basically accurate, the only thing is that I think that we do
need an
> answer for handling the VCS side of things, but I think that we should
defer
> it until after this PEP. Right now I think the PEP is trying to tackle two
> different problems at once because on the surface they look similar but
IMO
> a VCS/arbitrary directory and a sdist are two entirely different things
that
> are only similar on the surface.
>
> So Yes the path would be:
>
> VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install
> \
>  \-> "In Place" install (aka Development Install)
>
> Right now, we have the Wheel -> Install part, and I'd like for this PEP to
> focus on the sdist -> Wheel part, and then a future PEP focus on the VCS
part.

thanks! is there a diagram of this somewhere?

Is this basically a FSM with URIs for each resource and transformation,
where more specific attributes are added (e.g. to pydist.jsonld)? [PEP 426
JSONLD]

differences between VCS and sdist:
* MANIFEST.in
* setup.py
* egg-info metadata

PEP 426 JSONLD: https://github.com/pypa/interoperability-peps/issues/31

>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151003/2cc6722e/attachment.html>