[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Paul Moore]

On 2 October 2015 at 20:45, Donald Stufft <donald at stufft.io> wrote:
> n October 2, 2015 at 2:41:20 PM, Brett Cannon (brett at python.org) wrote:
>> > Is there another proposal I'm unaware for the sdist -> wheel
>> step that is build tool-agnostic? I'm all for going with the best
>> solution but there has to be an actual alternative to compare
>> against and I don't know of any others right now and this proposal
>> does seem to move things forward in a reasonable fashion.
>
> As Marcus said, it was touched on in PEP 426 but there isn’t a fully fleshed
> out PEP (or even a sort of fleshed out PEP) but the high level idea isn’t super
> hard, take PEP 426 metadata + some sdist specific metadata and stick it in a
> tarball or something and then you have sdist 2.0. Part of that sdist specific
> metadata would be describing in some way how to build a wheel out of this sdist
> (rather than just assuming distutils/setuptools).
>
> I think we need to push back against partial solutions to problems that we
> have without at least some semblance of a plan for how to solve the rest of
> the problems we have, or to explicitly declare them unsolvable. That doesn't
> mean that we need to turn every thing into an overengineered mess where we try
> to solve every problem ever, but we need to be careful that we minimize churn
> where possible. An example just recently where we had this was we accepted
> PEP 438 2 years ago as a partial solution to try and "move things forward" and
> then it turned out that solution was, well partial, and when we removed the
> partial solution for a full solution we had the folks who decided to rely on
> the partial solution upset.
>
> IOW we need to balance between avoiding churn and making a change "worth" it,
> and in my opinion this idea here doesn't solve enough problems to make it
> worth it.

That's a fair point. But I don't see any reason Nathaniel's proposal
*couldn't* be that solution. I'd want to see the sdist format required
to include static metadata, and the metadata format to be PEP 426, but
neither of those seem incompatible with the ideas behind the proposal.

Maybe I'm missing something massive, but I don't see a *huge* gap
between this proposal and the basic ideas behind the metabuild concept
- the sdist should define how to build it (in terms of required
packages/tools for the build, and commands/hooks to call to do the
specific build steps - make a wheel, do an in-place install). The
biggest sticking point would be if Nathaniel is adamantly opposed to
static metadata.

Is there a good enumeration anywhere of the problems sdist 2.0 / the
metabuild system needs to solve? I can think of:

1. Metadata needs to be discoverable without running code.
2. Packages should be able to specify the tools needed to build them.
3. Installers should only interact with sdists through well-defined
entry points.
4. Build processes required are (a) create a wheel, (b) do a "develop"
style in-place installation.
5. (Maybe) There needs to be better means of handling build errors.

The proposal here seems to at least move towards better solutions for
2, 3, and 4.

Paul