[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Paul Moore]

On 2 October 2015 at 21:19, Nathaniel Smith <njs at pobox.com> wrote:
>> One of the problems with the current system, is that we have no mechanism by
>> which to determine dependencies of a source distribution without downloading
>> the file and executing some potentially untrusted code. This makes dependency
>> resolution harder and much much slower than if we could read that information
>> statically from a source distribution. This PEP doesn't offer anything in the
>> way of solving this problem.
>
> What are the "dependencies of a source distribution"? Do you mean the
> runtime dependencies of the wheels that will be built from a source
> distribution?
>
> If you need that metadata to be statically in the sdist, then you
> might as well give up now because it's simply impossible.
>
> As the very simplest example, every package that uses the numpy C API
> gets a runtime dependency on "numpy >= [whatever version happened to
> be installed on the *build* machine]". There are plenty of more
> complex examples too (e.g. ones that involve build/configure-time
> decisions about whether to rely on particular system libraries, or
> build/configure-time decisions about whether particular packages
> should even be built).

I'm really not at all clear what you're saying here. It's quite
possible that those of us who don't understand the complexities of the
scientific/numpy world are missing something important, but if so it
would be useful if you could spell out the problems in detail.

>From my point of view, it's not a source distribution or a binary
distribution that depends on something (numpy or whatever) - it's the
*project*. If project foo needs numpy to work, it depends on numpy. If
it depends on features in numpy 1.9, it depends on numpy>=1.9.
Optional dependencies are covered by extras, and environment specific
dependencies are covered by environment markers.[1] That remains true
for all wheels that are built from that project, for whatever platform
using whatever tools. It should also be true for the source
distribution, precisely *because* it's independent of the build
process.

I can understand that a binary wheel may need a certain set of
libraries installed - but that's about the platform tags that are part
of the wheel definition, not about dependencies. Platform tags are an
ongoing discussion, and a good example of a partial solution that
needs to be extended, certainly, but they aren't really relevant in
any way that I can see to how the build chain works.

You seem to be saying that wheels need a dependency on "the version of
numpy they were built against". That sounds to me like a binary
compatibility requirement that platform tags are intended to cover. It
may well be a requirement that platform tags need significant
enhancement (maybe even redesign) to cover, but it's not a dependency
in the sense that pip and the packaging PEPs use the term. And if my
understanding is correct, I'm against trying to fit that information
into a dependency simply to work around the current limitations of the
platform tag mechanism.

I'm all in favour of new initiatives to make progress in areas that
are currently stalled (we definitely need people willing to
contribute) but we really don't have the resources to throw away the
progress we've already made. Even if some of the packaging PEPs are
still works in progress, what is there represents an investment we
need to build on, not bypass.

Paul

[1] If extras and environment markers don't cover the needs of
scientific modules, we need some input into their design from the
scientific community. But again, let's not throw away the work that's
already done.