[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Paul Moore]

On 8 October 2015 at 13:39, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> On 8 October 2015 at 12:46, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 8 October 2015 at 11:18, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>>>
>>> A concrete example would be whether or not the numpy source wheel
>>> depends on pyopenblas. Depending on how numpy is built the binary
>>> wheel may or may not depend on pyopenblas. It doesn't make any sense
>>> to say that the numpy source release depends on pyopenblas so what
>>> should be the dependencies of the source wheel?
>>
>> Well, I said this previously but I don't have any objections to the
>> idea that binary wheels have additional dependencies - so the source
>> wheel doesn't depend on pyopenblas but the binary does.
>
> Okay, I guess I'm confused by what you mean when you say that a source
> wheel (or sdist) should have a "one-to-one" correspondence with a
> binary wheel.

OK, let me try to clarify:

1. The identifying data is name and version. There should only ever be
*one* source wheel with a given name/version.
2. It's possible to have multiple binary wheels derived from a source
wheel, but they must all have the same name/version as the source
wheel, and only one will ever be considered as the "right" one for a
given system. (I.e., compatibility matching takes place and a single
best match is selected).

The key things here is that binary wheels that differ only in
compatibility tags are considered "the same" in this sense (think
equivalence class, if you have a mathematical background).

Things that may have muddied the water:

1. When I say "all metadata in a source wheel must be static".
Technically, tools (pip) don't care about anything other than name and
version. The rest is mostly for things like publishing on PyPI and
distro packaging, which is not a "core" use.
2. The whole pyopenblas question, which as I understand it is part of
the effort to work around the limitations of the compatibility tag
mechanism by making binary library dependencies into package
dependencies. That approach looks promising, but it moves some of the
"what is the appropriate wheel for this environment" question away
from compatibility tags and into dependency checking. Which is a
non-trivial change to the model.

I'd like to see the "binary dependency as package dependency" approach
explored properly, but it's completely orthogonal to the question of
how source wheels should work. You'd have the same issues with the
current sdist format. I'd suggest that the work around this area be
separated out from the revamped source distribution discussion. It
could probably even be prototyped without getting involved with numpy
- how about creating a test project that linked either statically or
dynamically to a dummy external library, and then sorting out the
wheel issues with that (project testproj depends on project testlib if
it's built to link dynamically, and doesn't if it's built statically)?
You could do that without anyone needing to understand about BLAS, or
numpy, and get much better informed feedback from the wider
distutils-sig than you currently get (because you keep needing to
explain numpy build processes every time anyone asks a question).

For the purposes of this discussion, however, I'd suggest that we
assume that building a binary wheel from a source wheel *doesn't* add
any new dependencies. Apart from the pyopenblas case, I don't know of
any other case where that would happen. (If making that restriction
makes the whole discussion trivial, then that's sort of my point - I
don't think there *is* a big debate to be had outside of the
pyopenblas question).

Paul