[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Oscar Benjamin]

On 8 October 2015 at 12:46, Paul Moore <p.f.moore at gmail.com> wrote:
> On 8 October 2015 at 11:18, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>>
>> A concrete example would be whether or not the numpy source wheel
>> depends on pyopenblas. Depending on how numpy is built the binary
>> wheel may or may not depend on pyopenblas. It doesn't make any sense
>> to say that the numpy source release depends on pyopenblas so what
>> should be the dependencies of the source wheel?
>
> Well, I said this previously but I don't have any objections to the
> idea that binary wheels have additional dependencies - so the source
> wheel doesn't depend on pyopenblas but the binary does.

Okay, I guess I'm confused by what you mean when you say that a source
wheel (or sdist) should have a "one-to-one" correspondence with a
binary wheel.

> But as I understand it, this is currently theoretical - there isn't
> yet any pyopenblas validate these speculations against?

I don't think pyopenblas is ready yet but it is being developed with
specifically this scenario in mind.

<snip>
> So unless I'm mistaken about what you're saying, I don't see any issue
> here. Unless you're saying that you're not willing to work under some
> of the constraints I describe above

As an aside : I'm not a contributor to numpy. I just use it a lot and
teach people how to use it (which is where the packaging problems come
in).

> - but in that case, you need pip's
> compatibility matching, dependency resolution, or automated wheel
> build processes to change. That's fine but to move the discussion
> forwards, we'd then need to understand (and agree with) whatever
> changes you need in pip. At the moment, I'm not aware that anyone has
> asked for substantive changes to pip's behaviour in these areas as
> part of this proposal.

I don't think anyone is suggesting significant changes to pip's
dependency resolution. Compatibility matching does need improvement
IMO. Also the automated build process does need to be changed -
specifically we need build-requires so that third party build tools
can work. I didn't think improving the build process was
controversial...

<snip>
>> I think this satisfies all of the requirements for static metadata and
>> one-to-one correspondence of source wheels and binary wheels. If numpy
>> followed this then I imagine that there would be a single source wheel
>> on PyPI corresponding to the one configuration that would be used
>> consistently there. However numpy still needs to separately release
>> the code in a form that is also usable in all of the many other
>> contexts that it is already used. IOW they will need to continue to
>> issue source releases in more or less the same form as today. It makes
>> sense for PyPI to host the source release archives on the project page
>> even if pip will simply ignore them.
>
> So you're talking about numpy only supporting one configuration via
> PyPI, and expecting any other configurations to be made available only
> via other channels? I guess you could do that, but I hope you won't.
> It feels to me like giving up before we've properly tried to
> understand the issues.

Okay so again I'm not a numpy dev.

Numpy already supports being used in lots of setups that are not via
pypi. Apart from Cristoph's builds you have all kinds of people
building on all kinds of OSes and linking with different BLAS
libraries in different ways. Some people will compile numpy statically
with CPython. If you follow the discussions about numpy development
it's clear that the numpy devs don't know all of the ways that numpy
is built and used.

Clearly pip/PyPI cannot be used to statically link numpy with CPython
or for all of the different (often non-redistributable) BLAS libraries
so numpy will support some setups that are not possible through pip.
That's fine, I don't see the problem with that. At the moment an sdist
is the same thing is a source release. If you propose to change it so
that projects should upload source wheels and then make source wheels
something tightly defined (e.g. a zip file containing exactly two
directories setup to build for one particular configuration) then
there needs to be a separate way to simply release the code in
traditional format as is done now.

--
Oscar