[Distutils] Handling the binary dependency management problem

[Oscar Benjamin]

On 2 December 2013 13:54, Paul Moore <p.f.moore at gmail.com> wrote:
>
> If the named projects provided Windows binaries, then there would be
> no issue with Christoph's stuff. But AFAIK, there is no place I can
> get binary builds of matplotlib *except* from Christoph. And lxml
> provides limited sets of binaries - there's no Python 3.3 version, for
> example. I could continue :-)

The matplotlib folks provide a list of binaries for Windows and OSX
hosted by SourceForge:
http://matplotlib.org/downloads.html

So do numpy and scipy.

> Oh, and by the way, in what sense do you mean "cross-platform" here?
> Win32 and Win64? Maybe I'm being narrow minded, but I tend to view
> "cross platform" as meaning "needs to think about at least two of
> Unix, Windows and OSX". The *platform* issues on Windows (and OSX, I
> thought) are solved - it's the ABI issues that we've ignored thus far
> (successfully till now :-))

Exactly. A python extension that uses Fortran needs to indicate which
of the two Fortran ABIs it uses. Scipy must use the same ABI as the
BLAS/LAPACK library that numpy was linked with. This is core
compatibility data but there's no way to communicate it to pip.
There's no need to actually provide downloadable binaries for both
ABIs but there is a need to be able to detect incompatibilities.

Basically if
1) There is at least one single consistent set of built wheels for
Windows/OSX for any popular set of binary-interdependent packages.
2) A way to automatically detect incompatibilities and to
automatically find compatible built wheels.
then *a lot* of packaging problems have been solved.

Part 1 already exists. There are multiple consistent sets of built
installers (not wheels yet) for many hard to build packages. Part 2
requires at least some changes in pip/PyPI.

I read somewhere that numpy is the most frequently cited dependency on
PyPI. It can be built in multiple binary-incompatible ways. If there
is at least a way for the installer to know that it was built in "the
standard way" (for Windows/OSX) then there can be a set of binaries
built to match that. There's no need for a combinatorial explosion of
compatibility tags - just a single set of compatibility tags that has
complete binaries (where the definition of complete obviously depends
on your field).

People who want to build in different incompatible ways can do so
themselves, although it would still be nice to get an install time
error message when you subsequently try to install something
incompatible.

For Linux this problem is basically solved as far as beginners are
concerned because they can just use apt.


Oscar