[Numpy-discussion] new mingw-w64 based numpy and scipy wheel (still experimental)

[Nathaniel Smith]

On 25 Jan 2015 18:46, "Carl Kleffner" <cmkleffner at gmail.com> wrote:
>
> 2015-01-25 16:46 GMT+01:00 Nathaniel Smith <njs at pobox.com>:
>>
>> On Sat, Jan 24, 2015 at 5:29 PM, Carl Kleffner <cmkleffner at gmail.com>
wrote:
>> >
>> > 2015-01-23 0:23 GMT+01:00 Nathaniel Smith <njs at pobox.com>:
>> >>
>> >> On Thu, Jan 22, 2015 at 9:29 PM, Carl Kleffner <cmkleffner at gmail.com>
>> >> wrote:
>> >> > OpenBLAS is deployed as part of the numpy wheel. That said, the
scipy
>> >> > wheels
>> >> > mentioned above are dependant on the installation of the OpenBLAS
based
>> >> > numpy and won't work i.e. with an installed  numpy-MKL.
>> >>
>> >> This sounds like it probably needs to be fixed before we can recommend
>> >> the scipy wheels for anyone? OTOH it might be fine to start
>> >> distributing numpy wheels first.
>> >
>> >
>> > I very much prefer dynamic linking to numpy\core\libopenblas.dll
instead of
>> > static linking to avoid bloat. This matters, because libopenblas.dll
is a
>> > heavy library (around 30Mb for amd64). As a consequence all packages
with
>> > dynamic linkage to OpenBLAS depend on numpy-openblas. This is not
different
>> > to scipy-MKL that has a dependancy to numpy-MKL - see C. Gohlke's site.
>>
>> The difference is that if we upload this as the standard scipy wheel,
>> and then someone goes "hey, look, a new scipy release just got
>> announced, 'pip upgrade scipy'", then the result will often be that
>> they just get random unexplained crashes. I think we should try to
>> avoid that kind of outcome, even if it means making some technical
>> compromises. The whole idea of having the wheels is to make fetching
>> particular versions seamless and robust, and the other kinds of builds
>> will still be available for those willing to invest more effort.
>>
>> One solution would be for the scipy wheel to explicitly depend on a
>> numpy+openblas wheel, so that someone doing 'pip install scipy' also
>> forced a numpy upgrade. But I think we should forget about trying this
>> given the current state of python packaging tools: pip/setuptools/etc.
>> are not really sophisticated enough to let us do this without a lot of
>> kluges and compromises, and anyway it is nicer to allow scipy and
>> numpy to be upgraded separately.
>
>
> I've learned, that mark numpy with something like numpy+openblas is
called "local version identifier":
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
> These identifieres are not allowed for Pypi however.

Right, it's fine for the testing wheels, but even if it were allowed on
pypi then it still wouldn't let us specify the correct dependency -- we'd
have to say that scipy build X depends on exactly numpy 1.9.1+openblas, not
numpy <anything>+openblas. So then when a new version of numpy was uploaded
it'd be impossible to upgrade without also rebuilding numpy.

Alternatively pip would be within its rights to simply ignore the local
version part, because "Local version identifiers are used to denote fully
API (and, if applicable, ABI) compatible patched versions of upstream
projects." Here the +openblas is exactly designed to communicate ABI
incompatibility.

Soooooo yeah this is ugly all around. Pip and friends are getting better
but they're just not up to this kind of thing.

>> Another solution would be to just include openblas in both. This
>> bloats downloads, but I'd rather waste 30 MiB then waste users' time
>> fighting with random library incompatibility nonsense that they don't
>> care about.
>>
>> Another solution would be to split the openblas library off into its
>> own "python package", that just dropped the binary somewhere where it
>> could be found later, and then have both the numpy and scipy wheels
>> depend on this package.
>
>
> Creating a dedicated OpenBLAS package and adding this package as an
dependancy to numpy/scipy would also allow independant upgrade paths to
OpenBLAS, numpy and scipy. The API of OpenBLAS seems to be stable enough to
allow for that. Having an additional package dependancy is a minor problem,
as pip can handle this automatically for the user.

Exactly.

We might even want to give it a tiny python wrapper, e.g. you do
  import openblas
  openblas.add_to_library_path()
and that would be a little function that modifies LD_LIBRARY_PATH or calls
AddDllDirectory etc. as appropriate, so that code linking to openblas can
ignore all these details.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20150125/4d02edaf/attachment.html>