On Sun, Jan 25, 2015 at 12:23 PM, Nathaniel Smith firstname.lastname@example.org wrote:
On 25 Jan 2015 18:46, "Carl Kleffner" email@example.com wrote:
2015-01-25 16:46 GMT+01:00 Nathaniel Smith firstname.lastname@example.org:
On Sat, Jan 24, 2015 at 5:29 PM, Carl Kleffner email@example.com wrote:
2015-01-23 0:23 GMT+01:00 Nathaniel Smith firstname.lastname@example.org:
On Thu, Jan 22, 2015 at 9:29 PM, Carl Kleffner email@example.com wrote:
OpenBLAS is deployed as part of the numpy wheel. That said, the scipy wheels mentioned above are dependant on the installation of the OpenBLAS based numpy and won't work i.e. with an installed numpy-MKL.
This sounds like it probably needs to be fixed before we can recommend the scipy wheels for anyone? OTOH it might be fine to start distributing numpy wheels first.
I very much prefer dynamic linking to numpy\core\libopenblas.dll instead of static linking to avoid bloat. This matters, because libopenblas.dll is a heavy library (around 30Mb for amd64). As a consequence all packages with dynamic linkage to OpenBLAS depend on numpy-openblas. This is not different to scipy-MKL that has a dependancy to numpy-MKL - see C. Gohlke's site.
The difference is that if we upload this as the standard scipy wheel, and then someone goes "hey, look, a new scipy release just got announced, 'pip upgrade scipy'", then the result will often be that they just get random unexplained crashes. I think we should try to avoid that kind of outcome, even if it means making some technical compromises. The whole idea of having the wheels is to make fetching particular versions seamless and robust, and the other kinds of builds will still be available for those willing to invest more effort.
One solution would be for the scipy wheel to explicitly depend on a numpy+openblas wheel, so that someone doing 'pip install scipy' also forced a numpy upgrade. But I think we should forget about trying this given the current state of python packaging tools: pip/setuptools/etc. are not really sophisticated enough to let us do this without a lot of kluges and compromises, and anyway it is nicer to allow scipy and numpy to be upgraded separately.
I've learned, that mark numpy with something like numpy+openblas is called "local version identifier": https://www.python.org/dev/peps/pep-0440/#local-version-identifiers These identifieres are not allowed for Pypi however.
Right, it's fine for the testing wheels, but even if it were allowed on pypi then it still wouldn't let us specify the correct dependency -- we'd have to say that scipy build X depends on exactly numpy 1.9.1+openblas, not numpy <anything>+openblas. So then when a new version of numpy was uploaded it'd be impossible to upgrade without also rebuilding numpy.
Alternatively pip would be within its rights to simply ignore the local version part, because "Local version identifiers are used to denote fully API (and, if applicable, ABI) compatible patched versions of upstream projects." Here the +openblas is exactly designed to communicate ABI incompatibility.
Soooooo yeah this is ugly all around. Pip and friends are getting better but they're just not up to this kind of thing.
I agree, that shipping openblas with both numpy and scipy seems perfectly reasonable to me - I don't think anyone will much care about the 30M, and I think our job is to make something that works with the least complexity and likelihood of error.
It would be good to rename the dll according to the package and version though, to avoid a scipy binary using a pre-loaded but incompatible 'libopenblas.dll'. Say something like openblas-scipy-0.15.1.dll - on the basis that there can only be one copy of scipy loaded at a time.