[Numpy-discussion] Numpy, BLAS, and CBLAS questions

[Nathaniel Smith]

On Jul 13, 2015 11:48 AM, "Eric Martin" <eric at ericmart.in> wrote:
>
> Nathaniel and Ian both mention adding code to Numpy to explicitly support
Eigen.
>
> It seems to me that a potentially better route than "add code to Numpy to
support BLAS library" for each library is to make Numpy easy to configure
to compile with an arbitrary BLAS library (like what I've been doing).

I think that if you can make this work then it would be great (who doesn't
like less code that does more?), but that as a practical matter
accomplishing this may be difficult. AFAIK there simply is no way to write
generic code to probe the system for "any BLAS", except by explicitly
having a list of supported BLASes and checking for each in sequence. Given
that new BLAS libraries appear at a rate of < 1/year, and that we already
support most of the viable ones, the current approach is practically
reasonable even if non-ideal in principle.

> Reasoning behind this:
> * numpy.distutils.system_info is already 2300 lines of code. This code
(and the full process of linking to BLAS) are not very well documented or
flexible, as evidenced by the trouble I had linking against custom BLAS and
the fact that changes to numpy's source were required. Adding more code to
this seems to make things worse rather than better in my opinion.
>
> * I think the "support arbitrary BLAS library" option is better plane on
the "ease of use" to "robustness" than adding support for particular
libraries.
>
> * BLAS is an interface. Shouldn't we (as Numpy developers) take full
advantage of the interface and the generality it allows?
>
> As I mentioned in my first email, the current user experience of linking
Numpy to arbitrary BLAS consists of (0) compile/acquire BLAS binary, (1)
modify Numpy source, (2) compile CBLAS against your BLAS, and (3) modify
site.cfg and build. Step 0 is outside the scope of Numpy, and none of steps
1-3 were very well documented. I think the ideal user experience would
consist solely of steps 0 and 3, that is just acquiring a BLAS and then
configuring Numpy to build against that BLAS, and then building Numpy. I'm
not sure if it's technically possible, but it would be an improvement on
this if the user could change the BLAS for a numpy install without
rebuilding (ie Numpy optionally loads some site.cfg equivalent each time it
opens and dynamically links based off of that, or something clever with
symlinks is done). I believe this is technically possible with symlinks
(see Ubuntu's update-alternatives program).

Keep in mind that any solution needs to support weird systems too,
including Windows. I'm not sure we can assume that all BLAS libraries are
ABI compatible either. Debian/Ubuntu make sure that this is true for the
ones they ship, but not all systems are so carefully managed. For LAPACK in
particular I'm pretty sure there are multiple ABIs in active use that scipy
supports via code in numpy.distutils. (Hopefully someone with more
expertise will speak up.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20150713/111b9749/attachment.html>