Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas
Hi Nathaniel. Thanks for your prompt reply. I think numpy is a wonderful project, and you all do a great job moving it forward. If you ask what would my vision for maturing numpy, I would like to see a grouping of linalg matrix-operation functionality into a python level package, exactly the opposite of more tightly tying linalg into the core of numpy. The orthagonality would allow goups like PyOpenCL to reuse the matrix operations on data located off the CPU's RAM, just to give one example; and make it easier for non-numpy developers to create a complete replacement of lapack with other implementations. Much of the linalg package would of course be implemented in c or fortran, but the interface to ndarray would use the well-established idea of contiguous matrices with shapes, strides, and a single memory store, supporting only numeric number types. I suggested cffi since it provides a convienent and efficient interface to ndarray. Thus python could remain as a thin wrapper over the calls out to c-based libraries much like lapack_lite does today, but at the python level rather that the capi level. Yes, a python-based interface would slows the code down a bit, but I would argue that 1. the current state of lapack_litemodule.c and umath_linalg.c.src, with its myriad of compile-time macros and complex code paths, scares people away from contributing to the ongoing maintenance of the library while tying the code very closely to the lapack routines, and 2. matrices larger than 3x3 or so should be spending most of the computation time in the underlying lapack/blas library irregardless of whether the interface is python-based or capi-based. Matti On 10/08/2014 8:00 PM, numpy-discussion-request@scipy.org wrote:
Date: Sat, 9 Aug 2014 21:11:19 +0100 From: Nathaniel Smith <njs@pobox.com> Subject: Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas To: Discussion of Numerical Python <numpy-discussion@scipy.org>
On Sat, Aug 9, 2014 at 8:35 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi. I am working on numpy in pypy. It would be much more challenging for me if you merged more code into the core of numpy, Hi Matti,
I can definitely see how numpy changes cause trouble for you, and sympathize. But, can you elaborate on what kind of changes would make your life easier *that also* help make numpy proper better in their own right? Because unfortunately, I don't see how we can reasonably pass up on improvements to numpy if the only justification is to make numpypy's life easier. (I'd also love to see pypy become usable for general numerical work, but not only is it not there now, I don't see how numpypy will ultimately get us there even if we do help it along -- almost none of the ecosystem can get by numpy's python-level APIs alone.) But obviously if there are changes that are mutually beneficial, well then, that's a lot easier to justify :-)
-n
Matti Picus <matti.picus@gmail.com> wrote:
Thanks for your prompt reply. I think numpy is a wonderful project, and you all do a great job moving it forward. If you ask what would my vision for maturing numpy, I would like to see a grouping of linalg matrix-operation functionality into a python level package, exactly the opposite of more tightly tying linalg into the core of numpy.
But with the @ operator in Python 3.5 it makes sence to have both matrix multiplication and linear algebra solvers in the core of NumP. Just consider: A @ B A.LazyInverse @ B A @ B.LazyInverse (Lazy matrix inversion with O(1) complexity does not exist right now, but could be added in the future.) To implement this efficiently, we need BLAS and LAPACK in the core of NumPy. It does not mean there would not be a linalg namespace for LU, SVD, et al. Sturla
Hi Matt, On Mon, Aug 11, 2014 at 10:46 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Nathaniel. Thanks for your prompt reply. I think numpy is a wonderful project, and you all do a great job moving it forward. If you ask what would my vision for maturing numpy, I would like to see a grouping of linalg matrix-operation functionality into a python level package, exactly the opposite of more tightly tying linalg into the core of numpy.
As I understood it (though I admit Chuck was pretty terse, maybe he'll correct me :-)), what he was proposing was basically just a build system reorganization -- it's much easier to call between C functions that are in the same Python module than C functions that are in different modules, so we end up with lots of boilerplate gunk for the latter. I don't think it would involve any tighter coupling than we already have in practice.
The orthagonality would allow goups like PyOpenCL to reuse the matrix operations on data located off the CPU's RAM, just to give one example; and make it easier for non-numpy developers to create a complete replacement of lapack with other implementations.
I guess I don't really understand what you're suggesting. If we have a separate package that is the same as current np.linalg, then how does that allow PyOpenCL to suddenly run the np.linalg code on the GPU? What kind of re-use are you envisioning? The important kind of re-use that comes to mind for me is that I should be able to write code that can accept either a RAM matrix or a GPU matrix and works the same. But the key feature to enable this is that there should be a single API that works on both types of objects -- e.g. np.dot(a, b) should work even if a, b are on the GPU. But this is exactly what __numpy_ufunc__ is designed to enable, and that has nothing to do with splitting linalg off into a separate package... And of course if someone has a better idea about how to implement lapack, then they should do that work in the numpy repo so everyone can benefit, not go off and reimplement their own version from scratch that no-one will use :-).
Much of the linalg package would of course be implemented in c or fortran, but the interface to ndarray would use the well-established idea of contiguous matrices with shapes, strides, and a single memory store, supporting only numeric number types.
It's actually possible today for third-party users to add support for third-party dtypes to most linalg operations, b/c most linalg operations are implemented using the numpy ufunc machinery.
I suggested cffi since it provides a convienent and efficient interface to ndarray. Thus python could remain as a thin wrapper over the calls out to c-based libraries much like lapack_lite does today, but at the python level rather that the capi level. Yes, a python-based interface would slows the code down a bit, but I would argue that 1. the current state of lapack_litemodule.c and umath_linalg.c.src, with its myriad of compile-time macros and complex code paths, scares people away from contributing to the ongoing maintenance of the library while tying the code very closely to the lapack routines, and
I agree that simple is better than complex, but I don't see how moving those macros and code paths into a separate package decreases complexity. If anything it would increase complexity, because now we have two repos instead of one, two release schedules instead of one, and n^2 combinations of (linalg version, numpy version) to test against. -n
2. matrices larger than 3x3 or so should be spending most of the computation time in the underlying lapack/blas library irregardless of whether the interface is python-based or capi-based. Matti
On 10/08/2014 8:00 PM, numpy-discussion-request@scipy.org wrote:
Date: Sat, 9 Aug 2014 21:11:19 +0100 From: Nathaniel Smith <njs@pobox.com> Subject: Re: [Numpy-discussion] NumPy-Discussion OpenBLAS and dotblas To: Discussion of Numerical Python <numpy-discussion@scipy.org>
On Sat, Aug 9, 2014 at 8:35 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi. I am working on numpy in pypy. It would be much more challenging for me if you merged more code into the core of numpy,
Hi Matti,
I can definitely see how numpy changes cause trouble for you, and sympathize. But, can you elaborate on what kind of changes would make your life easier *that also* help make numpy proper better in their own right? Because unfortunately, I don't see how we can reasonably pass up on improvements to numpy if the only justification is to make numpypy's life easier. (I'd also love to see pypy become usable for general numerical work, but not only is it not there now, I don't see how numpypy will ultimately get us there even if we do help it along -- almost none of the ecosystem can get by numpy's python-level APIs alone.) But obviously if there are changes that are mutually beneficial, well then, that's a lot easier to justify :-)
-n
-- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
On Tue, Aug 12, 2014 at 8:26 AM, Nathaniel Smith <njs@pobox.com> wrote:
Hi Matt,
On Mon, Aug 11, 2014 at 10:46 PM, Matti Picus <matti.picus@gmail.com> wrote:
Hi Nathaniel. Thanks for your prompt reply. I think numpy is a wonderful project, and you all do a great job moving it forward. If you ask what would my vision for maturing numpy, I would like to see a grouping of linalg matrix-operation functionality into a python level package, exactly the opposite of more tightly tying linalg into the core of numpy.
As I understood it (though I admit Chuck was pretty terse, maybe he'll correct me :-)), what he was proposing was basically just a build system reorganization -- it's much easier to call between C functions that are in the same Python module than C functions that are in different modules, so we end up with lots of boilerplate gunk for the latter. I don't think it would involve any tighter coupling than we already have in practice.
I'm trying to think of the correct sequence of moves. Here are my current thoughts. - Move _dotblas down into multiarray 1. When there is cblas, add cblas implementations of decr->f->dot. 2. Reimplement API matrixproduct2 3. Make ndarray.dot a first class method and use it for numpy.dot. - Implement matmul 1. Add matrixmultiply (matmul?) to the numpy API 2. Implement __matmul__ method. 3. Add functions to linalg for stacked vectors. 4. Make sure __matmul__ works with __numpy_ufunc__ - Consider using blas_lite instead of cblas, but that is now independent of the previous steps. <snip> Chuck
Charles R Harris <charlesr.harris@gmail.com> wrote:
- Consider using blas_lite instead of cblas, but that is now independent of the previous steps.
It should also be possible to build reference cblas on top of blas_lite. (Or just create a wrapper for the parts of cblas we need.) Sturla
Charles R Harris <charlesr.harris@gmail.com> wrote:
- Move _dotblas down into multiarray 1. When there is cblas, add cblas implementations of decr->f->dot. 2. Reimplement API matrixproduct2 3. Make ndarray.dot a first class method and use it for numpy.dot. - Implement matmul 1. Add matrixmultiply (matmul?) to the numpy API 2. Implement __matmul__ method. 3. Add functions to linalg for stacked vectors. 4. Make sure __matmul__ works with __numpy_ufunc__ - Consider using blas_lite instead of cblas, but that is now independent of the previous steps.
We could consider to have a linalg._linalg module that just stores BLAS and LAPACK function pointer values as read-only integer attributes. This way we could move _dotblas into the core without actually having linalg in the core. linalg._linalg would just sit there and own BLAS and LAPACK, and no other part of NumPy would need build dependencies on these libraries. When _dotblas is imported it just imports linalg._linalg and reads whatever function pointer value it needs. It would also make it possible to remove BLAS and LAPACK build dependencies from SciPy, as long as we export most or all of BLAS and LAPACK. Sturla
On Tue, Aug 12, 2014 at 9:24 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
Charles R Harris <charlesr.harris@gmail.com> wrote:
- Move _dotblas down into multiarray 1. When there is cblas, add cblas implementations of decr->f->dot. 2. Reimplement API matrixproduct2 3. Make ndarray.dot a first class method and use it for numpy.dot. - Implement matmul 1. Add matrixmultiply (matmul?) to the numpy API 2. Implement __matmul__ method. 3. Add functions to linalg for stacked vectors. 4. Make sure __matmul__ works with __numpy_ufunc__ - Consider using blas_lite instead of cblas, but that is now independent of the previous steps.
We could consider to have a linalg._linalg module that just stores BLAS and LAPACK function pointer values as read-only integer attributes. This way we could move _dotblas into the core without actually having linalg in the core. linalg._linalg would just sit there and own BLAS and LAPACK, and no other part of NumPy would need build dependencies on these libraries.
Note that those dependencies are optional now.
When _dotblas is imported it just imports linalg._linalg and reads whatever function pointer value it needs. It would also make it possible to remove BLAS and LAPACK build dependencies from SciPy, as long as we export most or all of BLAS and LAPACK.
That's not possible. The only way you can do that is move the hard dependency on BLAS & LAPACK to numpy, which we don't want to do. Ralf
On Tue, Aug 12, 2014 at 10:53 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
Ralf Gommers <ralf.gommers@gmail.com> wrote:
That's not possible. The only way you can do that is move the hard dependency on BLAS & LAPACK to numpy, which we don't want to do.
But NumPy already depends on BLAS and LAPACK, right?
No. Numpy uses those libs when they're detected, but it falls back on its own dot implementation if they're not found. From first bullet under http://scipy.org/scipylib/building/linux.html#generic-instructions: "BLAS and LAPACK libraries (optional but strongly recommended for NumPy, required for SciPy)". BLAS/LAPACK are heavy dependencies that often give problems, which is why you don't want to require them for the casual user that only needs numpy arrays to make some plots for examples. Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Ralf Gommers <ralf.gommers@gmail.com> wrote:
No. Numpy uses those libs when they're detected, but it falls back on its own dot implementation if they're not found. From first bullet under <a href="http://scipy.org/scipylib/building/linux.html#generic-instructions:">http://scipy.org/scipylib/building/linux.html#generic-instructions:</a> "BLAS and LAPACK libraries (optional but strongly recommended for NumPy, required for SciPy)".
BLAS/LAPACK are heavy dependencies that often give problems, which is why you don't want to require them for the casual user that only needs numpy arrays to make some plots for examples.
Maybe we are not talking about the same thing, but isn't blas_lite.c and lapack_lite.c more or less f2c'd versions of reference BLAS and reference LAPACK? Sturla
On Tue, Aug 12, 2014 at 10:35 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
Ralf Gommers <ralf.gommers@gmail.com> wrote:
No. Numpy uses those libs when they're detected, but it falls back on its own dot implementation if they're not found. From first bullet under <a href="http://scipy.org/scipylib/building/linux.html#generic-instructions:">http://scipy.org/scipylib/building/linux.html#generic-instructions:</a> "BLAS and LAPACK libraries (optional but strongly recommended for NumPy, required for SciPy)".
BLAS/LAPACK are heavy dependencies that often give problems, which is why you don't want to require them for the casual user that only needs numpy arrays to make some plots for examples.
Maybe we are not talking about the same thing, but isn't blas_lite.c and lapack_lite.c more or less f2c'd versions of reference BLAS and reference LAPACK?
Not all of them, no. Just the routines that numpy itself uses. Hence, "lite". -- Robert Kern
Robert Kern <robert.kern@gmail.com> wrote:
BLAS/LAPACK are heavy dependencies that often give problems, which is why you don't want to require them for the casual user that only needs numpy arrays to make some plots for examples.
Maybe we are not talking about the same thing, but isn't blas_lite.c and lapack_lite.c more or less f2c'd versions of reference BLAS and reference LAPACK?
Not all of them, no. Just the routines that numpy itself uses. Hence, "lite".
I thought it got the 'lite' name because Netlib up to LAPACK 3.1.1 had packages named 'lapack-lite-3.1.1.tgz' in addition to 'lapack-3.1.1.tgz'. (I am not sure what the differences between the packages were.) The lapack_lite.c file looks rather complete, but it seems the build process somehow extracts only parts of it. Sturla
On Wed, Aug 13, 2014 at 12:47 AM, Sturla Molden <sturla.molden@gmail.com> wrote:
Robert Kern <robert.kern@gmail.com> wrote:
BLAS/LAPACK are heavy dependencies that often give problems, which is why you don't want to require them for the casual user that only needs numpy arrays to make some plots for examples.
Maybe we are not talking about the same thing, but isn't blas_lite.c and lapack_lite.c more or less f2c'd versions of reference BLAS and reference LAPACK?
Not all of them, no. Just the routines that numpy itself uses. Hence, "lite".
I thought it got the 'lite' name because Netlib up to LAPACK 3.1.1 had packages named 'lapack-lite-3.1.1.tgz' in addition to 'lapack-3.1.1.tgz'. (I am not sure what the differences between the packages were.)
No. https://github.com/numpy/numpy/blob/master/numpy/linalg/lapack_lite/README
The lapack_lite.c file looks rather complete, but it seems the build process somehow extracts only parts of it.
I assume you mean dlapack_lite.c? It is incomplete. It is the end product of taking the full LAPACK 3.0 distribution, stripping out the routines that are not used in numpy, and f2cing the subset. Go ahead and look for the routines in LAPACK 3.0 systematically, and you will find many of them missing. -- Robert Kern
Charles, Nothing I've seen so far envisages disturbing the existing, in my opinion flawed, Matrix Class. I trust that I have not missed anything. Compilation is a complex press for a person unfamiliar with the C. Anything you could do to simplify that would be welcome. Colin W. On 12/08/2014 1:50 PM, Charles R Harris wrote:
On Tue, Aug 12, 2014 at 8:26 AM, Nathaniel Smith <njs@pobox.com <mailto:njs@pobox.com>> wrote:
Hi Matt,
On Mon, Aug 11, 2014 at 10:46 PM, Matti Picus <matti.picus@gmail.com <mailto:matti.picus@gmail.com>> wrote: > Hi Nathaniel. > Thanks for your prompt reply. I think numpy is a wonderful project, and you > all do a great job moving it forward. > If you ask what would my vision for maturing numpy, I would like to see a > grouping of linalg matrix-operation functionality into a python level > package, exactly the opposite of more tightly tying linalg into the core of > numpy.
As I understood it (though I admit Chuck was pretty terse, maybe he'll correct me :-)), what he was proposing was basically just a build system reorganization -- it's much easier to call between C functions that are in the same Python module than C functions that are in different modules, so we end up with lots of boilerplate gunk for the latter. I don't think it would involve any tighter coupling than we already have in practice.
I'm trying to think of the correct sequence of moves. Here are my current thoughts.
* Move _dotblas down into multiarray 1. When there is cblas, add cblas implementations of decr->f->dot. 2. Reimplement API matrixproduct2 3. Make ndarray.dot a first class method and use it for numpy.dot. * Implement matmul 1. Add matrixmultiply (matmul?) to the numpy API 2. Implement __matmul__ method. 3. Add functions to linalg for stacked vectors. 4. Make sure __matmul__ works with __numpy_ufunc__ * Consider using blas_lite instead of cblas, but that is now independent of the previous steps.
<snip>
Chuck
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Aug 12, 2014 at 4:19 PM, cjw <cjw@ncf.ca> wrote:
Charles,
Nothing I've seen so far envisages disturbing the existing, in my opinion flawed, Matrix Class.
I trust that I have not missed anything.
Compilation is a complex press for a person unfamiliar with the C. Anything you could do to simplify that would be welcome.
We aren't talking about the matrix class, but rather the new '@' operator to be used with arrays. The implementation of that operator could depend on routines defined in other modules, as does the current dot method, or we can move the overall implementation down into the multiarray module. The question of using blas_lite come up because some things would be a bit simpler is we didn't need to make the code depend on whether or not there was a cblas library. <snip> Chuck
participants (7)
-
Charles R Harris -
cjw -
Matti Picus -
Nathaniel Smith -
Ralf Gommers -
Robert Kern -
Sturla Molden