Expose Lapack for Cython
Hi all, I have put together pxd files that allow direct access from Cython to the the blas and lapack functions wrapped in scipy.linalg.blas and scipy.linalg.lapack. The idea is to make these low level functions available in Cython without the corresponding Python level function call that is necessary when using the wrappers already in SciPy. They are currently in my github repository https://github.com/insertinterestingnamehere/cylinalg. The approach is based on the gist https://gist.github.com/pv/5437087. I think this would be a good addition to SciPy since it is really only another interface to the wrappers that are already there. Where would be the best place to put this? Also, what sorts of tests should be included? Any other input is welcome too. Thanks! -Ian Henriksen
Ian Henriksen <insertinterestingnamehere@gmail.com> wrote:
Hi all, I have put together pxd files that allow direct access from Cython to the the blas and lapack functions wrapped in scipy.linalg.blas and scipy.linalg.lapack. The idea is to make these low level functions available in Cython without the corresponding Python level function call that is necessary when using the wrappers already in SciPy. They are currently in my github repository <a href="https://github.com/insertinterestingnamehere/cylinalg.">https://github.com/insertinterestingnamehere/cylinalg.</a> The approach is based on the gist <a href="https://gist.github.com/pv/5437087.">https://gist.github.com/pv/5437087.</a> I think this would be a good addition to SciPy since it is really only another interface to the wrappers that are already there.
I would recommend storing the pointers globally in a module (instead of populating a struct) together with references to the LAPACK and BLAS modules in SciPy. LAPACK and BLAS extension modules should not be unloaded as long as the pointers are available, or you risk creating dangling function pointers.
Where would be the best place to put this? Also, what sorts of tests should be included? Any other input is welcome too.
You can check that they create the same output as calls to the Python interface. Or you can use codes from the LAPACK test programs to check the sanity of the output. Sturla
Hmm. It's not clear to me exactly how to make a custom Python module with additional C level pointers like that that. I'm in the process of looking through the C api to see how something like that might be done, but an example would be helpful. That would certainly be a good way to take care of references though. My end goal is to make these all available in a single pxd file. It will be necessary to initialize the struct/module from the wrappers in scipy, but other than that I'd like to make the interface as simple as possible. I don't think using the lapack test programs is feasible without a great deal of trouble. My main concern is what sort of testing needs to be done for its inclusion in scipy. I could write examples that call each of them, but that would be a rather lengthy project in its own right. I really just need a way to show that the signatures are correct. Does anyone have any suggestions as to how/where this should be included? Thanks! -Ian On Sun, Sep 7, 2014 at 10:21 AM, Sturla Molden <sturla.molden@gmail.com> wrote:
Ian Henriksen <insertinterestingnamehere@gmail.com> wrote:
Hi all, I have put together pxd files that allow direct access from Cython to the the blas and lapack functions wrapped in scipy.linalg.blas and scipy.linalg.lapack. The idea is to make these low level functions available in Cython without the corresponding Python level function call that is necessary when using the wrappers already in SciPy. They are currently in my github repository <a href="https://github.com/insertinterestingnamehere/cylinalg."> https://github.com/insertinterestingnamehere/cylinalg.</a> The approach is based on the gist <a href="https://gist.github.com/pv/5437087."> https://gist.github.com/pv/5437087.</a> I think this would be a good addition to SciPy since it is really only another interface to the wrappers that are already there.
I would recommend storing the pointers globally in a module (instead of populating a struct) together with references to the LAPACK and BLAS modules in SciPy. LAPACK and BLAS extension modules should not be unloaded as long as the pointers are available, or you risk creating dangling function pointers.
Where would be the best place to put this? Also, what sorts of tests should be included? Any other input is welcome too.
You can check that they create the same output as calls to the Python interface. Or you can use codes from the LAPACK test programs to check the sanity of the output.
Sturla
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Ian Henriksen <insertinterestingnamehere@gmail.com> wrote:
Hmm. It's not clear to me exactly how to make a custom Python module with additional C level pointers like that that.
The function pointer instances goes in a lapack.pyx file and the declarations goes in a corresponding lapack.pxd file. It is similar to what you would do in C when you have corresponding.c and .h files. The lapack module must be loaded from SciPy when its __init__.py is executed. This also initializes the pointers. Just let your lapack.pyx initialize its own pointers when it is loaded. Then any other Cython module can "cimport lapack". (In reality you don't need to use the f2py objects either, if you link lapack.pyx with LAPACK when Scipy is built.)
I'm in the process of looking through the C api to see how something like that might be done, but an example would be helpful. That would certainly be a good way to take care of references though. My end goal is to make these all available in a single pxd file.
It is better to have a .pyx file with a corresponding .pxd file.
It will be necessary to initialize the struct/module from the wrappers in scipy,
No, this can be avoided if you have corresponding .pyx and .pxd files. You should only need to "cimport lapack" if you do it right. (Actually it would be something like "cimport scipy.linalg.lapack.cylapack as lapack".) See here: http://docs.cython.org/src/userguide/sharing_declarations.html#sharing-c-fun... Sturla
Hi, On Sat, Sep 6, 2014 at 9:09 PM, Ian Henriksen <insertinterestingnamehere@gmail.com> wrote:
Hi all, I have put together pxd files that allow direct access from Cython to the the blas and lapack functions wrapped in scipy.linalg.blas and scipy.linalg.lapack. The idea is to make these low level functions available in Cython without the corresponding Python level function call that is necessary when using the wrappers already in SciPy. They are currently in my github repository https://github.com/insertinterestingnamehere/cylinalg. The approach is based on the gist https://gist.github.com/pv/5437087. I think this would be a good addition to SciPy since it is really only another interface to the wrappers that are already there.
Where would be the best place to put this? Also, what sorts of tests should be included? Any other input is welcome too.
Just to say - thanks very much for doing this - it will be very useful indeed. Cheers, Matthew
On Thu, Sep 11, 2014 at 9:24 AM, Gael Varoquaux < gael.varoquaux@normalesup.org> wrote:
On Thu, Sep 11, 2014 at 11:07:48AM -0400, Matthew Brett wrote:
Just to say - thanks very much for doing this - it will be very useful indeed.
+1000!
Gaël
PS: Note that I haven't posted on this list for at least a year. This tells you how enthousiastic I am. _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Thanks. I'm glad to hear it will be helpful. Sturla: Thanks, I see what you were saying now. I'll work on getting a pull request up in the next couple of days. Thanks! -Ian Henriksen
Ian Henriksen <insertinterestingnamehere@gmail.com> wrote:
Sturla: Thanks, I see what you were saying now. I'll work on getting a pull request up in the next couple of days.
Thanks, it was a bit convoluted to explain. (This way of using Cython modules are not so well known, perhaps.) Anyhow, I suggest we make two Cython modules with corresponding .pxd files that can be cimported like this: cimport scipy.linalg.lapack.cylapack as lapack cimport scipy.linalg.lapack.cyblas as blas After these statements it should be possible to just call blas.dgemm, lapack.dpotrf, etc. (Also ignore what I said in my earlier post about not using the f2py wrappers. Because of the f2c ABI in Apple Accelerate framework only the f2py wrappers in SciPy have a known C API. So use the f2py wrappers like you currently do.) Sturla
On Thu, Sep 11, 2014 at 9:57 AM, Ian Henriksen < insertinterestingnamehere@gmail.com> wrote:
On Thu, Sep 11, 2014 at 9:24 AM, Gael Varoquaux < gael.varoquaux@normalesup.org> wrote:
On Thu, Sep 11, 2014 at 11:07:48AM -0400, Matthew Brett wrote:
Just to say - thanks very much for doing this - it will be very useful indeed.
+1000!
Gaël
PS: Note that I haven't posted on this list for at least a year. This tells you how enthousiastic I am. _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Thanks. I'm glad to hear it will be helpful.
Sturla: Thanks, I see what you were saying now. I'll work on getting a pull request up in the next couple of days.
Thanks!
-Ian Henriksen
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome. Thanks! -Ian
On 25/09/14 23:22, Ian Henriksen wrote:
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome.
I think there is a minor bug causing a build failure. But apart from that it looks rather complete and well written. :-) Sturla
On 26/09/14 03:47, Sturla Molden wrote:
On 25/09/14 23:22, Ian Henriksen wrote:
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome.
I think there is a minor bug causing a build failure. But apart from that it looks rather complete and well written. :-)
Documentation is needed though. Sturla
On Thu, Sep 25, 2014 at 6:53 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
On 26/09/14 03:47, Sturla Molden wrote:
On 25/09/14 23:22, Ian Henriksen wrote:
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome.
It would be great to have this in Scipy - we have done the same thing, but on an ad hoc basis, in Statsmodels. It would be much preferred to have it centralized. Thanks for the work.
We are only talking about making LAPACK available for Cython though. C and C++ extension modules would have to take care of this themselves using the Python C API. Perhaps we should provide an example on how to do that?
Does anyone have information on how to do this? It would be very much appreciated. -Robert On Wed, Oct 1, 2014 at 11:12 AM, Chad Fulton <chadfulton@gmail.com> wrote:
On Thu, Sep 25, 2014 at 6:53 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
On 26/09/14 03:47, Sturla Molden wrote:
On 25/09/14 23:22, Ian Henriksen wrote:
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome.
It would be great to have this in Scipy - we have done the same thing, but on an ad hoc basis, in Statsmodels. It would be much preferred to have it centralized. Thanks for the work.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Mon, Nov 17, 2014 at 5:23 PM, Robert McGibbon <rmcgibbo@gmail.com> wrote:
We are only talking about making LAPACK available for Cython though. C and C++ extension modules would have to take care of this themselves using the Python C API. Perhaps we should provide an example on how to do that?
Does anyone have information on how to do this? It would be very much appreciated.
Thanks for reminding me about this. The easiest solution for SciPy itself would be to declare all the function pointers as public and have Cython generate a header file to go along with the module so that C and C++ modules could use the declarations with less effort. I worked on that a little, but ran in to some trouble getting complex types to work properly. I'll revisit that in the next few days and see if I can get it working. I don't currently have a full working example on hand, but I'll get back to you if I get a good one put together.
-Robert
On Wed, Oct 1, 2014 at 11:12 AM, Chad Fulton <chadfulton@gmail.com> wrote:
On Thu, Sep 25, 2014 at 6:53 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
On 26/09/14 03:47, Sturla Molden wrote:
On 25/09/14 23:22, Ian Henriksen wrote:
I've opened this pull request <https://github.com/scipy/scipy/pull/4021> that implements this. There are still several things I'd like to finish before it gets merged, but the basic functionality is there. Feedback is welcome.
It would be great to have this in Scipy - we have done the same thing, but on an ad hoc basis, in Statsmodels. It would be much preferred to have it centralized. Thanks for the work.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
In working on the pull request to implement this, I've run in to some trouble in picking the right module structure for the new Cython modules. I'm posting here to see if we can get a few better ideas that what I currently have. In the long run, I'd like to leave space so that wrappers that operate on memoryviews instead of just pointers can be added to scipy.linalg. This would, ultimately mean that there would be several new modules added in scipy.linalg. One for each the sets of function pointers corresponding to BLAS and LAPACK, and one for each of the modules defining routines that operate on memoryviews. I'm fairly certain that the C-compiled tests that are currently there should not be included in the tests directories. I've never seen that done anywhere else and was unable to get it working when I tried here. One option would be to rewrite the tests so that they use Python callable versions of the new Cython wrappers to test for correctness. That will take me a little more time, but it is probably the best solution. My current favorite idea for the structure is to do this linalg/ cython_blas_pointers (function pointers, already written) cython_blas (memory view versions, to be added later) cython_lapack_pointers (function pointers, already written) cython_lapack (memory view versions, to be added later) and then rewrite the tests in pure Python using function calls to cython_blas and cython_lapack for the functions that I have tests for. The names here are up for discussion too. The fact that these are function pointers is really just an implementation detail, but I'm not really sure what else to call the raw function pointer interface. It seems to be the most descriptive choice, since these are function pointers that operate on pointers to arrays. If we wanted shorter module names we could do linalg/ cyblas cyblas_ptrs cylapack cylapack_ptrs Another alternative would be to put these modules all in a new submodule of scipy.linalg. The current version in the pull request does this, but the submodule is called "cython" and that is causing name collision issues. The submodule could instead be called "cylinalg" or "cython_wrappers", or something else along those lines. What are your thoughts? Thanks! -Ian
On 19/11/14 20:27, Ian Henriksen wrote:
Another alternative would be to put these modules all in a new submodule of scipy.linalg. The current version in the pull request does this, but the submodule is called "cython" and that is causing name collision issues. The submodule could instead be called "cylinalg" or "cython_wrappers", or something else along those lines.
I think that special memoryview versions will be a hard nut to crack because there it is not obvious what the API should be. A memory has metadata while a Fortran 77 array has not, so metadata is typically passed along with the array in BLAS and LAPACK subroutines. memoryvrappers would probably also be expected to ensure correct alignment of the array and ensure it is Fortran contiguous. C pointer versions can easily be used with typed memoryviews, just pass the address of the first element. The important part is to get the versions with C pointers working. I am not sure the hypothetical memoryview versions, for which API is not even thought through, will ever be realized. If you use the metadata in typed memoryviews the API deviate so much from LAPACK that we might just as well use scipy.linalg Python functions. If you don't use the metadata they don't add any value beyond the C pointer versions. All in all, I think what you should do is ignore memoryviews and just focus on getting the C pointers versions ready for SciPy 0.15.0. The ideas of creating a typed memoryview abstraction for LAPACK is distracting the important work. Sturla
Yeah this is really useful Ian! Thanks for working on it. On 11 September 2014 16:24, Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:
On Thu, Sep 11, 2014 at 11:07:48AM -0400, Matthew Brett wrote:
Just to say - thanks very much for doing this - it will be very useful indeed.
+1000!
Gaël
PS: Note that I haven't posted on this list for at least a year. This tells you how enthousiastic I am. _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:
PS: Note that I haven't posted on this list for at least a year. This tells you how enthousiastic I am.
Yes, many projects depending on SciPy would benefit from access to SciPy's LAPACK and BLAS. Having to link BLAS and LAPACK again is a PITA, not to mention the pain of building ATLAS or OpenBLAS correctly on your own. Matlab has always provided access to its BLAS and LAPACK for Fortran and C MEX files. 15 years ago, when I was writing MEX files extensively I used this option a lot. It takes away a lot of pain... We are only talking about making LAPACK available for Cython though. C and C++ extension modules would have to take care of this themselves using the Python C API. Perhaps we should provide an example on how to do that? BTW: SciPy already provides access to BLAS and LAPACK for Fortran extension modules through f2py. This is a fantastic feature, but unfortunately few seem to be aware of it. Now we are actually just piggybacking the Cython code on the f2py machinery for this feature. So Pearu Peterson and the other authors of f2py deserve to be credited. Sturla
participants (7)
-
Aaron O'Leary -
Chad Fulton -
Gael Varoquaux -
Ian Henriksen -
Matthew Brett -
Robert McGibbon -
Sturla Molden