Would you consider making ccallback part of the public scipy interface?
Hi, This is my first message on the mailing list. The question was originally asked in the following github issue, https://github.com/scipy/scipy/issues/19148 . Here is the gist of it: I've been developing a symplectic ODE solver in Cython. Aiming for efficiency, the solver hugely benefits from scipy providing a public cython interface to blas/lapack. In other words, I'm very happy to use scipy.linalg.cython_blas.dgemm and such. For small systems, I clearly measure a bottleneck in the python machinery during the call to the user provided callback defining the ODE. I had left it at that for a few months till I looked at the input for scipy.integrate.quad and discovered the existence of scipy.LowLevelCallable. I tested it in my own code and it so happens that yes, it eliminates the bottleneck Iwas experiencing. Great. However, in order to access the pointer to the a c function in a scipy.LowLevelCallable, following https://github.com/scipy/scipy/blob/main/scipy/_lib/_ccallback_c.pyx , I need to import the follwoing: from .ccallback cimport (ccallback_t, ccallback_prepare, ccallback_release, CCALLBACK_DEFAULTS, ccallback_signature_t) Those functions are not part of the public Cython interface in scipy as far as I can tell https://docs.scipy.org/doc/scipy/dev/contributor/public_cython_api.html , so I had to copy paste the files https://github.com/scipy/scipy/blob/main/scipy/_lib/src/ccallback.h and https://github.com/scipy/scipy/blob/main/scipy/_lib/ccallback.pxd into my own project. This felt a bit hacky. Hence the question: would you consider making ccallback.pxd part of the public Cython API ?
On Tue, Aug 29, 2023 at 11:48 AM Gabriel Fougeron < gabriel.fougeron@hotmail.fr> wrote:
Hi,
This is my first message on the mailing list. The question was originally asked in the following github issue, https://github.com/scipy/scipy/issues/19148 .
Here is the gist of it: I've been developing a symplectic ODE solver in Cython. Aiming for efficiency, the solver hugely benefits from scipy providing a public cython interface to blas/lapack. In other words, I'm very happy to use scipy.linalg.cython_blas.dgemm and such.
For small systems, I clearly measure a bottleneck in the python machinery during the call to the user provided callback defining the ODE. I had left it at that for a few months till I looked at the input for scipy.integrate.quad and discovered the existence of scipy.LowLevelCallable.
I tested it in my own code and it so happens that yes, it eliminates the bottleneck Iwas experiencing. Great. However, in order to access the pointer to the a c function in a scipy.LowLevelCallable, following https://github.com/scipy/scipy/blob/main/scipy/_lib/_ccallback_c.pyx , I need to import the follwoing:
from .ccallback cimport (ccallback_t, ccallback_prepare, ccallback_release, CCALLBACK_DEFAULTS, ccallback_signature_t)
Those functions are not part of the public Cython interface in scipy as far as I can tell https://docs.scipy.org/doc/scipy/dev/contributor/public_cython_api.html , so I had to copy paste the files https://github.com/scipy/scipy/blob/main/scipy/_lib/src/ccallback.h and https://github.com/scipy/scipy/blob/main/scipy/_lib/ccallback.pxd into my own project. This felt a bit hacky.
Hence the question: would you consider making ccallback.pxd part of the public Cython API ?
Hi Gabriel, thanks for the suggestion! The reply is a bit late because the question is a little hairy - both because it's a tricky bit of machinery, and because there is no obvious place to expose this new Cython API. For the `linalg` and `special` submodules we have `cython_linalg` and `cython_special`, but we can't follow that pattern here because `_lib` is private. If we'd follow the `cython_*` pattern, the logical place would be `scipy.cython_scipy` next to LowlevelCallable. But I'm not sure I like that:) I think your request is very reasonable in principle though. Maybe you or someone else have a better idea about where this should live. Also, I don't think you shared your own code or a small self-contained example on the GitHub issue - that could be instructive to see. Cheers, Ralf
Hey Ralf, Thank you so much for your answer. scipy.cython_scipy sounds good to me, but TBH I care most about the fact that the functionality exists, so I might not be the best to juge. Else, how about scipy.cython_lib ? Or just scipy.cython since LowLevelCallable is exposed right at the root of scipy (that's quite unique afaik btw) My own code lives at https://github.com/gabrielfougeron/choreo/tree/main/choreo/scipy_plus/cython . I wrote it mimicking what I read in scipy.integrate.quad. The purpose of the code is to provide symplectic integrators for partitioned hamiltonian systems following this issue https://github.com/scipy/scipy/issues/12690 Would you rather continue this discussion here or on github ? (I'm asking in particular because I don't get notified here) Cheers, Gabriel
participants (2)
-
Gabriel Fougeron -
Ralf Gommers