[SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU

[Ralf Gommers]

On Fri, May 21, 2021 at 2:11 AM Neil Godber <njgodber at gmail.com> wrote:

> Hi dev-list,
>
> Further last, realised that I failed to link to the issue itself despite
> linking to virtually everything else:
>
> Replace superLU sequential with superLU dist · Issue #14096 · scipy/scipy
> (github.com) <https://github.com/scipy/scipy/issues/14096>
>
> Cheers,
> Neil
>
> On Fri, 21 May 2021 at 10:04, Neil Godber <njgodber at gmail.com> wrote:
>
>> Hi dev-list,
>>
>
Thanks for the proposal Neil.


>> I've logged an enhancement request on github to bring the
>> parallel versions of SuperLU into scipy. As requested I'm circulating the
>> enhancement here to solicit comment/feedback/engagement.
>>
>> Content of request is as follows (with light editing and links):
>>
>> "Is your feature request related to a problem? Please describe.
>>
>> The direct solution of sparse matrices is a common problem that arises
>> across many domains. Current scipy.sparse.linalg provides spsolve and splu
>> to solve such classes of problems. Neither solution appears to utilise more
>> than one or two threads leaving a lot of the available compute on modern
>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore
>> processors but are either minimally supported and/or locked behind
>> proprietary libraries.
>>
>> Describe the solution you'd like
>> Fortunately SuperLU has two multithreaded implementations superLU_mt and
>> superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov)
>> <https://portal.nersc.gov/project/sparse/superlu/>). Ideally SciPy would
>> replace the current sequential implementation with one of the parallel
>> implementations to yield large performance improvements.
>>
> The relevant one seems to be superLU_mt. The `_dt` flavor is for
distributed computing, which is out of scope for SciPy. superLU_mt says it
has both pthreads and OpenMP interfaces - we'd prefer pthreads (and
actually forbid OpenMP within SciPy).

The last release of superLU_mt is v3.1, from March 2016. Our current copy
of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches
applied to it, and then https://github.com/scipy/scipy/pull/12243 did a
sync with SuperLU master in May 2020. So it would be necessary to figure
out if we can get superLU_mt from upstream master as well, and that we
don't lose bug fixes we applied since 2016. Could you look into that Neil?
And do you know if the parallel and sequential versions are developed in
sync or not?

Cheers,
Ralf


Describe alternatives you've considered
>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines
>> (github.com) <https://github.com/dwfmarchant/pyMKL>) - MKL with issues
>> associated with that for non Intel processors, unmaintained relatively to
>> scipy, PyTrilinos (implements all three superLU types but only available on
>> osx and linux PyTrilinos | Trilinos
>> <https://trilinos.github.io/pytrilinos.html>), petsc (petsc4py PETSc /
>> petsc · GitLab <https://gitlab.com/petsc/petsc>) (implements all three
>> superLU types but only available on osx and linux). All are relatively
>> inaccessible compared to ubiquity and accessibility of scipy.
>>
>> Additional context (e.g. screenshots)
>> Anecdotally large number of projects use various non scipy sparse solvers
>> to achieve high level of performance. This comes with considerable time and
>> maintenance investment. It would be fantastic for scipy to provide a
>> 'performance competitive' sparse direct solver out of the hood as it does
>> for dense matrices."
>>
>>
>> Cheers,
>>
>> Neil
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210523/77d22c80/attachment.html>