[SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU

[Neil Godber]

Hi Ralf + ScipyDev,

Dr Li has helpfully uploaded the SuperLU_mt code base to github
https://github.com/xiaoyeli/superlu_mt . Dr Li indicated that most
development effort is focused on _dt, but that sequential and mt remain
supported and bug reports will be attended to.

Cheers,
Neil

On Mon, 24 May 2021 at 11:09, Neil Godber <njgodber at gmail.com> wrote:

> Hi Ralf,
>
> Thanks for your email and happy to assist in whatever way I can. I've
> reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the
> principal maintainer of the code base with a request to host the _mt code
> on github. Given the other two versions are already hosted I can't see that
> being an issue. I've also queried whether developments are done in sync and
> if so, if there is a pending patching for mt in the similar fashion to
> sequential.
>
> I will revert with further information as I receive it.
>
> Cheers,
> Neil
>
> On Mon, 24 May 2021 at 04:33, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>>
>>
>> On Fri, May 21, 2021 at 2:11 AM Neil Godber <njgodber at gmail.com> wrote:
>>
>>> Hi dev-list,
>>>
>>> Further last, realised that I failed to link to the issue itself despite
>>> linking to virtually everything else:
>>>
>>> Replace superLU sequential with superLU dist · Issue #14096 ·
>>> scipy/scipy (github.com) <https://github.com/scipy/scipy/issues/14096>
>>>
>>> Cheers,
>>> Neil
>>>
>>> On Fri, 21 May 2021 at 10:04, Neil Godber <njgodber at gmail.com> wrote:
>>>
>>>> Hi dev-list,
>>>>
>>>
>> Thanks for the proposal Neil.
>>
>>
>>>> I've logged an enhancement request on github to bring the
>>>> parallel versions of SuperLU into scipy. As requested I'm circulating the
>>>> enhancement here to solicit comment/feedback/engagement.
>>>>
>>>> Content of request is as follows (with light editing and links):
>>>>
>>>> "Is your feature request related to a problem? Please describe.
>>>>
>>>> The direct solution of sparse matrices is a common problem that arises
>>>> across many domains. Current scipy.sparse.linalg provides spsolve and splu
>>>> to solve such classes of problems. Neither solution appears to utilise more
>>>> than one or two threads leaving a lot of the available compute on modern
>>>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore
>>>> processors but are either minimally supported and/or locked behind
>>>> proprietary libraries.
>>>>
>>>> Describe the solution you'd like
>>>> Fortunately SuperLU has two multithreaded implementations superLU_mt
>>>> and superLU_dt, both actively maintained (SuperLU: Home Page
>>>> (nersc.gov) <https://portal.nersc.gov/project/sparse/superlu/>).
>>>> Ideally SciPy would replace the current sequential implementation with one
>>>> of the parallel implementations to yield large performance improvements.
>>>>
>>> The relevant one seems to be superLU_mt. The `_dt` flavor is for
>> distributed computing, which is out of scope for SciPy. superLU_mt says it
>> has both pthreads and OpenMP interfaces - we'd prefer pthreads (and
>> actually forbid OpenMP within SciPy).
>>
>> The last release of superLU_mt is v3.1, from March 2016. Our current copy
>> of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches
>> applied to it, and then https://github.com/scipy/scipy/pull/12243 did a
>> sync with SuperLU master in May 2020. So it would be necessary to figure
>> out if we can get superLU_mt from upstream master as well, and that we
>> don't lose bug fixes we applied since 2016. Could you look into that Neil?
>> And do you know if the parallel and sequential versions are developed in
>> sync or not?
>>
>> Cheers,
>> Ralf
>>
>>
>> Describe alternatives you've considered
>>>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL
>>>> routines (github.com) <https://github.com/dwfmarchant/pyMKL>) - MKL
>>>> with issues associated with that for non Intel processors, unmaintained
>>>> relatively to scipy, PyTrilinos (implements all three superLU types but
>>>> only available on osx and linux PyTrilinos | Trilinos
>>>> <https://trilinos.github.io/pytrilinos.html>), petsc (petsc4py PETSc /
>>>> petsc · GitLab <https://gitlab.com/petsc/petsc>) (implements all three
>>>> superLU types but only available on osx and linux). All are relatively
>>>> inaccessible compared to ubiquity and accessibility of scipy.
>>>>
>>>> Additional context (e.g. screenshots)
>>>> Anecdotally large number of projects use various non scipy sparse
>>>> solvers to achieve high level of performance. This comes with considerable
>>>> time and maintenance investment. It would be fantastic for scipy to provide
>>>> a 'performance competitive' sparse direct solver out of the hood as it does
>>>> for dense matrices."
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Neil
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210529/5ffb1c63/attachment-0001.html>