[SciPy-Dev] Numba as a dependency for SciPy?

[Ralf Gommers]

On Tue, Mar 6, 2018 at 12:02 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
wrote:

> Interesting discussion. Would our plan be to support both side-by-side for
> a while & just see what happens with the evolution of the ecosystem?
>

Indeed. I expect we would get lots of easy wins - stuff that's slow now and
we can speed up by simply adding @jit and that no one wants to port to
Cython because that is a lot of work.


> If there's no clear winner in the short-term would we discourage PRs that
> simply migrate from Cython to numba for say 1.5 x performance increase?
>

Yes that's not the best idea. I'd say start carefully, just adding some jit
calls. Then it's also easy to reverse if something goes wrong.


> What about an algorithm that mixes the two approaches -- some numba and
> some Cython components for whatever reason -- is that discouraged?
>

That seems like a recipe for disaster.


>
> It looks like numba plays ok with airspeed velocity -- presumably mixing
> Cython / numba in our suite will be ok?
>

They're both okay, asv will be completely agnostic to implementation
language.

Ralf


>
>
>
> On 5 March 2018 at 21:06, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>> Hi all,
>>
>> Goal of this email: start a discussion to decide whether we'd be okay
>> with relying on Numba as a dependency, now or in 1-2 years' time.
>>
>> Context: in https://github.com/pydata/sparse/issues/126 a discussion is
>> ongoing about whether to adopt Cython or Numba, with Numba being preferred
>> by the majority. That `sparse` package is meant to provide sparse *arrays*
>> that down the line should either be replacing our current sparse *matrices*
>> or at least be integrated in scipy.sparse in addition to them. See
>> https://github.com/scipy/scipy/issues/8162 and
>> https://github.com/hameerabbasi/sparse-ndarray-protocols for more
>> details on that.
>>
>> Also related is the question from Serge Guelton some weeks ago about
>> whether we'd want to rely on Pythran: https://mail.python.org/piperm
>> ail/scipy-dev/2018-January/022325.html
>>
>> On that Pythran thread I commented that we'd want to take these aspects
>> into account:
>> - portability
>> - performance
>> - maturity
>> - maintenance status (active devs, how quick do bugs get fixed after a
>> release with an issue)
>> - ease of use (@jit vs. Pythran comments vs. translate to .pyx syntax)
>> - size of generated binaries
>> - templating support for multiple dtypes
>> - debugging and optimization experience/tool
>>
>> Debugging is one of the ones where I'd say Numba is still worse than
>> Cython, however that's being resolved as we speak:
>> https://github.com/numba/numba/issues/2788
>>
>> One thing I missed in the above list is dependencies: while our use of
>> Cython only adds a build-time dependency, Numba would add a run-time
>> dependency. Given that binary wheels and conda packages for all major
>> platforms are available that's not a showstopper, but it matters.
>>
>> Overall I'd say that:
>> - Numba is better than Cython at: performance, ease of use, size of
>> generated binaries, and templating support for multiple dtypes. Possibly
>> also maintenance status right now.
>> - Numba and Cython are about equally good at portability (I think, not
>> much data about exotic platforms for Numba).
>> - Cython is better than Numba at: maturity, debugging (but not for long
>> anymore probably), dependencies.
>>
>> I'm usually pretty conservative in these things, but considering the
>> above I'm leaning towards saying use of Numba should be allowed in the
>> future. The added run-time dependency is the one major downside that's
>> going to stay, however compared to our Fortran headaches that's a
>> relatively small issue.
>>
>> Thoughts?
>>
>> Cheers,
>> Ralf
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180306/545e4e7c/attachment-0001.html>