[SciPy-Dev] Numba as a dependency for SciPy?

[Ralf Gommers]

On Wed, Mar 7, 2018 at 3:09 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Hi,
>
> Ralf Gommers kirjoitti 06.03.2018 klo 05:06:
>
>> Goal of this email: start a discussion to decide whether we'd be okay with
>> relying on Numba as a dependency, now or in 1-2 years' time.
>>
>
> I think the main concerns indeed are portability and maturity.
>
> The advantages of Numba on the other hand are clear, as it mostly avoids
> the multiple-language mess.
>
> We might also want to consider using cffi at some point. Currently, we
> basically just have Cython, f2py, and hand-written C code for ffi.
>
> I guess LLVM supports most of the architectures we would be interested in,
> but e.g. the fact that ARM does not work yet is not so nice. Moreover,
> libLLVM is 50+ megabyte blob, but maybe today when people run text editors
> on web browsers instead of vice versa that's not a big deal.
>
> The idea that we could use `scipy._lib.jit` that's either a noop or Numba
> jit does not sound good in practice: for code where the JIT is wanted, the
> performance without it is likely unacceptable.


I don't think it's performance here that matters. For >99.x% of our users
Numba will install just fine, the exceptions being the exotic architectures
like POWER8. On those, having SciPy import and work as expected is enough;
those functions that are implemented with Numba then run slower - that's at
the start <1% of functions we offer for <1% of our userbase.

Also, I don't think performance will necessarily be unacceptable. There are
a bunch of places in the existing code base where we can throw in @jit and
get speedups basically for free. Performance in the noop case will then be
what it is today - not great, but apparently also not enough of a problem
that someone has attempted to go to Cython.

Ralf



> (For PyPy, the no-op decorator in principle could be acceptable, but only
> with numpypy which IIUC is not production ready currently, cpyext+numpy
> likely won't be faster than CPython.) Moreover, we presumably would like to
> use features such as `numba.cfunc`. So as I see it, either Numba is a hard
> dependency, or we don't use it.
>
> On maturity: I don't know what is the API stability status for Numba,
> presumably the basic API is stable.
>
> Numba debugging also in my experience has several paper cuts, e.g., the
> compilation and runtime errors are cryptic --- they assume you know how
> numba works, and don't include such niceties as line numbers or useful
> tracebacks, etc. Maybe this will improve in coming years.
>
>         Pauli
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180307/8ffddab5/attachment.html>