[SciPy-Dev] Numba as a dependency for SciPy?

[Stanley Seibert]

A couple notes of clarification:

The way llvmlite (Numba's Python wrapper around LLVM) is designed, we
statically link to the LLVM library for several reasons:

   - LLVM changes its C++ API and its IR syntax frequently enough that it
   is never safe to assume Linux distributions ship the version Numba/llvmlite
   requires, or that it was compiled with the options we need.

   - This allows llvmlite to contain only the subset of LLVM that is needed
   for the JIT.  Your 50MB estimate is about right for the install size of
   llvmlite (54 MB on Linux x86_64, 30 MB on macOS 64-bit), but a full install
   of LLVM has more like 165MB of library code.

I just wanted to call all this out in case someone is assuming that Numba
requires an installation of LLVM.  It does not, and in fact deliberately
ignores any system installation of LLVM on purpose.

Regarding the API stability: Numba's API is stable in practice because the
most common interface (the compiler decorators, like @jit) tend to be very
simple.  We are planning to release 1.0 this year, at which point we will
clearly identify in the documentation which APIs and options are stable and
which we consider experimental.  That will also identify a clear point
where we think that Numba is safe for core projects to take as a dependency.

Debugging support continues to be an area we work on, so your criticism is
fair.

On Wed, Mar 7, 2018 at 5:09 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Hi,
>
> Ralf Gommers kirjoitti 06.03.2018 klo 05:06:
>
>> Goal of this email: start a discussion to decide whether we'd be okay with
>> relying on Numba as a dependency, now or in 1-2 years' time.
>>
>
> I think the main concerns indeed are portability and maturity.
>
> The advantages of Numba on the other hand are clear, as it mostly avoids
> the multiple-language mess.
>
> We might also want to consider using cffi at some point. Currently, we
> basically just have Cython, f2py, and hand-written C code for ffi.
>
> I guess LLVM supports most of the architectures we would be interested in,
> but e.g. the fact that ARM does not work yet is not so nice. Moreover,
> libLLVM is 50+ megabyte blob, but maybe today when people run text editors
> on web browsers instead of vice versa that's not a big deal.
>
> The idea that we could use `scipy._lib.jit` that's either a noop or Numba
> jit does not sound good in practice: for code where the JIT is wanted, the
> performance without it is likely unacceptable. (For PyPy, the no-op
> decorator in principle could be acceptable, but only with numpypy which
> IIUC is not production ready currently, cpyext+numpy likely won't be faster
> than CPython.) Moreover, we presumably would like to use features such as
> `numba.cfunc`. So as I see it, either Numba is a hard dependency, or we
> don't use it.
>
> On maturity: I don't know what is the API stability status for Numba,
> presumably the basic API is stable.
>
> Numba debugging also in my experience has several paper cuts, e.g., the
> compilation and runtime errors are cryptic --- they assume you know how
> numba works, and don't include such niceties as line numbers or useful
> tracebacks, etc. Maybe this will improve in coming years.
>
>         Pauli
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180307/4620fba8/attachment.html>