[SciPy-Dev] Numba as a dependency for SciPy?

[Juan Luis Cano]

Hi all,

Thanks Jaime for pinging me, I don't follow scipy-dev regularly.

My frustration with the first release of poliastro, a Python library for
astrodynamics that I created some years ago, was that it mixed MATLAB
(Octave), FORTRAN (all caps) and Python code and that it was a mess to
distribute (this was 2013). I had no experience with C so Cython looked
intimidating to me (I even tried to do some debugging recently and simply
couldn't cope with cython-gdb, see https://groups.google.com/d/
msg/cython-users/SCk2IDG9M5g/muhUhmw9AwAJ). When I saw that I could achieve
a decent performance with numba, I threw away all the FORTRAN and now the
library is pure Python.

For some releases, my approach was exactly what Ralf said: have a `@jit`
decorator that selected what to do depending on whether numba was
installed. This was nice because there was no easy way to install numba
with pip when I started, and in doing so I guaranteed performance with
`conda install poliastro` and a working system with `pip install poliastro`.

https://github.com/poliastro/poliastro/blob/0.6.x/src/poliastro/jit.py

However, now there are wheels on PyPI for both numba and llvmlite, so I
removed the conditional `@jit` and included numba as a required dependency
(except on PyPy):

https://github.com/poliastro/poliastro/blob/master/setup.py#L40

As a last note, numba works beautifully with C code through CFFI:

https://www.anaconda.com/blog/developer-blog/calling-c-
libraries-numba-using-cffi/
http://old.pybonacci.org/2016/02/07/como-crear-extensiones-
en-c-para-python-usando-cffi-y-numba/ [my take, in Spanish]

I understand the concerns about numba because it used to work only with
conda and "it comes from a company" (which some people consider a bad
thing). numba is no doubt a complex project, but the devs are committed to
it and the installation and packaging are now few (specially compared to
the situation we had in 2013). Also, the Julia folks will drive the LLVM
ecosystem to more platforms I would say.

Hope this helps!

On Wed, Mar 7, 2018 at 9:40 AM, Jaime Fernández del Río <
jaime.frio at gmail.com> wrote:

> On Wed, Mar 7, 2018 at 9:02 AM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Tue, Mar 6, 2018 at 1:55 PM, Stephan Hoyer <shoyer at gmail.com> wrote:
>>
>>> Numba does have a mechanism for exporting pre-compiled code:
>>> http://numba.pydata.org/numba-doc/dev/user/pycc.html
>>>
>>> It would be interesting to see if those limitations are flexible enough
>>> for SciPy.
>>>
>>
>> I suspect that (a) we're then going to run into more Numba bugs because
>> pre-compilation is not well-tested, and (b) we throw away some of the
>> advantages of Numba, e.g. we then get back the binary size explosion for
>> multiple dtype templating.
>>
>>
>>> On Tue, Mar 6, 2018 at 1:21 PM CJ Carey <perimosocordiae at gmail.com>
>>> wrote:
>>>
>>>> I think adding a required runtime dependency may be overly restrictive,
>>>> given scipy's position near(-ish) the base of the scientific computing
>>>> pyramid.
>>>>
>>>> Would it be possible to run numba-optimized code on systems with numba
>>>> installed without impacting "vanilla" users?
>>>>
>>>
>> It's worth thinking about. We could put a jit decorator in scipy._lib
>> that becomes numba @jit if numba is installed and is do-nothing otherwise.
>>
>
> I'll admit I have a "fear of the unknown" mistrust of numba, but reading
> through this thread I was thinking of something like this as being
> something even I would have no problem with.
>
> Juan Luis Cano, who probably reads this, but who I have just in case CCed
> in this e-mail, is the author of https://github.com/poliastro/poliastro,
> a numerical library that gave up on Fortran/Cython/C by design and embraced
> Numba from the start, would be nice to hear his take on making it a
> dependency.
>
> Jaime
>
>
>>
>> Ralf
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
> --
> (\__/)
> ( O.o)
> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
> de dominación mundial.
>



-- 
Juan Luis Cano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180307/c480e1bb/attachment-0001.html>