[SciPy-Dev] Using Pythran to compile some of the scipy internals

[Serge Guelton]

Hi Scipy-dev,

= Context

I've started the experiment of compiling some of the scipy kernels with
the Pythran[0] compiler.

The approach was basically to pick some easy .pyx file, turn them back
to high-level Python + Numpy files then processing them with Pythran.
This results in easier to maintain, Python-compatible files that run at
the same speed as their Cython alternative up to 10 times faster
depending on the kernels and whether SIMD instructions are activated or
not (I did not play with parallelization in that experiment).

This required a few improvement to Pythran, but not that much, see

    https://github.com/serge-sans-paille/pythran/pull/770

for the associated PR on pythran, which includes the following scipy
kernels:

    hausdorff.py
    max_len_seq_inner.py
    solve_toeplitz.py
    spectral.py

= Why bother?

According to that first round of experiments, using Pythran can bring
extra speed improvements while remaining at a higher level than Cython.
Both points looks like an improvement to me.

= Cost?

This adds an extra dependency on Pythran, which uses C++ as backend.
This increases the failure surface. Although alive since 2012 and being
tested a lot [1] on Linux (but scarcely on Windows), its is obviously
less mature than cython

= Alternatives

There is an experimental Pythran mode in cython[2] that uses Pythran as
a backend for numpy operations. Unfortunately it is still at early
stages and cannot translate calls to ``np.roll`` or ``np.sum(a[i,:] - b[j, :])`` while Pythran supports it.

Instead of translating Cython files, I could also focus on some
pure-python functions. I tested Pythran on the rosenbrock function and I
get good speedup (from 1.5x to 4x depending on vectorization being
enabled or not) there too.

So yeah, that's a rather long introduction to probe the interest here
around that idea :-)

Best,
Serge

PS: I started https://github.com/scipy/scipy/pull/8306 meanwhile, but
let's discuss here first :-)



[0] https://github.com/serge-sans-paille/pythran
[1] https://travis-ci.org/serge-sans-paille/pythran/builds/330052310
[2] http://cython.readthedocs.io/en/latest/src/userguide/numpy_pythran.html