[SciPy-Dev] Using Pythran to compile some of the scipy internals

[Ralf Gommers]

On Sat, Jan 20, 2018 at 11:10 PM, Serge Guelton <serge.guelton at telecom-
bretagne.eu> wrote:

> Hi Scipy-dev,
>
> = Context
>
> I've started the experiment of compiling some of the scipy kernels with
> the Pythran[0] compiler.
>
> The approach was basically to pick some easy .pyx file, turn them back
> to high-level Python + Numpy files then processing them with Pythran.
> This results in easier to maintain, Python-compatible files that run at
> the same speed as their Cython alternative up to 10 times faster
> depending on the kernels and whether SIMD instructions are activated or
> not (I did not play with parallelization in that experiment).
>
> This required a few improvement to Pythran, but not that much, see
>
>     https://github.com/serge-sans-paille/pythran/pull/770
>
> for the associated PR on pythran, which includes the following scipy
> kernels:
>
>     hausdorff.py
>     max_len_seq_inner.py
>     solve_toeplitz.py
>     spectral.py
>
> = Why bother?
>
> According to that first round of experiments, using Pythran can bring
> extra speed improvements while remaining at a higher level than Cython.
> Both points looks like an improvement to me.
>

Another potential benefit is to decrease the size of the binary
distribution of SciPy. Cython extensions are quite expensive in this
respect. Do you have an idea of how Pythran compares? Both in case of only
float64 inputs, and with templated inputs? A good example of the latter is
scipy.ndimage.label, which is a straightforward function that ends up being
a 700kb .so


> = Cost?
>
> This adds an extra dependency on Pythran, which uses C++ as backend.
>

Only a build-time dependency. That's not my major worry from the
maintenance point of view.


> This increases the failure surface. Although alive since 2012 and being
> tested a lot [1] on Linux (but scarcely on Windows), its is obviously
> less mature than cython
>

This is a worry. We need Windows, Linux, macOS (officially supported, also
the 32-bit flavors) as well as less commonly used Unix/BSD-like platforms.

I guess Windows needs separate testing, both with gcc and MSVC. But for all
other platforms, can you say something about portability based on the kind
of C++ Pythran generates?


> = Alternatives
>
> There is an experimental Pythran mode in cython[2] that uses Pythran as
> a backend for numpy operations. Unfortunately it is still at early
> stages and cannot translate calls to ``np.roll`` or ``np.sum(a[i,:] - b[j,
> :])`` while Pythran supports it.
>
> Instead of translating Cython files, I could also focus on some
> pure-python functions. I tested Pythran on the rosenbrock function and I
> get good speedup (from 1.5x to 4x depending on vectorization being
> enabled or not) there too.
>
> So yeah, that's a rather long introduction to probe the interest here
> around that idea :-)
>

Interest in general, but there's a long way to go - our requirements are
pretty demanding. I have seen some comparisons between Cython, Pythran and
Numba in terms of performance and ease of use, but never a comprehensive
comparison from the point of view of library authors. I know Travis O. has
an interest in seeing Numba being adopted more widely, which will also need
such a comparison. It should cover at least:

- portability
- performance
- maturity
- maintenance status (active devs, how quick do bugs get fixed after a
release with an issue)
- ease of use (@jit vs. Pythran comments vs. translate to .pyx syntax)
- size of generated binaries
- templating support for multiple dtypes
- debugging and optimization experience/tools

Is anyone aware of such a comparison? Or interested in putting it together?

Cheers,
Ralf



> Best,
> Serge
>
> PS: I started https://github.com/scipy/scipy/pull/8306 meanwhile, but
> let's discuss here first :-)
>
>
>
> [0] https://github.com/serge-sans-paille/pythran
> [1] https://travis-ci.org/serge-sans-paille/pythran/builds/330052310
> [2] http://cython.readthedocs.io/en/latest/src/userguide/numpy_p
> ythran.html
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/ffb2c3b9/attachment-0001.html>