[Numpy-discussion] Proposed Roadmap Overview

[Nathaniel Smith]

On Tue, Feb 21, 2012 at 4:04 AM, Travis Oliphant <travis at continuum.io> wrote:
> It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do the
> translation from Python byte-code to LLVM.   This LLVM can then be "JIT"ed.
>   I have several applications that I would like to use this for.   It would
> be possible to write "more of NumPy" using this approach.     Initially, it
> makes it *very* easy to create a machine-code ufunc from Python code.
> There are other use-cases of having loops written in Python and plugged in
> to a calculation, filtering, or indexing framework that this system will be
> useful for.

Very neat!

It's interesting that you decided to use Python bytecode as your
source representation. I'm curious what your strategy is for
overcoming all the challenges that have plagued previous attempts to
efficiently compile "real Python"? (Unladen Swallow, PyPy, etc.) Just
support some subset of the language that's easy to handle and do type
inference over? Or do you plan to continue using Python as your input
language?

I guess the conventional wisdom would be that there's a lot of
potential for using LLVM to generate efficient specialized loops for
numpy on the fly (cf. llvm-pipe for a similar and successful project),
but that the key would be to use a more specialized representation
than Python bytecode -- one that left out hard/irrelevant parts of the
language, that had richer type information, that didn't change around
for different Python releases, etc.

-- Nathaniel