[Cython] Wacky idea: proper macros

Dag Sverre Seljebotn d.s.seljebotn at astro.uio.no
Tue May 1 10:32:48 CEST 2012

On 05/01/2012 10:29 AM, Dag Sverre Seljebotn wrote:
> On 04/30/2012 11:36 PM, William Stein wrote:
>> On Mon, Apr 30, 2012 at 2:32 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> Wes McKinney<wesmckinn at gmail.com> wrote:
>>>> On Mon, Apr 30, 2012 at 4:55 PM, Nathaniel Smith<njs at pobox.com> wrote:
>>>>> On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>>> JIT is really the way to go. It is one thing that a JIT could
>>>> optimize the
>>>>>> case where you pass a callback to a function and inline it run-time.
>>>> But
>>>>>> even if it doesn't get that fancy, it'd be great to just be able to
>>>> write
>>>>>> something like "cython.eval(s)" and have that be compiled (I guess
>>>> you could
>>>>>> do that now, but the sheer overhead of the C compiler and all the
>>>> .so files
>>>>>> involved means nobody would sanely use that as the main way of
>>>> stringing
>>>>>> together something like pandas).
>>>>> The overhead of running a fully optimizing compiler over pandas on
>>>>> every import is pretty high, though. You can come up with various
>>>>> caching mechanisms, but they all mean introducing some kind of
>>>> compile
>>>>> time/run time distinction. So I'm skeptical we'll just be able to get
>>>>> rid of that concept, even in a brave new LLVM/PyPy/Julia world.
>>>>> -- Nathaniel
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>> I'd be perfectly OK with just having to compile pandas's "data engine"
>>>> and generate loads of C/C++ code. JIT-compiling little array
>>>> expressions would be cool too. I've got enough of an itch that I might
>>>> have to start scratching pretty soon.
>>> I think a good start is:
>>> Myself I'd look into just using Jinja2 to generate all the Cython
>>> code, rather than those horrible Python interpolated strings...that
>>> should give you something that's at least rather pleasant for you to
>>> work with once you are used to it (even if it is a bit horrible to
>>> newcomers to the code base).
>>> You can even check in the generated sources.
>>> And we've discussed letting cython be smart with templating languages
>>> and error report on a line in the original template, such features
>>> will certainly accepted once somebody codes it up.
>>> (I can give you me breakdown of how I eliminate other templating
>>> languages than Jinja2 for this purpose tomorrow if you are interested).
>> Can you point us to a good example of you using jinja2 for this purpose?
> Sure, I just needed some sleep...
> I only use it for C code, haven't used it for Cython so far (I tend to
> write things in C and wrap it in Cython).
> 1)
> https://github.com/dagss/elemental4py/blob/master/src/elemental_wrapper.cpp.in
> (work-in-progress) Here I use Jinja2 to write a C wrapper around
> Elemental (Elemental is a library for dense linear algebra over MPI).
> The C++ library is a heavy user of templates, I replace the templates
> with run-time dispatches using if-tests, so that rather than
> "DistMatrix<double, MC, MR>" you have an elem_matrix struct with
> 2)
> https://github.com/wavemoth/wavemoth/blob/master/src/legendre_transform.c.in
> This is a numerical kernel where I do loop unrolling etc. using
> metaprogramming (with Tempita, not Jinja2).
> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
> 3)
> https://github.com/wavemoth/wavemoth/blob/cuda/wavemoth/cuda/legendre_transform.cu.in
> Numerical kernel in templated CUDA using Tempita.
> On templating languages I tried)
> I've scanned through a few and actually tried Tempita, Mako, Jinja2.
> The features I need:
> - Pythonic syntax and ability to embed arbitrary Python code
> - A "call-block", such as this
> {% call catch('A->grid->ctx') %}
> {% endcall %}
> i.e. in Jinja 2, one of the arguments to the function "catch" here is
> "caller", which when called invokes the body (and can be called multiple
> times with different arguments)
> I started out with Tempita because it's so simple to ship, but the lack
> of a call-block construct + the inability to break lines where I wanted
> drove me crazy.
> Then I tried Mako, because it has the largest set of features, but the
> syntax was simply too gruesome. I first tried to ignore this, but simply
> couldn't, it made my code totally unreadable.
> Finally, Jinja2 has most of what I need. Slight disadvantage is it tries
> to be "pure" and not allow too much arbitrary Python,

[sorry:] Slight disadvantage is it tries to be "pure" and not allow too 
much arbitrary Python, but one can work around that by using an 
auxiliary Python module and pass that Python module to the template when 
instantiating it -- so one kind of can use arbitrary Python in the 
templating process, one just need to edit separate files. (Which is 
perhaps better -- I'm unable to make up my mind on such trivial issues.)


> Ideally what I'd like is something like Tempita but developed further to
> allow line-breaks and call-blocks, but lacking that I use Jinja2.
> I don't remember why I didn't like Cheetah (perhaps it doesn't do
> call-blocks?)
> Dag

More information about the cython-devel mailing list