[Cython] RFC: an inline_ function that dumps c/c++ code to the code emitter

Robert Bradshaw robertwb at gmail.com
Fri Aug 19 05:36:59 EDT 2016

On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion at gmail.com> wrote:
> Accidentally posted to an already-opened tab for the cython-users ML
> yesterday, moving to here. Following up from a github opened issue here:
> https://github.com/cython/cython/issues/1440
> I was hoping we could give a way to drop straight into C/C++ inside of
> Cython pyx files.
> Why?
> -It [helps] avoid needing to declare every class/function in cython, a
> somewhat daunting/impossible task that I think everyone hates.  Have you
> libraries like Eigen or others that use complex template based techniques?
> How about those with tons of [member] functions to boot or getting C++
> inheritance involved.

I agree that this is a pain, and better tooling should be developed to
(mostly?) eliminate this (e.g. see the recent thread at
https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).

Of course having symbols magically appear (e.g. due some #Include, who
knows which) has its downsides too, which is why import * is often
discouraged in Python too.

> -It works around having the Cython compiler know about all of C++'s nuances
> - as an advanced C++ developer these are painful and it is a 2nd class
> citizen to Cython's simpler C-support - that's no good.  Just right now I
> was bitten by yet another template argument bug and it's clear C++ template
> arguments have been kind of dicy since support appeared.

Yes, C++ is an extraordinarily complicated language, and exposing all
of that would add significant amounts of complexity to Cython itself,
and perhaps more importantly increase the barrier of entry to reading
Cython code. One of the guiding principles is that we try to stay as
close to Python as possible (if you know Python, you can probably read
Cython, and with a minimal amount of C knowledge start writing it) and
much of C++ simply isn't Pythonic.

Though, as you discovered, there are some basic things like non-type
template arguments that we would like to have. If there are other
specific C++ constructs that are commonly used but impossible to
express in Cython it'd be useful to know.

> -It would allow single source files - I think this is important for runtime
> compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide

Not quite following what you're saying here.

> The idea is that Cython glue makes the playing field for extracting data
> easy, but that once it's extracted to a cdef variable for instance, cython
> doesn't need to know what happens.  Maybe in a way sort of like the GCC asm
> extension.  Hopefully simpler variable passing though.

Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any
possible conflicts. Specifying what/how to mangle could get as ugly as
GCC's asm variable passing. And embedded variable declarations, let
alone control flow statements (especially return, break, ...) could
get really messy. It obscures analysis Cython can do on the code, such
as whether variables are used or what values they may take. Little
code snippets are not always local either, e.g. do they often need to
refer to variables (or headers) referenced elsewhere. And they must
all be mutually compatible.

That's aside from the jarring nature of interleaving Python and C++
code. Would semicolons be required? How would parsers (including IDEs)
handle the C++ snippets? Or would they be placed in opaque
strings/comments (which I'd rather avoid)?

Feels like you want PyInline or weave.inline, but at compile time.

> The alternative to not having this construct is a concrete wall.  I'll have
> to segment up files for C++ for the rest of time until I get function forms
> that Cython can handle. At that point I just say screw it and use boost
> python.  Of course cython does the definitions, data extraction, and
> compilation so much easier than boost.python.  It would be a shame to not
> consider this plight C++ developers have been cornered into and you can't
> say the C++ libraries are broken, it is the strategy Cython is taking that
> cannot work.

Some C++ libraries are not very amenable to being used outside a C++
context. They don't expose an "FFI-friendly" interface and calling
them from other languages is more painful. That doesn't mean they're
broken, just C++ centric.

The strategy that Cython uses is that you use (as close to possible)
Python structures and syntax to write your bindings. This is great for
people who know and understand Python, but for those who would rather
write their code (including bindings) in C++ there's Boost.Python and
others (including writing C extensions by hand).

If your library is full of C++isms, another option is to create a C
wrapper, and expose that to Python (which is what people did before
there was any C++ support). If exposing these as Python functions,
with automatic conversion to/from Python types is the primary value
you're getting from Cython, simply declare these as cpdef functions in
your extern blocks and wrappers will automatically be created. If your
C wrappers are simple enough, you might even be able to auto-generate
the .pxd and .pyx files with some of the tools listed at
https://github.com/cython/cython/wiki/AutoPxd . Yes, you'd have your
C++ code in a C++ file and "Python" code in a .pyx file rather than
interleaving them, but the amount of code you write would be the same.

> I did some examination of how this could be implemented - my idea was to
> base on the print/exec statements handling and simply emit their arguments
> from the Nodes to the code generator.  Proposing inline_c or inline_ as the
> statement.  Such a statement could be used to do macros, pull in includes,
> and modify/declare c-variables.
> -Jason
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

More information about the cython-devel mailing list