RFC: an inline_ function that dumps c/c++ code to the code emitter
Accidentally posted to an already-opened tab for the cython-users ML yesterday, moving to here. Following up from a github opened issue here: https://github.com/cython/cython/issues/1440 I was hoping we could give a way to drop straight into C/C++ inside of Cython pyx files. Why? -It [helps] avoid needing to declare every class/function in cython, a somewhat daunting/impossible task that I think everyone hates. Have you libraries like Eigen or others that use complex template based techniques? How about those with tons of [member] functions to boot or getting C++ inheritance involved. -It works around having the Cython compiler know about all of C++'s nuances - as an advanced C++ developer these are painful and it is a 2nd class citizen to Cython's simpler C-support - that's no good. Just right now I was bitten by yet another template argument bug and it's clear C++ template arguments have been kind of dicy since support appeared. -It would allow single source files - I think this is important for runtime compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though. The alternative to not having this construct is a concrete wall. I'll have to segment up files for C++ for the rest of time until I get function forms that Cython can handle. At that point I just say screw it and use boost python. Of course cython does the definitions, data extraction, and compilation so much easier than boost.python. It would be a shame to not consider this plight C++ developers have been cornered into and you can't say the C++ libraries are broken, it is the strategy Cython is taking that cannot work. I did some examination of how this could be implemented - my idea was to base on the print/exec statements handling and simply emit their arguments from the Nodes to the code generator. Proposing inline_c or inline_ as the statement. Such a statement could be used to do macros, pull in includes, and modify/declare c-variables. -Jason
On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion@gmail.com> wrote:
Accidentally posted to an already-opened tab for the cython-users ML yesterday, moving to here. Following up from a github opened issue here:
https://github.com/cython/cython/issues/1440
I was hoping we could give a way to drop straight into C/C++ inside of Cython pyx files.
Why?
-It [helps] avoid needing to declare every class/function in cython, a somewhat daunting/impossible task that I think everyone hates. Have you libraries like Eigen or others that use complex template based techniques? How about those with tons of [member] functions to boot or getting C++ inheritance involved.
I agree that this is a pain, and better tooling should be developed to (mostly?) eliminate this (e.g. see the recent thread at https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ). Of course having symbols magically appear (e.g. due some #Include, who knows which) has its downsides too, which is why import * is often discouraged in Python too.
-It works around having the Cython compiler know about all of C++'s nuances - as an advanced C++ developer these are painful and it is a 2nd class citizen to Cython's simpler C-support - that's no good. Just right now I was bitten by yet another template argument bug and it's clear C++ template arguments have been kind of dicy since support appeared.
Yes, C++ is an extraordinarily complicated language, and exposing all of that would add significant amounts of complexity to Cython itself, and perhaps more importantly increase the barrier of entry to reading Cython code. One of the guiding principles is that we try to stay as close to Python as possible (if you know Python, you can probably read Cython, and with a minimal amount of C knowledge start writing it) and much of C++ simply isn't Pythonic. Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
-It would allow single source files - I think this is important for runtime compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
Not quite following what you're saying here.
The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though.
Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any possible conflicts. Specifying what/how to mangle could get as ugly as GCC's asm variable passing. And embedded variable declarations, let alone control flow statements (especially return, break, ...) could get really messy. It obscures analysis Cython can do on the code, such as whether variables are used or what values they may take. Little code snippets are not always local either, e.g. do they often need to refer to variables (or headers) referenced elsewhere. And they must all be mutually compatible. That's aside from the jarring nature of interleaving Python and C++ code. Would semicolons be required? How would parsers (including IDEs) handle the C++ snippets? Or would they be placed in opaque strings/comments (which I'd rather avoid)? Feels like you want PyInline or weave.inline, but at compile time.
The alternative to not having this construct is a concrete wall. I'll have to segment up files for C++ for the rest of time until I get function forms that Cython can handle. At that point I just say screw it and use boost python. Of course cython does the definitions, data extraction, and compilation so much easier than boost.python. It would be a shame to not consider this plight C++ developers have been cornered into and you can't say the C++ libraries are broken, it is the strategy Cython is taking that cannot work.
Some C++ libraries are not very amenable to being used outside a C++ context. They don't expose an "FFI-friendly" interface and calling them from other languages is more painful. That doesn't mean they're broken, just C++ centric. The strategy that Cython uses is that you use (as close to possible) Python structures and syntax to write your bindings. This is great for people who know and understand Python, but for those who would rather write their code (including bindings) in C++ there's Boost.Python and others (including writing C extensions by hand). If your library is full of C++isms, another option is to create a C wrapper, and expose that to Python (which is what people did before there was any C++ support). If exposing these as Python functions, with automatic conversion to/from Python types is the primary value you're getting from Cython, simply declare these as cpdef functions in your extern blocks and wrappers will automatically be created. If your C wrappers are simple enough, you might even be able to auto-generate the .pxd and .pyx files with some of the tools listed at https://github.com/cython/cython/wiki/AutoPxd . Yes, you'd have your C++ code in a C++ file and "Python" code in a .pyx file rather than interleaving them, but the amount of code you write would be the same.
I did some examination of how this could be implemented - my idea was to base on the print/exec statements handling and simply emit their arguments from the Nodes to the code generator. Proposing inline_c or inline_ as the statement. Such a statement could be used to do macros, pull in includes, and modify/declare c-variables.
-Jason
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Hi Robert, Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are: 1. Non-type template parameters. 2. "rvalue" references for standalone functions and class member functions: void foo[T]( T && t ) 3. Variadic templates (member and non-member). More generally, "parameter packs": http://en.cppreference.com/w/cpp/language/parameter_pack #2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location. #3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file. Support for "auto" would be nice, but perhaps unrealistic. I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly. Thanks for all your hard work, Kevin On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt@uci.edu> wrote:
Hi Robert,
Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are:
1. Non-type template parameters.
This is actually probably the hardest :). There is a PR that's stalled out, probably needs some attention by both the author and us; it'd be great to get this in.
2. "rvalue" references for standalone functions and class member functions:
void foo[T]( T && t )
From a callers perspective, one can just declare an extern foo as foo(T) in Cython, it doesn't care if the actual call is made by value or by reference--the calling code is exactly the same.
Actually implementing it for Cython-defined functions (as the callee) would require additional support, but would probably be pretty easy (implement a new type, with parsing, which generally behaves the same as a reference or value from Python's perspective--the actual heavy lifting happens in the C compiler). Want to file an issue on github for this?
3. Variadic templates (member and non-member). More generally, "parameter packs":
This'd be interesting (especially as to how it relates to changing the arity of functions for concrete instantiations). There's also a question of syntax. Maybe void foo[A, int N, ... Types](A a, Types t): ... It'd likely require #1, e.g. to be able to declare things like tuple_element::type such that std::tuple.get<N> would be correctly typed. Unfortunately seems declaring the (somewhat intuitive) API of std::tuple requires being able to declare a fair amount of its implementation. Still likely feasible though, and I think the notion of variadic templates is natural enough in a Python setting (which has had heterogeneously typed tuples since its inception, though of course not statically typed).
#2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location.
It's interesting to call this an essential feature, given that C++ itself didn't have it until recently :). Not that it can't be a big win, and libraries will more and more probably be designed assuming it.
#3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file.
Support for "auto" would be nice, but perhaps unrealistic.
Just leave the cdef out and you basically get auto, i.e. my_var = expr gives my_var the type of expr. If multiple assignments are made to my_var, it attempts to find a suitable union type that can hold all assignments. The one exception is for integers, where it infers the type to be object iff my_var is used in arithmetic expressions (that it can't deduce would not overflow with the finite-sized C integer type).
I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly.
Thanks for all your hard work,
Kevin
On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt@uci.edu> wrote:
Hi Robert,
Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are:
1. Non-type template parameters.
This is actually probably the hardest :). There is a PR that's stalled out, probably needs some attention by both the author and us; it'd be great to get this in.
2. "rvalue" references for standalone functions and class member functions:
void foo[T]( T && t )
From a callers perspective, one can just declare an extern foo as foo(T) in Cython, it doesn't care if the actual call is made by value or by reference--the calling code is exactly the same.
Actually implementing it for Cython-defined functions (as the callee) would require additional support, but would probably be pretty easy (implement a new type, with parsing, which generally behaves the same as a reference or value from Python's perspective--the actual heavy lifting happens in the C compiler).
Want to file an issue on github for this?
Sure--will do.
3. Variadic templates (member and non-member). More generally, "parameter packs":
This'd be interesting (especially as to how it relates to changing the arity of functions for concrete instantiations). There's also a question of syntax. Maybe
void foo[A, int N, ... Types](A a, Types t): ...
It'd likely require #1, e.g. to be able to declare things like tuple_element::type such that std::tuple.get<N> would be correctly typed. Unfortunately seems declaring the (somewhat intuitive) API of std::tuple requires being able to declare a fair amount of its implementation.
Still likely feasible though, and I think the notion of variadic templates is natural enough in a Python setting (which has had heterogeneously typed tuples since its inception, though of course not statically typed).
#2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location.
It's interesting to call this an essential feature, given that C++ itself didn't have it until recently :). Not that it can't be a big win, and libraries will more and more probably be designed assuming it.
Also, libraries not designed assuming it can still take advantage of it in many contexts. For lots of objects, the move constructor can be implicitly created by the constructor. Also, this would solve some quirks for Cython. Right now, std::unique_ptr is supported, but the object is non-copyable. One needs std::move in order to have this type be fully-functional.
#3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file.
Support for "auto" would be nice, but perhaps unrealistic.
Just leave the cdef out and you basically get auto, i.e.
my_var = expr
gives my_var the type of expr. If multiple assignments are made to my_var, it attempts to find a suitable union type that can hold all assignments. The one exception is for integers, where it infers the type to be object iff my_var is used in arithmetic expressions (that it can't deduce would not overflow with the finite-sized C integer type).
This is surprising to me. Would my_var be passable to a "cdef nogil" function? I'll have to give this a try--thanks for the tip!
I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly.
Thanks for all your hard work,
Kevin
On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt@uci.edu> wrote:
Hi Robert,
Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are:
1. Non-type template parameters.
This is actually probably the hardest :). There is a PR that's stalled out, probably needs some attention by both the author and us; it'd be great to get this in.
2. "rvalue" references for standalone functions and class member functions:
void foo[T]( T && t )
From a callers perspective, one can just declare an extern foo as foo(T) in Cython, it doesn't care if the actual call is made by value or by reference--the calling code is exactly the same.
Actually implementing it for Cython-defined functions (as the callee) would require additional support, but would probably be pretty easy (implement a new type, with parsing, which generally behaves the same as a reference or value from Python's perspective--the actual heavy lifting happens in the C compiler).
Want to file an issue on github for this?
3. Variadic templates (member and non-member). More generally, "parameter packs":
This'd be interesting (especially as to how it relates to changing the arity of functions for concrete instantiations). There's also a question of syntax. Maybe
void foo[A, int N, ... Types](A a, Types t): ...
It'd likely require #1, e.g. to be able to declare things like tuple_element::type such that std::tuple.get<N> would be correctly typed. Unfortunately seems declaring the (somewhat intuitive) API of std::tuple requires being able to declare a fair amount of its implementation.
Still likely feasible though, and I think the notion of variadic templates is natural enough in a Python setting (which has had heterogeneously typed tuples since its inception, though of course not statically typed).
#2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location.
It's interesting to call this an essential feature, given that C++ itself didn't have it until recently :). Not that it can't be a big win, and libraries will more and more probably be designed assuming it.
#3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file.
Support for "auto" would be nice, but perhaps unrealistic.
Just leave the cdef out and you basically get auto, i.e.
my_var = expr
gives my_var the type of expr. If multiple assignments are made to my_var, it attempts to find a suitable union type that can hold all assignments. The one exception is for integers, where it infers the type to be object iff my_var is used in arithmetic expressions (that it can't deduce would not overflow with the finite-sized C integer type).
Wow--that does work (at least for simple cases). It may be nice to have a paragraph about this in the dox? Something like "Emulating auto in C++"? We've been doing a lot of copy/pasting of type names b/c we weren't aware of this.
I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly.
Thanks for all your hard work,
Kevin
On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
I actually just realized another nagging issue with C++/Cython: member typedefs are not supported: cdef vector[T]: ctypedef size_t size_type These "scoped" typedefs are very widely-used. We can live without it, and we have been, but while we're making a list.. :) --Kevin On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt@uci.edu> wrote:
Hi Robert,
Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are:
1. Non-type template parameters.
This is actually probably the hardest :). There is a PR that's stalled out, probably needs some attention by both the author and us; it'd be great to get this in.
2. "rvalue" references for standalone functions and class member functions:
void foo[T]( T && t )
From a callers perspective, one can just declare an extern foo as foo(T) in Cython, it doesn't care if the actual call is made by value or by reference--the calling code is exactly the same.
Actually implementing it for Cython-defined functions (as the callee) would require additional support, but would probably be pretty easy (implement a new type, with parsing, which generally behaves the same as a reference or value from Python's perspective--the actual heavy lifting happens in the C compiler).
Want to file an issue on github for this?
3. Variadic templates (member and non-member). More generally, "parameter packs":
This'd be interesting (especially as to how it relates to changing the arity of functions for concrete instantiations). There's also a question of syntax. Maybe
void foo[A, int N, ... Types](A a, Types t): ...
It'd likely require #1, e.g. to be able to declare things like tuple_element::type such that std::tuple.get<N> would be correctly typed. Unfortunately seems declaring the (somewhat intuitive) API of std::tuple requires being able to declare a fair amount of its implementation.
Still likely feasible though, and I think the notion of variadic templates is natural enough in a Python setting (which has had heterogeneously typed tuples since its inception, though of course not statically typed).
#2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location.
It's interesting to call this an essential feature, given that C++ itself didn't have it until recently :). Not that it can't be a big win, and libraries will more and more probably be designed assuming it.
#3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file.
Support for "auto" would be nice, but perhaps unrealistic.
Just leave the cdef out and you basically get auto, i.e.
my_var = expr
gives my_var the type of expr. If multiple assignments are made to my_var, it attempts to find a suitable union type that can hold all assignments. The one exception is for integers, where it infers the type to be object iff my_var is used in arithmetic expressions (that it can't deduce would not overflow with the finite-sized C integer type).
I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly.
Thanks for all your hard work,
Kevin
On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Mon, Aug 22, 2016 at 7:44 AM, Kevin Thornton <krthornt@uci.edu> wrote:
I actually just realized another nagging issue with C++/Cython: member typedefs are not supported:
cdef vector[T]: ctypedef size_t size_type
These "scoped" typedefs are very widely-used. We can live without it, and we have been, but while we're making a list.. :)
Yes, that's a good one. Worth an issue :).
On Sat, Aug 20, 2016 at 11:15 PM Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Aug 19, 2016 at 9:34 AM, Kevin Thornton <krthornt@uci.edu> wrote:
Hi Robert,
Related to this point, I think there are three important features that Cython would benefit from, as far as C++11 and newer are concerned. In order of what I'm guessing to be increased complexity are:
1. Non-type template parameters.
This is actually probably the hardest :). There is a PR that's stalled out, probably needs some attention by both the author and us; it'd be great to get this in.
2. "rvalue" references for standalone functions and class member functions:
void foo[T]( T && t )
From a callers perspective, one can just declare an extern foo as foo(T) in Cython, it doesn't care if the actual call is made by value or by reference--the calling code is exactly the same.
Actually implementing it for Cython-defined functions (as the callee) would require additional support, but would probably be pretty easy (implement a new type, with parsing, which generally behaves the same as a reference or value from Python's perspective--the actual heavy lifting happens in the C compiler).
Want to file an issue on github for this?
3. Variadic templates (member and non-member). More generally, "parameter packs":
This'd be interesting (especially as to how it relates to changing the arity of functions for concrete instantiations). There's also a question of syntax. Maybe
void foo[A, int N, ... Types](A a, Types t): ...
It'd likely require #1, e.g. to be able to declare things like tuple_element::type such that std::tuple.get<N> would be correctly typed. Unfortunately seems declaring the (somewhat intuitive) API of std::tuple requires being able to declare a fair amount of its implementation.
Still likely feasible though, and I think the notion of variadic templates is natural enough in a Python setting (which has had heterogeneously typed tuples since its inception, though of course not statically typed).
#2 is important for efficiency reasons--it isn't just a convenience feature. Without it, Cython cannot support all the member function of std::vector, etc. The implication is that some expensive operations, like moving a big object into a vector, require extra copies in Cython while a C++ program would just move the object into the new location.
It's interesting to call this an essential feature, given that C++ itself didn't have it until recently :). Not that it can't be a big win, and libraries will more and more probably be designed assuming it.
#3 seems like the hardest, but would enable std::tuple, std::bind, and other handy things to be used in a .pyx file.
Support for "auto" would be nice, but perhaps unrealistic.
Just leave the cdef out and you basically get auto, i.e.
my_var = expr
gives my_var the type of expr. If multiple assignments are made to my_var, it attempts to find a suitable union type that can hold all assignments. The one exception is for integers, where it infers the type to be object iff my_var is used in arithmetic expressions (that it can't deduce would not overflow with the finite-sized C integer type).
I know that PRs have been submitted, and since stalled out, on a couple of these features. I've tried to look at the Cython parser myself, but I'm not able to follow it, sadly.
Thanks for all your hard work,
Kevin
On Fri, Aug 19, 2016 at 2:39 AM Robert Bradshaw <robertwb@gmail.com> wrote:
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Aug 19, 2016 at 5:36 AM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion@gmail.com> wrote:
Accidentally posted to an already-opened tab for the cython-users ML yesterday, moving to here. Following up from a github opened issue here:
https://github.com/cython/cython/issues/1440
I was hoping we could give a way to drop straight into C/C++ inside of Cython pyx files.
Why?
-It [helps] avoid needing to declare every class/function in cython, a somewhat daunting/impossible task that I think everyone hates. Have you libraries like Eigen or others that use complex template based techniques? How about those with tons of [member] functions to boot or getting C++ inheritance involved.
I agree that this is a pain, and better tooling should be developed to (mostly?) eliminate this (e.g. see the recent thread at https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).
Of course having symbols magically appear (e.g. due some #Include, who knows which) has its downsides too, which is why import * is often discouraged in Python too.
-It works around having the Cython compiler know about all of C++'s nuances - as an advanced C++ developer these are painful and it is a 2nd class citizen to Cython's simpler C-support - that's no good. Just right now I was bitten by yet another template argument bug and it's clear C++ template arguments have been kind of dicy since support appeared.
Yes, C++ is an extraordinarily complicated language, and exposing all of that would add significant amounts of complexity to Cython itself, and perhaps more importantly increase the barrier of entry to reading Cython code. One of the guiding principles is that we try to stay as close to Python as possible (if you know Python, you can probably read Cython, and with a minimal amount of C knowledge start writing it) and much of C++ simply isn't Pythonic.
Maybe it's off topic but I debate the guiding principle of Cython - I was not able to comprehend Cython before reading tutorials and this is not my first time looking at it, I had a couple runins over the last 5 years with it on projects such as h5py easily got lost on what was going on between all the wrapper-wrappers (pure py code wrapping py-invokable code) and re-declarations. These projects complied with Cython's current philosophy to the degradation of clarity, context, and overall idea of how code was hooked up. Perhaps Cython should take the lessons learned from it's inception, time, and the results of the state of the c-python userbase to guide us into a new philosophy.
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
I haven't had the capability to use Cython sufficiently to learn more of them because it currently can't solve my problems. From prior SWIG et al experiences, my outlook is that it is treacherous path to walk and unless you brought in llvm/clang into the project for parsing/AST, I'd hold onto that outlook.
-It would allow single source files - I think this is important for runtime compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
Not quite following what you're saying here.
Maybe PyInline was a better example off the bat for you, but something a little more flexible but also with less work is needed. Compare with PyOpenCL: https://documen.tician.de/pyopencl/ - check out some examples. There is a c runtime api between the contexts hooking things up (this is the OpenCL runtime part) - it's a pretty similar story to PyCuda (and by the same author, execpt for that project has to jump out to nvcc and cache kernel compilation like the inline function implementation does). There's no limit to the number of functions you can declare though and the OpenCL side is kept simple - things are generally pretty typesafe/do what you would expect on dispatch. PyInline for comparison looks like it might lean on the Python c-api for it's work more and maybe limited in the number of functions per snippet it can declare. I don't expect to be able to work with Numpy ndarray data easily with it.
The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though.
Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any possible conflicts. Specifying what/how to mangle could get as ugly as GCC's asm variable passing. And embedded variable declarations, let alone control flow statements (especially return, break, ...) could get really messy. It obscures analysis Cython can do on the code, such as whether variables are used or what values they may take. Little code snippets are not always local either, e.g. do they often need to refer to variables (or headers) referenced elsewhere. And they must all be mutually compatible.
Like gcc's asm, let's let adults do what they want and let them worry about the consequences of flow control/stray includes. I'm not even sure how most of this would be an issue (switch/break/if) if you are properly nesting pyxd output. The only thing I think is an issue here is mangled names. I haven't yet figured out why (cdef) variable names must be mangled. Can you explain? Maybe we add an option to allow it to be unmangled in their declaration? C++ has extern "C" for example.
That's aside from the jarring nature of interleaving Python and C++ code. Would semicolons be required? How would parsers (including IDEs) handle the C++ snippets? Or would they be placed in opaque strings/comments (which I'd rather avoid)?
Opaque strings. It's a good and time tested solution to the issue. I'm very happy with it in the contexts I use it in.
Feels like you want PyInline or weave.inline, but at compile time.
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors and I'm not even sure it had the power to do what I wanted anyway because Cython provides a language that eases the data-extraction/typecasting part of inlining C/C++.
The alternative to not having this construct is a concrete wall. I'll have to segment up files for C++ for the rest of time until I get function forms that Cython can handle. At that point I just say screw it and use boost python. Of course cython does the definitions, data extraction, and compilation so much easier than boost.python. It would be a shame to not consider this plight C++ developers have been cornered into and you can't say the C++ libraries are broken, it is the strategy Cython is taking that cannot work.
Some C++ libraries are not very amenable to being used outside a C++ context. They don't expose an "FFI-friendly" interface and calling them from other languages is more painful. That doesn't mean they're broken, just C++ centric.
The strategy that Cython uses is that you use (as close to possible) Python structures and syntax to write your bindings. This is great for people who know and understand Python, but for those who would rather write their code (including bindings) in C++ there's Boost.Python and others (including writing C extensions by hand).
As a multi-year user and contributor to the project of Boost.Python, it's not all it's cracked up to be - Cython with inline C/C++ is often a better approach (and better for scientific code acceleration/glue). Boost.Python's autoconversion magic is very unpredictable/broken - and this is required. I'm interested in Pybind for large projects but not as much as in Cython for numeric processing. Writing C extensions by hand is usually a waste of time, which is why so many people tried to automate it. Cython's audience *I believe* is not people not knowing C/C++ and only familiar with python but is C/C++ users trying to wrap stuff with Cython. Further, I don't see how this is detrimental - it's segmented. For small stuff the cognitive burden of finding and going to an external file is far higher which is the way I've seen it end up so far.
If your library is full of C++isms, another option is to create a C wrapper, and expose that to Python (which is what people did before there was any C++ support). If exposing these as Python functions, with automatic conversion to/from Python types is the primary value you're getting from Cython, simply declare these as cpdef functions in your extern blocks and wrappers will automatically be created. If your C wrappers are simple enough, you might even be able to auto-generate the .pxd and .pyx files with some of the tools listed at https://github.com/cython/cython/wiki/AutoPxd . Yes, you'd have your C++ code in a C++ file and "Python" code in a .pyx file rather than interleaving them, but the amount of code you write would be the same.
So many nopes to this approach. I listed it myself effectively above, just not the autopxd part. No I don't want to make C bindings for all of C++ libraries, I don't buy this saved anything and only creates a ridiculous amount of work again where the cost (far) exceeds the gains. -Jason
On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion@gmail.com> wrote:
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors [...]
May I ask why "any other python driven way to compile c-code in the spirit these projects do is deprecated/dead?" I'm curious since when I started Sage (and Cython based on forking Pyrex), it was because none of the other approaches seemed like they would work for the developer base I envisioned growing for Sage. That was a long time ago, and I'm always pleasantly surprised that Cython has become very popular. However, I didn't realize the other approaches were deprecated/dead. William -- William (http://wstein.org)
On Fri, Aug 19, 2016 at 3:19 PM, William Stein <wstein@gmail.com> wrote:
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python -
On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion@gmail.com> wrote: pybind11 is
the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors [...]
May I ask why "any other python driven way to compile c-code in the spirit these projects do is deprecated/dead?" I'm curious since when I started Sage (and Cython based on forking Pyrex), it was because none of the other approaches seemed like they would work for the developer base I envisioned growing for Sage. That was a long time ago, and I'm always pleasantly surprised that Cython has become very popular. However, I didn't realize the other approaches were deprecated/dead.
PyInline's last news update was in 2004 where the author gives a "Hats of[f] to PyRex", prior to that only a few blog entries in 2001/2002, I've not come across any project using it but maybe that is not sufficient to call it deprecated/dead? Does it work with Python 3? http://pyinline.sourceforge.net/ Pyrex no longer has a userbase. Last ML post was 2014. Weave only supports python2, got ripped out of Scipy and also directs to check out Cython and strongly implies the project is all but dead/maintenance mode: https://github.com/scipy/weave Did I miss any of the python driven ways? -Jason
On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton <nevion@gmail.com> wrote:
On Fri, Aug 19, 2016 at 3:19 PM, William Stein <wstein@gmail.com> wrote:
On Fri, Aug 19, 2016 at 8:19 AM, Jason Newton <nevion@gmail.com> wrote:
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors [...]
May I ask why "any other python driven way to compile c-code in the spirit these projects do is deprecated/dead?" I'm curious since when I started Sage (and Cython based on forking Pyrex), it was because none of the other approaches seemed like they would work for the developer base I envisioned growing for Sage. That was a long time ago, and I'm always pleasantly surprised that Cython has become very popular. However, I didn't realize the other approaches were deprecated/dead.
PyInline's last news update was in 2004 where the author gives a "Hats of[f] to PyRex", prior to that only a few blog entries in 2001/2002, I've not come across any project using it but maybe that is not sufficient to call it deprecated/dead? Does it work with Python 3? http://pyinline.sourceforge.net/
Pyrex no longer has a userbase. Last ML post was 2014.
Weave only supports python2, got ripped out of Scipy and also directs to check out Cython and strongly implies the project is all but dead/maintenance mode: https://github.com/scipy/weave
Did I miss any of the python driven ways?
Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw another project pop up several years ago that was a lot like what you're suggesting, e.g. a listing of def foo(a, b): [c code using a and b] I don't recall how the types for a and b were declared/converted, and I don't think it did much non-trivial stuff yet, but I can't find a trace of it now (but it's hard to search for laking a name).
Robert Bradshaw schrieb am 21.08.2016 um 11:11:
On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton wrote:
PyInline's last news update was in 2004 where the author gives a "Hats of[f] to PyRex", prior to that only a few blog entries in 2001/2002, I've not come across any project using it but maybe that is not sufficient to call it deprecated/dead? Does it work with Python 3? http://pyinline.sourceforge.net/
Pyrex no longer has a userbase. Last ML post was 2014.
Weave only supports python2, got ripped out of Scipy and also directs to check out Cython and strongly implies the project is all but dead/maintenance mode: https://github.com/scipy/weave
Did I miss any of the python driven ways?
Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw another project pop up several years ago that was a lot like what you're suggesting, e.g. a listing of
def foo(a, b): [c code using a and b]
I don't recall how the types for a and b were declared/converted, and I don't think it did much non-trivial stuff yet, but I can't find a trace of it now (but it's hard to search for laking a name).
This page lists some, although I can't see any real candidates for what you describe: https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages And most of those are dead links and/or projects. Stefan
ctypes/cffi aren't in the same class for comparison. Just at the start of the list is basically a restriction from C++ name mangling fouling things up. Pybindgen uses a programmable model to generate bindings for existing functions/classes only. It's a fair bit different from anything I had in mind for the python side and even in the C++ binding side (boost.python class) or pyinline class - no additional code of any kind allowed. Last activity 2014 - I don't really have a handle on how alive it is, but I think it's inactive. On Sun, Aug 21, 2016 at 5:38 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw schrieb am 21.08.2016 um 11:11:
On Fri, Aug 19, 2016 at 12:32 PM, Jason Newton wrote:
PyInline's last news update was in 2004 where the author gives a "Hats of[f] to PyRex", prior to that only a few blog entries in 2001/2002, I've not come across any project using it but maybe that is not sufficient to call it deprecated/dead? Does it work with Python 3? http://pyinline.sourceforge.net/
Pyrex no longer has a userbase. Last ML post was 2014.
Weave only supports python2, got ripped out of Scipy and also directs to check out Cython and strongly implies the project is all but dead/maintenance mode: https://github.com/scipy/weave
Did I miss any of the python driven ways?
Pybindgen? And of course there's ctypes/cffi. I could have sworn I saw another project pop up several years ago that was a lot like what you're suggesting, e.g. a listing of
def foo(a, b): [c code using a and b]
I don't recall how the types for a and b were declared/converted, and I don't think it did much non-trivial stuff yet, but I can't find a trace of it now (but it's hard to search for laking a name).
This page lists some, although I can't see any real candidates for what you describe:
https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages
And most of those are dead links and/or projects.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Aug 19, 2016 at 11:19 AM, Jason Newton <nevion@gmail.com> wrote:
On Fri, Aug 19, 2016 at 5:36 AM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton <nevion@gmail.com> wrote:
Accidentally posted to an already-opened tab for the cython-users ML yesterday, moving to here. Following up from a github opened issue here:
https://github.com/cython/cython/issues/1440
I was hoping we could give a way to drop straight into C/C++ inside of Cython pyx files.
Why?
-It [helps] avoid needing to declare every class/function in cython, a somewhat daunting/impossible task that I think everyone hates. Have you libraries like Eigen or others that use complex template based techniques? How about those with tons of [member] functions to boot or getting C++ inheritance involved.
I agree that this is a pain, and better tooling should be developed to (mostly?) eliminate this (e.g. see the recent thread at https://groups.google.com/forum/#!topic/cython-users/c8ChI6jERzY ).
Of course having symbols magically appear (e.g. due some #Include, who knows which) has its downsides too, which is why import * is often discouraged in Python too.
-It works around having the Cython compiler know about all of C++'s nuances - as an advanced C++ developer these are painful and it is a 2nd class citizen to Cython's simpler C-support - that's no good. Just right now I was bitten by yet another template argument bug and it's clear C++ template arguments have been kind of dicy since support appeared.
Yes, C++ is an extraordinarily complicated language, and exposing all of that would add significant amounts of complexity to Cython itself, and perhaps more importantly increase the barrier of entry to reading Cython code. One of the guiding principles is that we try to stay as close to Python as possible (if you know Python, you can probably read Cython, and with a minimal amount of C knowledge start writing it) and much of C++ simply isn't Pythonic.
Maybe it's off topic but I debate the guiding principle of Cython - I was not able to comprehend Cython before reading tutorials and this is not my first time looking at it, I had a couple runins over the last 5 years with it on projects such as h5py easily got lost on what was going on between all the wrapper-wrappers (pure py code wrapping py-invokable code) and re-declarations.
In my experience Cython has generally been fairly easy to pick up for people who already know Python. And Python often easy to pick up for people who already know C/C++. Of course for many wrappings it often takes non-trivial knowledge of the wrapped library itself too, but typically at the same level as would be required to grok code written against that same library directly from C/C++. Yes, there's bad code out there in any language (no offense meant towards h5py--I haven't looked at that project myself). Much of it due to cargo-cult perpetuations or archaic (or simply flat-out-wrong) contortions due to historical limitations (e.g. creating a Python module to wrap an _extension module, avoiding all C++ features with extensive C wrappers, ...). (You're familiar with C++, so likely no stranger to this effect.)
These projects complied with Cython's current philosophy to the degradation of clarity, context, and overall idea of how code was hooked up. Perhaps Cython should take the lessons learned from it's inception, time, and the results of the state of the c-python userbase to guide us into a new philosophy.
I fail to see how "staying close to Python" caused "degradation of clarity, context, etc." If anything, the lessons learned over time have validated this philosophy. More on this later.
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
I haven't had the capability to use Cython sufficiently to learn more of them because it currently can't solve my problems. From prior SWIG et al experiences, my outlook is that it is treacherous path to walk and unless you brought in llvm/clang into the project for parsing/AST, I'd hold onto that outlook.
I agree that any efforts to trying to parsing C++ without building on an actual compiler are fraught with danger. That's not the case with generating C++ code, which is the direction we're going. In particular, our goal is to understand C++ enough to invoke it, which allows us to be much less pedantic. A that automatically extract definitions (or even wrappings, even partially) from C++ headers is another topic, and should almost certainly lean on an existing compiler.
-It would allow single source files - I think this is important for runtime compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
Not quite following what you're saying here.
Maybe PyInline was a better example off the bat for you, but something a little more flexible but also with less work is needed. Compare with PyOpenCL: https://documen.tician.de/pyopencl/ - check out some examples. There is a c runtime api between the contexts hooking things up (this is the OpenCL runtime part) - it's a pretty similar story to PyCuda (and by the same author, execpt for that project has to jump out to nvcc and cache kernel compilation like the inline function implementation does). There's no limit to the number of functions you can declare though and the OpenCL side is kept simple - things are generally pretty typesafe/do what you would expect on dispatch. PyInline for comparison looks like it might lean on the Python c-api for it's work more and maybe limited in the number of functions per snippet it can declare. I don't expect to be able to work with Numpy ndarray data easily with it.
The fact that you're essentially defining a kernel/ufunc with a well defined API in another language makes this a somewhat more natural and tractable case than freely sliding back and forth between C and Python data structures, function calls, etc. multiple times in a function's body. (Not to say that these aren't quite sophisticated projects--they tackle more interesting difficulties elsewhere.)
The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though.
Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any possible conflicts. Specifying what/how to mangle could get as ugly as GCC's asm variable passing. And embedded variable declarations, let alone control flow statements (especially return, break, ...) could get really messy. It obscures analysis Cython can do on the code, such as whether variables are used or what values they may take. Little code snippets are not always local either, e.g. do they often need to refer to variables (or headers) referenced elsewhere. And they must all be mutually compatible.
Like gcc's asm, let's let adults do what they want and let them worry about the consequences of flow control/stray includes. I'm not even sure how most of this would be an issue (switch/break/if) if you are properly nesting pyxd output. The only thing I think is an issue here is mangled names. I haven't yet figured out why (cdef) variable names must be mangled. Can you explain? Maybe we add an option to allow it to be unmangled in their declaration? C++ has extern "C" for example.
Name mangling is done for the standard reasons--to avoid possible conflicts with all other symbols that may be defined. E.g. We don't want things to suddenly break if I happen to create a variable called "PyNone." Or "__pyx_something_we_defined_implicitly." And of course we want to mangle globals, function names, etc. lest they conflict with some otherwise irrelevant symbol defined in some (possibly recursively) included header somewhere. Again, you could just say "Don't name things like that." This exposes some more guiding principles. (1) If it's valid Python, it should be valid Cython and (2) we always try to produce valid C code--if you haven't lied to us (too much) about your external declarations, a successful Cython compilation results in a valid C/C++ output. Also (3) you shouldn't have to read or understand the generated C and the Python/C API to use, let alone debug, Cython (though you're happy to do so if you want, like Java developers sometimes read bytecodes, but not usually, though understanding implementation can sometimes be helpful when chasing performance (for all languages)). There's an obvious tension between giving users all the rope they want vs. providing an API that is possibly more restrictive, but inherently correct by construction. I'll concede that Cython necessarily has pointers, so I'll give that there's plenty of room for foot-shooting (and better interfacing with modern C++ would be good help there), but the kind of errors one runs into by injecting arbitrary code snippets take things to a whole new level (and specifically violate (3) when developing and debugging). The escape hatch is to wrap the C++ in an actual C++ file and invoke the wrapping. Typically this is the minority of one's code--if it's the "whole library" then you probably have an API that's only understandable to someone well versed in C++ anyways. You've given a single example (non-type template arguments) that we would like to support that's blocking you.
Why is allowing arbitrary code inside not a good idea? We're not talking something necessarily like eval here and the reputation it got,
Actually, I think it's a whole lot like eval. It's taking an opaque (string) chunk of data and executing it as code. But potentially worse as it's in a different language and evaluated in a transformed (even if the names were unmangled) context. If we were to go this direction, I might go with a function call (like weave, maybe even follow it) rather than a new statement as the latter is difficult to extend with the myriad of optional configuration parameters, etc. that would beg to follow.
we're talking C/C++ glue and for the scale of Cython's development effort, proper C++ support is insurmountable simply because the language is that hard to work with on the compiler side. You can't just go and take something that is readily in the process of evolving (as newer standards actively come out), that very slowly achieves full compiler support in it's own domain, and pretend/aspire you can work like that though high level constructs in your language - this project and none before it have not accomplished anywhere near that and simply don't have the man-years/decades to contribute. I truly think it is irresponsible to think a) we should do nothing or b) delude ourselfs that we can make high level language level features to get around this.
As Ian said, we can declare the interface, not necessarily the implementation, and when things get so bad that you need to express them in C++, express them in direct C++ in a separate file and invoke the nicer interface from Cython. C++ is evolving, but most (not all) of that is happening at the level of implementation details and much of the API changes are in terms of syntactic sugar/new features that look a lot like old features. Cython only needs concern itself with that which commonly affects a library's API. As I said before, if there are specific missing features that block wrapping much of C++ code, it's worth looking at how they could fit into the language. We aim for a clean API, but also strive to be pragmatic. There is much room for improvement.
That's aside from the jarring nature of interleaving Python and C++ code. Would semicolons be required? How would parsers (including IDEs) handle the C++ snippets? Or would they be placed in opaque strings/comments (which I'd rather avoid)?
Opaque strings. It's a good and time tested solution to the issue. I'm very happy with it in the contexts I use it in.
Feels like you want PyInline or weave.inline, but at compile time.
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors and I'm not even sure it had the power to do what I wanted anyway because Cython provides a language that eases the data-extraction/typecasting part of inlining C/C++.
You seem to be repeatedly bringing up the points * Many (most?) of these string-based approaches are essentially dead, often pointing people to Cython instead, but * Cython should adopt the string-embedding approach of these earlier projects. You ask at the beginning of the email whether time has vindicated our philosophy. I think, based on the mindshare vs. these other attempts at integrating with C, in large part it has. It has served us and our users well; we will strive to stay close to Python. Tight interleaving of multiple languages in is cute for making a polyglot script, but I do not think it leads to legible code. An "eval_cpp" operator would be a lot like the builtin eval--it'd be really tempting to do the "quick and easy" hack of dropping in some executable string instead of thinking how to structure things such that that could be avoided, but putting in this effort leads to more comprehensible code. It's hard to say "no" to features, but I think such an introduction would fundamentally change Cython and how it's written for the worse. - Robert
Robert Bradshaw schrieb am 21.08.2016 um 11:30:
You ask at the beginning of the email whether time has vindicated our philosophy. I think, based on the mindshare vs. these other attempts at integrating with C, in large part it has. It has served us and our users well; we will strive to stay close to Python.
Tight interleaving of multiple languages in is cute for making a polyglot script, but I do not think it leads to legible code. An "eval_cpp" operator would be a lot like the builtin eval--it'd be really tempting to do the "quick and easy" hack of dropping in some executable string instead of thinking how to structure things such that that could be avoided, but putting in this effort leads to more comprehensible code.
+1
It's hard to say "no" to features, but I think such an introduction would fundamentally change Cython and how it's written for the worse.
... and restrict the further development of Cython to anything that doesn't break what users have relied on in their interwoven code. I agree with all of your arguments. But I'd like to note that it should still be quite possible for the OP to write a preprocessor for pyx-ish files that extracts any contained C/C++ code snippets (e.g. from specially marked Python strings with a bit of a metadata DSL), drops them into an external C/C++ header file as functions etc., and then replaces them with external function calls in the original file, before passing it through Cython. That's essentially how the described feature would work anyway, but it would keep Cython itself free from any eval-my-string special handling and user code integration overhead, while allowing users to play around with the possibilities of keeping everything in one source file. If that turns out to become the Next Great Thing, we will at least have learned something from real usage by the time we get there, rather than building something now that eventually everyone (but a few) will hate and not use. I've seen some people turn their back on Cython because they didn't understand it and instead started their own projects their own way. It's obviously everyone's good right to do with their free time whatever they want, but most of those attempts have visibly failed by now. They usually didn't even survive the first couple of months or weeks. It often seems so much easier (and certainly more fun, right?) to start something out in the open countryside that never gets anywhere, than to build on something existing which not quite but almost does what you need. I'd be happy to help making it easier for people to integrate their own tooling with Cython. Not everything needs to be an integrated part of the compiler itself, and certainly not everything should be maintained or developed by us. Stefan
On Sun, Aug 21, 2016 at 5:30 AM, Robert Bradshaw <robertwb@gmail.com> wrote:
In my experience Cython has generally been fairly easy to pick up for people who already know Python. And Python often easy to pick up for people who already know C/C++. Of course for many wrappings it often takes non-trivial knowledge of the wrapped library itself too, but typically at the same level as would be required to grok code written against that same library directly from C/C++.
Is your experience drawn from binding moderately complex libraries or play code (of complexity like from a tutorial). Bottom up or top down? Sorry if this sounds asinine to you. To clarify, I'm coming from the case where I didn't read the whole tutorial/docs before being faced with pyx in the projects I previously mentioned, while on tight turn around time - I was not able to grok in that context. For myself and any other ML user who comes across this thread - can you list a few libraries that do things the right way?
Yes, there's bad code out there in any language (no offense meant towards h5py--I haven't looked at that project myself). Much of it due to cargo-cult perpetuations or archaic (or simply flat-out-wrong) contortions due to historical limitations (e.g. creating a Python module to wrap an _extension module, avoiding all C++ features with extensive C wrappers, ...). (You're familiar with C++, so likely no stranger to this effect.)
These projects complied with Cython's current philosophy to the degradation of clarity, context, and overall idea of how code was hooked up. Perhaps Cython should take the lessons learned from it's inception, time, and the results of the state of the c-python userbase to guide us into a new philosophy.
I fail to see how "staying close to Python" caused "degradation of clarity, context, etc." If anything, the lessons learned over time have validated this philosophy. More on this later.
My point was that multifile multi-level wrapper that I mentioned earlier - if you're saying that those projects did Cython extensions wrong, then I'm incorrect at faulting Cython and should fault the libraries using it. I didn't say staying close to python caused $blurb. I don't know in a situation as confusing as to all the binding projects if this should be taken as validation of philosophy either - I think it is reasonable to consider the attrition of these projects as a function of manpower, number of early on project supporters/authors, and if a project (like sage) indirectly, through dependency, kept the project alive. And good old fashioned luck. I noted most of them don't use distutils and something custom but less capable instead which maybe plays a roll in how mature/usable/smalltime they where/are.
I agree that any efforts to trying to parsing C++ without building on an actual compiler are fraught with danger. That's not the case with generating C++ code, which is the direction we're going. In particular, our goal is to understand C++ enough to invoke it, which allows us to be much less pedantic.
I understand and agree with the logic in stating it's a less complicated goal but what comparable success stories exist? I strongly think "devils in the details" in correctly making that work and that they will be tough solvable problems. And then you're going and promising on unfamiliar territory. But what the ultimate takeaway for me is that you won't have it ready in any near term. Do you have the skills and resources to implement this in under 2 years? And then the other question is are you and the team reasonably confident you will have it working and usable by then. Otherwise you are not being pragmatic. On the other hand, if it was reasonably simple as many of your other points in future emails point out, I'd really like to know why you hadn't addressed them earlier.
A that automatically extract definitions (or even wrappings, even partially) from C++ headers is another topic, and should almost certainly lean on an existing compiler.
-It would allow single source files - I think this is important for runtime compiled quasi-JIT/AOC fragments, like OpenCL/PyOpenCL/PyCUDA provide
Not quite following what you're saying here.
Maybe PyInline was a better example off the bat for you, but something a little more flexible but also with less work is needed. Compare with PyOpenCL: https://documen.tician.de/pyopencl/ - check out some examples. There is a c runtime api between the contexts hooking things up (this is the OpenCL runtime part) - it's a pretty similar story to PyCuda (and by the same author, execpt for that project has to jump out to nvcc and cache kernel compilation like the inline function implementation does). There's no limit to the number of functions you can declare though and the OpenCL side is kept simple - things are generally pretty typesafe/do what you would expect on dispatch.
The fact that you're essentially defining a kernel/ufunc with a well defined API in another language makes this a somewhat more natural and tractable case than freely sliding back and forth between C and Python data structures, function calls, etc. multiple times in a function's body. (Not to say that these aren't quite sophisticated projects--they tackle more interesting difficulties elsewhere.)
After you get used to the mental gymnastics you do there, in that kind of development (CUDA/OpenCL), this is pretty simple by comparison. It does feels like some of the same areas of my brain would be activated though in this case. :-) Minor quibble - no you're not defining a ufunc... even a kernel is misnomer, it's a full program on the other end with no main.
The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though.
Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any possible conflicts. Specifying what/how to mangle could get as ugly as GCC's asm variable passing. And embedded variable declarations, let alone control flow statements (especially return, break, ...) could get really messy. It obscures analysis Cython can do on the code, such as whether variables are used or what values they may take. Little code snippets are not always local either, e.g. do they often need to refer to variables (or headers) referenced elsewhere. And they must all be mutually compatible.
Like gcc's asm, let's let adults do what they want and let them worry about the consequences of flow control/stray includes. I'm not even sure how most of this would be an issue (switch/break/if) if you are properly nesting pyxd output. The only thing I think is an issue here is mangled names. I haven't yet figured out why (cdef) variable names must be mangled. Can you explain? Maybe we add an option to allow it to be unmangled in their declaration? C++ has extern "C" for example.
Name mangling is done for the standard reasons--to avoid possible conflicts with all other symbols that may be defined. E.g. We don't want things to suddenly break if I happen to create a variable called "PyNone." Or "__pyx_something_we_defined_implicitly." And of course we want to mangle globals, function names, etc. lest they conflict with some otherwise irrelevant symbol defined in some (possibly recursively) included header somewhere.
Again, you could just say "Don't name things like that." This exposes some more guiding principles. (1) If it's valid Python, it should be valid Cython and (2) we always try to produce valid C code--if you haven't lied to us (too much) about your external declarations, a successful Cython compilation results in a valid C/C++ output. Also (3) you shouldn't have to read or understand the generated C and the Python/C API to use, let alone debug, Cython (though you're happy to do so if you want, like Java developers sometimes read bytecodes, but not usually, though understanding implementation can sometimes be helpful when chasing performance (for all languages)).
There's an obvious tension between giving users all the rope they want vs. providing an API that is possibly more restrictive, but inherently correct by construction. I'll concede that Cython necessarily has pointers, so I'll give that there's plenty of room for foot-shooting (and better interfacing with modern C++ would be good help there), but the kind of errors one runs into by injecting arbitrary code snippets take things to a whole new level (and specifically violate (3) when developing and debugging).
I think injecting arbitrary code snippets has a reasonably good probably of not breaking 3 in your above, provided we have a way to get at unmangled identifiers (*or* document and stick to the mangling strategy, assuming it's easy) - that or we scan the snippet code and replace identifiers (significantly more complex, instincts make me think fragile for a while until it's gotten right - esp without LLVM). Perhaps syntax errors would be an issue if you're just coding things up... but again there's the opt-in to this construct and we could make life easier by annotating the snippet in the output - to help localize the user.
The escape hatch is to wrap the C++ in an actual C++ file and invoke the wrapping. Typically this is the minority of one's code--if it's the "whole library" then you probably have an API that's only understandable to someone well versed in C++ anyways. You've given a single example (non-type template arguments) that we would like to support that's blocking you.
My lack of examples is due to insufficient time playing with Cython - I hit nonstarters so I stop and abandon; as I said, to date, Cython has never been able to solve my C++ problems and none of them seem extraordinary. I think you've got alot of more unknown-unknowns here than you give credit to but we can't discover that until you at least fix that template bug (properly). I'm still not looking forward to forward declaring every identifier/function/whatever from C++ land in Cython though and I still strongly dislike that there's no single source way of doing Cython with something like a kernel/ufunc that needs to escape to C/C++. This makes doing something like the mako based templates I mentioned in the OP email much more cumbersome/hard and Cython would provide no built in mechanism (like inline/inline_module) for making that work.
Why is allowing arbitrary code inside not a good idea? We're not talking something necessarily like eval here and the reputation it got,
Actually, I think it's a whole lot like eval. It's taking an opaque (string) chunk of data and executing it as code. But potentially worse as it's in a different language and evaluated in a transformed (even if the names were unmangled) context.
If we were to go this direction, I might go with a function call (like weave, maybe even follow it) rather than a new statement as the latter is difficult to extend with the myriad of optional configuration parameters, etc. that would beg to follow.
My point with eval is it's bad reputation was mostly due to security vulnerabilities and it joined the league of evil like goto and other tools that are great in the right hands and times. The string is known and fixed at pyx body cython compile time which is one of the things that made me think it had to be a statement rather than a function. But again, I'm not a cython developer. If you think an inlining function that takes in all the args it needs to inline successfully is the mechanism that achieves the effect on pyx compile, I don't think I'd mind. It sure is a privileged and weird function though, to be able to emit code and not be a runtime statement. 2 Examples I wanted to pull out real quick are: https://github.com/scipy/weave/blob/master/examples/wx_example.py https://github.com/scipy/weave/blob/master/examples/binary_search.py But I think I detect magic going on here for adding includes in the wx example and that is not usable/reliable approach. The cython.inline implementation also did include magic for numpy variables... Just a warning. I don't think weave failed because of Python 3 support, I think it was because it was too limited to be useful because of that magic and the walls around getting something say like Eigen in, so nobody used it.
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors and I'm not even sure it had the power to do what I wanted anyway because Cython provides a language that eases the data-extraction/typecasting part of inlining C/C++.
You seem to be repeatedly bringing up the points[:]
* Many (most?) of these string-based approaches are essentially dead, often pointing people to Cython instead, but * Cython should adopt the string-embedding approach of these earlier projects.
Hoho - zing! No that is not a conclusion you should be drawing. Your faults here are to imply those projects failed because they used string-embedding approaches and to imply string-embedding based approaches are the approaches that failed - *most* have failed over a variety of implementations both Python driven and not. I restricted to python-driven for the sake of brevity and mentioned the selection. As I tried to hint earlier, the several other projects failure happened because of any number of unrelated reasons to it being a string based approach. I believe additionally that there were too many options (confusion) and too small a potential userbase (at least in those years) to bolster and attract blood to each of the projects and make them thrive. Probably something akin to the ton of orphaned projects on pipi, it doesn't mean it happened because the approach was wrong. One thing I did want you to take away is that Cython needs to absorb the responsibility of it's reputation and status - the last survivor of a somewhat diverse class with different capabilities, if you will, that went outside of your original usage.
You ask at the beginning of the email whether time has vindicated our philosophy. I think, based on the mindshare vs. these other attempts at integrating with C, in large part it has. It has served us and our users well; we will strive to stay close to Python.
Tight interleaving of multiple languages in is cute for making a polyglot script, but I do not think it leads to legible code. An "eval_cpp" operator would be a lot like the builtin eval--it'd be really tempting to do the "quick and easy" hack of dropping in some executable string instead of thinking how to structure things such that that could be avoided, but putting in this effort leads to more comprehensible code.
It's served your *C* and faster-python users well. If you had proper constructs, I'm sure people wouldn't choose to do it with inline_c unless there was a compelling solid reasoning. I'm not saying you can't make those constructs and that users wouldn't use them when they appear, but you are not being pragmatic again. You currently don't have all capabilities and are on risky turf for supporting all c++ standards for the rest of time. I'm betting against that you will produce in a useful timeframe (maybe this a 5 year scale?) the usable constructs needed - which I equate as turning your head away and giving the middle finger to C++ developers. inline_c would allow a forwards compatible way to use anything the target c++ compiler allows with some very minimal guarantees on Cython's side. It is a very elegant and capable solution to a hard problem.
It's hard to say "no" to features, but I think such an introduction would fundamentally change Cython and how it's written for the worse.
I agree with the statement but I don't think you've classified the feature correctly. I watched one of your old talks for Sage Days 29 and at the bottom of a slide you have "Cython is a very pragmatic project, driven by user needs". I'm calling foul. Go watch that video again and tell me what's changed since 2011. -Jason
On Sun, Aug 21, 2016 at 7:26 AM, Jason Newton <nevion@gmail.com> wrote:
On Sun, Aug 21, 2016 at 5:30 AM, Robert Bradshaw <robertwb@gmail.com> wrote:
In my experience Cython has generally been fairly easy to pick up for people who already know Python. And Python often easy to pick up for people who already know C/C++. Of course for many wrappings it often takes non-trivial knowledge of the wrapped library itself too, but typically at the same level as would be required to grok code written against that same library directly from C/C++.
Is your experience drawn from binding moderately complex libraries or play code (of complexity like from a tutorial). Bottom up or top down? Sorry if this sounds asinine to you.
A whole variety: from simple string manipulation libraries to complex ML and big data processing frameworks. And second hand with complete amateurs and students to senior software engineers.
To clarify, I'm coming from the case where I didn't read the whole tutorial/docs before being faced with pyx in the projects I previously mentioned, while on tight turn around time - I was not able to grok in that context.
Understood--I am short on time too. Though your self-admitted lack of familiarity doesn't bolster your argument that Cython's doing it wrong, we need this new feature.
For myself and any other ML user who comes across this thread - can you list a few libraries that do things the right way?
Yes, there's bad code out there in any language (no offense meant towards h5py--I haven't looked at that project myself). Much of it due to cargo-cult perpetuations or archaic (or simply flat-out-wrong) contortions due to historical limitations (e.g. creating a Python module to wrap an _extension module, avoiding all C++ features with extensive C wrappers, ...). (You're familiar with C++, so likely no stranger to this effect.)
These projects complied with Cython's current philosophy to the degradation of clarity, context, and overall idea of how code was hooked up. Perhaps Cython should take the lessons learned from it's inception, time, and the results of the state of the c-python userbase to guide us into a new philosophy.
I fail to see how "staying close to Python" caused "degradation of clarity, context, etc." If anything, the lessons learned over time have validated this philosophy. More on this later.
My point was that multifile multi-level wrapper that I mentioned earlier - if you're saying that those projects did Cython extensions wrong, then I'm incorrect at faulting Cython and should fault the libraries using it. I didn't say staying close to python caused $blurb.
I don't know in a situation as confusing as to all the binding projects if this should be taken as validation of philosophy either - I think it is reasonable to consider the attrition of these projects as a function of manpower, number of early on project supporters/authors, and if a project (like sage) indirectly, through dependency, kept the project alive. And good old fashioned luck. I noted most of them don't use distutils and something custom but less capable instead which maybe plays a roll in how mature/usable/smalltime they where/are.
Certainly the success of a project depends on many external factors, and even raw luck plays a part. But the approach and philosophy taken to attack a problem and guard its API (and the users/contributors that such decisions attract or repel) can't be discounted either, especially when taken over a long timeframe. But if anyone wants to believe that Cython's become popular because of pure luck despite wrongheaded guiding principles or philosophies, it'll be difficult to persuade them otherwise.
I agree that any efforts to trying to parsing C++ without building on an actual compiler are fraught with danger. That's not the case with generating C++ code, which is the direction we're going. In particular, our goal is to understand C++ enough to invoke it, which allows us to be much less pedantic.
I understand and agree with the logic in stating it's a less complicated goal but what comparable success stories exist? I strongly think "devils in the details" in correctly making that work and that they will be tough solvable problems. And then you're going and promising on unfamiliar territory. But what the ultimate takeaway for me is that you won't have it ready in any near term. Do you have the skills and resources to implement this in under 2 years? And then the other question is are you and the team reasonably confident you will have it working and usable by then. Otherwise you are not being pragmatic.
On the other hand, if it was reasonably simple as many of your other points in future emails point out, I'd really like to know why you hadn't addressed them earlier.
Other higher priority items for limited resources. And non-type template args are not necessarily that simple given the way things are structured now.
The idea is that Cython glue makes the playing field for extracting data easy, but that once it's extracted to a cdef variable for instance, cython doesn't need to know what happens. Maybe in a way sort of like the GCC asm extension. Hopefully simpler variable passing though.
Cython uses "mangled" names (e.g. with a __pyx prefix) to avoid any possible conflicts. Specifying what/how to mangle could get as ugly as GCC's asm variable passing. And embedded variable declarations, let alone control flow statements (especially return, break, ...) could get really messy. It obscures analysis Cython can do on the code, such as whether variables are used or what values they may take. Little code snippets are not always local either, e.g. do they often need to refer to variables (or headers) referenced elsewhere. And they must all be mutually compatible.
Like gcc's asm, let's let adults do what they want and let them worry about the consequences of flow control/stray includes. I'm not even sure how most of this would be an issue (switch/break/if) if you are properly nesting pyxd output. The only thing I think is an issue here is mangled names. I haven't yet figured out why (cdef) variable names must be mangled. Can you explain? Maybe we add an option to allow it to be unmangled in their declaration? C++ has extern "C" for example.
Name mangling is done for the standard reasons--to avoid possible conflicts with all other symbols that may be defined. E.g. We don't want things to suddenly break if I happen to create a variable called "PyNone." Or "__pyx_something_we_defined_implicitly." And of course we want to mangle globals, function names, etc. lest they conflict with some otherwise irrelevant symbol defined in some (possibly recursively) included header somewhere.
Again, you could just say "Don't name things like that." This exposes some more guiding principles. (1) If it's valid Python, it should be valid Cython and (2) we always try to produce valid C code--if you haven't lied to us (too much) about your external declarations, a successful Cython compilation results in a valid C/C++ output. Also (3) you shouldn't have to read or understand the generated C and the Python/C API to use, let alone debug, Cython (though you're happy to do so if you want, like Java developers sometimes read bytecodes, but not usually, though understanding implementation can sometimes be helpful when chasing performance (for all languages)).
There's an obvious tension between giving users all the rope they want vs. providing an API that is possibly more restrictive, but inherently correct by construction. I'll concede that Cython necessarily has pointers, so I'll give that there's plenty of room for foot-shooting (and better interfacing with modern C++ would be good help there), but the kind of errors one runs into by injecting arbitrary code snippets take things to a whole new level (and specifically violate (3) when developing and debugging).
I think injecting arbitrary code snippets has a reasonably good probably of not breaking 3 in your above, provided we have a way to get at unmangled identifiers (*or* document and stick to the mangling strategy, assuming it's easy) - that or we scan the snippet code and replace identifiers (significantly more complex, instincts make me think fragile for a while until it's gotten right - esp without LLVM). Perhaps syntax errors would be an issue if you're just coding things up... but again there's the opt-in to this construct and we could make life easier by annotating the snippet in the output - to help localize the user.
You're missing the point of (3). The fact that we're generating C code, and not fortran or directly assembly, should mostly be an implementation detail. It's not realistic to embed snippets without caring about the surrounding context in all but the simplest of cases. And there'd be feature creep here--you're in the middle of a C snippet and want to report an error, or access a Python object, or ...
The escape hatch is to wrap the C++ in an actual C++ file and invoke the wrapping. Typically this is the minority of one's code--if it's the "whole library" then you probably have an API that's only understandable to someone well versed in C++ anyways. You've given a single example (non-type template arguments) that we would like to support that's blocking you.
My lack of examples is due to insufficient time playing with Cython - I hit nonstarters so I stop and abandon; as I said, to date, Cython has never been able to solve my C++ problems and none of them seem extraordinary. I think you've got alot of more unknown-unknowns here than you give credit to but we can't discover that until you at least fix that template bug (properly).
It would be helpful for you to enumerate the (implied many) nonstarters and problems you've had, to at least get the known-unknowns out on the table.
I'm still not looking forward to forward declaring every identifier/function/whatever from C++ land in Cython though
This is largely a separate issue, and I agree a big pain point.
and I still strongly dislike that there's no single source way of doing Cython with something like a kernel/ufunc that needs to escape to C/C++. This makes doing something like the mako based templates I mentioned in the OP email much more cumbersome/hard and Cython would provide no built in mechanism (like inline/inline_module) for making that work.
The issue here that you want to inline C code snippets into your inlined Cython code snippet?
Why is allowing arbitrary code inside not a good idea? We're not talking something necessarily like eval here and the reputation it got,
Actually, I think it's a whole lot like eval. It's taking an opaque (string) chunk of data and executing it as code. But potentially worse as it's in a different language and evaluated in a transformed (even if the names were unmangled) context.
If we were to go this direction, I might go with a function call (like weave, maybe even follow it) rather than a new statement as the latter is difficult to extend with the myriad of optional configuration parameters, etc. that would beg to follow.
My point with eval is it's bad reputation was mostly due to security vulnerabilities and it joined the league of evil like goto and other tools that are great in the right hands and times. The string is known and fixed at pyx body cython compile time which is one of the things that made me think it had to be a statement rather than a function. But again, I'm not a cython developer. If you think an inlining function that takes in all the args it needs to inline successfully is the mechanism that achieves the effect on pyx compile, I don't think I'd mind. It sure is a privileged and weird function though, to be able to emit code and not be a runtime statement.
My intuition is that the "interface" would need more complicated specification which will grow over time--weave.inline itself takes 27 parameters.
2 Examples I wanted to pull out real quick are:
https://github.com/scipy/weave/blob/master/examples/wx_example.py https://github.com/scipy/weave/blob/master/examples/binary_search.py
But I think I detect magic going on here for adding includes in the wx example and that is not usable/reliable approach. The cython.inline implementation also did include magic for numpy variables... Just a warning. I don't think weave failed because of Python 3 support, I think it was because it was too limited to be useful because of that magic and the walls around getting something say like Eigen in, so nobody used it.
You must realize that almost any other python driven way to compile c-code in the spirit these projects do is deprecated/dead. Cython has absorbed all the reputation and users that didn't go to pure-c/boost.python - pybind11 is the new kid on the block there so I'm not including it (I'm of the opinion that SWIG users stayed unchanged). Community belief/QA/designers/google all think of Cython first. Weave has effectively closed up it's doors and I'm not even sure it had the power to do what I wanted anyway because Cython provides a language that eases the data-extraction/typecasting part of inlining C/C++.
You seem to be repeatedly bringing up the points[:]
* Many (most?) of these string-based approaches are essentially dead, often pointing people to Cython instead, but * Cython should adopt the string-embedding approach of these earlier projects.
Hoho - zing! No that is not a conclusion you should be drawing. Your faults here are to imply those projects failed because they used string-embedding approaches and to imply string-embedding based approaches are the approaches that failed - *most* have failed over a variety of implementations both Python driven and not. I restricted to python-driven for the sake of brevity and mentioned the selection. As I tried to hint earlier, the several other projects failure happened because of any number of unrelated reasons to it being a string based approach. I believe additionally that there were too many options (confusion) and too small a potential userbase (at least in those years) to bolster and attract blood to each of the projects and make them thrive. Probably something akin to the ton of orphaned projects on pipi, it doesn't mean it happened because the approach was wrong. One thing I did want you to take away is that Cython needs to absorb the responsibility of it's reputation and status - the last survivor of a somewhat diverse class with different capabilities, if you will, that went outside of your original usage.
I'm not saying that these projects died because of the string-based approach they took, rather that if this "embed strings" approach were so critical, so superior, it should at least kept one of them alive. Or a new project could have formed around this approach (e.g. letting all the executable code be C++, with Python syntax for the structure, could be an interesting point in the design space). It has its pros and cons. I think for Cython it's the wrong direction. But I'm in favor of letting many flowers bloom--and we're in luck that these are all open source to boot.
You ask at the beginning of the email whether time has vindicated our philosophy. I think, based on the mindshare vs. these other attempts at integrating with C, in large part it has. It has served us and our users well; we will strive to stay close to Python.
Tight interleaving of multiple languages in is cute for making a polyglot script, but I do not think it leads to legible code. An "eval_cpp" operator would be a lot like the builtin eval--it'd be really tempting to do the "quick and easy" hack of dropping in some executable string instead of thinking how to structure things such that that could be avoided, but putting in this effort leads to more comprehensible code.
It's served your *C* and faster-python users well. If you had proper constructs, I'm sure people wouldn't choose to do it with inline_c unless there was a compelling solid reasoning.
I'm not saying you can't make those constructs and that users wouldn't use them when they appear, but you are not being pragmatic again. You currently don't have all capabilities and are on risky turf for supporting all c++ standards for the rest of time. I'm betting against that you will produce in a useful timeframe (maybe this a 5 year scale?) the usable constructs needed - which I equate as turning your head away and giving the middle finger to C++ developers. inline_c would allow a forwards compatible way to use anything the target c++ compiler allows with some very minimal guarantees on Cython's side. It is a very elegant and capable solution to a hard problem.
It's hard to say "no" to features, but I think such an introduction would fundamentally change Cython and how it's written for the worse.
I agree with the statement but I don't think you've classified the feature correctly.
My conclusion is based on thinking a lot about where this feature would lead us, not just the immediate possibilities it would open up.
I watched one of your old talks for Sage Days 29 and at the bottom of a slide you have "Cython is a very pragmatic project, driven by user needs". I'm calling foul. Go watch that video again and tell me what's changed since 2011.
Embedding code is not a need, it's a means to an end. This is why I've been asking for concrete features that are the most important to try to support. - Robert
On Mon, Aug 22, 2016 at 3:26 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
To clarify, I'm coming from the case where I didn't read the whole tutorial/docs before being faced with pyx in the projects I previously mentioned, while on tight turn around time - I was not able to grok in that context.
Understood--I am short on time too. Though your self-admitted lack of familiarity doesn't bolster your argument that Cython's doing it wrong, we need this new feature.
Bare in mind that was then, and while I'm not Cython developer level I do have a better idea of how Cython works now, possibly better than your average user. I've thought about this problem and the state of the overall C++ python binding capability alot as I've had to reevaluate it possibly more than most, and I've used them more deeply than playcode.
For myself and any other ML user who comes across this thread - can you list a few libraries that do things the right way?
if you're saying that those projects did Cython extensions wrong, then I'm incorrect at faulting Cython and should fault the libraries using it. I didn't say staying close to python caused $blurb.
I don't know in a situation as confusing as to all the binding projects if this should be taken as validation of philosophy either - I think it is reasonable to consider the attrition of these projects as a function of manpower, number of early on project supporters/authors, and if a project (like sage) indirectly, through dependency, kept the project alive. And good old fashioned luck. I noted most of them don't use distutils and something custom but less capable instead which maybe plays a roll in how mature/usable/smalltime they where/are.
My point was that multifile multi-level wrapper that I mentioned earlier
Certainly the success of a project depends on many external factors, and even raw luck plays a part. But the approach and philosophy taken to attack a problem and guard its API (and the users/contributors that such decisions attract or repel) can't be discounted either, especially when taken over a long timeframe.
But if anyone wants to believe that Cython's become popular because of pure luck despite wrongheaded guiding principles or philosophies, it'll be difficult to persuade them otherwise.
You snidely play with extremes, I never said pure luck was the main factor - I listed several factors of very high significance. Further, Cython branched off of pyrex - why do you think that project failed (you might actually know)? Did your philosophy carry separate the two or was it because you needed things to work (for sage), had a larger set of smart developers and improve it faster than pyrex would allow? I think it's the bustling of life that got Cython where it is, I don't believe it's philosophy in this case - I think it was lack of feature competitiveness and Cython probably worked better (more bugs/cases fixed).
I agree that any efforts to trying to parsing C++ without building on an actual compiler are fraught with danger. That's not the case with generating C++ code, which is the direction we're going. In particular, our goal is to understand C++ enough to invoke it, which allows us to be much less pedantic.
I understand and agree with the logic in stating it's a less complicated goal but what comparable success stories exist? I strongly think "devils in the details" in correctly making that work and that they will be tough solvable problems. And then you're going and promising on unfamiliar territory. But what the ultimate takeaway for me is that you won't have it ready in any near term. Do you have the skills and resources to implement this in under 2 years? And then the other question is are you and the team reasonably confident you will have it working and usable by then. Otherwise you are not being pragmatic.
On the other hand, if it was reasonably simple as many of your other points in future emails point out, I'd really like to know why you hadn't addressed them earlier.
Other higher priority items for limited resources. And non-type template args are not necessarily that simple given the way things are structured now.
There's always limited resources and priorities but hen it sounds like it's going to be a very (ever)? long time until everything is working great unless the need is becoming a clear driver or this email thread has reassigned priorities. Do you think you can sketch out a time table for the known issues? Maybe not in this thread but I'd appreciate a back of the envelope calculation - realistic time of usefulness is part of being pragmatic.
You're missing the point of (3). The fact that we're generating C code, and not fortran or directly assembly, should mostly be an implementation detail. It's not realistic to embed snippets without caring about the surrounding context in all but the simplest of cases. And there'd be feature creep here--you're in the middle of a C snippet and want to report an error, or access a Python object, or ...
I know you took a different philosophy than pyrex, but I don't understand why. In that sage days presentation, you mentioned pyrex chose to distinguish itself as a way to write c extensions, emphasizing it (I don't know to the degree if it offered what wish cython offered, doubted). It's pretty clear to me you wanted mostly accelerated python with some c-trampolining capability through externally defined functions. I *really* think you should remember your root's goals though.
The escape hatch is to wrap the C++ in an actual C++ file and invoke the wrapping. Typically this is the minority of one's code--if it's the "whole library" then you probably have an API that's only understandable to someone well versed in C++ anyways. You've given a single example (non-type template arguments) that we would like to support that's blocking you.
My lack of examples is due to insufficient time playing with Cython - I hit nonstarters so I stop and abandon; as I said, to date, Cython has never been able to solve my C++ problems and none of them seem extraordinary. I think you've got alot of more unknown-unknowns here than you give credit to but we can't discover that until you at least fix that template bug (properly).
It would be helpful for you to enumerate the (implied many) nonstarters and problems you've had, to at least get the known-unknowns out on the table.
Honestly I can't remember the other 2 times I reevaluated its applicability (we're talking a few days each over the last ~10 years), it was too long ago and capability of Cython has grown right? My answer in each of those times, originally coming from SWIG, was boost.python... and in the mean time it's looking like it might be pybind11 (I don't have the luxury of waiting for what I'm working on, but I'm not forgetting about Cython either).
I'm still not looking forward to forward declaring every identifier/function/whatever from C++ land in Cython though
This is largely a separate issue, and I agree a big pain point.
and I still strongly dislike that there's no single source way of doing Cython with something like a kernel/ufunc that needs to escape to C/C++. This makes doing something like the mako based templates I mentioned in the OP email much more cumbersome/hard and Cython would provide no built in mechanism (like inline/inline_module) for making that work.
The issue here that you want to inline C code snippets into your inlined Cython code snippet?
Multiple files is more a PITA to making the mako-templated strategy work - I didn't say impossible, just more a PITA. I learned the JIT-module compilation technique from OpenCL/PyCUDA and found it makes a metric ton of sense to do, the performance boost can be massive from using -DDEFINES and embedding runtime variables/constants right into C (recall the point of Cython seems to be for speed :-). This is one of the reasons why I'd like to stick with a python driven way of doing C++ bindings (which hopefully can use C++ directly). It's alot easier for a single file to work and simply pass a string to something like inline which does all the file writing, and module placement.
I'm not saying that these projects died because of the string-based approach they took, rather that if this "embed strings" approach were so critical, so superior, it should at least kept one of them alive. Or a new project could have formed around this approach (e.g. letting all the executable code be C++, with Python syntax for the structure, could be an interesting point in the design space). It has its pros and cons. I think for Cython it's the wrong direction. But I'm in favor of letting many flowers bloom--and we're in luck that these are all open source to boot.
I think it's because of their as-is-limitations, not of the technique they implemented but as a whole system. I looked at the source for pyinline the other day and it looked like it only supports primitive types and would choke on C++ for instance and doesn't really offer convenience to extract data from python objects like Cython can so easily (the whole system, including it's custom C (no ++) lexer is ~700 lines). I tend to think of a few/most of them are just passed the working proof of concept stage and were significantly lacking in overall capability. Most python projects (on PyPi) are single contributors with short lifespans too which seems to be the case for those projects. I'm not sure when/how Pyrex started out (1 guy at Uni who set up a full website?) but how did it happen with Cython?
I watched one of your old talks for Sage Days 29 and at the bottom of a slide you have "Cython is a very pragmatic project, driven by user needs". I'm calling foul. Go watch that video again and tell me what's changed since 2011.
Embedding code is not a need, it's a means to an end. This is why I've been asking for concrete features that are the most important to try to support.
I have another idea/iteration to run by you then. One of your chief quibbles, although I don't think it's your underlying one, is Cython must understand what's going on. So how about we support a block of C/C++ code as a proper construct. Same name but now, I guess braces maybe get involved (since semicolons do too). Then to transform (mangle) the nested identifiers in accordance with scoping/shadowing rules we pull in the python bindings to llvm/Clang to parse that complicated code, work with the AST (or further down the rabbit-hole) appropriately and spit it back out. Obviously you could detect and analyze embedded statements to your hearts content to decide to do something smart about that (you mentioned exception handling and return statement as concerning items). The potential hazards here are that LLVM/Clang is brought in (LLVM is already on ~ everybody's linux boxes as a graphics shader dependency) and it's [python] api changes somewhat often. I've used it before with success for python driven C++ POD boiler plate (IO, size checks, visitors methods etc) autocoding. If you can understand the underlying C/C++ and get over that Cython will always run on a C/C++ compiler in our universe, and make that a feature to embrace, not a detail to hide (who actually benefits in the later case?) - are you still against the significant convenience - and potentially only way to make things work when Cython is not supporting a C++ way of doing things (say things aren't finished/working yet)? -Jason
On Mon, Aug 22, 2016 at 4:48 PM, Jason Newton <nevion@gmail.com> wrote:
Bare in mind that was then, and while I'm not Cython developer level I do have a better idea of how Cython works now, possibly better than your average user. I've thought about this problem and the state of the overall C++ python binding capability alot as I've had to reevaluate it possibly more than most, and I've used them more deeply than playcode.
Good to know.
My point was that multifile multi-level wrapper that I mentioned earlier - if you're saying that those projects did Cython extensions wrong, then I'm incorrect at faulting Cython and should fault the libraries using it. I didn't say staying close to python caused $blurb.
I don't know in a situation as confusing as to all the binding projects if this should be taken as validation of philosophy either - I think it is reasonable to consider the attrition of these projects as a function of manpower, number of early on project supporters/authors, and if a project (like sage) indirectly, through dependency, kept the project alive. And good old fashioned luck. I noted most of them don't use distutils and something custom but less capable instead which maybe plays a roll in how mature/usable/smalltime they where/are.
Certainly the success of a project depends on many external factors, and even raw luck plays a part. But the approach and philosophy taken to attack a problem and guard its API (and the users/contributors that such decisions attract or repel) can't be discounted either, especially when taken over a long timeframe.
But if anyone wants to believe that Cython's become popular because of pure luck despite wrongheaded guiding principles or philosophies, it'll be difficult to persuade them otherwise.
You snidely play with extremes, I never said pure luck was the main factor - I listed several factors of very high significance. Further, Cython branched off of pyrex - why do you think that project failed (you might actually know)? Did your philosophy carry separate the two or was it because you needed things to work (for sage), had a larger set of smart developers and improve it faster than pyrex would allow? I think it's the bustling of life that got Cython where it is, I don't believe it's philosophy in this case - I think it was lack of feature competitiveness and Cython probably worked better (more bugs/cases fixed).
I wouldn't say that Pyrex failed--it gave birth to Cython. At a high level where I'd say they diverged is that Pyrex was viewed as a tool to write C extensions, whereas Cython was viewed as a Python compiler. The latter meant prioritizing things like being an optimizing compiler, full coverage of the Python API (e.g. generators), numpy/buffer integration, and Python 3 support were higher priority, as opposed to being non-goals for Pyrex. But the key insight of using Python, or at least Python-inspired, syntax to declare and invoke C constructs lives on. Which is quite relevant to this thread. The fact that Greg didn't seem to have the time/interest in corralling an open source project factored in here as well. If you have anything to add, Greg, always happy to have you chime in.
I agree that any efforts to trying to parsing C++ without building on an actual compiler are fraught with danger. That's not the case with generating C++ code, which is the direction we're going. In particular, our goal is to understand C++ enough to invoke it, which allows us to be much less pedantic.
I understand and agree with the logic in stating it's a less complicated goal but what comparable success stories exist? I strongly think "devils in the details" in correctly making that work and that they will be tough solvable problems. And then you're going and promising on unfamiliar territory. But what the ultimate takeaway for me is that you won't have it ready in any near term. Do you have the skills and resources to implement this in under 2 years? And then the other question is are you and the team reasonably confident you will have it working and usable by then. Otherwise you are not being pragmatic.
On the other hand, if it was reasonably simple as many of your other points in future emails point out, I'd really like to know why you hadn't addressed them earlier.
Other higher priority items for limited resources. And non-type template args are not necessarily that simple given the way things are structured now.
There's always limited resources and priorities but hen it sounds like it's going to be a very (ever)? long time until everything is working great unless the need is becoming a clear driver or this email thread has reassigned priorities. Do you think you can sketch out a time table for the known issues? Maybe not in this thread but I'd appreciate a back of the envelope calculation - realistic time of usefulness is part of being pragmatic.
For a small project like this run completely by volunteers, the only way to set a timetable is to to the work yourself :). However, this discussion has re-motivated me to iron some of these issues out, and I might (surprisingly) have some time on my hands to do so.
You're missing the point of (3). The fact that we're generating C code, and not fortran or directly assembly, should mostly be an implementation detail. It's not realistic to embed snippets without caring about the surrounding context in all but the simplest of cases. And there'd be feature creep here--you're in the middle of a C snippet and want to report an error, or access a Python object, or ...
I know you took a different philosophy than pyrex, but I don't understand why. In that sage days presentation, you mentioned pyrex chose to distinguish itself as a way to write c extensions, emphasizing it (I don't know to the degree if it offered what wish cython offered, doubted). It's pretty clear to me you wanted mostly accelerated python with some c-trampolining capability through externally defined functions. I *really* think you should remember your root's goals though.
Yep. "C extension" is an implementation detail. (One can't always get away from implementation details leaking though, but I rather like to avoid prominently baking them in.)
The issue here that you want to inline C code snippets into your inlined Cython code snippet?
Multiple files is more a PITA to making the mako-templated strategy work - I didn't say impossible, just more a PITA. I learned the JIT-module compilation technique from OpenCL/PyCUDA and found it makes a metric ton of sense to do, the performance boost can be massive from using -DDEFINES and embedding runtime variables/constants right into C (recall the point of Cython seems to be for speed :-). This is one of the reasons why I'd like to stick with a python driven way of doing C++ bindings (which hopefully can use C++ directly). It's alot easier for a single file to work and simply pass a string to something like inline which does all the file writing, and module placement.
I just had one idea to ease the pain--cython.inline[_module] could take a "extra C source" parameter that would be #included in the file. I'm much more comfortable with this over making changes to the language itself.
I'm not saying that these projects died because of the string-based approach they took, rather that if this "embed strings" approach were so critical, so superior, it should at least kept one of them alive. Or a new project could have formed around this approach (e.g. letting all the executable code be C++, with Python syntax for the structure, could be an interesting point in the design space). It has its pros and cons. I think for Cython it's the wrong direction. But I'm in favor of letting many flowers bloom--and we're in luck that these are all open source to boot.
I think it's because of their as-is-limitations, not of the technique they implemented but as a whole system. I looked at the source for pyinline the other day and it looked like it only supports primitive types and would choke on C++ for instance and doesn't really offer convenience to extract data from python objects like Cython can so easily (the whole system, including it's custom C (no ++) lexer is ~700 lines). I tend to think of a few/most of them are just passed the working proof of concept stage and were significantly lacking in overall capability. Most python projects (on PyPi) are single contributors with short lifespans too which seems to be the case for those projects. I'm not sure when/how Pyrex started out (1 guy at Uni who set up a full website?) but how did it happen with Cython?
I watched one of your old talks for Sage Days 29 and at the bottom of a slide you have "Cython is a very pragmatic project, driven by user needs". I'm calling foul. Go watch that video again and tell me what's changed since 2011.
Embedding code is not a need, it's a means to an end. This is why I've been asking for concrete features that are the most important to try to support.
I have another idea/iteration to run by you then. One of your chief quibbles, although I don't think it's your underlying one, is Cython must understand what's going on. So how about we support a block of C/C++ code as a proper construct. Same name but now, I guess braces maybe get involved (since semicolons do too). Then to transform (mangle) the nested identifiers in accordance with scoping/shadowing rules we pull in the python bindings to llvm/Clang to parse that complicated code, work with the AST (or further down the rabbit-hole) appropriately and spit it back out. Obviously you could detect and analyze embedded statements to your hearts content to decide to do something smart about that (you mentioned exception handling and return statement as concerning items).
The potential hazards here are that LLVM/Clang is brought in (LLVM is already on ~ everybody's linux boxes as a graphics shader dependency) and it's [python] api changes somewhat often. I've used it before with success for python driven C++ POD boiler plate (IO, size checks, visitors methods etc) autocoding.
If you can understand the underlying C/C++ and get over that Cython will always run on a C/C++ compiler in our universe, and make that a feature to embrace, not a detail to hide (who actually benefits in the later case?) - are you still against the significant convenience - and potentially only way to make things work when Cython is not supporting a C++ way of doing things (say things aren't finished/working yet)?
That would help solve the analysis, though enormous cost in complexity (one now has two ASTs one must understand and reconcile, and a huge amount of C++ knowledge must be baked in to understand the AST (or even lower levels) and it'd be neigh impossible to do "partial analysis" for features we don't yet understand, probably harder than putting them into Cython itself modulo parsing) and making LLVM/Clang a dependency for Cython is probably a non-starter as well. But you're right, it's not my primary quibble, which is that a tight alternating of C++ and Python statements seems, well, not very Pythonic and would IMHO lead to hard to read and debug code. That and I think the interface (including expectations and dependencies, possibly non-local) between the two would grow a lot messier than a question of how to resolve name mangling. I don't think snippets would solve the fundamental differences like about scoping discrepancies between Python and C++ (e.g. for RAII). And it would be a step back for the nice auto-conversion facilities you care about, e.g. py_func(c_func(py_value)) would become c_value = py_value cdef some_type c_ret inline_cpp "c_ret = c_func(c_value)" py_func(c_ret) It's just choosing a smaller unit (line vs. file) to swap between languages. - Robert
On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
And it would be a step back for the nice auto-conversion facilities you care about, e.g.
py_func(c_func(py_value))
would become
c_value = py_value cdef some_type c_ret inline_cpp "c_ret = c_func(c_value)" py_func(c_ret)
It's just choosing a smaller unit (line vs. file) to swap between languages.
And on this note it would also encourage use of the Python/C API (over swapping "back" to Python) which is often a dangerous step backwards.
Watching this thread for a long time now, and I must say I absolutely love how someone just barges in «oh just add parsing of C++ AST». C++ is a language with an extremely complex context-dependent grammar, much more so than C. Not to mention they took to updating the standard with new awesome features every 3 or 4 years, which all seem to absolutely need their own syntactic support.
23 авг. 2016 г., в 8:03, Robert Bradshaw <robertwb@gmail.com> написал(а):
On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
And it would be a step back for the nice auto-conversion facilities you care about, e.g.
py_func(c_func(py_value))
would become
c_value = py_value cdef some_type c_ret inline_cpp "c_ret = c_func(c_value)" py_func(c_ret)
It's just choosing a smaller unit (line vs. file) to swap between languages.
And on this note it would also encourage use of the Python/C API (over swapping "back" to Python) which is often a dangerous step backwards. _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Just trying and improve the state of the art. Yea maybe we have to add some tricks to separate the two grammar wise, so they can play together. Besides, I didn't start with that complexity, it evolved into seeing if that was a viable path forward. On Tue, Aug 23, 2016 at 5:15 AM, Alex S. <alex0player@gmail.com> wrote:
Watching this thread for a long time now, and I must say I absolutely love how someone just barges in «oh just add parsing of C++ AST». C++ is a language with an extremely complex context-dependent grammar, much more so than C. Not to mention they took to updating the standard with new awesome features every 3 or 4 years, which all seem to absolutely need their own syntactic support.
23 авг. 2016 г., в 8:03, Robert Bradshaw <robertwb@gmail.com> написал(а):
On Mon, Aug 22, 2016 at 9:44 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
And it would be a step back for the nice auto-conversion facilities you care about, e.g.
py_func(c_func(py_value))
would become
c_value = py_value cdef some_type c_ret inline_cpp "c_ret = c_func(c_value)" py_func(c_ret)
It's just choosing a smaller unit (line vs. file) to swap between languages.
And on this note it would also encourage use of the Python/C API (over swapping "back" to Python) which is often a dangerous step backwards. _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
are you still against the significant convenience - and potentially only way to make things work when Cython is not supporting a C++ way of doing things (say things aren't finished/working yet)?
I have another idea/iteration to run by you then. One of your chief quibbles, although I don't think it's your underlying one, is Cython must understand what's going on. So how about we support a block of C/C++ code as a proper construct. Same name but now, I guess braces maybe get involved (since semicolons do too). Then to transform (mangle) the nested identifiers in accordance with scoping/shadowing rules we pull in the python bindings to llvm/Clang to parse that complicated code, work with the AST (or further down the rabbit-hole) appropriately and spit it back out. Obviously you could detect and analyze embedded statements to your hearts content to decide to do something smart about that (you mentioned exception handling and return statement as concerning items).
The potential hazards here are that LLVM/Clang is brought in (LLVM is already on ~ everybody's linux boxes as a graphics shader dependency) and it's [python] api changes somewhat often. I've used it before with success for python driven C++ POD boiler plate (IO, size checks, visitors methods etc) autocoding.
If you can understand the underlying C/C++ and get over that Cython will always run on a C/C++ compiler in our universe, and make that a feature to embrace, not a detail to hide (who actually benefits in the later case?)
That would help solve the analysis, though enormous cost in complexity (one now has two ASTs one must understand and reconcile, and a huge amount of C++ knowledge must be baked in to understand the AST (or even lower levels) and it'd be neigh impossible to do "partial analysis" for features we don't yet understand, probably harder than putting them into Cython itself modulo parsing) and making LLVM/Clang a dependency for Cython is probably a non-starter as well.
Well, one thing occurred to me is that leaning on Clang may be the only way you'll be able to remove/improve the DRY violating Cython side declaration of C/C++ land stuff. -Jason
Robert Bradshaw wrote:
Name mangling is done for the standard reasons--to avoid possible conflicts with all other symbols that may be defined.
The main reason for mangling names is that top-level declarations in Cython belong to a module namespace, whereas C just has a single namespace for all globals. So some way is needed to distinguish things with the same name in different modules. Also, a single Cython declaration often leads to several different related things being declared in C, e.g. a class has a PyType instance, a struct declaration, and possibly a vtable, all of which need distinct names in the C code, so prefixes are used to distinguish them. It might be possible to get away without mangling a few things, such as local variables, but for simplicity they're treated the same as everything else. -- Greg
Robert Bradshaw schrieb am 19.08.2016 um 11:36:
On Thu, Aug 18, 2016 at 12:05 PM, Jason Newton wrote:
I was hoping we could give a way to drop straight into C/C++ inside of Cython pyx files.
Why? [...] -It works around having the Cython compiler know about all of C++'s nuances - as an advanced C++ developer these are painful and it is a 2nd class citizen to Cython's simpler C-support - that's no good. Just right now I was bitten by yet another template argument bug and it's clear C++ template arguments have been kind of dicy since support appeared.
Yes, C++ is an extraordinarily complicated language, and exposing all of that would add significant amounts of complexity to Cython itself, and perhaps more importantly increase the barrier of entry to reading Cython code. One of the guiding principles is that we try to stay as close to Python as possible (if you know Python, you can probably read Cython, and with a minimal amount of C knowledge start writing it) and much of C++ simply isn't Pythonic.
Though, as you discovered, there are some basic things like non-type template arguments that we would like to have. If there are other specific C++ constructs that are commonly used but impossible to express in Cython it'd be useful to know.
Especially with respect to C++, we've always tried to make things safe and helpful that Cython can support directly, and to make things possible that are too complex to handle safely. But I agree with Robert that allowing arbitrary C++ code snippets inside of Cython code is not a good idea. If there are code constructs that we can add to improve the C++ integration then we should do that, even if it means that Cython would (initially?) not be able to validate that code. But we should otherwise keep the language such that it allows us to get back to a state where Cython can validate and understand what it sees. Stefan
On Sat, Aug 20, 2016 at 1:19 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Especially with respect to C++, we've always tried to make things safe and helpful that Cython can support directly, and to make things possible that are too complex to handle safely. But I agree with Robert that allowing arbitrary C++ code snippets inside of Cython code is not a good idea.
Why is allowing arbitrary code inside not a good idea? We're not talking something necessarily like eval here and the reputation it got, we're talking C/C++ glue and for the scale of Cython's development effort, proper C++ support is insurmountable simply because the language is that hard to work with on the compiler side. You can't just go and take something that is readily in the process of evolving (as newer standards actively come out), that very slowly achieves full compiler support in it's own domain, and pretend/aspire you can work like that though high level constructs in your language - this project and none before it have not accomplished anywhere near that and simply don't have the man-years/decades to contribute. I truly think it is irresponsible to think a) we should do nothing or b) delude ourselfs that we can make high level language level features to get around this.
If there are code constructs that we can add to improve the C++ integration then we should do that, even if it means that Cython would (initially?) not be able to validate that code. But we should otherwise keep the language such that it allows us to get back to a state where Cython can validate and understand what it sees.
I'm not going to argue against you making C++ developer life easier but I have no idea how you could wrangle the beast in another way than letting the C++ compiler handle things. I don't think it's important that Cython compiler can validate / understand everything it sees - I'm not saying that by default, but just like the various flags you can use with cpdef, strategic sacrifices are acceptable for consenting adults. Otherwise you are still saying you will limit capability, which many can't agree with and avoid like the plague. Remember - cython is "the way" that comes up for how to glue C/C++ into python these days, not just the program in cish python frontend for speed.
Stefan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Remember - cython is "the way" that comes up for how to glue C/C++ into python these days, not just the program in cish python frontend for speed.
There's some truth to the issues you raise, but there are a lot of nasty details behind the scenes. Do you have some more concrete details for what you're proposing? In particular, Cython mangles all of its variable and function names. How do you want to work around that? How should exception forwarding and throwing work? You're right that it's unrealistic to ever expect Cython to support a full set of C++ like features, but, at the same time, often C++ libraries use many of the fancier features to create higher level interfaces that, conceptually, have a much simpler interface than we give them credit for. In many cases, the metaprogramming, though a part of the "interface" in C++ really is more of an implementation detail and is hidden from Cython. I wouldn't expect Cython to ever properly implement things like SFINAE, expression SFINAE, or even perfect forwarding, but it often still makes sense for it to interact with interfaces that are written using these features. Given that not all features actually need to be implemented in Cython, the question becomes more about what is good enough to support most interfaces nicely. Regardless of exactly which C++ features need to be supported right now, here are a few ways you can do things like this right now. - You can define inline functions, macros, or templates in separate headers. Though this does introduce additional files, it routes things through fairly standard code paths and makes things pretty reasonable. This is what I prefer to do when need more C++ features that Cython provides. - Tell Cython how the interface you're using is used, not how it is constructed. C style variadic function calls are especially good for this since they essentially bypass all type checking on a given function's arguments. - You can currently exploit user-specified C names of functions and variables to inject expressions into compiled code. This is an awful hack that happens to combine supported features in a useful way. You can inject a wide variety of expressions into the generated C++ code, but it's painful enough that doing something else is almost always preferable. It totally kills readability. All that said, I really think that there are more useful and concrete things to work on at the moment. There's plenty of work to do to make Cython up to scratch for wrapping arbitrary (reasonably well designed) C++ interfaces. Inlining C++ code into Cython seems like a nice idea up front, but in practice it's less clear what that would mean and how exactly all the accompanying semantics would work. Thanks for raising a lot of these issues though. Interfacing between Python and modern C++ is definitely worth the discussion. Best, Ian Henriksen
On Sun, Aug 21, 2016 at 1:36 AM, Ian Henriksen < insertinterestingnamehere@gmail.com> wrote:
Remember - cython is "the way" that comes up for how to glue C/C++ into python these days, not just the program in cish python frontend for speed.
There's some truth to the issues you raise, but there are a lot of nasty details behind the scenes. Do you have some more concrete details for what you're proposing? In particular, Cython mangles all of its variable and function names. How do you want to work around that? How should exception forwarding and throwing work?
Recall I'm not one of the Cython devs :-) I think it is reasonable without knowing detail to the internals of Cython that if mangling must be done, there can be an opt-out specifier to the mangling done on cdef'd functions and variables. Variables are where the meat is at for me but there could be some benefit to functions as well if there's no reason they couldn't come along for the ride if they don't add significant complication. I think for exception forwarding and throwing, the responsibility would lay to the wrapped code to guarantee either no thrown exceptions (you see this in multithreaded software all the time) or they must code the exception support in themselves. However easier user-handled translations patterns apply to make this a bit easier - set a cdef'd flag and a message and check that flag after the eval'd portion - then decode and possibly raise an exception.
participants (8)
-
Alex S. -
Greg Ewing -
Ian Henriksen -
Jason Newton -
Kevin Thornton -
Robert Bradshaw -
Stefan Behnel -
William Stein