Mailman 3 November 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

cython --cplus --embed generates invalid code
by Michael Enßlin Dec. 1, 2014

Dec. 1, 2014

Hi, on my system, --embed does not work with --cplus. How to reproduce: Any valid pyx file works: $ rm -f test.pyx; touch test.pyx $ cython --embed --cplus test.pyx $ g++ -c test.cpp -I/usr/include/python3.4m test.cpp: In function ‘wchar_t* __Pyx_char2wchar(char*)’: test.cpp:945:41: error: invalid conversion from ‘void*’ to ‘wchar_t*’ [-fpermissive] res = malloc(argsize*sizeof(wchar_t)); $ clang++ -c test.cpp -I/usr/include/python3.4m test.cpp:945:9: error: assigning to 'wchar_t *' from incompatible type 'void *' res = malloc(argsize*sizeof(wchar_t)); The issue can easily be fixed by manually casting the malloc result to (wchar_t *). In C it is not recommended to cast the malloc result, while in C++ it is required (and malloc is discouraged). System info: $ cython --version Cython version 0.21.1 $ g++ --version g++ (Debian 4.9.2-2) 4.9.2 $ python3 --version Python 3.4.2 $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux unstable (sid) Release: unstable Codename: sid ~ Michael

2 1

Location where extern declarations are parsed
by Favian Contreras Nov. 15, 2014

Nov. 15, 2014

Hello all, Does anyone know where the (c++) external declarations are parsed? I am specifically looking for where external function pointers are parsed by the compiler. I noticed that all functions are parsed in some way in Symtab.py, but this includes all of the inherited methods from python and the c files within Cython, and I'm thinking there is a more specific place where external function pointers parsed. Cheers, Favian Contreras

2 1

Cython sometimes fails to build on Travis
by Ondrej Certik Nov. 14, 2014

Nov. 14, 2014

Hi, Cython sometimes fails to build on Travis for various projects. If it fails, it always ends with this error: https://gist.github.com/certik/08f16dd572170c17d956 Here are some instances of this failure in other projects: https://github.com/sympy/csympy/issues/308 https://github.com/scipy/scipy/pull/4126#issuecomment-62162812 Any ideas what is going on? Thanks, Ondrej

5 6

Re: [Cython] [cython-users] Re: New function (pointer) syntax.
by Ian Henriksen Nov. 11, 2014

Nov. 11, 2014

On Thursday, November 6, 2014 9:48:53 PM UTC-7, 1989lzhh wrote: > > > > > > 在 Nov 7, 2014，02:56，Robert Bradshaw <robe...(a)gmail.com <javascript:>> > 写道： > > > > [Cc'ing elsewhere for more feedback. Also top-posting for initial > > impressions before the discussion.] > > > > Here's some proposed function pointer syntaxes; which are the most > > obvious to understand/read/remember? Can you figure them out? > > > > cdef float (*F)(float) > > cdef float (*G)(float (*)(float), float, float) > > cdef float ((*H)(char*))(float (*)(float), float, float) > > > > vs > > > > cdef float -> float F > > cdef (float -> float, float, float) -> float G > > cdef (char*) -> (float -> float, float, float) -> float H > > > > vs > > > > cdef lambda float: float F > > cdef lambda (lambda float: float), float, float: float G > > cdef lambda (char*): lambda: (lambda float: float), float, float: > float H > > > > > > If you want a hint, the last is something that returns numerical > > integration algorithm given a string name. Yes, you could use > > typedefs, but you shouldn't have to. especially for the first. > > > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. > > Regards, > Liu zhenhai > Here are my thoughts, for what they're worth. I actually really like the arrow syntax. It matches the way that the domains and ranges of functions are often written mathematically. On the other hand, it does conflict with the syntax from C. The syntax from Numba is a really good way to do this as well. It is just as clear and doesn't conflict with C. Strictly speaking, it isn't Python, but it would be nice to have similarities in the syntax for the different packages. The C syntax is not intuitive. The only benefit there is the overlap with C. For something this simple, learning the new syntax is easy enough. It will also The lambda syntax strikes me as an intuitive solution, but it isn't particularly clean or concise. If the syntax is going to change, it would be good to change it to something nicer than that. I don't think this one would be worth making the change. All things considered, the Numba syntax looks like the best idea. The arrows are intuitive as well, but the overlap with the syntax from C could be a bad thing. Thanks for looking into this! -Ian Henriksen > > > >> On Thu, Nov 6, 2014 at 9:59 AM, Stefan Behnel <stef...(a)behnel.de > <javascript:>> wrote: > >> Robert Bradshaw schrieb am 06.11.2014 um 18:15: > >>>> On Thu, Nov 6, 2014 at 12:29 AM, Stefan Behnel wrote: > >>>> Robert Bradshaw schrieb am 06.11.2014 um 08:34: > >>>>> I'd like to propose a more pythonic way to declare function pointer > >>>>> types, namelye > >>>>> > >>>>> type0 (*[ident])(type1, type2, type3) > >>>>> > >>>>> would instead become > >>>>> > >>>>> (type1, type2, type3) -> type0 [ident] > >>>> > >>>> Not convinced. Looks quite magic, very different from the C that > people > >>>> would know, and takes a while reading from left to right until you > see the > >>>> "->" which is essentially the only indication of what is happening > here. > >>> > >>> I suppose it depends on whether one knows just C, is familiar with a > >>> wide variety of other (more functional) languages. I think it's a lot > >>> easier for people to read who only know Python, or just a smattering > >>> of C. > >> > >> The thing is, in order to use Cython, you currently only need to > understand > >> Python, some C/C++ and a minor bit of special Cython syntax. The more > new > >> syntax we add outside of Python/C/C++, the higher the entry bar is > raised > >> for both the "mostly Python" and "I know C" users. > > > > True. However, " -> return_annotation" is still Python syntax. > > > >>>> Also, "->" has a widely known meaning in C/C++ which makes the above > very > >>>> confusing. I think in C when thinking about function pointers, and > there is > >>>> no "def ..." keyword indication here that would switch me back to > Python > >>>> (as for signature annotations). > >>>> > >>>> I do see the problem of "where did I have to put that '*' again in > the C > >>>> declaration?", but this looks like it will produce more confusion > than it > >>>> avoids. > >>> > >>> Hmm, I hadn't thought of that meaning of ->. I don't think it would be > >>> as confusing in context, which will be immediately following cdef or > >>> in a function argument, e.g. one would write > >>> > >>> cdef (float -> float, float, float) -> float integrator = > >>> get_integrator(...) > >> > >> My brain can't even parse that. Does it join as > >> > >> (float -> (float, float, float)) > >> > >> or > >> > >> ((float -> float), float, float) > >> > >> ? > > > > The comma token remains lower precedence than about everything else. > > It's just like a parameter list > > > > cdef integrate(float -> float f, float a, float b): > > ... > > > > f is something that takes floats to floats. > > > >> A ctypedef would definitely help here. > >> > >> > >>> which doesn't to me look like member access, and is much clear than > >>> > >>> cdef float (*integrator)(float (*)(float), float, float) = > >>> get_integrator(...) > >> > >> Apart from the (*), at least this makes it clear what gets passed into > what. > >> > >> > >>> This becomes especially clear for return types, e.g. > >>> > >>> cdef ((float -> float, float, float) -> float) > >>> get_integrator(char* algorithm): > >>> if algorithm == ... > >>> > >>> vs > >>> > >>> cdef float (*get_integrator(char* algorithm))(float (*)(float), > >>> float, float): > >>> if algorithm == ... > >> > >> Same thing as above. The nesting is clearer in the latter. > > > > Clearly we think differently... I can pick apart the latter > > definition, but I have to admit it took a lot longer to type than the > > former, and I still see "cdef float blah(...., float, float): ..." > > when in fact I'm declaring something that takes a char* (or should > > that read, "takes something that dereferences to a char" :-)). > > > > I am curious, when you read "cdef int * p" do you parse this as "cdef > > (int*) p" or "cdef int (*p)" 'cause for me no matter how well I know > > it's the latter, I think the former (i.e. I think "I'm declaring a > > variable of type p that's of int pointer type.") > > > >>>> Many (or most?) function pointer type declarations will come from > some C > >>>> library's header file, so it means making it more difficult for users > to > >>>> copy over the declaration. > >>> > >>> I really hope, in the long run, people won't have to manually do these > >>> declarations. > >> > >> They'll definitely stay. At least as an advanced feature and for the > simple > >> (entry) cases, they provide both an easily understandable mapping and a > >> handy way to tune the declarations to make their usage friendlier in > Cython > >> code. > > > > I should have said they'll be less common, at least for the "copy > > header file" in which I'd rather point to the header file and list > > members to declare. For advance use (e.g. you want to mis-declare > > types) we'd still need this. > > > >>> As well as making things easier to read, part of the motivation is to > >>> come up with a syntax for declaring this type without having to add > >>> the notion of declarators (empty or not) to the grammar. Essentially > >>> everyone thinks "cdef type var" even though that's not currently the > >>> true grammar. > >> > >> True. But why not require at least a ctypedef for these things? Ad-hoc > >> declaration of C function types is just bound to get in the way of > readability. > > > > Not being able to declare this without a ctypedef would be a weakness > > in the language, especially as ctypedefs can't be declared locally. > > Also, it just defers the ugliness to elsewhere, e.g. > > > > ctypedef float (*FloatFunc)(float) > > ctypedef float (*IntegrateFunc)(FloatFunc, float, float) > > > > The reason ctypedefs help, and are so commonly used for with function > > pointers, is because the existing syntax is just so horrendously bad. > > If there's a clear way to declare a function taking a single float and > > returning a single float, no typedef needed. > > > >>> Another option that just occurred to me would be the syntax "lambda > >>> type1, type2, type3: type0" so for this example one would have > >>> > >>> cdef lambda (lambda float: float), float, float: float integrator > >>> = get_integrator(...) > >>> > >>> doesn't read as nice to me, but thought I'd throw it out there. > >> > >> It's funny that lambda uses the same expression separator as the > signature > >> annotations in Py3. Hadn't even noticed that before. > >> > >> Looks a bit funny at first sight, but definitely clearer than any of > the above. > >> > >> Stefan > >> > >> _______________________________________________ > >> cython-devel mailing list > >> cython...(a)python.org <javascript:> > >> https://mail.python.org/mailman/listinfo/cython-devel > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "cython-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to cython-users...(a)googlegroups.com <javascript:>. > > For more options, visit https://groups.google.com/d/optout. >

3 3

New function (pointer) syntax.
by Robert Bradshaw Nov. 11, 2014

Nov. 11, 2014

I'd like to propose a more pythonic way to declare function pointer types, namelye type0 (*[ident])(type1, type2, type3) would instead become (type1, type2, type3) -> type0 [ident] I have a pull request up at https://github.com/cython/cython/pull/333; what do people think? - Robert

6 17

Re: [Cython] [cython-users] Re: New function (pointer) syntax.
by Robert Bradshaw Nov. 10, 2014

Nov. 10, 2014

On Sun, Nov 9, 2014 at 1:29 PM, Nils Bruin <bruin.nils(a)gmail.com> wrote: > On Saturday, November 8, 2014 12:15:02 PM UTC-8, Sturla Molden wrote: >> >> Presumably Cython >> needs C and C++ function and method pointer syntax for wrapping C and C++ >> libraries, so why add another one? > > > I think that's a valid point. The reality is that one of cython's strength > is that interfacing with existing C/C++ libraries is extremely > straightforward. At some point this involves translating a ".h" into a > ".pxd" file. Having many C/C++ type designations be valid in cython is > therefore a generally desirable property (especially for the inscrutable > C/C++ ones!). Thus I don't think you'd want to *deprecate* type definitions > that are presently valid in C/C++ as well as in cython. Even better would be a tool that automatically translates an .h file into a .pxd file (if the .pxd file is needed at all), pulling out all or specifically enumerated declarations. They'd only be edited by hand by experts. One could support something like from "signal.h" cimport signal rather than re-declare it. > The bar for *adding* > syntax for things that can already be expressed should be really quite high. > I'm not sure the proposal meets that bar for me. The proposal would > definitely benefit from a motivation that goes beyond "C/C++ is ugly and > hard to understand", because the reality is that cython won't be able to get > away from that entirely anyway. If you're wrapping C libraries, yes, but if you want to pass around a cdef function as a first class object as easily as you pass a standard Python callable, for people who don't already know C well. That is the use case I'm targeting, and I think the C function pointer syntax is particularly poorly suited for that. Again, I agree the "wrapping C libraries" case as a very important one we want to support, but disagree that a better syntax would be an undue burden (especially if most cases were automated away). This came up recently because I have also been toying with a formal grammar for Cython: https://github.com/robertwb/cython/blob/grammar/Cython/Parser/Grammar . Adding declarators here greatly complicates things (though feel free to try and do it cleanly). - Robert

1 0

Compiler crash in cython 0.21.1 DebugTransform
by Rich Leigh Nov. 10, 2014

Nov. 10, 2014

Hi all - I get the following compiler crash from cython 0.21.1 (from pip) when compiling a C++ module with debugging. It's fine without the --gdb option. [1/3] Compiling Cython CXX source for [project]... FAILED: cd [project_build] && cython --cplus -I [project_src] -2 --gdb --output-file [project_build]/[project].cxx [project_src]/[project].pyx Error compiling Cython file: ------------------------------------------------------------ ... def merge(self): cdef cpp_string ofnametmp = self.ofname + ".tmp" merge[cpp_vector[cpp_string]](self.sources, ofnametmp.c_str()) ^ ------------------------------------------------------------ [project_src]/[project].pyx:241:28: Compiler crash in DebugTransform ModuleNode.body = StatListNode([project].pyx:1:0) StatListNode.stats[12] = StatListNode([project].pyx:227:5) StatListNode.stats[0] = CompilerDirectivesNode([project].pyx:227:5) CompilerDirectivesNode.body = StatListNode([project].pyx:227:5) StatListNode.stats[0] = CClassDefNode([project].pyx:227:5, as_name = u'Merge', class_name = u'Merge', decorators = [...]/0, module_name = u'', visibility = u'private') CClassDefNode.body = StatListNode([project].pyx:228:4) StatListNode.stats[1] = DefNode([project].pyx:238:4, modifiers = [...]/0, module_name = u'[Project]', name = u'merge', num_required_args = 1, py_wrapper_required = True, qualname = u'Merge.merge', reqd_kw_flags_cname = '0', used = True) DefNode.body = StatListNode([project].pyx:239:8) StatListNode.stats[0] = ExprStatNode([project].pyx:240:41) ExprStatNode.expr = SimpleCallNode([project].pyx:240:41, analysed = True, is_temp = 1, use_managed_ref = True) SimpleCallNode.function = IndexNode([project].pyx:240:17, is_called = 1, is_subscript = True, result_is_used = True, type_indices = [...]/1, use_managed_ref = True) IndexNode.index = IndexNode([project].pyx:240:28, is_subscript = True, result_is_used = True, use_managed_ref = True) IndexNode.base = NameNode([project].pyx:240:28, cf_maybe_null = True, is_name = True, name = u'cpp_vector', result_is_used = True, use_managed_ref = True) Compiler crash traceback from this point on: File "Cython/Compiler/Visitor.py", line 173, in Cython.Compiler.Visitor.TreeVisitor._visit (cython/Cython/Compiler/Visitor.c:4449) return handler_method(obj) File "python2.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2937, in visit_NameNode node.type.is_cfunction and AttributeError: 'NoneType' object has no attribute 'is_cfunction' In my pyx file, cpp_vector and cpp_string are as follows: from libcpp.vector cimport vector as cpp_vector from libcpp.string cimport string as cpp_string and merge (my C++ function, though I've also got the python def above of a method with the same name) has this signature: cdef void merge[T](const T&, const char *) except+ I've managed to fix it by turning line 2937 in ParseTreeTransforms.py from 2937: node.type.is_cfunction and to 2937: getattr(node.type, 'is_cfunction', False) and which seems to do the trick. I did a quick google to see if anybody else had this issue, but it seems like a fresh bug. Not sure if this is the correct solution though, or whether I've simply buried an underlying issue.

1 0

Re: [Cython] New function (pointer) syntax.
by Stefan Behnel Nov. 7, 2014

Nov. 7, 2014

1989lzhh schrieb am 07.11.2014 um 05:48: >> 在 Nov 7, 2014，02:56，Robert Bradshaw 写道： >> Here's some proposed function pointer syntaxes; which are the most >> obvious to understand/read/remember? Can you figure them out? >> >> cdef float (*F)(float) >> cdef float (*G)(float (*)(float), float, float) >> cdef float ((*H)(char*))(float (*)(float), float, float) >> >> vs >> >> cdef float -> float F >> cdef (float -> float, float, float) -> float G >> cdef (char*) -> (float -> float, float, float) -> float H >> >> vs >> >> cdef lambda float: float F >> cdef lambda (lambda float: float), float, float: float G >> cdef lambda (char*): lambda: (lambda float: float), float, float: float H >> >> >> If you want a hint, the last is something that returns numerical >> integration algorithm given a string name. Yes, you could use >> typedefs, but you shouldn't have to. especially for the first. >> > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. I find it clearer than any of the above. And it still allows the use of named arguments, i.e. you could say cdef char*(float x, int y) F = ... F(y=1, x=2.0) Drawback is that you have to parse up to the name in order to distinguish it from cdef char*(float x, int y): ... Then there's also cdef char*(float x, int y) nogil F And we have to disallow cdef char*(float x, int y) with gil F It gets a bit less readable when you take a proper lower-case variable name, e.g. cdef char*(float x, int y) nogil get_gil We could fix some of that by allowing cdef nogil: cdef char*(float x, int y) get_gil But then, that's adding yet another special case. Stefan

1 1

Re: [Cython] [sage-devel] Re: New function (pointer) syntax.
by Robert Bradshaw Nov. 7, 2014

Nov. 7, 2014

Thanks for all the feedback! On Thu, Nov 6, 2014 at 7:13 PM, Travis Scrimshaw <tscrim(a)ucdavis.edu> wrote: > Here's my 2 cents. >> >> >> cdef float (*F)(float) >> cdef float (*G)(float (*)(float), float, float) >> cdef float ((*H)(char*))(float (*)(float), float, float) > > > I prefer this one because it mimics C function pointers (at least how I > remember, I'm somewhat rusty). It is C function pointer syntax... if you consider that a plus (aside from compatibility). > When I saw the "->", I thought of pointer > redirection (without reading the rest of the post which mentioned it) and > believe it could be a major source of confusion. I'm surprised so many people are noting this; good feedback. > However since cython is > suppose to mimic python as much as possible (please correct me if I'm wrong > here), perhaps the lambda one is better? Yeah, it's a bit odd, but less confusing. A goal is a syntax such that when you look at it you know what it means, unlike, say, the C syntax that you have to learn how to decode. When I first saw it in Miranda ages ago, but clearly others think of C first when they think of something of type "float -> float." On Thu, Nov 6, 2014 at 8:06 PM, Peter Schay <pschay(a)gmail.com> wrote: > > The C function pointer syntax is clever and makes a lot of sense but > reading/writing a complicated declaration usually feels like I'm solving a > puzzle. Exactly what I want to avoid. Certainly violates several points of the Zen of Python. > IANACG (I am not a compiler guy) so I am sure there must be some flaw in the > following but I can't help asking whether there would be anything wrong with > this form, which reads more like a sentence ("a pointer to a function F with > args x which returns y"): > > cdef lambda F(float): float > cdef lambda G( > (lambda (float): float), > float, > float > ): float > cdef lambda H(char *): (lambda (float, float, float): float) In Python "lambda (a, b): ..." with the parentheses means a function taking one argument that's unpacked into two variables. > One could also use an underscore for the unnamed lambdas, maybe. One of the main goals is to move away from C style declarators, i.e. actually have "cdef [type] [name]" instead of "cdef [basetype] [mix_of_names_and_type_info]. So one would write cdef (float -> float) f, g, h to declare three variables of type float -> float. Then we might have hope of an actual, comprehensible grammar for Cython. (Few folks think in terms of declarators, so one would write "int[10] x" to declare an array of size 10 rather than "int x[10].") On Thu, Nov 6, 2014 at 8:46 PM, Nathann Cohen <nathann.cohen(a)gmail.com> wrote: > Hello ! > > To me the second looks the most clear. Except that I would have written > > cdef (char*) -> ((float -> float, float, float) -> float H) instead of > > cdef (char*) -> (float -> float, float, float) -> float H > > But well, one gets used to things like that. Yeah, cdef ( (char*) -> ((float -> float, float, float) -> float) ) H would be just fine as well for those who like parentheses. > It is clearer but indeed is it surprising to find such a syntax in Cython... > Actually like Travis I would expect Cython to support the first, even though > it is the least obvious... It does now. > If both are implemented I will definitele use the second whenever I can :-) Me too. On Thu, Nov 6, 2014 at 8:48 PM, 1989lzhh <1989lzhh(a)gmail.com> wrote: > > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. That's an interesting alternative. Would float(float)* be a pointer to a function pointer then? (float*)(float) something that returned a pointer to a float? - Robert

1 0