Mailman 3 August 2014 - cython-devel

Towards a formal Cython grammar
by Robert Bradshaw Aug. 24, 2014

Aug. 24, 2014

I've started playing around with writing a grammar for Cython. As well as formally defining the language (in particular, with respect to Python) this should allow us to eventually move to using parser-generators rather than ad-hoc hand written code and be useful to external tools (e.g. IDEs, linters, syntax highlighting, etc.) I've posted what I have at https://github.com/robertwb/cython/tree/grammar , in particular https://github.com/robertwb/cython/blob/grammar/Cython/Parser/Grammar https://github.com/robertwb/cython/compare/robertwb:3aa9056943f83a68cc9d933… Nothing beyond generating the raw parse tree is hooked up yet, and even that requires an extra directive (formal_grammar=True). Building and using the parser requires a source-built Python (it looks for the pgen artifact Python uses to compile the grammar). We may or may not want to stick with this approach long term (though if we do we might ship the generated .c files). This parse tree is still pretty low-level, we might want to also create something like Python.asdl to give us a (closer) AST. The grammar isn't complete, but should cover a most of the language (over 3/4 of the test suite passes, and that explores a lot of the corner cases). The most notable omissions are that it's using Python's lexer, so doesn't have a token for '?' or the additional literal string prefixes/int literal suffixes. Also, as the lexer clearly doesn't understand includes, these are handled by inlining in a preparsing step (which messes with line numbers). The grammar could be tightened up as well. For example, this grammar doesn't distinguish between valid pxd vs. pyx constructs, and allows cdef statements within if statements (or even normal class declarations). I tried to restrict the existing grammar as little as possible, in particular the only new illegal identifiers are 'cdef', 'ctypedef' and (for ambiguity reasons new' and 'sizeof'). Also, the "cython" keywords may not be used for identifiers that might be typed (e.g. "def foo(int): ..." is not allowed). The most significant departure from the existing "grammar" is that rather than using C-style declarators, cdef declarations are of the form "cdef [type] name." Thus one write the (already legal) cdef double[3] loc and cdef double* a, b declares two pointers. How to handle function pointers is still up in the air, but I wouldn't be opposed to moving to a new syntax (e.g. "(double, void*) -> double" inspired by Py3) for those. It disallows empty "declarators" for parameters of function declarations (though we could consider adding this back). i think this would also be a nice chance to simplify the grammar, so there are some intentional ommisions. Most notably, there are several modifiers (e.g the __cdecl, __stdcall, __fastcall callspecs, maybe inline, maybe even "with nogil", and the "cdef class Foo [object object_struct_name, type type_object_name ]" spec for external classes) that would make more sense as decorators. This would be backwards incompatible, but these are not commonly used features and fair warning (or even translators, I included a sed script to deal with the common case of declarators mentioned above) could be given and I think could be worth the simplification. Thoughts? - Robert

1 0

build failure on windows with 0.21b1 windows py27 x64
by Ian Bell Aug. 21, 2014

Aug. 21, 2014

A snippet of my failure: lex\Scanners.c(6876) : error C2121: '#' : invalid character : possibly the result of a macro expansion lex\Scanners.c(6876) : error C2146: syntax error : missing ')' before identifier 'ifdef' lex\Scanners.c(6876) : error C2121: '#' : invalid character : possibly the result of a macro expansion lex\Scanners.c(6878) : error C2059: syntax error : 'else' lex\Scanners.c(6888) : error C2059: syntax error : '}' lex\Scanners.c(6902) : error C2121: '#' : invalid character : possibly the result of a macro expansion lex\Scanners.c(6902) : error C2121: '#' : invalid character : possibly the result of a macro expansion soft Visual Studio 9.0\VC\BIN\amd64\cl.exe"' failed with exit status 2 Anyone run into similar problems?

1 0

bug: constructor declarations not working
by Marcus Brinkmann Aug. 17, 2014

Aug. 17, 2014

Hi, I want to declare and define C++ classes from Cython. Sure, I could write them in native C++, but it's just small glue code, and I prefer to let Cython handle the GIL and all the other good stuff. First, for reference, the following works (c.f. tests/run/cpp_classes_def.pyx): == cython --cplus example1.pyx ============ cdef cppclass Foo "Foo": int _foo __init__(int foo) nogil: this._foo = foo __dealloc__() nogil: pass =========================================== This will emit a declaration for Foo and the implementations of the constructor and destructor (output is truncated to show relevant parts only): struct Foo { int _foo; Foo(int); virtual ~Foo(void); }; Foo::Foo(int __pyx_v_foo) { this->_foo = __pyx_v_foo; } Foo::~Foo(void) { } Now, I want to export the class to other cython files. The following does not work, and I think it's a bug: == example2.pxd =========================== cdef extern cppclass Foo: int _foo __init__(int foo) __dealloc__() =========================================== == cython --cplus example2.pyx ============ cdef cppclass Foo: __init__(int foo) nogil: this._foo = foo __dealloc__() nogil: pass =========================================== == cython --cplus example3.pyx ============ from example2 cimport Foo cdef Foo* foo_p = new Foo(2) =========================================== This fails for example2 with an obscure error message: $ cython --cplus example2.pyx example2.pyx:3:8: undeclared name not builtin: this ...more spurious errors... For example3, Cython runs through but trying to compile shows the actual problem (which you can also see in example2 if you shift things around a bit): $ g++ -fPIC -shared -o example3.so -I /usr/include/python2.7 example3.cpp example3.cpp: In function ‘void initexample3()’: example3.cpp:775:38: error: no matching function for call to ‘Foo::Foo(int)’ __pyx_v_8example3_foo_p = new Foo(2); This is in the generated code: /*--- Type declarations ---*/ struct Foo; struct Foo { int _foo; virtual ~Foo(void); }; It's missing the constructor declaration! I traced this a bit down the Compiler/Symtab.py and found this part in CppClassScope::declare_var: 4e07fc52 (Robert Bradshaw 2012-08-21 00:46:00 -0700 2112) if name != "this" and (defining or name != "<init>"): 4e07fc52 (Robert Bradshaw 2012-08-21 00:46:00 -0700 2113) self.var_entries.append(entry) Here defining is 0, so the declaration is skipped. I don't know Cython internals, so I am hoping someone who does can fix this easily given the above information or point me in the right direction. Thanks a lot for Cython, it's very useful! Marcus Brinkmann

2 2

python-ideas discussion on "multiple dispatch", also operator overloading
by Stefan Behnel Aug. 16, 2014

Aug. 16, 2014

Hi, there's a discussion going on on python-ideas about generic functions and multiple dispatch. Might be interesting for the fused functions implementation in Cython. http://thread.gmane.org/gmane.comp.python.ideas/28790/focus=28848 Cython can definitely improve its own dispatch implementation. The reply I linked to is by Nick Coghlan, who also makes an interesting comment on operator overloading methods and NotImplemented: """ A *lot* of folks make the mistake of raising TypeError or NotImplementedError directly in their operator overload implementations, rather than returning the NotImplemented singleton that tells the interpreter to try the other type. Even some of the CPython builtins get that wrong, since the sq_concat and sq_repeat slots in C don't properly support the type coercion dance, so you *have* to raise the exception yourself if you're only implementing those without implementing nb_add and nb_mul (types defined in Python automatically populate both sets of C level slots if you define __add__ or __mul__). """ I think Cython could help its users here to a certain extent, if only by warning about raising TypeError explicitly in these methods and encouraging better behaviour. Stefan

1 0

Guido on type annotations in Py3
by Stefan Behnel Aug. 14, 2014

Aug. 14, 2014

He's picked up the idea from a recent keynote talk at EuroPython that there's this thing called mypy that provides type annotations. http://thread.gmane.org/gmane.comp.python.ideas/28619 Stefan

1 0

[Re] OpenMP thread private variable not recognized (bug report + discussion)
by Leon Bottou Aug. 13, 2014

Aug. 13, 2014

> > I am making heavy uses of OpenBlas which also uses OpenMP. > > Using the same queue manager prevents lots of CPU provisioning problem. > > Using multiple queue managers in the same code does not work as well > > because they are not aware of what the other one is doing. > > Normally OpenBLAS is built without OpenMP. Also, OpenMP is not fork safe > (cf. multiprocessing) but OpenBLAS' own threadpool is. So it is recommended > to build OpenBLAS without OpenMP dependency. > > That is: If you build OpenBLAS with OpenMP, numpy.dot will hang if used > together with multiprocessing. I am effectively using a version of openblas built with openmp because Debian used to compile openblas this way. They seem to have reverted now. Note than I cannot use python multiprocessing because my threads work on a very large state vector. My current solution is to use python threading and nogil cython compiled routines but this sometimes lead to weird effects provisioning threads. This is why I wanted to try the pure openmp solution and found the aforementioned bug in cython.parallel. Is there somebody actively trying to make cython.parallel work correctly? - If yes, then my bug report should be of interest to this person. - If no, then one should avoid (and possibly deprecate) cython.parallel and find other ways to do things. Thanks to the replies. - L.

1 0

[Re] OpenMP thread private variable not recognized (bug report + discussion)
by Leon Bottou Aug. 13, 2014

Aug. 13, 2014

On Tue, 12 Aug 2014 14:26:31, Sturla Molden wrote: > Cython does not do an error here:[... > - i is recognized as private > - r is recognized as reduction > - w is (correctly) recognized as shared Not according to the documentation. http://docs.cython.org/src/userguide/parallelism.html documentation for cython.parallel.parallel says "A contained prange will be a worksharing loop that is not parallel, so any variable assigned to in the parallel section is also private to the prange. Variables that are private in the parallel block are unavailable after the parallel block.". Variable w is such a variable. Furthermore, if cython is correct, why does GCC report an error on the cython generated C code? My point here is that there is a bug because (a) cython does not behave as documented, and (b) it generates invalid C code despite not reporting an error. > Personally I prefer to avoid OpenMP and just use Python threads and an > internal function (closure) or an internal class. If you start to use OpenMP, > Apple's libdispatch ("GCD"), Intel TBB, or Intel clikplus, you will soon discover > that they are all variations over the same theme: a thread pool and a closure. I am making heavy uses of OpenBlas which also uses OpenMP. Using the same queue manager prevents lots of CPU provisioning problem. Using multiple queue managers in the same code does not work as well because they are not aware of what the other one is doing. - L.

3 3

OpenMP thread private variable not recognized (bug report + discussion)
by Leon Bottou Aug. 12, 2014

Aug. 12, 2014

The attached cython program uses an extension class to represent a unit of work. The with parallel block temporarily gets the gil to allocate the object, then release the gil and performs the task with a for i in prange(...) statement. My expectation was to have w recognized as a thread private variable. with nogil, parallel(): with gil: w = Worker(n) # should be thread private with nogil: for i in prange(0,m): r += w.run() # should be reduction w = None # is this needed? Cythonize (0.20.2) works without error but produces an incorrect C file. hello.c: In function ‘__pyx_pf_5hello_run’: hello.c:2193:42: error: expected identifier before ‘)’ token hello.c: At top level: The erroneous line is: #pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5, __pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2) private(__pyx_filename, __pyx_lineno, __pyx_clineno) shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value, __pyx_parallel_exc_tb) where you can see that the first private() clause has no argument. The variable __pyx_v_w is not declared as private either as I would expect. I believe that the problem comes from line 7720 in Cython/Compiler/Node.py if self.privates: privates = [e.cname for e in self.privates if not e.type.is_pyobject] code.put('private(%s)' % ', '.join(privates)) And I further believe that the clause "if not e.type.is_pyobject" has been added because nothing would decrements the reference count of the thread private worker object when leaving the parallel block. My quick fix would be to remove this clause and make sure that my program contains the line "w = None" before leaving the thread. But I realize that this is not sufficient for you. Note that the temporary python objects generated by the call to the Worker construction are correctly recognized as thread private and their reference count is correctly decremented when they are no longer needed. The problem here is the clash between the python scoping rules and the semantics of thread private variables. This is one of these cases where I would have liked to be able to write with nogil, parallel(): with gil: cdef w = Worker(n) # block-scoped cdef with nogil: for i in prange(0,m): r += w.run() with an understanding that the scope of the cdef variable is limited to the block where the cdef appears. But when you try this, cython tells you that cdefs are not legal there.

3 4

Re: [Cython] aritmetic with arrays in Cython
by Ian Henriksen Aug. 6, 2014

Aug. 6, 2014

Message: 3 Date: Tue, 05 Aug 2014 21:38:38 +0200 From: Julian Taylor <jtaylor.debian(a)googlemail.com> To: Core developer mailing list of the Cython compiler <cython-devel(a)python.org> Subject: Re: [Cython] aritmetic with arrays in Cython Message-ID: <53E132BE.3070207(a)googlemail.com> Content-Type: text/plain; charset="windows-1252" >On 04.08.2014 21:11, Ian Henriksen wrote: >> Hi all, >> I noticed that some time ago there was a pull request >> (https://github.com/cython/cython/pull/144) open that was trying to >> implement basic arithmetic operations with arrays. This seems to have >> also been proposed in CEP 518 >> (https://github.com/cython/cython/wiki/enhancements-simd). What is the >> status on this? >> >> This is a feature I'd really like to see in Cython. I'd be happy to help >> with it, though I don't currently know enough about the internals of >> Cython to do it without some guidance. What would be the best way to do >> this? I've had three ideas thus far that might allow something like this >> to work, but I'd really like to get some second opinions before I get >> too far with any of them. >> >> 1. Make these array operations available as a part of Cython's memory >> view objects. >> >> 2. Write a Cython wrapper around a NumPy-like C++ >> library. https://github.com/ContinuumIO/libdynd seems to be a good >> candidate, though I'm not as familiar with it. (This doesn't strike me >> as something that would be a part of the main Cython project, but it >> could be a good way to make this sort of functionality available. It >> seems like a good way to avoid a lot of duplicate work though.) >> >> 3. Write a Fortran backend for Cython that uses Fortran's arrays >> wherever memory views are used. >> (This could be a bit crazy, but with the iso_c_binding module, it might >> be possible. It could also allow better optimizations for arrays to be >> applied by the compiler.) >> >> What would be the best way to go about this? >> > >4. Use numpy itself. numpy since 1.8 has vectorized base math functions, >though for large arrays you have to apply manual blocking and inplace >operations to see significant gains. That is where cython can help. >Possibly numpy is also open to expose the raw data buffer functions in >its api to make blocking more efficient. Thank you for pointing that out. Using the machinery that is already built in to numpy would be ideal. I hadn't realized that was a part of numpy's API. If I understand correctly, you're referring to https://github.com/numpy/numpy/blob/master/doc/neps/generalized-ufuncs.rst. Is that the right portion of the API? I'll try and figure out how to use it. Any working examples would be greatly appreciated. Other suggestions are also still welcome Thanks! -Ian Henriksen

1 0

Test/example for cpdef enum
by Ian Bell Aug. 5, 2014

Aug. 5, 2014

Are there any tests/docs/examples showing how to use the cpdef enum? This is a feature I have been looking for for a long time and I am hoping to use it very soon in my own project. I have a windows box that I can run the alpha test suite on with the full complement of python versions. Let me know if there is interest and I can run the tests. Alternatively, if you are still doing a buildbot (the link at https://github.com/cython/cython/wiki/HackerGuide#buildbot is broken) , we could configure a nightly slave to run the tests.

3 10