Mailman 3 November 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Cython 0.17.2 released
by Stefan Behnel Jan. 8, 2013

Jan. 8, 2013

Hi everyone, I'm happy to announce the release of Cython 0.17.2. This is a (mostly) bug fix release for the stable 0.17 release series. http://pypi.python.org/pypi/Cython/0.17.2 Direct downloads: http://cython.org/release/Cython-0.17.2.tar.gz http://cython.org/release/Cython-0.17.2.zip Github version: https://github.com/cython/cython/commit/10183d61eb5cf33e6912dec2ab09740498f… Have fun, Stefan Complete changelog: 0.17.2 (2012-11-20) =================== Features added -------------- * ``cythonize()`` gained a best effort compile mode that can be used to simply ignore .py files that fail to compile. Bugs fixed ---------- * Replacing an object reference with the value of one of its cdef attributes could generate incorrect C code that accessed the object after deleting its last reference. * C-to-Python type coercions during cascaded comparisons could generate invalid C code, specifically when using the 'in' operator. * "obj[1,]" passed a single integer into the item getter instead of a tuple. * Cyclic imports at module init time did not work in Py3. * The names of C++ destructors for template classes were built incorrectly. * In pure mode, type casts in Cython syntax and the C ampersand operator are now rejected. Use the pure mode replacements instead. * In pure mode, C type names and the sizeof() function are no longer recognised as such and can be used as normal Python names. * The extended C level support for the CPython array type was declared too late to be used by user defined classes. * C++ class nesting was broken. * Better checking for required nullary constructors for stack-allocated C++ instances. * Remove module docstring in no-docstring mode. * Fix specialization for varargs function signatures. * Fix several compiler crashes.

3 6

memoryview of extension types: warnings with gcc, errors with g++
by Matěj Laitl Nov. 30, 2012

Nov. 30, 2012

Hi list and Mark, it seems that C code with questionable casts is generated when using memory views of extension types. I get following warnings from gcc: extension_type_memoryview.c: In function ‘__pyx_pf_25extension_type_memoryview_test_getitem’: extension_type_memoryview.c:1468:15: warning: assignment from incompatible pointer type extension_type_memoryview.c: In function ‘__pyx_pf_25extension_type_memoryview_2test_getitem_typed’: extension_type_memoryview.c:1565:15: warning: assignment from incompatible pointer type extension_type_memoryview.c:1568:18: warning: assignment from incompatible pointer type And following errors if compiling in C++ mode with g++: extension_type_memoryview.c: In function ‘PyObject* __pyx_pf_25extension_type_memoryview_test_getitem(PyObject*)’: extension_type_memoryview.c:1468:213: error: cannot convert ‘__pyx_obj_25extension_type_memoryview_ExtensionType*’ to ‘PyObject*’ in assignment extension_type_memoryview.c: In function ‘PyObject* __pyx_pf_25extension_type_memoryview_2test_getitem_typed(PyObject*)’: extension_type_memoryview.c:1565:213: error: cannot convert ‘__pyx_obj_25extension_type_memoryview_ExtensionType*’ to ‘PyObject*’ in assignment extension_type_memoryview.c:1568:20: error: cannot convert ‘PyObject*’ to ‘__pyx_obj_25extension_type_memoryview_ExtensionType*’ in assignment I get exactly the same error when --cplus is passed to Cython and extension_type_memoryview.cpp is generated/compiled. ...which currently prevents my project combining C++ code with extension types and memory views to compile. :-( Test-case is attached. Cython 0.17.2. Regards, Matěj

2 1

bug report: compiler crash when assigning from function call returning ndarray
by Lars Buitinck Nov. 19, 2012

Nov. 19, 2012

Hi all, I just got a compiler crash (Cython 0.17, 0.17.1 and current master) when trying to compile a call to a function returning an np.ndarray: cimport numpy as np cdef np.ndarray[np.float64_t, ndim=2] trivial(X): return X def test(): # Uncommenting the following line, or removing the assignment, # prevents the crash: #cdef np.ndarray[np.float64_t, ndim=2] Z Z = trivial(np.zeros(4)) I'm aware that the program contains an error and it's not much of a hindrance to the project I'm working on, but the error message just isn't very friendly: Compiler crash traceback from this point on: File "/home/lars/src/cython/Cython/Compiler/ExprNodes.py", line 1528, in analyse_target_types Buffer.used_buffer_aux_vars(self.entry) File "/home/lars/src/cython/Cython/Compiler/Buffer.py", line 280, in used_buffer_aux_vars buffer_aux.buflocal_nd_var.used = True AttributeError: 'NoneType' object has no attribute 'buflocal_nd_var' More details at https://gist.github.com/4114677. I looked at Buffer.py and ExprNodes.py, but I couldn't figure out what exactly was going on. Could someone please look into this? TIA, -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam

1 0

Re: [Cython] [cython-users] Test case that will segfault
by Stefan Behnel Nov. 17, 2012

Nov. 17, 2012

Hi, thanks for the test case. Ewald de Wit, 16.11.2012 23:09: > I have attached some code below that will lead to a segmentation fault on > Linux. The code iterates over a linked list of nodes and removes a node > while doing so. Adding an extra ref to the removed node will prevent the > crash, so the problem might be a refcount going to zero before its time. > This is with Cython 0.17.1 - not sure if bug or known limitation. > ------------------------------------------------------------------------------------------------------------------------- > cdef class Node: > > cdef public int value > cdef public Node nextNode > > def __init__(self, value): > self.value = value > self.nextNode = None > > > cdef class Chain: > > cdef public Node firstNode > > def __init__(self): > self.firstNode = None Note that Python object attributes are automatically initialised to None, so the above line is redundant. > def __iter__(self): > n = self.firstNode > while n is not None: > yield n > n = n.nextNode > > def add(self, Node node): > if self.firstNode is None: > self.firstNode = node > else: > for n in self: pass > n.nextNode = node > > def remove(self, Node node): > if node is self.firstNode: > self.firstNode = node.nextNode > else: > for n in self: > if node is n.nextNode: > n.nextNode = node.nextNode > break > > def dump(self): > for n in self: > print n.value > > > def test(): > c = Chain() > c.add(Node(1)) > c.add(Node(2)) > c.add(Node(3)) > c.add(Node(4)) > n = c.firstNode > while n is not None: > if n.value == 3: > c.remove(n) > #nn = n # Adding extra ref will prevent crash > n = n.nextNode # Crash will happen here at value 3 The C code generated for the last line is as follows: __Pyx_INCREF(((PyObject *)__pyx_v_n->nextNode)); __Pyx_DECREF(((PyObject *)__pyx_v_n)); __pyx_v_n = __pyx_v_n->nextNode; That's definitely unhealthy code. For a test case, the following would be enough: """ cdef class Node: cdef public Node next def test(): cdef Node n = Node() n.next = Node() n = n.next test() """ However, this doesn't crash for me, likely because the DECREF() above doesn't free the memory, just release it to CPython's memory allocator. So we're just lucky here. Stefan

1 1

incorrect codegen for nogil functions calling nogil except+ functions
by Gerald Dalley Nov. 16, 2012

Nov. 16, 2012

When a nogil function calls a nogil except+ function, Cython generates C code that requires the Py_BEGIN_ALLOW_THREADS macro to be evaluated, but it doesn't inject the code to evaluate the Py_BEING_ALLOW_THREADS macro. Consider the following C header file (nogil_except.h) inline int foo() { return 1; } and a corresponding Cython source file (nogil_except.pyx) cdef extern from "nogil_except.h": cdef int foo() nogil except+ cdef int bar() nogil: return foo() print bar() Cython's ExprNodes.py generates code that calls the Py_BLOCKS_THREADS macro, but it doesn't ensure that Py_BEGIN_ALLOW_THREADS has been injected into an enclosing scope. The latter macro defines a variable, "_save" that's needed by the Py_BLOCKS_THREADS macro: $ cython nogil_except.pyx && /usr/bin/g++ ... -c nogil_except.cpp nogil_except.c: In function ‘int __pyx_f_12nogil_except_bar()’: nogil_except.c:537: error: ‘_save’ was not declared in this scope Here's the applicable block of generated code: $ awk -F, '{if (($2=="") && (NR >= 531) && (NR < 540)) printf "%-4d %s\n", NR,$0}' nogil_except.c 531 /* "nogil_except.pyx":4 532 * cdef int foo() nogil except+ 533 * cdef int bar() nogil: 534 * return foo() # <<<<<<<<<<<<<< 535 * print bar() 536 */ 537 try {__pyx_t_1 = foo();} catch(...) { Py_BLOCK_THREADS; __Pyx_CppExn2PyErr(); Py_UNBLOCK_THREADS; {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1_error;}} 538 __pyx_r = __pyx_t_1; 539 goto __pyx_L0; It seems like either ExprNodes.py should avoid injecting Py_BLOCK_THREADS and Py_UNBLOCK_THREADS when operating outside an OpenMP context and/or something should guarantee that Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS get injected as needed into the code stream. I've tested Cython 0.17b2 and 0.17b3 and both exhibit this problem. -- --Gerald Dalley dalleyg(a)ieee.org

1 0

Re: [Cython] [cython-users] Re: Release candidate 0.17.2rc1
by Stefan Behnel Nov. 15, 2012

Nov. 15, 2012

Jasmine Sandhu, 15.11.2012 19:36: > Thanks for the heads up Stefan. > We are developing on the Mac and Windows and using Cython to wrap C++ code. > I tested the new Cython on windows against our code base. The setup and > install was easy. > > I build our application with the new Cython on Windows. It builds fine and > passes the unit tests. > No problems or issues thus far. Thanks for testing, especially on so many platforms! Stefan

1 0

Release candidate 0.17.2rc1
by Stefan Behnel Nov. 14, 2012

Nov. 14, 2012

Hi, here is a release candidate for 0.17.2. http://cython.org/release/Cython-0.17.2rc1.tar.gz Please give it a try. If there are no (serious) regressions by the end of the week, this will be our next release. Stefan

1 0

Any more fixes for 0.17.2?
by Stefan Behnel Nov. 12, 2012

Nov. 12, 2012

Hi, the 0.17 branch has gained a couple of fixes, so I'd like to release a 0.17.2 some time next week. In case you have other things to add to it within the next few days, please do, or send a quick reply if you'd like to have more time. Stefan

3 14

redefining PyIndex_Check
by Stefan Behnel Nov. 11, 2012

Nov. 11, 2012

Hi, instead of overriding an existing definition of "PyIndex_Check" like this: """ +#if PY_VERSION_HEX < 0x02050000 + /* NumPy headers define PyIndex_Check incorrectly */ + #undef PyIndex_Check + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && !PyComplex_Check(o)) +#endif """ which may or may not have come from NumPy, shouldn't we be using our own definition of a "__Pyx_PyIndex_Check" in our code instead? Stefan

2 4