Mailman 3 November 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

"embedsignature" and "autotestdict" features in Py3.4
by Stefan Behnel Jan. 30, 2014

Jan. 30, 2014

Hi, recent changes in Py3.4 affect the functionality of the two directives "embedsignature" and "autotestdict". The so-called "argument clinic" extracts any embedded signatures from docstrings and moves them into a new property "__text_signature__". This is done at runtime, i.e. when either "__doc__" or "__text_signature__" are requested. http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182 I personally consider this a feature (and it works nicely with the signatures that Cython embeds), but you may or may not agree. It broke some of our own doctests, at least, because the "__doc__" value that we tested for was no longer the same. Regarding the "autotestdict", CPython also got smarter and now finds doctests embedded in C implemented functions all by itself. This is clearly an improvement compared to the previous behaviour. However, this also means that it now executes both the doctest of the function itself and its copy in the generated "__test__" dict (stored in the module under that name), meaning that it executes all function doctests twice. This also broke some of Cython's own doctests for us, which modified global state and thus failed the second time. I'm yet unsure if we should try to do something about this. Currently, the explicit doctest dict is generated by default, so all code that uses doctests in function docstrings suffers from this. We might be able to make Cython a bit smarter so that these docstrings are left out of the test dict when (C-)compiling in CPython 3.4. Deciding which ones are still needed might be a bit tricky, though... Stefan

2 2

BUG: assignment to typed memoryview
by Daniele Nicolodi Dec. 4, 2013

Dec. 4, 2013

Hello, I believe there is a bug in how assignment to typed memoryviews is handled in the compiler. I think the following code should be valid code: cdef double[::1] a = np.arange(10, dtype=np.double) cdef double[::1] b = np.empty(a.size // 2) b[:] = a[::2] However, the Cython compiler complains with: Memoryview 'double[:]' not conformable to memoryview 'double[::1]'. A similar thing happens with memoryview copies: cdef double[::1] a = np.arange(10, dtype=np.double) cdef double[::1] b b = a[::2].copy() In both cases I would expect Cython to copy the non contiguous elements of the source array to the destination array (or to a newly created array in the second case). The copy() method is defined here https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#… and (while I haven't spent much time analyzing it in detail) it seems to do the right thing. Indeed, if I short-circuit the src_conforms_to_dst() function https://github.com/cython/cython/blob/master/Cython/Compiler/MemoryView.py#… the code compiles and runs correctly in both cases. I don't know much about how the Cython compiler works, therefore I don't know where to dig for the source of this problem. Can someone point me in the right direction? Thanks. Cheers, Daniele

2 2

'Referenced before assignment' warning triggered due to OpenMP directives?
by Yury V. Zaytsev Nov. 29, 2013

Nov. 29, 2013

Hi, I'm trying to find out whether the number of threads has been actually set to what I wanted it to be, but the compiler tells me that I'm trying to reference a variable before assignment. This doesn't happen when I simply release the GIL, but only within a parallel section. Is it a problem with Cython or I'm missing something obvious? Thanks! $ cat ompt.pyx cimport openmp from cython.parallel import parallel, prange cdef Py_ssize_t x = 1 with nogil, parallel(): x = openmp.omp_get_num_threads() print("Number of OpenMP threads is '{}'!".format(x)) $ cython ompt.pyx Error compiling Cython file: ------------------------------------------------------------ ... cdef Py_ssize_t x = 1 with nogil, parallel(): x = openmp.omp_get_num_threads() print("Number of OpenMP threads is '{}'!".format(x)) ^ ------------------------------------------------------------ ompt.pyx:9:50: local variable 'x' referenced before assignment -- Sincerely yours, Yury V. Zaytsev

1 0

Cython crash when using forward declarations
by Vinay Sajip Nov. 25, 2013

Nov. 25, 2013

The following Cython source: cdef extern from "header": cdef cppclass String cdef cppclass List[T] cdef cppclass StringList(List[String]) cdef cppclass String: String() int length() cdef cppclass List[T]: List() int length() cdef cppclass StringList(List[String]): StringList() void sort() causes a crash: Error compiling Cython file: ------------------------------------------------------------ ... cdef cppclass List[T]: List() int length() cdef cppclass StringList(List[String]): ^ ------------------------------------------------------------ s.pyx:14:9: Cannot inherit from incomplete type Error compiling Cython file: ------------------------------------------------------------ ... cdef cppclass List[T]: List() int length() cdef cppclass StringList(List[String]): StringList() ^ ------------------------------------------------------------ s.pyx:15:8: Compiler crash in AnalyseDeclarationsTransform File 'ModuleNode.py', line 101, in analyse_declarations: ModuleNode(s.pyx:1:0, full_module_name = 's') File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:1:5) File 'Nodes.py', line 438, in analyse_declarations: CDefExternNode(s.pyx:1:5, include_file = u'header') File 'Nodes.py', line 382, in analyse_declarations: StatListNode(s.pyx:2:4) File 'Nodes.py', line 1313, in analyse_declarations: CppClassNode(s.pyx:14:9, base_classes = [...]/1, name = u'StringList', visibility = u'extern') File 'Nodes.py', line 1203, in analyse_declarations: CVarDefNode(s.pyx:15:8, modifiers = [...]/0, visibility = u'extern') Compiler crash traceback from this point on: File ".../Cython/Compiler/Nodes.py", line 1203, in analyse_declarations api = self.api, modifiers = self.modifiers) File ".../Cython/Compiler/Symtab.py", line 2131, in declare_cfunction self.check_base_default_constructor(pos) File ".../Cython/Compiler/Symtab.py", line 2108, in \ check_base_default_constructor temp_entry = base_class.scope.lookup_here("<init>") AttributeError: 'NoneType' object has no attribute 'lookup_here' There seem to be two problems here: the AttributeError crash, but also the earlier "Cannot inherit from incomplete type" message. None of the types look incomplete; if you comment out the forward declarations, no error is reported: so it looks as if the forward declarations aren't being tied up with the actual declarations. In the actual use case, of course, I need the forward decls. Regards, Vinay Sajip

1 0

Fwd: [cython-users] How cythonize support c++?
by Stefan Behnel Nov. 10, 2013

Nov. 10, 2013

Given how often this has come up already, we should consider it a major usability issue. IMHO, stricter argument validation in cythonize() would help, as would better documentation about which option to use where. Stefan -------- Original-Message -------- Subject: [cython-users] How cythonize support c++? Date: Sun, 10 Nov 2013 19:19:48 +0800 To: cython-users(a)googlegroups.com Hi All I encountered a problem on writing a setup.py for cython extension. Following a tutorial from cython.org I wrote some codes like: from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext from Cython.Build import cythonize setup(cmdclass={'build_ext':build_ext}, ext_modules = cythonize("my.pyx", language="c++")) when I typed python setup.py build_ext --inplace on windows 7 with cython 0.19.2 It complained a lot of error. I founded out if I write setup file like this: extModules = [Extension("my", sources=["my.pyx"],language="c++")] setup(cmdclass={'build_ext':build_ext}, ext_modules=extModules) It ran fine without any problem. Do I miss sth so that cythonize can't support c++ ? Regards gelin yan

1 0

question on Get/SetItemInt*() utility functions
by Stefan Behnel Nov. 9, 2013

Nov. 9, 2013

Hi, the Get/SetItemInt helper macros usually look something like this: """ #define __Pyx_GetItemInt(o, i, size, to_py_func, is_list, wraparound, boundscheck) \ (((size) <= sizeof(Py_ssize_t)) ? \ __Pyx_GetItemInt_Fast(o, i, is_list, wraparound, boundscheck) : \ __Pyx_GetItemInt_Generic(o, to_py_func(i))) """ e.g. here: https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f2605… https://github.com/cython/cython/blob/59c3408c054bbbb7334ccca7d76bed93f2605… The "generic" part will almost never be called, except when the index *type* is really huge. However, if that happens, there's a rather large impact due to object creation, even if the value in question is tiny. Is there a way to safely extend this compile time size check with checks that compare the actual runtime value with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX ? Within those bounds, a cast would do the trick just fine, and we could still use the fast path. If it fails that bounds check, however, and the type we are processing is a builtin container, we'd already know that an exception is going to be raised and could simply drop the generic implementation, replacing it with an appropriate IndexError directly. Obviously, non-builtins would still need to fall back to the generic implementation, but there is a lot of special casing code for builtins that (IMHO) could be removed and replaced by a simple exception. I'm aware that the C comparison rules for unsigned vs. signed might make this tricky, but I also know that some on this list know their C better than I do. :) BTW, I also wonder if the above sizeof() comparison is safe enough - what if "i" has an unsigned type of the same size as Py_ssize_t and its value is larger than PY_SSIZE_T_MAX ? Wouldn't it wrap around in that case? Stefan

1 3

Bug Report: can't assign zero length array to contiguous 2D memoryview
by Josh Ayers Nov. 6, 2013

Nov. 6, 2013

Assigning a NumPy array with a zero length dimension to a memoryview fails if the memoryview is declared contiguous in the zero-length dimension. I'm getting a run-time error that the buffer is not C-contiguous or Fortran contiguous. See examples below. Thanks, Josh Ayers cdef float [:, :] arr1 = numpy.empty((2, 0), 'f4') # works cdef float [:, :] arr2 = numpy.empty((0, 2), 'f4') # works cdef float [:, ::1] arr3 = numpy.empty((2, 0), 'f4') # fails (buffer not C-contiguous) cdef float [:, ::1] arr4 = numpy.empty((0, 2), 'f4') # works cdef float [:, :] arr5 = numpy.empty((2, 0), 'f4', order='F') # works cdef float [:, :] arr6 = numpy.empty((0, 2), 'f4', order='F') # works cdef float [::1, :] arr7 = numpy.empty((2, 0), 'f4', order='F') # works cdef float [::1, :] arr8 = numpy.empty((0, 2), 'f4', order='F') # fails (buffer not Fortran contiguous)

1 0

Supporting new and old NumPy C API
by Frédéric Bastien Nov. 5, 2013

Nov. 5, 2013

Hi, As you probably know, cython generated c code use the old NumPy C API, so when we compile them with newer version of NumPy this generate warnings. In Theano we compile many different module during the user script. So this cause many warning being printed to the users, so we updated Theano to work with newer and older NumPy without warning. As we disable the old NumPy C API, we are not able to compile the cython generated code. So I manually updated the cython code to work with the new and old api. I did a post on the numpy mailing list that explain how I did this, in case someone want to tackle this problem in Cython: http://mail.scipy.org/pipermail/numpy-discussion/2013-November/068209.html Frédéric Bastien

1 0

Bug report: some incorrect typed memoryview indexing operations cause crashes
by James Sanders Nov. 5, 2013

Nov. 5, 2013

If I try to compile the following with the current master version: def buggy(int N, double[:] x): x[0, N-1] I get a crash. This only seems to happen when too many indices are used, and at least one of them is an expression involving a variable (for example, things like x[0, N] and x[0, 2-1] don't cause crashes). Here is the full output (this probably isn't related, but I'm also a bit confused about all these "non-trivial type declarators" warnings, which keep appearing in code I thought was working correctly): warning: View.MemoryView:410:43: Non-trivial type declarators in shared declaration. warning: View.MemoryView:583:32: Non-trivial type declarators in shared declaration. warning: View.MemoryView:588:32: Non-trivial type declarators in shared declaration. warning: View.MemoryView:1020:20: Non-trivial type declarators in shared declaration. warning: View.MemoryView:1020:28: Non-trivial type declarators in shared declaration. warning: View.MemoryView:1020:38: Non-trivial type declarators in shared declaration. Error compiling Cython file: ------------------------------ ------------------------------ ... def buggy(int N, double[:] x): x[0, N-1] ^ ------------------------------------------------------------ bug.pyx:2:10: Too many indices specified for type double[:] Error compiling Cython file: ------------------------------------------------------------ ... def buggy(int N, double[:] x): x[0, N-1] ^ ------------------------------------------------------------ bug.pyx:2:10: Compiler crash in OptimizeBuiltinCalls ModuleNode.body = StatListNode(bug.pyx:1:0) StatListNode.stats[0] = DefNode(bug.pyx:1:0, modifiers = [...]/0, name = 'buggy', num_required_args = 2, py_wrapper_required = True, reqd_kw_flags_cname = '0', used = True) DefNode.body = StatListNode(bug.pyx:2:5) StatListNode.stats[0] = ExprStatNode(bug.pyx:2:5) ExprStatNode.expr = IndexNode(bug.pyx:2:5, use_managed_ref = True) IndexNode.index = TupleNode(bug.pyx:2:5, is_sequence_constructor = 1, result_is_used = True, use_managed_ref = True) TupleNode.args[1] = SubNode(bug.pyx:2:10, infix = True, operator = '-', result_is_used = True, use_managed_ref = True) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/home/.../cython/Cython/Compiler/Visitor.c:4122) File "Visitor.py", line 505, in Cython.Compiler.Visitor.MethodDispatcherTransform.visit_BinopNode (/home/.../cython/Cython/Compiler/Visitor.c:9459) File "Visitor.py", line 516, in Cython.Compiler.Visitor.MethodDispatcherTransform._visit_binop_node (/home/.../cython/Cython/Compiler/Visitor.c:9613) AttributeError: 'NoneType' object has no attribute 'is_builtin_type'

1 0