Mailman 3 December 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

"embedsignature" and "autotestdict" features in Py3.4
by Stefan Behnel Jan. 30, 2014

Jan. 30, 2014

Hi, recent changes in Py3.4 affect the functionality of the two directives "embedsignature" and "autotestdict". The so-called "argument clinic" extracts any embedded signatures from docstrings and moves them into a new property "__text_signature__". This is done at runtime, i.e. when either "__doc__" or "__text_signature__" are requested. http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182 I personally consider this a feature (and it works nicely with the signatures that Cython embeds), but you may or may not agree. It broke some of our own doctests, at least, because the "__doc__" value that we tested for was no longer the same. Regarding the "autotestdict", CPython also got smarter and now finds doctests embedded in C implemented functions all by itself. This is clearly an improvement compared to the previous behaviour. However, this also means that it now executes both the doctest of the function itself and its copy in the generated "__test__" dict (stored in the module under that name), meaning that it executes all function doctests twice. This also broke some of Cython's own doctests for us, which modified global state and thus failed the second time. I'm yet unsure if we should try to do something about this. Currently, the explicit doctest dict is generated by default, so all code that uses doctests in function docstrings suffers from this. We might be able to make Cython a bit smarter so that these docstrings are left out of the test dict when (C-)compiling in CPython 3.4. Deciding which ones are still needed might be a bit tricky, though... Stefan

2 2

Seemingly incorrect specialization of templates
by Vinay Sajip Dec. 17, 2013

Dec. 17, 2013

Given this code in Cython/Includes/libcpp/utility.pxd: cdef extern from "<utility>" namespace "std": cdef cppclass pair[T, U]: T first U second # rest omitted when the following code in Cython/Includes/libcpp/map.pxd is seen: from utility cimport pair cdef extern from "<map>" namespace "std": cdef cppclass map[T, U]: cppclass iterator: pair[T, U]& operator*() nogil # rest omitted The line "pair[T, U]& operator*() nogil" causes a specialization of pair[T, U] to be created, but using placeholders rather than actual types. This seems wrong - can someone here confirm whether this is intentional? This led to a problem where the generated C++ code for a genuine specialisation instead generated a declaration like "pair<T, U>" instead of using the actual arguments. Given this code in Cython/Includes/libcpp/utility.pxd: cdef extern from "<utility>" namespace "std": cdef cppclass pair[T, U]: T first U second # rest omitted when the following code in Cython/Includes/libcpp/map.pxd is seen: from utility cimport pair cdef extern from "<map>" namespace "std": cdef cppclass map[T, U]: cppclass iterator: pair[T, U]& operator*() nogil # rest omitted The line "pair[T, U]& operator*() nogil" causes a specialization of pair[T, U] to be created, but using placeholders rather than actual types. This seems wrong - can someone here confirm whether this is intentional? This caused a problem where the generated C++ code for a genuine specialisation instead generated a declaration like "pair<T, U>" instead of using the actual arguments.Given this code in Cython/Includes/libcpp/utility.pxd: cdef extern from "<utility>" namespace "std": cdef cppclass pair[T, U]: T first U second # rest omitted when the following code in Cython/Includes/libcpp/map.pxd is seen: from utility cimport pair cdef extern from "<map>" namespace "std": cdef cppclass map[T, U]: cppclass iterator: pair[T, U]& operator*() nogil # rest omitted The line "pair[T, U]& operator*() nogil" causes a specialization of pair[T, U] to be created, but using placeholders rather than actual types. This seems wrong - can someone here confirm whether this is intentional? This caused a problem where the generated C++ code for a genuine specialisation instead generated a declaration like "pair<T, U>" instead of using the actual arguments. The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the template_types passed to specialize are actually all instances of TemplatePlaceholderType. Regards, Vinay Sajip Given this code in Cython/Includes/libcpp/utility.pxd: cdef extern from "<utility>" namespace "std": cdef cppclass pair[T, U]: T first U second # rest omitted when the following code in Cython/Includes/libcpp/map.pxd is seen: from utility cimport pair cdef extern from "<map>" namespace "std": cdef cppclass map[T, U]: cppclass iterator: pair[T, U]& operator*() nogil # rest omitted The line "pair[T, U]& operator*() nogil" causes a specialization of pair[T, U] to be created, but using placeholders rather than actual types. This seems wrong - can someone here confirm whether this is intentional? This caused a problem where the generated C++ code for a genuine specialisation instead generated a declaration like "pair<T, U>" instead of using the actual arguments. This, of course, failed in g++. The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the template_types passed to specialize are actually all instances of TemplatePlaceholderType. It seems like just the base_type could be used. Regards, Vinay Sajip The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the template_types passed to specialize are actually all instances of TemplatePlaceholderType. Regards, Vinay Sajip The specialize call occurs in Nodes.py:TemplatedTypeNode.analyse, where the template_types passed to specialize are actually all instances of TemplatePlaceholderType. Regards, Vinay Sajip

1 1

BUG: assignment to typed memoryview
by Daniele Nicolodi Dec. 4, 2013

Dec. 4, 2013

Hello, I believe there is a bug in how assignment to typed memoryviews is handled in the compiler. I think the following code should be valid code: cdef double[::1] a = np.arange(10, dtype=np.double) cdef double[::1] b = np.empty(a.size // 2) b[:] = a[::2] However, the Cython compiler complains with: Memoryview 'double[:]' not conformable to memoryview 'double[::1]'. A similar thing happens with memoryview copies: cdef double[::1] a = np.arange(10, dtype=np.double) cdef double[::1] b b = a[::2].copy() In both cases I would expect Cython to copy the non contiguous elements of the source array to the destination array (or to a newly created array in the second case). The copy() method is defined here https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#… and (while I haven't spent much time analyzing it in detail) it seems to do the right thing. Indeed, if I short-circuit the src_conforms_to_dst() function https://github.com/cython/cython/blob/master/Cython/Compiler/MemoryView.py#… the code compiles and runs correctly in both cases. I don't know much about how the Cython compiler works, therefore I don't know where to dig for the source of this problem. Can someone point me in the right direction? Thanks. Cheers, Daniele

2 2