Mailman 3 August 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Memory views: dereferencing pointer does break strict-aliasing rules
by Yury V. Zaytsev Oct. 11, 2013

Oct. 11, 2013

Hi, The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc: def hello(int [:] a): print(a, "world") If I translate it with the latest released version of Cython like this: cython cpp.pyx cython --cplus cpp.pyx and compile like this: gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o I get lots of warnings (see attached). It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules. I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results. Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that? Thanks! -- Sincerely yours, Yury V. Zaytsev

4 6

Should we monkey patch inspect.isfunction() ?
by Stefan Behnel Aug. 31, 2013

Aug. 31, 2013

Hi, currently, this fails in Cython compiled code: def cyfunc(): pass import inspect assert inspect.isfunction(cyfunc) There isn't really much of a reason why this *should* fail, except for the implementation of inspect.isfunction(), which tests for isinstance(obj, types.FunctionType) where types.FunctionType is type(some_python_function). In the benchmarks that we run on Jenkins, I'm patching into this as follows: """ def patch_inspect_isfunction(): import inspect orig_isfunction = inspect.isfunction def isfunction(obj): return (orig_isfunction(obj) or type(obj).__name__ == 'cython_function_or_method') isfunction._orig_isfunction = orig_isfunction inspect.isfunction = isfunction patch_inspect_isfunction() """ For example, Django uses inspect.isfunction() in some dynamic code and requires this fix to run properly when compiled. Now that we have a function type that is shared between different Cython modules, I think it's worth considering the addition of a proper type test in the inspect module as a general 'fix'. Meaning that we could patch the inspect module automatically whenever we insert the Cython function type into the shared Cython types module. I'm aware that this is a bit evil and hackish, but it usually works quite nicely. I'm also aware that the proper way to do this would eventually be the ABC mechanism as defined by PEP 3119, but Python's function type doesn't currently support the ABC protocol. We could additionally lobby for getting that changed. http://www.python.org/dev/peps/pep-3119/#overloading-isinstance-and-issubcl… Opinions? Stefan

2 1

Problem with final cdef methods
by Stefan Behnel Aug. 29, 2013

Aug. 29, 2013

Hi, I noticed two problems with final cdef methods. When overriding a normal cdef method with a final cdef method, you currently get a C compiler warning about a call with the wrong 'self' type, and when you call them before you declare them in your code, you get an error because the method is not forward declared. http://trac.cython.org/cython_trac/ticket/819 I think this suggests that we should try to come up with a cleaner way to integrate this feature into the function overloading architecture rather than just remembering their cname and call that. Stefan

3 6

Re: [Cython] [cython-users] Re: wiki spam
by Robert Bradshaw Aug. 28, 2013

Aug. 28, 2013

On Wed, Aug 28, 2013 at 11:53 AM, William Stein <wstein(a)gmail.com> wrote: > On Wed, Aug 28, 2013 at 10:12 AM, Lars Buitinck <L.J.Buitinck(a)uva.nl> wrote: >> 2013/8/28 <cfkaran2(a)gmail.com>: >>> On Wednesday, August 28, 2013 10:29:35 AM UTC-4, Josh Ayers wrote: >>>> >>>> I just noticed one of the pages on the Cython wiki was changed to spam. >>>> See the page history below. I reverted it, but it will probably happen >>>> again. It may be time to upgrade the captcha used for registering new >>>> accounts. >> >> Man, there's a lot of spam on that wiki (see >> http://wiki.cython.org/RecentChanges). Did you consider moving to a >> GitHub wiki instead? > > As the owner of the hardware on which the cython wiki is running, I > think this is well worth considering. If somebody wants to do it, > I'm happy to tar up the actual wiki install for them, so they can > somehow try to auto-convert all the pages... +1 Spam has increased a lot lately. What I really would like is a publicly editable wiki whose changes have to go through an approval first (e.g. a pull request). Unfortunately, despite expectations, github wikis don't work like this. I do really like the idea of the wiki being backed by a repo rather than some opaque format though. Anyone know of anything like that? We could also look at locking down the wiki to trac account owners only. This should probably be done right now. (I'm not sure how to do it though.) Also, I think much of the wiki content should be moved into the docs instead. - Robert

1 0

A bug when declare array.
by yi huang Aug. 27, 2013

Aug. 27, 2013

I use cython 0.19.1, when compile following code: cdef void some_function(): cdef char buf[sizeof(long)*8/3+6] Got error message: Error compiling Cython file: ------------------------------------------------------------ ... cdef void some_function(): cdef char buf[sizeof(long)*8/3+6] ^ ------------------------------------------------------------ /tmp/test.pyx:2:17: Variable type 'char []' is incomplete

3 3

bug in cythonize dependency tracking?
by Felix Salfelder Aug. 26, 2013

Aug. 26, 2013

Hi there. Lets say, i have foo.pyx. and use cythonize() to generate foo.c. Calling cythonize() a second time will correctly do nothing, because foo.c has been modified after foo.pyx. If I now add foo.pxd, cythonize will correctly update foo.c. In general foo.c now looks different. But: Deleting foo.pxd and running cythonize another time will fail (=do nothing) and leave foo.c in the wrong state. This might be a bug, depending on whether cythonize is meant to refresh output files consistently. regards felix

2 3

Likely bug: compilation failure on aliased cimport + subclass interaction
by Matěj Laitl Aug. 22, 2013

Aug. 22, 2013

Hi, it seems that when I do: > cimport foo as f > > cdef class DerivedClass(f.BaseClass): > pass I get following compilation errors: > 3:5: 'BaseClass' is not declared > 3:5: 'f.pxd' not found which looks like a bug to me. When I use plain "cimport foo" + "foo.BaseClass" or "from foo cimport BaseClass" + "BaseClass", it works as expected. Cython version 0.20dev add8091340e (git describe: 0.19-324-gadd8091) I'm attaching a more complete testcase which I'll also submit as a review request for convenience. Regards, Matěj Laitl

1 1

Re: [Cython] [cython-users] Confused by user's guide
by Robert Bradshaw Aug. 14, 2013

Aug. 14, 2013

On Mon, Aug 12, 2013 at 1:37 AM, Zak <cython(a)m.allo.ws> wrote: > I bet some people are still confused, because I would have been confused if > I read that a few months ago. Let me explain with code: > > # This is a Python function: > def do_math(top, bottom): > return top / bottom > > # This is also a Python function: > def do_math2(int top, int bottom): > return top / bottom > > # The function do_math2 above is "actually" doing > # something like this: > def do_math3(top_temp, bottom_temp): > cdef int top = top_temp > cdef int bottom = bottom_temp > return top / bottom > > All 3 functions above are Python functions, they take Python objects as > arguments and they return a Python object. The static type annotations in > do_math2 can be thought of as syntactic sugar - these type annotations do > not change the call signature of the function, they just tell the function > to immediately convert those arguments into C integers. When the function is > called, it is called with Python objects for top and bottom. But inside the > function, it will use fast C integers for top and bottom. Without this > syntactic sugar, you would be forced to do a renaming dance like that shown > in do_math3. Good explanation. These days do_mat2 actually equivalent to def do_math2(top_temp, bottom_temp): cdef int top = top_temp cdef int bottom = bottom_temp return do_math_c(top, bottom) cdef do_math_c(int top, int bottom): return top / bottom which doesn't really matter to the user except that there may be some cases (in the future) where we can extract and call do_math_c directly. Of course, this is exactly what cpdef is, which has got me thinking whether we need a separate keyword for this, or if we could simply make all def functions cpdef automatically. One place that's different is for exported cdef classes where c(p)def methods must be declared in the .pxd file, but we could perhaps handle this case by allowing them to be declared as plain cdef (or "def") methods there. Does anyone see any drawbacks to this? (I suppose there's a little overhead in the check-if-overridden dispatch.) - Robert

2 2

new "cythonize" script
by Stefan Behnel Aug. 11, 2013

Aug. 11, 2013

Hi, I added a script as "bin/cythonize" that acts as a frontend for the cythonize() compilation tool (and distutils' build_ext). https://github.com/cython/cython/blob/master/bin/cythonize One cool feature is that you can use it to recursively compile packages: cythonize -i -k Cython will translate, compile and link any .py files it finds in the Cython package in-place (-i), ignoring anything that fails to compile (-k). If you pass "--lenient", it ignores some compile time errors in favour of Python compatibility. It also works on glob patterns etc. I'm sure there's lots of features missing to make it nicely usable, so please give it a try and speak up if you miss something. Improvements (in the form of ideas and/or pull requests) are warmly appreciated. Stefan

2 1