Mailman 3 October 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

"Stack" checker for undefined behaviour in C code
by Stefan Behnel Nov. 2, 2013

Nov. 2, 2013

Hi, I just came across this paper: http://pdos.csail.mit.edu/~xi/papers/stack-sosp13.pdf They describe an analysis tool that checks C code for bugs that exploit undefined behaviour and that are thus up to the mercy of compiler assumptions and "optimisations" to do the right thing or not. They made it available on github: https://github.com/xiw/stack/ If anyone wants to take the time to set it up for checking some Cython generated code, I'd be interested to see if it finds something. Stefan

2 6

Cython wiki
by Robert Bradshaw Oct. 17, 2013

Oct. 17, 2013

I've cleaned out the spam and moved the wiki to https://github.com/cython/cython/wiki Don't edit it just yet (as I have at lest one more round of force-pushing conversion from moinmoin files), but please let me know if something doesn't look right or it's missing pages you expect to see. (Yes, I know some pages are failing to render; still looking into that, but if you can see what the error is please let me know.) - Robert

1 0

Cython 0.19.2 released
by Stefan Behnel Oct. 15, 2013

Oct. 15, 2013

Hi all, I released Cython 0.19.2 today as a bug-fix release for the 0.19 series. You can get it from here: http://cython.org/release/Cython-0.19.2.tar.gz http://cython.org/release/Cython-0.19.2.zip Or from PyPI: https://pypi.python.org/pypi/Cython/0.19.2 Release notes: https://github.com/cython/cython/blob/c2ddf294e025646132f7c29fae609c2f3c010… Documentation: http://docs.cython.org/ Have fun, Stefan

3 2

State of PyPy compatibility wrt. arrays, incl. NumPy
by Yury V. Zaytsev Oct. 14, 2013

Oct. 14, 2013

Hi, I've been playing with my Cython-generated extension on PyPy, trying to load it through CPyExt, and, surprisingly, after a few fixes to PyPy, it generally seems to work. However, it looks like the situation with whatever kind of arrays is actually pretty gloomy at the moment. First, PyPy supports Python array.arrays, but doesn't replicate Python internals, so the extension even fails to load, since in PyPy tp_basicsize == 16 and not 56. Not sure what to make of this one... for my needs, that would be enough. I guess PyPy developers have to be approached to see if it's practical to expose CPython structure, or agree on a different API? Second, they say that the new-style buffer access is broken and it's not clear if they will support it at all & when this is going to happen. Third, apparently, there is already some limited support for NumPy C-API in NumPyPy, and it seems that there is interest in improving this support to make it usable: http://docs.scipy.org/doc/numpy/reference/c-api.array.html Finally, GetItem on NumPyPy arrays, which will be very slow, but at least should work, rather than not, is also broken. I think this will be easiest of all to fix. So my question is, has anyone been playing with PyPy recently? Is there anyone interested in making Cython-generated code to work nicely with CPyExt? What are the plans w.r.t. the points I mentioned? I've noticed that PyPy jobs in Jenkins are disabled... not sure whether this really means anything or not. Thanks! -- Sincerely yours, Yury V. Zaytsev

2 2

preparing 0.19.2 for this weekend
by Stefan Behnel Oct. 12, 2013

Oct. 12, 2013

Hi, I've started preparing a bug-fix release on the 0.19.x branch. If you have any further changes that should go in, please speak up now. I'd like to get it out by this Sunday at the latest (before PyCon-DE starts on Monday). Stefan

2 2

Memory views: dereferencing pointer does break strict-aliasing rules
by Yury V. Zaytsev Oct. 11, 2013

Oct. 11, 2013

Hi, The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc: def hello(int [:] a): print(a, "world") If I translate it with the latest released version of Cython like this: cython cpp.pyx cython --cplus cpp.pyx and compile like this: gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o I get lots of warnings (see attached). It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules. I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results. Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that? Thanks! -- Sincerely yours, Yury V. Zaytsev

4 6

Declaration syntax change
by Robert Bradshaw Oct. 4, 2013

Oct. 4, 2013

In thinking about 1.0, this would be a chance to make backwards-incompatible changes. One thing that has always bothered me is the C-delclarator decoration for specifying complex types. Specifically, cdef int *a, b, c, *d[3] is IMHO quite ugly but also adds a lot of complexity to the parser. What if instead we required cdef int* a cdef int b, c cdef int[3]* d In this case our grammar could be almost identical to the Python grammar, with the addition of a "type declaration" that could occur in certain situations plus a new type token. Does anyone see any issues with this? We could have a period where we warned if declarators were used, or simply disallow them outright before changing the meaning. It'd be easy to create a tool one could run on your sources that would "expand" such declarations as well. - Robert

5 13

Memory leak when using Typed Memory View and np array of objects
by Haijie Gu Oct. 2, 2013

Oct. 2, 2013

Hi, I'm new to cython's typed memory view, and found some cases where the function that uses typed memory view has memory leaking. The leak happens when I pass a numpy array of objects, where each object itself is a numpy array.( You can get this 'weird' object from construction a pandas Series with a list of numpy arrays). Please see the following code snippet or use the attached code to reproduce the case. I appreciate any help and suggestions in advance! (I also posted on the cython-users google group. Apologize for the redundancy.) # BEGIN CONTENT OF test.pyx # this does not leak cpdef int do_nothing(arr): return 0 # this does leak cpdef int do_nothing_typed(double[:] arr): return 0 # this does leak cpdef int do_nothing_but_copy(arr): cdef double[:] _arr = arr return 0 # END CONTENT OF test.pyx # BEGIN CONTENT OF runtest.py # ... omit all the imports here def gc_obj_hist(): """ Returns a sorted map from type to the counts of in memory objects with the type """ hst = defaultdict(lambda: 0) for v in gc.get_objects(): hst[type(v)] += 1 l = sorted(hst.iteritems(), key=operator.itemgetter(1), reverse=True) return l # NOT LEAK def test1(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing(s[i]) print "Top 5 object types after test 1: " + str(gc_obj_hist()[:5]) # LEAK def test2(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_typed(s[i]) print "Top 5 object types after test 2: " + str(gc_obj_hist()[:5]) # LEAK def test3(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_but_copy(s[i]) print "Top 5 object types after test 3: " + str(gc_obj_hist()[:5]) # NOT LEAK def test4(n=10000): s = pd.Series([np.random.randn(10) for i in range(n)]) for i in range(n): do_nothing_but_copy(np.array(s[i])) print "Top 5 object types after test 4: " + str(gc_obj_hist()[:5]) if __name__ == "__main__": n = 100000 test1(n) test2(n) test3(n) test4(n) # END CONTENT OF runtest.py Thanks, -jay

1 0