Mailman 3 September 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Memory views: dereferencing pointer does break strict-aliasing rules
by Yury V. Zaytsev Oct. 11, 2013

Oct. 11, 2013

Hi, The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc: def hello(int [:] a): print(a, "world") If I translate it with the latest released version of Cython like this: cython cpp.pyx cython --cplus cpp.pyx and compile like this: gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o I get lots of warnings (see attached). It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules. I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results. Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that? Thanks! -- Sincerely yours, Yury V. Zaytsev

4 6

Release plan for 0.20?
by Stefan Behnel Sept. 29, 2013

Sept. 29, 2013

Hi, I think 0.20 is shaping up nicely. Are there any further pending changes, or should we start preparing an alpha release? Stefan

1 0

cython wiki
by William Stein Sept. 26, 2013

Sept. 26, 2013

Hello, I've permanently disabled the Cython wiki, due to all the spam it contains, and that whenever I try to start it, the moinmoin/twisted/python processes instantly just burns lots of CPU and stops responding to requests. If anybody would like to move the wiki to github, google code, bitbucket, etc., please send me an email and I will make the complete tarball of the wiki available to you. (wstein(a)gmail.com) -- William -- William Stein Professor of Mathematics University of Washington http://wstein.org

1 0

Re: [Cython] Memory leak of memoryview attributes in cdef subclasses
by Daniel, Bruno Sept. 25, 2013

Sept. 25, 2013

Dear Paul, > Cannot reproduce the failure on Cython 0.19.1. If you are using an > earlier version, please try that instead. We also used Cython 0.19.1. If the unit test runs through, this means that the memory leak is there. I'm sorry for the ambiguity. Please change the lines if not cause_leak: n_memory_views_expexted -= 1 into just n_memory_views_expexted -= 1 Then the test will fail if the memory leak exists. I also just noticed that unittest.main() doesn't work in Cython files. The tests are just ignored. Please call the tests this way instead: python -m unittest memoryview_leak For your convenience, here's the updated code: -------- File memoryview_leak.pyx --------------------------------------------------------------- """ Cython module exposing memory leaks due to memoryview attributes in cdef classes in Cython 0.19.1 Compile:: python setup.py build_ext --inplace Run:: python -m unittest memoryview_leak """ from __future__ import print_function, division import unittest import gc import numpy as np cdef class LeakTestSuper: """ Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`. """ pass cdef class LeakTest1(LeakTestSuper): """ Class for memory leak testing. Holds a memoryview which is extracted from an :class:`numpy.ndarray` at initialization. This class avoids the leak by resetting the memoryview to None in the method :meth:`__dealloc__`. ``__del__()`` wouldn't work, because it is not supported in Cython's `cdef` classes. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) def __dealloc__(self): self._s = None cdef class LeakTest2(LeakTestSuper): """ Class for memory leak testing. Holds an array that is allocated at initialization. This class does not avoid the leak and thus exposes the problem with memoryviews in Cython. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) class TestLeakTest(unittest.TestCase): def n_objects_by_type(self, typename): return len([ obj for obj in gc.get_objects() if type(obj).__name__ == typename]) def test_gc(self): # Make sure to clear all old memoryviews (circularly referenced). gc.collect() # Now there mustn't be any memoryviews left. n_memory_views_expexted = 0 for cause_leak in [False, True, False, True]: # Create an object of LeakTest1 or LeakTest2 allocating a memoryview # internally. leaktest = LeakTest2() if cause_leak else LeakTest1() # Check the number of allocated memory views. n_memory_views_expexted += 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) # Delete the reference to leaktest and let the garbage collector do # its thing. del leaktest gc.collect() # Check for leaks by counting the memoryviews again. if True: # not cause_leak: n_memory_views_expexted -= 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) ----------------------------------------------------------------------------------------------------- Here's the corresponding setup file: -------- File setup.py ---------------------------------------------------------------------------- from __future__ import print_function, division from distutils.core import setup from Cython.Build import cythonize setup( name = "memoryview_leak", ext_modules = cythonize('memoryview_leak.pyx'), ) --------------------------------------------------------------------------------------------------------- Best regards, Bruno Daniel From: cython-devel [cython-devel-bounces+bruno.daniel=blue-yonder.com(a)python.org] on behalf of cython-devel-request(a)python.org [cython-devel-request(a)python.org] Sent: Wednesday, September 25, 2013 12:00 PM To: cython-devel(a)python.org Subject: cython-devel Digest, Vol 32, Issue 10 Send cython-devel mailing list submissions to cython-devel(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/cython-devel or, via email, send a message with subject or body 'help' to cython-devel-request(a)python.org You can reach the person managing the list at cython-devel-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cython-devel digest..." Today's Topics: 1. Re: Memory leak of memoryview attributes in cdef subclasses (Pauli Virtanen) ---------------------------------------------------------------------- Message: 1 Date: Tue, 24 Sep 2013 20:43:14 +0300 From: Pauli Virtanen <pav(a)iki.fi> To: cython-devel(a)python.org Subject: Re: [Cython] Memory leak of memoryview attributes in cdef subclasses Message-ID: <l1siva$rg5$1(a)ger.gmane.org> Content-Type: text/plain; charset=ISO-8859-1 Hi, 18.09.2013 16:14, Daniel, Bruno kirjoitti: > We encountered memory leaks when using memoryview attributes in cdef subclasses > in Cython code. They can be avoided by adding a dealloc method setting the value > of the memoryview attribute to None. The problem does not occur in topmost > cdef classes. > > Here's an example: (See the module docstring on how to compile and run the > example.) Cannot reproduce the failure on Cython 0.19.1. If you are using an earlier version, please try that instead. One bug with memory leaks in cdef classes was fixed in 0.18 IIRC. -- Pauli Virtanen ------------------------------ Subject: Digest Footer _______________________________________________ cython-devel mailing list cython-devel(a)python.org https://mail.python.org/mailman/listinfo/cython-devel ------------------------------ End of cython-devel Digest, Vol 32, Issue 10 ********************************************

1 0

Memory leak of memoryview attributes in cdef subclasses
by Daniel, Bruno Sept. 24, 2013

Sept. 24, 2013

Dear developers, We encountered memory leaks when using memoryview attributes in cdef subclasses in Cython code. They can be avoided by adding a dealloc method setting the value of the memoryview attribute to None. The problem does not occur in topmost cdef classes. Here's an example: (See the module docstring on how to compile and run the example.) --- File memoryview_leak.pyx -------------------------------------------------- """ Cython module exposing memory leaks due to memoryview attributes in cdef classes Compile:: python setup.py build_ext --inplace Run:: python -c "import memoryview_leak; memoryview_leak.main()" """ from __future__ import print_function, division import unittest import gc import numpy as np cdef class LeakTestSuper: """ Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`. """ pass cdef class LeakTest1(LeakTestSuper): """ Class for memory leak testing. Holds a memoryview which is extracted from an :class:`numpy.ndarray` at initialization. This class avoids the leak by resetting the memoryview to None in the method :meth:`__dealloc__`. ``__del__()`` wouldn't work, because it is not supported in Cython's `cdef` classes. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) def __dealloc__(self): self._s = None cdef class LeakTest2(LeakTestSuper): """ Class for memory leak testing. Holds an array that is allocated at initialization. This class does not avoid the leak and thus exposes the problem with memoryviews in Cython. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) class TestLeakTest(unittest.TestCase): def n_objects_by_type(self, typename): return len([ obj for obj in gc.get_objects() if type(obj).__name__ == typename]) def test_gc(self): # Make sure to clear all old memoryviews (circularly referenced). gc.collect() # Now there mustn't be any memoryviews left. n_memory_views_expexted = 0 for cause_leak in [False, True, False, True]: # Create an object of LeakTest1 or LeakTest2 allocating a memoryview # internally. leaktest = LeakTest2() if cause_leak else LeakTest1() # Check the number of allocated memory views. n_memory_views_expexted += 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) # Delete the reference to leaktest and let the garbage collector do # its thing. del leaktest gc.collect() # Check for leaks by counting the memoryviews again. if not cause_leak: n_memory_views_expexted -= 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) def main(): unittest.main() ------------------------------------------------------------------------------- This is the corresponding setup file: --- File setup.py ------------------------------------------------------------- from __future__ import print_function, division from distutils.core import setup from Cython.Build import cythonize setup( name = "memoryview_leak", ext_modules = cythonize('memoryview_leak.pyx'), ) ------------------------------------------------------------------------------- If the unit tests are all ok in this example, this means that the memory leak has been encountered in LeakTest2 and avoided in LeakTest1 as expected. Best regards, Bruno Daniel ----- Dr. Bruno Daniel Research and Development Blue Yonder, Karlsruhe, Germany

2 1

Improving compile times for Cython extensions with C sources
by Wes McKinney Sept. 17, 2013

Sept. 17, 2013

I haven't been able to find an easy answer to this question, but here is my problem: Cython file: extension.pyx Declarations / C imports: extension.pxd C dependencies: a.c b.c c.c These are all compiled together to produce extension.so. The problem is, when I modify extension.pyx, all 4 C files are recompiled, even if the 3 straight C files are untouched. Any way to fix the build system to not do this? Seems that the .o files from the C dependencies could be easily reused. thanks, Wes

2 2

Assign from array to scalar
by Emmanuel Gil Peyrot Sept. 12, 2013

Sept. 12, 2013

Hi, A few weeks ago I started an implementation of that simple usecase: cdef long a[2], x, y # set a[0] and a[1] to some useful values x, y = a It does works, but I’m not used to the interactions between the different parts of the code, I mainly hacked away where I could. I’m especially dissatisfied with the code duplication I came with (to check the correct types and size) and the location of the checks. I added testcases for both the array assignation and the slices of that same assignation I plan to implement, which seems to integrate badly with my current implementation, making the code even more unreadable. I tried to find a commit implementing a similar feature so I could get some inspiration but couldn’t find one. If someone has a hint about that I’d be glad. :) Btw, on a Cython.Compiler.ExprNodes.SliceIndexNode object, why does stop_code() give an int while start_code() gives a string? It’d be much better to return an int in both cases, so one could interpolate without conversion from both python or the generated C. Preliminary patch joined. -- Emmanuel Gil Peyrot XMPP: <linkmauve(a)linkmauve.fr> OpenPGP: 24B1D609

1 0

Metaclass bug corrupts the heap
by Dénes Vadász Sept. 12, 2013

Sept. 12, 2013

Hello, according to valgrind the following Cython fragment causes a heap corruption (with Python 2.7.5 and Cython 0.19.1 on Ubuntu): cdef class MetaClass(type): cdef int i class MyClass(object): __metaclass__ = MetaClass Please find below the result of a many hour investigation (originally triggered by random crashes and heap corruptions of a bigger Cython program that uses metaclasses). MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 200 byte structure on a 32-bit architecture): struct __pyx_obj_4meta_MetaClass { PyTypeObject __pyx_base; int i; }; Instances of PyTypeObject are supposed to be allocated statically when initializing extension modules, because the structure does not support (among others) garbage collection. However, when MyClass is created, an instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is dynamically allocated by the Python memory management machinery. The machinery then tries to initialize the allocated memory. The problem is that it expects that the first member of struct __pyx_obj_4meta_MetaClass is of type PyHeapTypeObject (a cc. 500 byte structure) and after a type cast it writes to the tail members of the assumed PyHeapTypeObject, i.e. way beyond the allocated cc. 200 + 4 bytes, corrupting the object heap. This corruption is nicely reported by valgrind. The proposed fix is to make __pyx_obj_4meta_MetaClass extend PyHeapTypeObject: struct __pyx_obj_4meta_MetaClass { PyHeapTypeObject __pyx_base; int i; }; This can be achieved by adding the below 2 lines to Compiler/Builtin.py: 382a383,384 > elif name == 'type': > objstruct_cname = 'PyHeapTypeObject' So that the init_builtin_types function becomes: def init_builtin_types(): global builtin_types for name, cname, methods in builtin_types_table: utility = builtin_utility_code.get(name) if name == 'frozenset': objstruct_cname = 'PySetObject' elif name == 'bool': objstruct_cname = None * elif name == 'type':* * objstruct_cname = 'PyHeapTypeObject'* else: objstruct_cname = 'Py%sObject' % name.capitalize() the_type = builtin_scope.declare_builtin_type(name, cname, utility, objstruct_cname) builtin_types[name] = the_type for method in methods: method.declare_in_type(the_type) After patching my Cython installation with the above, valgrind stopped to complain and there were no more crashes. Please consider adding this fix in the next release of Cython. Regards Dénes Vadász

3 2

memoryviews bloat?
by Robert Bradshaw Sept. 12, 2013

Sept. 12, 2013

>> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to the repo. Switched back to bare pointers to keep the compile times within bounds. > It would be interesting to hear the Cython team's point of view on that. @sturlamolden @robertwb @markflorisson88 @dagss The priority has always been to produce the most optimized runtime code, with compile time being way down on the list of priorities, if even thought about at all. This means that there's likely a lot of low-hanging fruit for those who find the compile time too slow (do you have some specific stats?). Also, while Cython generates very verbose C code, much of it is optimized away. While the lines-of-code and compile times are related, they should not be conflated. The recently added common_utility_include_dir option to cythonize(...) might help as well--it allows utility code to be shared across multiple generated files. Perhaps a faster -c dbg option for development would be handy. - Robert

3 3