Mailman 3 March 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

Fused type signature resolution failures
by Pauli Virtanen June 22, 2014

June 22, 2014

Hi, Here are the fused types + memoryview signature resolution failures I promised earlier (Cython 0.20.1): (A) TypeError: No matching signature ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused value_t: cnp.float32_t cnp.float64_t cpdef foo(value_t x): pass ----------------------------- quux.py import numpy as np import asd asd.foo(np.float32(1.0)) ----------------------------- (B) ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double' ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused idx_t: cnp.int32_t cnp.int64_t ctypedef fused value_t: cnp.int64_t cnp.float64_t cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x): pass ----------------------------- quux.py import numpy as np import asd i = np.zeros((3, 3), np.int64) j = np.zeros((3, 3), np.int64) x = np.zeros((3, 3), np.float64) asd.foo(i, j, x) ----------------------------- (C) Then some nasty platform-dependent failures: https://github.com/scipy/scipy/issues/3461 The relevant code is: https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L… The code looks nothing special. However, call to `lil_fancy_get` fails with "TypeError: No matching signature found" when the inputs have types <class 'int'> <class 'int'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> with ndarray dtypes: object object object object int32 int32 The failure occurs only on Christoph Gohlke's Win64 build, but not on Linux/OSX/MINGW32. This sounds like some integer size combination issue, but I'm far from sure. Unfortunately, I'm not easily able to isolate/understand what's going wrong here. -- Pauli Virtane

2 3

memoryview slice transpose
by Syam Gadde April 7, 2014

April 7, 2014

I wanted to re-offer for discussion a fix for a bug I reported a while back. The code that breaks is this: import numpy cimport numpy def testfunc(): a = numpy.arange(12).reshape([3,4]) cdef numpy.int_t[:,:] a_view = a cdef numpy.int_t[:,:] b = a_view[:,:].T One problem is that when a memoryview slice is decremented, the .memview and .data fields are not set to NULL, and any error that causes a jump to cleanup code will attempt to decref it again. The second problem (I think) is that assignment of a transpose of memoryview slices to a variable is not resulting in an incref on the underlying slice, even though the variable is not a temporary and continues to persist. Here is a patch that solves the problem for me. https://github.com/SyamGadde/cython/commit/5739d8b908f18c4fc9103ef04e39964d… The part I am least sure about is removing the "if" qualifications on the incref and decref: if self.obj.is_name or self.obj.is_attribute and self.obj.is_memslice_transpose: that was obviously there for a reason, but it causes problems in the above case. If there is a less extreme solution I'd be happy to take it. Thanks, -syam

2 2

line tracing/profiling code objects
by Syam Gadde March 27, 2014

March 27, 2014

Hi all, I tried using line tracing in conjunction with line_profiler/kernprof to attempt to see if line-based profiling works in Cython. I think it's pretty close. But I think the problem is that line_profiler is getting a different PyCodeObject when it wraps the function and when it actually gets the trace call. It adds the initial code object to a map, and later when it gets the trace call, decides that the trace call is not something it needs to pay attention to because the new code object that it gets is not the same as the original one. The first code object is created at function declaration by __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to a variable __pyx_k_codeobj_NNN. The second code object is created, essentially, during the first entry into the function (in __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting __pyx_frame_code to the initial code object before calling TraceCall() would fix this. Is this easy to do? I'd do it myself, but I'd need help figuring out how to get the name of the appropriate __pyx_k_codeobj_NNN variable from within FuncDefNode.generate_function_definitions(), which calls put_trace_call(). Thanks for your help... -syam

2 2

Too many instantiations with fused type memoryviews
by Pauli Virtanen March 22, 2014

March 22, 2014

Hi, FYI: Cython instantiates fused type routines with memoryview arguments unnecessarily many times. Example: ``` ctypedef fused idx_t: short long # Cython 0.20.1 instantiates this function 128 times, # even though only 2 would be needed def fubar_128(idx_t[:] a, idx_t[:] b, idx_t[:] c, idx_t[:] d, idx_t[:] e, idx_t[:] f, idx_t[:] g): print("HALLO") def fubar_2(idx_t a, idx_t b, idx_t c, idx_t d, idx_t e, idx_t f, idx_t g): print("HULLO") ``` $ cython cymod.pyx $ cython --version Cython version 0.20.1 $ grep -c 'print("HALLO")' cymod.c 128 $ grep -c 'print("HULLO")' _cymod.c 2 The n**m explosion starts to hurt quite quickly when there are several array arguments and more than one fused type. I think this issue is also accompanied by some signature resolution bugs (I'll try to come up with an example case). Cheers, -- Pauli Virtanen

3 6

0.20.2
by Stefan Behnel March 21, 2014

March 21, 2014

Hi, there are a couple of nice additions and fixes in the current master that could go out as a 0.20.2 some time soon. I suggest waiting another couple of days for more bug reports to come in and then just send it out. I'd like to also fix the buffer formatting bug by then, looks like I got a hold of it. Stefan

2 2

Re: [Cython] Numpy Deprecation Warnings
by Anthony Scopatz March 13, 2014

March 13, 2014

Forwarding this on to cython developers. On Sun, Mar 9, 2014 at 9:54 AM, Anthony Scopatz <scopatz(a)gmail.com> wrote: > Hello All, > > I continue to see numpy deprecation warnings. When I set the right flag > flag, the warnings go away but then the build fails because it seems Cython > *is* generating to the old API even though I am not using it! Is this what > is actually going on? And if so, is there a fix in the pipeline? > > Be Well > Anthony >

2 1

Crash with freelist() and __slots__
by Stefan Behnel March 5, 2014

March 5, 2014

Hi, I was made aware of crashes in the last lxml release, which turned out to be due to the use of freelists for types that could be subtyped from Python code. I was able to work around them in lxml, however, the real problem is in Cython. There was supposed to be a safe guard for that case in the freelist code based on the object struct size, which increases for subtypes that have a __dict__. However, if the Python subtype uses an empty __slots__ declaration, the object struct size will not increase, thus passing the guard. The correct fix is to also test if the type being instantiated lives on the heap and exclude it from the freelist if so. https://github.com/cython/cython/commit/d2aff824dd0900982a420ebaaa1dac6120a… This, together with the buffer/memory view related bugs I fixed since the last release, suggest (at least to me) that we shouldn't wait all too long with the next bug fix release. >From my POV, all changes in current master are safe enough to go out. Stefan

2 2