Mailman 3 May 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

Wrong order of __Pyx_DECREF when calling a function with an implicit str → char* conversion.
by Emmanuel Gil Peyrot July 6, 2014

July 6, 2014

Hi, I was testing my cython codebase on top of pypy/cpyext, and I found a memory corruption. After investigating it with the pypy guys (thanks arigato!), we identified it as a cython bug: cdef void test(char *string): print(string) def run(array): test(array[0]) This code works fine under cpython, but looking at the generated code for the last line, the call to test is done with a pointer (__pyx_t_2) to some potentially deallocated string (__pyx_t_1), which isn’t freed just yet on cpython but is on pypy: __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, …); __Pyx_GOTREF(__pyx_t_1); __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; __pyx_f_1a_test(__pyx_t_2); The obvious solution is to swap the last two lines, it then works fine on pypy (although not necessarily if the test function stores the pointer somewhere, but it’s not cython’s fault then). This issue can also happen with an explicit cast: pointer = <char *>array[0] test(pointer) I’m not sure if it should be addressed the same way, because that would mean keeping a reference to array[0] for all the lifetime of the current scope, but it could still prevent obscure bugs like the memory corruption I had. -- Emmanuel Gil Peyrot

3 5

Fused type signature resolution failures
by Pauli Virtanen June 22, 2014

June 22, 2014

Hi, Here are the fused types + memoryview signature resolution failures I promised earlier (Cython 0.20.1): (A) TypeError: No matching signature ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused value_t: cnp.float32_t cnp.float64_t cpdef foo(value_t x): pass ----------------------------- quux.py import numpy as np import asd asd.foo(np.float32(1.0)) ----------------------------- (B) ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double' ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused idx_t: cnp.int32_t cnp.int64_t ctypedef fused value_t: cnp.int64_t cnp.float64_t cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x): pass ----------------------------- quux.py import numpy as np import asd i = np.zeros((3, 3), np.int64) j = np.zeros((3, 3), np.int64) x = np.zeros((3, 3), np.float64) asd.foo(i, j, x) ----------------------------- (C) Then some nasty platform-dependent failures: https://github.com/scipy/scipy/issues/3461 The relevant code is: https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L… The code looks nothing special. However, call to `lil_fancy_get` fails with "TypeError: No matching signature found" when the inputs have types <class 'int'> <class 'int'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> with ndarray dtypes: object object object object int32 int32 The failure occurs only on Christoph Gohlke's Win64 build, but not on Linux/OSX/MINGW32. This sounds like some integer size combination issue, but I'm far from sure. Unfortunately, I'm not easily able to isolate/understand what's going wrong here. -- Pauli Virtane

2 3

Cython 0.20.2 beta
by Robert Bradshaw June 4, 2014

June 4, 2014

I've pushed a beta for the 0.20.2 can be found at http://cython.org/release/Cython-0.20rc1.tar.gz . This is a bugfix-only release, more development is going on for 0.21 which will also be released soon. Note, however, that the 0.20.x branch is the last to support older versions of Python (pre 2.6 or 3.2). Please try it out and report back. - Robert Features added Some optimisations for set/frozenset instantiation. Support for C++ unordered_set and unordered_map. Bugs fixed Access to attributes of optimised builtin methods (e.g. [].append.__name__) could fail to compile. Memory leak when extension subtypes add a memory view as attribute to those of the parent type without having Python object attributes or a user provided dealloc method. Compiler crash on readonly properties in "binding" mode. Auto-encoding with c_string_encoding=ascii failed in Py3.3. Crash when subtyping freelist enabled Cython extension types with Python classes that use __slots__. Freelist usage is restricted to CPython to avoid problems with other Python implementations. Memory leak in memory views when copying overlapping, contiguous slices. Format checking when requesting non-contiguous buffers from cython.array objects was disabled in Py3. C++ destructor calls in extension types could fail to compile in clang. Buffer format validation failed for sequences of strings in structs. Docstrings on extension type attributes in .pxd files were rejected.

1 1

undefined reference to `__Pyx_PyInt_As_unsigned_int error
by Joshua Adelman May 20, 2014

May 20, 2014

Hello all, I wanted to try to get some traction on this issue that I raised on the Cython-user google group list: https://groups.google.com/forum/#!topic/cython-users/l9wD4w2lcw8 The basic issue is that I get an error of "undefined reference to `__Pyx_PyInt_As_unsigned_int". The strange part is I then get non-deterministic codegen, wherein the first time I compile the code it fails, but if I delete the .c file and try again, it will then compile without errors. A minimal example is provided here: https://gist.github.com/synapticarbors/11060931 This issue seems to have cropped up somewhere between 0.19.2 (where it's fine) and 0.20.1. It also is still present in 0.21dev from github. The error seems to happen when running on a Windows machine (Windows 2008 server), but I can't reproduce the error on Mac OSX (running the same version of the Continuum Anaconda distribution). Any suggestions or confirmation of this issue would be appreciated. Thanks, Josh

1 0

Re: [Cython] GCC Pxd
by Philip Herron May 9, 2014

May 9, 2014

Just small update i think i have got most of C code working now, but working on C++ generation again its all shaping up to look pretty usable. I wonder by the off chance is there a python library that can parse the C preprocessor if anyone knows or not i am looking at https://code.google.com/p/pypreprocessor/ to try and parse cpp as this information is lost at the Middle-end level in GCC where the plugin runs. On 1 May 2014 14:01, Stefan Behnel <stefan_ml(a)behnel.de> wrote: > [forwarding Philip's mail to the Cython users mailing list] > > Hey all, > > Some of you may remember a project i worked on for GSoC 2012, i wasn't > really happy with the project over all for a long time it basicaly > required at the time people to be running GCC from latest SVN/GIT for > it to work correctly. > > Now gcc >= 4.7 is becoming more normal, so i went back into it and > rewrote it from scratch now and its shaping up to look fairly decent > and usable now. > > https://github.com/redbrain/cython/tree/master/Tools/gccpxd > > So the idea for this project is that you write a small configuration > file and it will integrate with your C/C++ build system for example > you could do: > > ./configure CC=cygcc ... > make > ... > > And you will get a project.pxd file after the build of your software. > the cygcc is simply a wrapper over GCC that shoves in the fplugin > arguments so you can run: > > cygcc -g -O2 -Wall -c test.c -o test.o > > So build systems wont really notice anything different it should just > think its gcc. The configuration file example in the folder .cygcc: > > $ cat .cygcc > header = 'test.h' # the header to get declaratiosn for > inputs = ['test.h', 'test.c'] # any locations of declarations that > would be involved > output = 'test.pxd' # the output > > The cygcc compiler looks for this in the current director or looks up > at the parent directory untill it finds it or it will fail with an > error message. > > It will only ouput once if the test.pxd file exists in the current > working directory it wont do anything. > > $ cat test.pxd > cdef extern from "test.h": > int test (int, int) > int global_variable > int static_variable # this is a bug but hey... > struct bla: > int a > float b > > So far for the example code in there its working pretty well the gcc > python plugin has stabalized alot and so has some of gcc's bits a > bobs. I will try and keep you all updated i think the plugin is doing > alot more work now so this doesnt have to do so much. > > Thanks to Robert Bradshaw for baring with me in the whole thing > because i probably wasn't the easiest to work with ;) lol. > > Thanks everyone cython is awesome. > > --Phil >

1 0

Bug in C code generation
by Ulf Worsoe May 7, 2014

May 7, 2014

Hi, I have encounterered a bug in the following situation with Cython 0.20.1: cdef funname(): cdef numpy.ndarray[numpy.int32_t,ndim=1] myvar if True: ... block that never initializes myvar else ... block that initializes myvar The resulting C code does not declare the variable for the buffer object of myvar (my guess is that cython detects that it is never used and drops it), but the cleanup part of the C function contains code that frees the buffer, and thus refers to a variable that was never declared. I have not posted the complete code since the .pyx is around 50k lines long, but the offending method looks like this, and the problematic variable is _3_perm in line 4: --------------- cdef _toStringArray__3JJ_3S(self,numpy.ndarray[numpy.int64_t,ndim=1] _0_subi,numpy.int64_t _1_dstidx,numpy.ndarray[object,ndim=1] _2_result): assert _0_subi is None or isinstance(_0_subi,numpy.ndarray) assert _2_result is None or isinstance(_2_result,numpy.ndarray) cdef numpy.ndarray[numpy.int64_t,ndim=1] _3_perm cdef numpy.float64_t _4_c cdef numpy.int32_t _5_b cdef numpy.int32_t _6_ni cdef numpy.int64_t _7_i cdef numpy.int32_t _8_p cdef mosek_fusion_Utils_StringBuffer _9_sb for _10_i in xrange(0,(_0_subi).shape[0]): if ((_0_subi[_10_i] < 0) or (_0_subi[_10_i] >= (self.shape.size))): raise mosek_fusion_IndexError_ctor_S("Constraint index out of bounds") if (((_0_subi).shape[0] + _1_dstidx) > (_2_result).shape[0]): raise mosek_fusion_LengthError_ctor_S("Result array is too small to hold the result") _9_sb=mosek.fusion.Utils.StringBuffer() _8_p=0 if True: for _11_k in xrange(0,(_0_subi).shape[0]): _7_i=_0_subi[_11_k] _6_ni=self.nativeindexes[_7_i] _5_b=_8_p while ((_8_p < ((self.cache.subi)).shape[0]) and ((self.cache.subi)[_8_p] == _7_i)): _8_p += 1 _9_sb._clear_()._a_S(self.name)._a_S("[")._a_S(self.shape.indexToString(_7_i))._a_S("] : ") for _12_j in xrange(_5_b,_8_p): _4_c=(self.cache.cof)[_12_j] if (_4_c > 0): _9_sb._a_S(" + ")._a_D(_4_c)._a_S(" ") elif (_4_c < 0): _9_sb._a_S(" - ")._a_D((- _4_c))._a_S(" ") self.model._nativeVarToStr_ILmosek_4fusion_4Utils_4StringBuffer_2((self.cache.subj)[_12_j],_9_sb) if (self.cache_bfix != None): if (self.cache_bfix[_7_i] > 0): _9_sb._a_S(" + ")._a_D(self.cache_bfix[_7_i]) elif (self.cache_bfix[_7_i] < 0): _9_sb._a_S(" + ")._a_D(self.cache_bfix[_7_i]) _9_sb._a_S(" ") self._domainToString_JLmosek_4fusion_4Utils_4StringBuffer_2(_7_i,_9_sb) _2_result[(numpy.int64(_11_k) + _1_dstidx)] = _9_sb._toString_() else: _3_perm=numpy.zeros(((_0_subi).shape[0],), dtype=numpy.dtype(numpy.int64)) mosek_fusion_CommonTools_argQsort__3J_3J_3JJJ(_3_perm,_0_subi,None,0l,(_0_subi).shape[0]) --------------- -- Ulf Worsøe Mosek ApS ulf.worsoe(a)mosek.com www.mosek.com

2 1