Mailman 3 April 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

Fused type signature resolution failures
by Pauli Virtanen June 22, 2014

June 22, 2014

Hi, Here are the fused types + memoryview signature resolution failures I promised earlier (Cython 0.20.1): (A) TypeError: No matching signature ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused value_t: cnp.float32_t cnp.float64_t cpdef foo(value_t x): pass ----------------------------- quux.py import numpy as np import asd asd.foo(np.float32(1.0)) ----------------------------- (B) ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double' ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused idx_t: cnp.int32_t cnp.int64_t ctypedef fused value_t: cnp.int64_t cnp.float64_t cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x): pass ----------------------------- quux.py import numpy as np import asd i = np.zeros((3, 3), np.int64) j = np.zeros((3, 3), np.int64) x = np.zeros((3, 3), np.float64) asd.foo(i, j, x) ----------------------------- (C) Then some nasty platform-dependent failures: https://github.com/scipy/scipy/issues/3461 The relevant code is: https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L… The code looks nothing special. However, call to `lil_fancy_get` fails with "TypeError: No matching signature found" when the inputs have types <class 'int'> <class 'int'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> with ndarray dtypes: object object object object int32 int32 The failure occurs only on Christoph Gohlke's Win64 build, but not on Linux/OSX/MINGW32. This sounds like some integer size combination issue, but I'm far from sure. Unfortunately, I'm not easily able to isolate/understand what's going wrong here. -- Pauli Virtane

2 3

Bug in C code generation
by Ulf Worsoe May 7, 2014

May 7, 2014

Hi, I have encounterered a bug in the following situation with Cython 0.20.1: cdef funname(): cdef numpy.ndarray[numpy.int32_t,ndim=1] myvar if True: ... block that never initializes myvar else ... block that initializes myvar The resulting C code does not declare the variable for the buffer object of myvar (my guess is that cython detects that it is never used and drops it), but the cleanup part of the C function contains code that frees the buffer, and thus refers to a variable that was never declared. I have not posted the complete code since the .pyx is around 50k lines long, but the offending method looks like this, and the problematic variable is _3_perm in line 4: --------------- cdef _toStringArray__3JJ_3S(self,numpy.ndarray[numpy.int64_t,ndim=1] _0_subi,numpy.int64_t _1_dstidx,numpy.ndarray[object,ndim=1] _2_result): assert _0_subi is None or isinstance(_0_subi,numpy.ndarray) assert _2_result is None or isinstance(_2_result,numpy.ndarray) cdef numpy.ndarray[numpy.int64_t,ndim=1] _3_perm cdef numpy.float64_t _4_c cdef numpy.int32_t _5_b cdef numpy.int32_t _6_ni cdef numpy.int64_t _7_i cdef numpy.int32_t _8_p cdef mosek_fusion_Utils_StringBuffer _9_sb for _10_i in xrange(0,(_0_subi).shape[0]): if ((_0_subi[_10_i] < 0) or (_0_subi[_10_i] >= (self.shape.size))): raise mosek_fusion_IndexError_ctor_S("Constraint index out of bounds") if (((_0_subi).shape[0] + _1_dstidx) > (_2_result).shape[0]): raise mosek_fusion_LengthError_ctor_S("Result array is too small to hold the result") _9_sb=mosek.fusion.Utils.StringBuffer() _8_p=0 if True: for _11_k in xrange(0,(_0_subi).shape[0]): _7_i=_0_subi[_11_k] _6_ni=self.nativeindexes[_7_i] _5_b=_8_p while ((_8_p < ((self.cache.subi)).shape[0]) and ((self.cache.subi)[_8_p] == _7_i)): _8_p += 1 _9_sb._clear_()._a_S(self.name)._a_S("[")._a_S(self.shape.indexToString(_7_i))._a_S("] : ") for _12_j in xrange(_5_b,_8_p): _4_c=(self.cache.cof)[_12_j] if (_4_c > 0): _9_sb._a_S(" + ")._a_D(_4_c)._a_S(" ") elif (_4_c < 0): _9_sb._a_S(" - ")._a_D((- _4_c))._a_S(" ") self.model._nativeVarToStr_ILmosek_4fusion_4Utils_4StringBuffer_2((self.cache.subj)[_12_j],_9_sb) if (self.cache_bfix != None): if (self.cache_bfix[_7_i] > 0): _9_sb._a_S(" + ")._a_D(self.cache_bfix[_7_i]) elif (self.cache_bfix[_7_i] < 0): _9_sb._a_S(" + ")._a_D(self.cache_bfix[_7_i]) _9_sb._a_S(" ") self._domainToString_JLmosek_4fusion_4Utils_4StringBuffer_2(_7_i,_9_sb) _2_result[(numpy.int64(_11_k) + _1_dstidx)] = _9_sb._toString_() else: _3_perm=numpy.zeros(((_0_subi).shape[0],), dtype=numpy.dtype(numpy.int64)) mosek_fusion_CommonTools_argQsort__3J_3J_3JJJ(_3_perm,_0_subi,None,0l,(_0_subi).shape[0]) --------------- -- Ulf Worsøe Mosek ApS ulf.worsoe(a)mosek.com www.mosek.com

2 1

GCC Pxd
by Philip Herron April 30, 2014

April 30, 2014

Hey all, Some of you may remember a project i worked on for GSoC 2012, i wasn't really happy with the project over all for a long time it basicaly required at the time people to be running GCC from latest SVN/GIT for it to work correctly. Now gcc >= 4.7 is becoming more normal, so i went back into it and rewrote it from scratch now and its shaping up to look fairly decent and usable now. https://github.com/redbrain/cython/tree/master/Tools/gccpxd So the idea for this project is that you write a small configuration file and it will integrate with your C/C++ build system for example you could do: ./configure CC=cygcc ... make ... And you will get a project.pxd file after the build of your software. the cygcc is simply a wrapper over GCC that shoves in the fplugin arguments so you can run: cygcc -g -O2 -Wall -c test.c -o test.o So build systems wont really notice anything different it should just think its gcc. The configuration file example in the folder .cygcc: $ cat .cygcc header = 'test.h' # the header to get declaratiosn for inputs = ['test.h', 'test.c'] # any locations of declarations that would be involved output = 'test.pxd' # the output The cygcc compiler looks for this in the current director or looks up at the parent directory untill it finds it or it will fail with an error message. It will only ouput once if the test.pxd file exists in the current working directory it wont do anything. $ cat test.pxd cdef extern from "test.h": int test (int, int) int global_variable int static_variable # this is a bug but hey... struct bla: int a float b So far for the example code in there its working pretty well the gcc python plugin has stabalized alot and so has some of gcc's bits a bobs. I will try and keep you all updated i think the plugin is doing alot more work now so this doesnt have to do so much. Thanks to Robert Bradshaw for baring with me in the whole thing because i probably wasn't the easiest to work with ;) lol. Thanks everyone cython is awesome. --Phil

1 0

MPI_Type_create_hindexed_block() segfaults
by Lisandro Dalcin April 21, 2014

April 21, 2014

I believe the problem is in the following source code line (file:ompi_datatype_args.c, line:221): https://bitbucket.org/ompiteam/ompi-svn-mirror/src/v1.8/ompi/datatype/ompi_… I think you should just remove that bogus line, and that's all. [dalcinl@kw2060 openmpi]$ cat type_hindexed_block.c #include <mpi.h> int main(int argc, char *argv[]) { MPI_Aint disps[] = {0}; MPI_Datatype datatype; MPI_Init(&argc, &argv); MPI_Type_create_hindexed_block(1, 1, disps, MPI_BYTE, &datatype); MPI_Finalize(); return 0; } [dalcinl@kw2060 openmpi]$ mpicc type_hindexed_block.c [dalcinl@kw2060 openmpi]$ ./a.out [kw2060:20304] *** Process received signal *** [kw2060:20304] Signal: Segmentation fault (11) [kw2060:20304] Signal code: Address not mapped (1) [kw2060:20304] Failing at address: 0x6a [kw2060:20304] [ 0] /lib64/libpthread.so.0[0x327c40f750] [kw2060:20304] [ 1] /lib64/libc.so.6[0x327bc94126] [kw2060:20304] [ 2] /home/devel/mpi/openmpi/1.8.0/lib/libmpi.so.1(ompi_datatype_set_args+0x7f1)[0x7f8f0158b62a] [kw2060:20304] [ 3] /home/devel/mpi/openmpi/1.8.0/lib/libmpi.so.1(MPI_Type_create_hindexed_block+0x24d)[0x7f8f015cedc8] [kw2060:20304] [ 4] ./a.out[0x40080c] [kw2060:20304] [ 5] /lib64/libc.so.6(__libc_start_main+0xf5)[0x327bc21d65] [kw2060:20304] [ 6] ./a.out[0x4006f9] [kw2060:20304] *** End of error message *** Segmentation fault (core dumped) -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 1

Move IPython's cython extension to cython
by Martín Gaitán April 14, 2014

April 14, 2014

As you probably know, IPython has a "cython magic" [1], an extension to interface ipython with cython, allowing to use cython code in the interactive session. Originally, few of these "language specific magics" (R, cython, octave, etc) were included builtin in IPython, but then the IPython core developers decided break out them (with fair arguments IMHO [2]), either to be included in the "language package" they depend on or as a third party extensions. At the moment, cython-magic is the only one language specific extension left to move. The rest were moved to the packages they depend on [3] So, this message is to ask if the Cython project would like to import this code, task that Brian Grangrer (the author of the cython-magic extension and IPython core dev) entrusted me [3]. Of course, I could help you preparing a PR that migrate the module, its tests and documentation, preserving the git history as far as possible. let me know your opinion/decision cheers Martin. [1] http://ipython.org/ipython-doc/dev/config/extensions/cythonmagic.html [2] https://github.com/ipython/ipython/issues/3150 [3] https://github.com/ipython/ipython/issues/3803 [4] https://github.com/ipython/ipython/issues/3150#issuecomment-37203217 -- mgaitan.github.io textosyprextextos.com.ar -- mgaitan.github.io textosyprextextos.com.ar

3 6

memoryview slice transpose
by Syam Gadde April 7, 2014

April 7, 2014

I wanted to re-offer for discussion a fix for a bug I reported a while back. The code that breaks is this: import numpy cimport numpy def testfunc(): a = numpy.arange(12).reshape([3,4]) cdef numpy.int_t[:,:] a_view = a cdef numpy.int_t[:,:] b = a_view[:,:].T One problem is that when a memoryview slice is decremented, the .memview and .data fields are not set to NULL, and any error that causes a jump to cleanup code will attempt to decref it again. The second problem (I think) is that assignment of a transpose of memoryview slices to a variable is not resulting in an incref on the underlying slice, even though the variable is not a temporary and continues to persist. Here is a patch that solves the problem for me. https://github.com/SyamGadde/cython/commit/5739d8b908f18c4fc9103ef04e39964d… The part I am least sure about is removing the "if" qualifications on the incref and decref: if self.obj.is_name or self.obj.is_attribute and self.obj.is_memslice_transpose: that was obviously there for a reason, but it causes problems in the above case. If there is a less extreme solution I'd be happy to take it. Thanks, -syam

2 2