Mailman 3 June 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

Cython bugfix release
by Robert Bradshaw July 22, 2014

July 22, 2014

I just pushed another bugfix release for the 0.20.x line, available on github, cython.org, or and pypi. == Features added == * Some optimisations for set/frozenset instantiation. * Support for C++ unordered_set and unordered_map. == Bugs fixed == * Access to attributes of optimised builtin methods (e.g. [].append.__name__) could fail to compile. * Memory leak when extension subtypes add a memory view as attribute to those of the parent type without having Python object attributes or a user provided dealloc method. * Compiler crash on readonly properties in "binding" mode. * Auto-encoding with c_string_encoding=ascii failed in Py3.3. * Crash when subtyping freelist enabled Cython extension types with Python classes that use __slots__. * Freelist usage is restricted to CPython to avoid problems with other Python implementations. * Memory leak in memory views when copying overlapping, contiguous slices. * Format checking when requesting non-contiguous buffers from cython.array objects was disabled in Py3. * C++ destructor calls in extension types could fail to compile in clang. * Buffer format validation failed for sequences of strings in structs. * Docstrings on extension type attributes in .pxd files were rejected. == Contributors == Andreas van Cranenburgh Ian Bell Lars Buitinck Martin Quarda Mikhail Korobov Robert Bradshaw Stefan Behnel

3 7

Wrong order of __Pyx_DECREF when calling a function with an implicit str → char* conversion.
by Emmanuel Gil Peyrot July 6, 2014

July 6, 2014

Hi, I was testing my cython codebase on top of pypy/cpyext, and I found a memory corruption. After investigating it with the pypy guys (thanks arigato!), we identified it as a cython bug: cdef void test(char *string): print(string) def run(array): test(array[0]) This code works fine under cpython, but looking at the generated code for the last line, the call to test is done with a pointer (__pyx_t_2) to some potentially deallocated string (__pyx_t_1), which isn’t freed just yet on cpython but is on pypy: __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, …); __Pyx_GOTREF(__pyx_t_1); __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; __pyx_f_1a_test(__pyx_t_2); The obvious solution is to swap the last two lines, it then works fine on pypy (although not necessarily if the test function stores the pointer somewhere, but it’s not cython’s fault then). This issue can also happen with an explicit cast: pointer = <char *>array[0] test(pointer) I’m not sure if it should be addressed the same way, because that would mean keeping a reference to array[0] for all the lifetime of the current scope, but it could still prevent obscure bugs like the memory corruption I had. -- Emmanuel Gil Peyrot

3 5

NumPy 1.7+ warnings
by Stefan Behnel June 25, 2014

June 25, 2014

Hi, recent NumPy versions are a bit unhappy about Cython code and generate deprecation warnings about it: .../numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" I don't know how much work it would be to port the C code properly, but it should be mostly about updating the implementation in Cython's "numpy/__init__.pxd", which is dated anyway. Given that NumPy 1.7 appeared not much more than a year ago, dropping support for older versions sounds a bit aggressive, but there should be a way to keep supporting them by defining some macros, similar to what we do for CPython in ModuleSetupCode.c. Any volunteers? The relevant bits of NumPy's C-API are described here: http://docs.scipy.org/doc/numpy-dev/reference/c-api.array.html My guess is that most of the work should be replacing direct access to ndarray struct fields with C macro calls. That won't easily fix user code, but it's the first major step on the way. Stefan

2 1

Fused type signature resolution failures
by Pauli Virtanen June 22, 2014

June 22, 2014

Hi, Here are the fused types + memoryview signature resolution failures I promised earlier (Cython 0.20.1): (A) TypeError: No matching signature ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused value_t: cnp.float32_t cnp.float64_t cpdef foo(value_t x): pass ----------------------------- quux.py import numpy as np import asd asd.foo(np.float32(1.0)) ----------------------------- (B) ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double' ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused idx_t: cnp.int32_t cnp.int64_t ctypedef fused value_t: cnp.int64_t cnp.float64_t cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x): pass ----------------------------- quux.py import numpy as np import asd i = np.zeros((3, 3), np.int64) j = np.zeros((3, 3), np.int64) x = np.zeros((3, 3), np.float64) asd.foo(i, j, x) ----------------------------- (C) Then some nasty platform-dependent failures: https://github.com/scipy/scipy/issues/3461 The relevant code is: https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L… The code looks nothing special. However, call to `lil_fancy_get` fails with "TypeError: No matching signature found" when the inputs have types <class 'int'> <class 'int'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> with ndarray dtypes: object object object object int32 int32 The failure occurs only on Christoph Gohlke's Win64 build, but not on Linux/OSX/MINGW32. This sounds like some integer size combination issue, but I'm far from sure. Unfortunately, I'm not easily able to isolate/understand what's going wrong here. -- Pauli Virtane

2 3

any more changes for 0.21?
by Stefan Behnel June 18, 2014

June 18, 2014

Hi, I think the master branch is good enough for at least a 0.21 alpha. Are there any pending changes that should go in before it can be released? Any pull requests that should be considered? We should get this in, I think: https://github.com/cython/cython/pull/284 And the gdb tests are still broken. Robert, could you take a look through the Sage test failures? Most of them might be issues in Sage rather than Cython, but it's better to make sure we can either fix or safely ignore them. Stefan

2 4

JyNI - C-API emulation for Jython
by Stefan Behnel June 16, 2014

June 16, 2014

Hi, it looks like there is a C-API emulation for Jython on the way: http://jyni.org/ https://github.com/Stewori/JyNI I haven't tried it yet, and I'm sure Cython modules won't work with its current state, but the effort I put into making them run in PyPy's cpyext should pay off here, so we might at some point have a third backend to target. This also reminded me of IronClad, the C-API layer for IronPython. I looked it up, but it seems to have died some years ago, with no current effort to improve it or even just make it work with the latest IronPython releases. https://groups.google.com/d/msg/c-extensions-for-ironpython/SrIesUIh1Gw/cBQ… https://code.google.com/p/ironclad/ Not sure if it's still worth trying to make that run in Cython. Stefan

2 1

Cython 0.20.2 beta
by Robert Bradshaw June 4, 2014

June 4, 2014

I've pushed a beta for the 0.20.2 can be found at http://cython.org/release/Cython-0.20rc1.tar.gz . This is a bugfix-only release, more development is going on for 0.21 which will also be released soon. Note, however, that the 0.20.x branch is the last to support older versions of Python (pre 2.6 or 3.2). Please try it out and report back. - Robert Features added Some optimisations for set/frozenset instantiation. Support for C++ unordered_set and unordered_map. Bugs fixed Access to attributes of optimised builtin methods (e.g. [].append.__name__) could fail to compile. Memory leak when extension subtypes add a memory view as attribute to those of the parent type without having Python object attributes or a user provided dealloc method. Compiler crash on readonly properties in "binding" mode. Auto-encoding with c_string_encoding=ascii failed in Py3.3. Crash when subtyping freelist enabled Cython extension types with Python classes that use __slots__. Freelist usage is restricted to CPython to avoid problems with other Python implementations. Memory leak in memory views when copying overlapping, contiguous slices. Format checking when requesting non-contiguous buffers from cython.array objects was disabled in Py3. C++ destructor calls in extension types could fail to compile in clang. Buffer format validation failed for sequences of strings in structs. Docstrings on extension type attributes in .pxd files were rejected.

1 1