Mailman 3 December 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Cythonize silently ignores nonexistent files
by Kevin Norris Jan. 9, 2015

Jan. 9, 2015

Is this behavior intentional? >>> from Cython.Build import cythonize >>> cythonize('/this/file/doesnt/exist.pyx') [] It would be *really* nice if Cython would raise an exception (or something) in this case. I just spent 20 minutes trying to puzzle out Cython/Setuptools compatibility when the actual problem was staring me in the face. -- Kevin Norris

3 2

Cython 0.21.2 released
by Stefan Behnel Dec. 27, 2014

Dec. 27, 2014

Hi everyone, I uploaded Cython 0.21.2. It adds some bug fixes and corrections for the 0.21.x release series. Downloads: https://pypi.python.org/pypi/Cython/0.21.2 http://cython.org/release/Cython-0.21.2.tar.gz http://cython.org/release/Cython-0.21.2.zip Changelog: https://github.com/cython/cython/blob/efdbeef340dfe8ad748ff0c305e1e424932e2… Documentation: http://docs.cython.org/ Stefan Bugs fixed ---------- * Crash when assigning a C value to both a Python and C target at the same time. * Automatic coercion from C++ strings to ``str`` generated incomplete code that failed to compile. * Declaring a constructor in a C++ child class erroneously required a default constructor declaration in the super class. * ``resize_smart()`` in ``cpython.array`` was broken. * Functions in ``libcpp.cast`` are now declared as ``nogil``. * Some missing C-API declarations were added. * Py3 main code in embedding program code was lacking casts. * Exception related to distutils "Distribution" class type in pyximport under latest CPython 2.7 and 3.4 releases when setuptools is being imported later.

1 0

Re: [Cython] [cython-users] is there a schedule for next cython update (0.21.2) ?
by Stefan Behnel Dec. 20, 2014

Dec. 20, 2014

stonebig34(a)gmail.com schrieb am 20.12.2014 um 10:14: > I see a lot of commits since 0.21.1. Well, there are about five commits in the 0.21.x branch. > Wouldn't it be time to do an incremental release ? Yes, getting these fixes out would be good. Also, the master branch looks pretty ok, it might be ready for a beta release as well. Stefan

1 0

GvR on type hinting in Python
by Stefan Behnel Dec. 20, 2014

Dec. 20, 2014

Looks like type annotations in Python are about to leave the "maybe one day" status: http://thread.gmane.org/gmane.comp.python.ideas/30432 Stefan

1 0

cygdb bugs when debugging python 3 extensions
by Ben Booth Dec. 12, 2014

Dec. 12, 2014

I'm trying to figure out how to use cygdb to debug extensions built with python 3. I've tested all the debugger commands listed in http://docs.cython.org/src/userguide/debugging.html <http://docs.cython.org/src/userguide/debugging.html> when debugging python 2 extensions and everything works great. But some of the commands seem to be broken when debugging python 3 extensions. Here are the debugger commands that I've confirmed as working with python 3 extensions: cy break cy step cy next cy run cy cont cy down cy finish .. works but prints error message cy exec cy list cy_lineno() cy_eval() cy_cname() Here are the commands which fail under python 3, and the error message that is given for each: cy up Python Exception <class 'gdb.error'> There is no member named length.: Error occurred in Python command: There is no member named length. cy finish .. works but gives error Python Exception <class 'gdb.error'> There is no member named length.: Error occurred in Python command: There is no member named length. cy bt Python Exception <class 'gdb.error'> There is no member named length.: Error occurred in Python command: There is no member named length. cy print Python Exception <class 'gdb.error'> There is no member named ma_mask.: Python Exception <class 'gdb.error'> Error occurred in Python command: There is no member named ma_mask.: Error occurred in Python command: Error occurred in Python command: There is no member named ma_mask. cy set x = 2 Python Exception <class 'gdb.error'> No symbol "__pyx_v_10helloworld_x" in current context.: Error occurred in Python command: No symbol "__pyx_v_10helloworld_x" in current context. cy locals Python Exception <class 'gdb.error'> There is no member named ma_mask.: Error occurred in Python command: There is no member named ma_mask. cy globals Python Exception <class 'gdb.error'> There is no member named ma_mask.: Error occurred in Python command: There is no member named ma_mask. cy_cvalue() Python Exception <class 'gdb.error'> There is no member named ma_mask.: Error occurred in Python convenience function: There is no member named ma_mask. In summary: - The "cy set" and "cy_cname()" commands seem to be giving incorrect C names for cython variables. I did a grep of the generated .c source file for "__pyx_v_10helloworld_x", and didn't get any hits. Luckily, the "cy exec" command works, so I can use that to get the value of variables as a workaround. - Any commands related to viewing or traversing the call stack seem to give an error related to a missing "length" member. - Commands related to printing the value of a variable complain about a missing "ma_mask" member. Here are the source files I used for the test: helloworld.py: import helloworld helloworld.pyx: x = 1 y = "a" print("Hello World 1") print("Hello World 2") print("Hello World 3") print("Hello World 4") print("Hello World 5") print("Hello World 6") print("Hello World 8") print("Hello World 9") print("Hello World 10") setup.py: from distutils.core import setup from Cython.Build import cythonize from distutils.extension import Extension extensions = [Extension("helloworld", ["helloworld.pyx"])] setup(ext_modules=cythonize(extensions, gdb_debug=True, output_dir=".")) To compile I ran: python setup.py build_ext --inplace To debug, I ran: cygdb -vv . -- --args python helloworld.py Here is my python version: Python 3.4.2 I used pyenv (https://github.com/yyuu/pyenv <https://github.com/yyuu/pyenv>) to build and install the latest Python 3 version with debugging symbols enabled. Ben

1 0

Cython <> pythran collaboration
by serge Guelton Dec. 12, 2014

Dec. 12, 2014

Hi Cython dev, I am one of the two core developers of the Pythran[0] compiler. We are planning to start a project where Cython would delegate the generation of optimized (i.e. parallel / vectorized / without temporary arrays) numpy expression when it finds one, instead of calling the native API. Something like the following meaningless example: import numpy as np cimport numpy as np def play_with_arrays(np.ndarray arr): cdef int N = arr.shape[0] cdef np.ndarray other_arr = np.empty([N], dtype=np.int) other_arr = arr ** 2 + 1 # pythran would handle this for i in range(N): other_arr[i] /= 2 # cython would handle this return other_arr Everything would live in a pyx file and it should be transparent to a cython user. Cython would drive the compilation process, using pythran as a numpy expression optimizer. Does that look feasible / appealing to you? [0] http://pythonhosted.org/pythran/

3 2

Can't assign numpy array to memoryview
by Dave Hirschfeld Dec. 8, 2014

Dec. 8, 2014

It appears that you can't assign an ndarray to a memoryview slice? Is this correct/expected behaviour? Whilst there are a couple of possible workarounds it would be nice if this just worked as it's a slightly surprising deviation from the ndarray behaviour. Broken example: ``` In [3]: %%cython ...: cimport cython ...: import numpy as np ...: cimport numpy as np ...: ...: cpdef np.ndarray[np.float64_t, ndim=2] f(): ...: cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], dtype=np.float64) ...: cdef double[:,:] y = np.zeros([3, 4], dtype=np.float64) ...: cdef np.ndarray[np.float64_t, ndim=1] row ...: cdef int idx ...: for idx, row in enumerate(x): ...: y[idx] = row ...: return np.asarray(y) In [4]: f() Traceback (most recent call last): File "<ipython-input-4-0ec059b9bfe1>", line 1, in <module> f() File "_cython_magic_5f2586693ddbf044815dae01d800bc0c.pyx", line 5, in _cython_magic_5f2586693ddbf044815dae01d800bc0c.f (C:\Users\dhirschfeld\.ipython\cython\_cython_magic_5f2586693ddbf044815d ae01d800bc0c.c:2086) File "_cython_magic_5f2586693ddbf044815dae01d800bc0c.pyx", line 11, in _cython_magic_5f2586693ddbf044815dae01d800bc0c.f (C:\Users\dhirschfeld\.ipython\cython\_cython_magic_5f2586693ddbf044815d ae01d800bc0c.c:1935) ``` Workarounds - either: 1. Type both lhs and rhs as ndarray 2. Create a 1D memoryview to use as a temporary container and assign that to the memoryview slice ``` In [5]: %%cython ...: cimport cython ...: import numpy as np ...: cimport numpy as np ...: ...: cpdef np.ndarray[np.float64_t, ndim=2] f(): ...: cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], dtype=np.float64) ...: cdef np.ndarray[np.float64_t, ndim=2] y = np.zeros([3, 4], dtype=np.float64) ...: cdef np.ndarray[np.float64_t, ndim=1] row ...: cdef int idx ...: for idx, row in enumerate(x): ...: y[idx] = row ...: return np.asarray(y) In [6]: f() Out[6]: array([[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]) In [7]: %%cython ...: cimport cython ...: import numpy as np ...: cimport numpy as np ...: ...: cpdef np.ndarray[np.float64_t, ndim=2] f(): ...: cdef np.ndarray[np.float64_t, ndim=2] x = np.ones([3, 4], dtype=np.float64) ...: cdef double[:,:] y = np.zeros([3, 4], dtype=np.float64) ...: cdef np.ndarray[np.float64_t, ndim=1] row ...: cdef double[:] tmp ...: cdef int idx ...: for idx, row in enumerate(x): ...: tmp = row ...: y[idx] = tmp ...: return np.asarray(y) In [8]: f() Out[8]: array([[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]) ```

1 1

Possible bug (or wrong documentation) WRT "not None"
by Samuele Kaplun Dec. 4, 2014

Dec. 4, 2014

Hi! I would like to report what I think is a possible bug (in cython or corresponding documentation). According to: <http://docs.cython.org/src/userguide/extension_types.html#extension-types-a…> [...] The self parameter of a method of an extension type is guaranteed never to be None. [...] However given this snippet: [...] cdef class test: def __sub__(self, test rhs not None): print "self is %s" % repr(self), "rhs is %s" % repr(rhs) [...] If I then load it python: In [1]: import test In [2]: x = test.test() In [3]: None - x self is None rhs is <test.test object at 0x7fdd9eeb7410> Thus showing that in this case "self" was actually initialized with None. If I explictly add "not None" as in: [...] cdef class test: def __sub__(self not None, test rhs not None): print "self is %s" % repr(self), "rhs is %s" % repr(rhs) [...] In [3]: None - x TypeError: Argument 'self' must not be None [...] Which is good. If I instead add "test self" as in: [...] cdef class test: def __sub__(test self not None, test rhs not None): print "self is %s" % repr(self), "rhs is %s" % repr(rhs) [...] In [3]: None - x TypeError: Argument 'self' has incorrect type (expected test.test, got NoneType) [...] What's worse is that if I use "test self" (without not None), the same TypeError is not raised: [...] cdef class test: def __sub__(test self, test rhs not None): print "self is %s" % repr(self), "rhs is %s" % repr(rhs) [...] In [3]: None - x self is None rhs is <test.test object at 0x7f17b36ff3f0> Thus we have: self -> not respecting None-checking (in disagreement with Documentation) self not None -> OK test self -> not respecting type-checking! test self non None -> OK (respecting type-checking, hence None-checking). Hope that helps. Cheers, Samuele -- Samuele Kaplun Invenio Developer ** <http://invenio-software.org/>

3 3

cython --cplus --embed generates invalid code
by Michael Enßlin Dec. 1, 2014

Dec. 1, 2014

Hi, on my system, --embed does not work with --cplus. How to reproduce: Any valid pyx file works: $ rm -f test.pyx; touch test.pyx $ cython --embed --cplus test.pyx $ g++ -c test.cpp -I/usr/include/python3.4m test.cpp: In function ‘wchar_t* __Pyx_char2wchar(char*)’: test.cpp:945:41: error: invalid conversion from ‘void*’ to ‘wchar_t*’ [-fpermissive] res = malloc(argsize*sizeof(wchar_t)); $ clang++ -c test.cpp -I/usr/include/python3.4m test.cpp:945:9: error: assigning to 'wchar_t *' from incompatible type 'void *' res = malloc(argsize*sizeof(wchar_t)); The issue can easily be fixed by manually casting the malloc result to (wchar_t *). In C it is not recommended to cast the malloc result, while in C++ it is required (and malloc is discouraged). System info: $ cython --version Cython version 0.21.1 $ g++ --version g++ (Debian 4.9.2-2) 4.9.2 $ python3 --version Python 3.4.2 $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux unstable (sid) Release: unstable Codename: sid ~ Michael

2 1