Mailman 3 December 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

releasing 0.18?
by Stefan Behnel Jan. 22, 2013

Jan. 22, 2013

Hi, the current master doesn't really have the one new, great&shiny feature, but I don't see a reason to keep the changes back that we have implemented so far. Robert's const support and the integer overflow checks (assuming that both are ready) are certainly worth it. Unless Mark's minivect changes are ready to merge soonish, I don't think we should keep waiting for them. So, what about getting the master ready for a release? Note that this means that we need a couple of more comments in the changelog, as well as documentation for the things that need it (const?). Stefan

4 7

Cython 0.17.2 released
by Stefan Behnel Jan. 8, 2013

Jan. 8, 2013

Hi everyone, I'm happy to announce the release of Cython 0.17.2. This is a (mostly) bug fix release for the stable 0.17 release series. http://pypi.python.org/pypi/Cython/0.17.2 Direct downloads: http://cython.org/release/Cython-0.17.2.tar.gz http://cython.org/release/Cython-0.17.2.zip Github version: https://github.com/cython/cython/commit/10183d61eb5cf33e6912dec2ab09740498f… Have fun, Stefan Complete changelog: 0.17.2 (2012-11-20) =================== Features added -------------- * ``cythonize()`` gained a best effort compile mode that can be used to simply ignore .py files that fail to compile. Bugs fixed ---------- * Replacing an object reference with the value of one of its cdef attributes could generate incorrect C code that accessed the object after deleting its last reference. * C-to-Python type coercions during cascaded comparisons could generate invalid C code, specifically when using the 'in' operator. * "obj[1,]" passed a single integer into the item getter instead of a tuple. * Cyclic imports at module init time did not work in Py3. * The names of C++ destructors for template classes were built incorrectly. * In pure mode, type casts in Cython syntax and the C ampersand operator are now rejected. Use the pure mode replacements instead. * In pure mode, C type names and the sizeof() function are no longer recognised as such and can be used as normal Python names. * The extended C level support for the CPython array type was declared too late to be used by user defined classes. * C++ class nesting was broken. * Better checking for required nullary constructors for stack-allocated C++ instances. * Remove module docstring in no-docstring mode. * Fix specialization for varargs function signatures. * Fix several compiler crashes.

3 6

Travis-CI builds for Cython
by Stefan Behnel Dec. 21, 2012

Dec. 21, 2012

Hi, I've set up travis builds for Cython's master branch: https://travis-ci.org/cython/cython There isn't currently any advantage over Jenkins (they run on 32bit Ubuntu Linux) and in fact, it's even less interesting because they don't support Py2.4 nor Py3.1, they only use CPython/PyPy release versions (not the latest developer versions as we do on Jenkins), and their Py2.5 installation even seems to be a bit broken (the embed test doesn't link against zlib automatically in that version). So, I'm not sure it's worth it, but it shouldn't hurt either. Personally, I find the Debian builds way more interesting, but their sources need to be updated manually: https://buildd.debian.org/status/package.php?p=cython&suite=experimental Stefan

1 0

Re: [Cython] [cython] Fix __Pyx_ImportType if #Py_LIMITED_API is defined. (#171)
by Stefan Behnel Dec. 20, 2012

Dec. 20, 2012

Hi Bradley, Bradley M. Froehle, 19.12.2012 21:31: > I was recently trying to use mpi4py in a hand-written Python 3 extension > module with `#define Py_LIMITED_API`. Work on getting Cython play nicely with Py_LIMITED_API is certainly appreciated. In most cases, it should be as simple as adding another #ifdef to utility functions that are already written in plain C anyway. A lot of the optimisations will need this. > One side effect of the limited > API is that `PyTypeObject` becomes an opaque type. > This means that our `__Pyx_ImportType` cannot just introspect > `tp_basicsize`. > > I've updated the code to extract the `tp_basicsize` value from the > `__basicsize__` attribute which should be available instead. > > I'd appreciate comments on the code style here... I'm not sure how to > write these sorts of calls most efficiently yet. I think it's ok. The only issue is that PyObject_GetAttrString() creates an intermediate string object from the char* you pass. Py3.3 (or 3.2+?) has a mechanism for avoiding this. We could start using that if available, or maybe even copy it over into Cython to use it in older versions as well. Anyway, this is code that runs at import time - I don't think it's any performance critical (and if a whole import works, the memory allocated for a tiny string can't be an issue either...). Stefan

2 1

Cython is not PEP-420 (Implicit Namespace Packages) compliant
by Alexey V Gorshkov Dec. 15, 2012

Dec. 15, 2012

Hi! Reporting about Cython PEP-420 (Implicit Namespace Packages) incompliance. (Sample package setup attached) (http://www.python.org/dev/peps/pep-0420/) Next package's tree will not work and will return error . ./setup.py ./firstlevel ./firstlevel/secondlevel ./firstlevel/secondlevel/mydearpackage ./firstlevel/secondlevel/mydearpackage/__init__.py ./firstlevel/secondlevel/mydearpackage/test.pyx ./firstlevel/secondlevel/mydearpackage/test_h.pxd ===== Error Message ===== > python3 ./setup.py build_ext --inplace running build_ext cythoning firstlevel/secondlevel/mydearpackage/test.pyx to firstlevel/secondlevel/mydearpackage/test.c Error compiling Cython file: ------------------------------------------------------------ ... cimport firstlevel.secondlevel.mydearpackage.test_h ^ ------------------------------------------------------------ firstlevel/secondlevel/mydearpackage/test.pyx:1:8: 'firstlevel.secondlevel.mydearpackage.test_h.pxd' not found building 'firstlevel.secondlevel.mydearpackage.test' extension creating build creating build/temp.linux-i686-3.3 creating build/temp.linux-i686-3.3/firstlevel creating build/temp.linux-i686-3.3/firstlevel/secondlevel creating build/temp.linux-i686-3.3/firstlevel/secondlevel/mydearpackage i486-pc-linux-gnu-gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python3.3m -c firstlevel/secondlevel/mydearpackage/test.c -o build/temp.linux-i686-3.3/firstlevel/secondlevel/mydearpackage/test.o firstlevel/secondlevel/mydearpackage/test.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation. error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 ===== Error Message End ===== Simply adding __init__.py to firstlevel and firstlevel/secondlevel, and there will no building error ===== Normal Success Message ===== >python3 ./setup.py build_ext --inplace running build_ext cythoning firstlevel/secondlevel/mydearpackage/test.pyx to firstlevel/secondlevel/mydearpackage/test.c building 'firstlevel.secondlevel.mydearpackage.test' extension i486-pc-linux-gnu-gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python3.3m -c firstlevel/secondlevel/mydearpackage/test.c -o build/temp.linux-i686-3.3/firstlevel/secondlevel/mydearpackage/test.o i486-pc-linux-gnu-gcc -pthread -shared build/temp.linux-i686-3.3/firstlevel/secondlevel/mydearpackage/test.o -L/usr/lib -lpython3.3m -o /mnt/sda3/home/agu/p/pep420test/firstlevel/secondlevel/mydearpackage/test.cpython-33m.so ===== Normal Success Message End =====

2 2

Cython 0.17.3 released
by Stefan Behnel Dec. 14, 2012

Dec. 14, 2012

Hi all, I'm happy to announce the next bug fix release of the current stable Cython 0.17 series: 0.17.3. It fixes two crash bugs, for weakrefs and for the cleanup code on interpreter shutdown, so if you are using these features, you might want to upgrade. http://cython.org/release/Cython-0.17.3.tar.gz http://cython.org/release/Cython-0.17.3.zip github: https://github.com/cython/cython/commit/1da2e2fa78504f2a01a999e6549d14219af… The release notes for the 0.17 series are here: https://github.com/cython/cython/blob/0.17/CHANGES.rst Have fun, Stefan Bugs fixed ---------- * During final interpreter cleanup (with types cleanup enabled at compile time), extension types that inherit from base types over more than one level that were cimported from other modules could lead to a crash. * Weak-reference support in extension types (with a "cdef __weakref__" attribute) generated incorrect deallocation code. * In CPython 3.3, converting a Unicode character to the Py_UNICODE type could fail to raise an overflow for non-BMP characters that do not fit into a wchar_t on the current platform. * Negative C integer constants lost their longness suffix in the generated C code.

1 0

unused function '__pyx_pw_5numpy_7ndarray_1__getbuffer__'
by Bradley M. Froehle Dec. 10, 2012

Dec. 10, 2012

I've been noticing some unused function warnings of the following variety: test.cpp:1189:12: warning: unused function > '__pyx_pw_5numpy_7ndarray_1__getbuffer__' [-Wunused-function] > static int __pyx_pw_5numpy_7ndarray_1__getbuffer__(PyObject *__pyx_v_self, > Py_buffer *__pyx_v_info, int __pyx_v_flags) { > ^ > test.cpp:2008:13: warning: unused function > '__pyx_pw_5numpy_7ndarray_3__releasebuffer__' [-Wunused-function] > static void __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject > *__pyx_v_self, Py_buffer *__pyx_v_info) { > ^ > 2 warnings generated. ---- test.pyx ---- from numpy cimport import_array, import_ufunc import_array(); import_ufunc() These warnings were once silenced by some code introduced in a previous commit: commit 6cc61605a91c8ebf16318a07439a206118d81614 > Author: Lisandro Dalcin <dalcinl(a)gmail.com> > Date: Wed Apr 28 16:18:43 2010 -0300 > introduce CYTHON_UNUSED macro to annotate functions and parametes > - Silent compiler warnings about unused function/parametes > - Defined to __attribute__((__unused__)) for GCC(>3.4) and ICC > - Applied to a __{get|release}buffer__ special methods > - Applied to a couple of (self,unused) parameter pairs But later reappeared: commit 519a19a9006494c6165ea481553a298d9bf9a008 > Author: Vitja Makarov <vitja.makarov(a)gmail.com> > Date: Sun Jan 8 19:22:38 2012 +0400 > DefNode: split into function and python wrapper I've introduced a pull request to re-silence these warnings: https://github.com/cython/cython/pull/168 -Brad

2 1

(no subject)
by Yaroslav Halchenko Dec. 10, 2012

Dec. 10, 2012

reproduced with cython 0.17.2 (+ few post release fixes), originally detected/reported [1] with 0.17.1 on Debian systems using debug build of Python: $> python-dbg -c 'import pyximport as pi; pi.install(); import weakfail; s=weakfail.foo(42)' python-dbg: ../Modules/gcmodule.c:366: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. [1] 12648 abort python-dbg -c $> cat weakfail.pyx import weakref foo_dict = weakref.WeakValueDictionary() cdef class Foo: cdef object __weakref__ def foo(key): obj = Foo() foo_dict[key] = obj return obj it seems to work fine with cython 0.15.2 . Any ideas? requires separate/new bug report? [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=692313 -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

4 9

Broken C++ compile test
by Stefan Behnel Dec. 7, 2012

Dec. 7, 2012

Hi, the question by Czarek Tomczak on the users list made me stumble into this old C++ test: https://github.com/cython/cython/blob/master/tests/compile/cpp_operators.pyx It's completely broken and only failed to fail because it was never executed. Apparently, the test runner shadowed it with the test of the same name in the tests/run/ directory. Stefan

2 2