Mailman 3 July 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Automatic conversion with fixed-size C arrays
by Kurt Smith Oct. 25, 2014

Oct. 25, 2014

Hi devs, Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. It seems like the following struct should be convertible safely: cdef struct state_t: int i, j, k float x[3] float v[3] Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well: def auto_convert(list ll): cdef int arr[10] = ll # ... return arr If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this. Thoughts? Kurt

4 17

Travis-ci builds of OSX wheels
by Matthew Brett Oct. 16, 2014

Oct. 16, 2014

Hi, We just got scikit-image OSX wheel builds more or less fully automated on travis-ci : https://travis-ci.org/scikit-image/scikit-image-wheels https://github.com/scikit-image/scikit-image-wheels Any interest in getting the same thing working for Cython? Cheers, Matthew

2 8

Cython 0.21 alpha 1 released
by Stefan Behnel Sept. 8, 2014

Sept. 8, 2014

Hi, I uploaded a first alpha release of Cython 0.21. http://cython.org/release/Cython-0.21a1.tar.gz http://cython.org/release/Cython-0.21a1.tar.gz.asc There were a couple of major changes in this version, including internal improvements to some basic syntax structures, so please give it a try with your code to help us fix some more bugs before the final release. Complete changelog: https://github.com/cython/cython/blob/deb348795f79342843e692358140ea4581985… Have fun, Stefan

2 8

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

heads-up on master branch status
by Stefan Behnel July 28, 2014

July 28, 2014

Hi, by applying some initially small but perpetually growing fixes, I pushed myself into a corner where the only escape was to rewrite BoolBinopNode (short-circuiting and/or expressions). This was long overdue anyway and simply wasn't done before because it's not trivial and quite risky. I think I got it working better than before, but there are still corner cases that require some further fighting. One of them currently crashes the pyregr tests in Py2.7, it's this code: [tuple((e is None and 'X' or e) for e in t) for t in target] Thank you, python-devs, for collecting these little gems in the test suite. >From the C code this generates, the problem seems to be related to type coercion in closures. I'm currently preparing my participation at EuroPython next week, so I can't tell for sure when I'll be able to dig into this. This means that current master may stay less stable than it should be for a couple of more days. Stefan

3 3

Compiler crash: forward declaration of cdef class with "public" attributes
by Alok Singhal July 23, 2014

July 23, 2014

Hi, I am getting a compiler crash with the following code: $ cat foo.pxd cdef class Foo: cdef public object x $ cat foo.pyx cdef class Foo cdef class Foo: pass $ ./cython.py foo.pyx Error compiling Cython file: ------------------------------------------------------------ ... cdef class Foo ^ ------------------------------------------------------------ foo.pyx:1:5: Compiler crash in AnalyseDeclarationsTransform ModuleNode.body = StatListNode(foo.pyx:1:0) StatListNode.stats[0] = CClassDefNode(foo.pyx:1:5, as_name = u'Foo', class_name = u'Foo', module_name = '', visibility = 'private') Compiler crash traceback from this point on: File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/Visitor.py", line 173, in _visit return handler_method(obj) File "/Users/alok/Downloads/repos/cython.git/Cython/Compiler/ParseTreeTransforms.py", line 1478, in visit_CClassDefNode node.body.stats += stats AttributeError: 'NoneType' object has no attribute 'stats' The problem seems to be due to having a .pxd file with the attributes of the cdef class in it. Since the attributes are declared "public", Cython is trying to generate property code for "Foo.x". But it's trying to do that at the point of forward declaration in the .pyx file, instead of the definition of the class. An easy fix seems to be to change AnalyseDeclarationsTransform.visit_CClassDefNode in ParseTreeTransforms.py to check for node.body not being none. I.e., the following patch seems to work: diff --git a/Cython/Compiler/ParseTreeTransforms.py b/Cython/Compiler/ParseTreeTransforms.py index f8c2efa..7811d56 100644 --- a/Cython/Compiler/ParseTreeTransforms.py +++ b/Cython/Compiler/ParseTreeTransforms.py @@ -1466,7 +1466,7 @@ if VALUE is not None: def visit_CClassDefNode(self, node): node = self.visit_ClassDefNode(node) - if node.scope and node.scope.implemented: + if node.body and node.scope and node.scope.implemented: stats = [] for entry in node.scope.var_entries: if entry.needs_property: Thanks, Alok

2 4

License information on each individual files
by Benjamin Lerman July 23, 2014

July 23, 2014

Hi, I'm planning to use cython in chromium. To be able to use cython as a third-party library, we need a license header on every files in the project (because we are creating source packages, and need to have clear licensing). Would cython accept to add such a copyright header on its files? Thanks in advance, -- Benjamin

6 11

Cython bugfix release
by Robert Bradshaw July 22, 2014

July 22, 2014

I just pushed another bugfix release for the 0.20.x line, available on github, cython.org, or and pypi. == Features added == * Some optimisations for set/frozenset instantiation. * Support for C++ unordered_set and unordered_map. == Bugs fixed == * Access to attributes of optimised builtin methods (e.g. [].append.__name__) could fail to compile. * Memory leak when extension subtypes add a memory view as attribute to those of the parent type without having Python object attributes or a user provided dealloc method. * Compiler crash on readonly properties in "binding" mode. * Auto-encoding with c_string_encoding=ascii failed in Py3.3. * Crash when subtyping freelist enabled Cython extension types with Python classes that use __slots__. * Freelist usage is restricted to CPython to avoid problems with other Python implementations. * Memory leak in memory views when copying overlapping, contiguous slices. * Format checking when requesting non-contiguous buffers from cython.array objects was disabled in Py3. * C++ destructor calls in extension types could fail to compile in clang. * Buffer format validation failed for sequences of strings in structs. * Docstrings on extension type attributes in .pxd files were rejected. == Contributors == Andreas van Cranenburgh Ian Bell Lars Buitinck Martin Quarda Mikhail Korobov Robert Bradshaw Stefan Behnel

3 7

buffmt.__test__.wrongsize gets stuck after common_include_dir?
by Yaroslav Halchenko July 22, 2014

July 22, 2014

while trying to re-test Debian package build, in an interactive shell, I get cython test stuck at: ... Doctest: buffmt.__test__.wrongsize ... ok runTest (__main__.EndToEndTest) End-to-end basic_cythonize ... ok runTest (__main__.EndToEndTest) End-to-end basic_distutils ... ok runTest (__main__.EndToEndTest) End-to-end build_dir ... ok runTest (__main__.EndToEndTest) End-to-end common_include_dir ... tests were executed using runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir any clues? is it multiprocessing to blame again? /bin/bash /tmp/hooks/C10shell /bin/bash /usr/bin/make -f debian/rules binary /usr/bin/perl -w /usr/bin/dh binary --with python2,python3 --buildsystem python_distutils /usr/bin/make -f debian/rules override_dh_auto_test /bin/sh -c set -e; for P in 2.7 3.4; do \ echo "PYTHON: $P"; \ export; \ PYTHONPATH=`/bin/ls -d /tmp/buildd/cython-0.20.2/build/lib.*-$P` \ /usr/bin/python$P runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir; \ done /usr/bin/python2.7 runtests.py --no-refnanny -v -v --exclude=parallel --work-dir=build/work-dir /bin/sh -c /usr/bin/python2.7 setup.py build_ext --inplace /usr/bin/python2.7 setup.py build_ext --inplace /usr/bin/python2.7 setup.py build_ext --inplace /usr/bin/python2.7 setup.py build_ext --inplace -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Research Scientist, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

1 0