Mailman 3 July 2015 - cython-devel

Non-type template parameters
by Ian Henriksen Oct. 1, 2015

Oct. 1, 2015

Hi all, I've spent a little time working on adding support for non-type template parameters. In doing this, it has been very easy to add support for doing something like the following: cdef extern from "add_const.hpp" nogil: int myfunc1[i](int) def test(): print myfunc1[2](a) The downside of this is that it does not let the Cython compiler distinguish between different kinds of template parameters. Stricter checking could be added using a syntax like this: cdef extern from "add_const.hpp" nogil: int myfunc1[int i](int) def test(): print myfunc1[2](a) The downsides here are that the syntax doesn't really match the existing template syntax. It will also complicate the Cython codebase since we'll have to go to greater lengths to allow or disallow all the different special cases for templates. Which version would be preferable? On a similar note, for variadic templates, would we prefer something like cdef extern from "my_variadic.hpp" nogil: T myfunc2[T,...](T, ...) or something like: cdef extern from "my_variadic.hpp" nogil: T myfunc2[T, Types...](T, Types... args) Again, the latter syntax is more explicit, but it will require much more complicated code in Cython. It also doesn't match the existing syntax very well. The former syntax matches the existing syntax for templates better, but will make it hard for Cython to raise errors early on in compilation. I'd greatly appreciate any input on the best syntax for either use-case. Regards, -Ian Henriksen

3 6

overflow bug?
by Mark Florisson Sept. 5, 2015

Sept. 5, 2015

Hi, I think this might be a bug (python 3.3 + cython 0.22.1): def f(term=b"12345"): val = int('987278186585') # The below line does not work, because it treats 1 as a constant integer # in the C code (32 bit on my machine). Using 1L does work however. val -= 1 << (len(term) * 8) return val print(f()) This works in pure-python, but Cython generates '1 << __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform). Cheers, Mark

3 2

Cython 0.23 beta 2 released
by Stefan Behnel Aug. 6, 2015

Aug. 6, 2015

Hi everyone, I just uploaded a second beta for the upcoming Cython 0.23. Please give it some more testing as there were substantial code changes since the first beta. You can get the signed release from here: http://cython.org/release/ http://cython.org/release/Cython-0.23b2.tar.gz http://cython.org/release/Cython-0.23b2.zip SHA1 sums: 1d12e335116ba3fcbe731d83ccc5ee5f0f6e5371 Cython-0.23b2.tar.gz 6f063e6b08b6dbecbadd44ed6c6150bc799fa230 Cython-0.23b2.zip Changes since beta 1 (in addition to general bug fixes): * The builtin ``type`` type is now declared as PyTypeObject in source, allowing for extern functions taking type parameters to have the correct C signatures. Note that this might break code that uses ``type`` just for passing around Python types in typed variables. Removing the type declaration provides a backwards compatible fix. * Support operator bool() for C++ classes so they can be used in if statements. * Installation under CPython 3.3+ no longer requires a pass of the 2to3 tool. This also makes it possible to run Cython in Python 3.3+ from a source checkout without installing it first. Patch by Petr Viktorin. * Coverage analysis and async/await were adapted to recent changes in coverage.py and CPython. Complete changelog follows below. Since we're in beta phase now, new features should not generally be added for the release any more, but we can make exceptions for currently existing pull requests if you can provide a good motivation. If you think we forgot something, sending a timely reply to this email or reviving an existing mailing list thread or pull request is a good idea. Have fun, Stefan Features added -------------- * PEP 492 (async/await) was implemented. See https://www.python.org/dev/peps/pep-0492/ * PEP 448 (Additional Unpacking Generalizations) was implemented. See https://www.python.org/dev/peps/pep-0448/ * Support for coverage.py 4.0+ can be enabled by adding the plugin "Cython.Coverage" to the ".coveragerc" config file. * Annotated HTML source pages can integrate (XML) coverage reports. * Tracing is supported in ``nogil`` functions/sections and module init code. * When generators are used in a Cython module and the module imports the modules "inspect" and/or "asyncio", Cython enables interoperability by patching these modules during the import to recognise Cython's internal generator and coroutine types. This can be disabled by C compiling the module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0" * When generators or coroutines are used in a Cython module, their types are registered with the ``Generator`` and ``Coroutine`` ABCs in the ``collections`` or ``collections.abc`` stdlib module at import time to enable interoperability with code that needs to detect and process Python generators/coroutines. These ABCs were added in CPython 3.5 and are available for older Python versions through the ``backports_abc`` module on PyPI. See https://bugs.python.org/issue24018 * Adding/subtracting/dividing/modulus and equality comparisons with constant Python floats and small integers are faster. * Binary and/or/xor/rshift operations with small constant Python integers are faster. * When called on generator expressions, the builtins ``all()``, ``any()``, ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()`` avoid the generator iteration overhead by inlining a part of their functionality into the for-loop. * Keyword argument dicts are no longer copied on function entry when they are not being used or only passed through to other function calls (e.g. in wrapper functions). * The ``PyTypeObject`` declaration in ``cpython.object`` was extended. * The builtin ``type`` type is now declared as PyTypeObject in source, allowing for extern functions taking type parameters to have the correct C signatures. Note that this might break code that uses ``type`` just for passing around Python types in typed variables. Removing the type declaration provides a backwards compatible fix. * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure Python mode. * Const iterators were added to the provided C++ STL declarations. * ``NULL`` is allowed as default argument when embedding signatures. This fixes ticket 843. * When compiling with ``--embed``, the internal module name is changed to ``__main__`` to allow arbitrary program names, including those that would be invalid for modules. Note that this prevents reuse of the generated C code as an importable module. * External C++ classes that overload the assignment operator can be used. Patch by Ian Henriksen. * Support operator bool() for C++ classes so they can be used in if statements. Bugs fixed ---------- * Calling "yield from" from Python on a Cython generator that returned a value triggered a crash in CPython. This is now being worked around. See https://bugs.python.org/issue23996 * Language level 3 did not enable true division (a.k.a. float division) for integer operands. * Functions with fused argument types that included a generic 'object' fallback could end up using that fallback also for other explicitly listed object types. * Relative cimports could accidentally fall back to trying an absolute cimport on failure. * The result of calling a C struct constructor no longer requires an intermediate assignment when coercing to a Python dict. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * ``cpdef void`` methods are now permitted. * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code in C++. Patch by David Vierra, original patch by Michael Enßlin. * Buffer index calculations using index variables with small C integer types could overflow for large buffer sizes. Original patch by David Vierra. * C unions use a saner way to coerce from and to Python dicts. * When compiling a module ``foo.pyx``, the directories in ``sys.path`` are no longer searched when looking for ``foo.pxd``. Patch by Jeroen Demeyer. * Memory leaks in the embedding main function were fixed. Original patch by Michael Enßlin. * Some complex Python expressions could fail to compile inside of finally clauses. Other changes ------------- * Changed mangling scheme in header files generated by ``cdef api`` declarations. * Installation under CPython 3.3+ no longer requires a pass of the 2to3 tool. This also makes it possible to run Cython in Python 3.3+ from a source checkout without installing it first. Patch by Petr Viktorin.

3 2

[RFE] Add dummy compiler directive decorators for pure python mode
by Carlos Pita July 28, 2015

July 28, 2015

Hi all, pure python mode will benefit from the addition of dummy decorators for compiler directives to Shadow.py, like in: @cython.boundscheck(False) def f(): pass AFAICS this is not currently valid inside the interpreter. Cheers -- Carlos

3 5

Cython 0.23 beta 1 released
by Stefan Behnel July 23, 2015

July 23, 2015

Hi everyone, I just uploaded a first beta for the upcoming Cython 0.23. There have been many major changes and fixes, so please give it some testing with your own code - especially on less commonly tested platforms, namely MS Windows. You can get the signed release from here: http://cython.org/release/ http://cython.org/release/Cython-0.23.beta1.tar.gz http://cython.org/release/Cython-0.23.beta1.zip SHA1 sums: 45347674bf14ebfce879dd1bf15af64387a39f8c Cython-0.23.beta1.tar.gz 0db13cdd8a11f83158e5c72f5e84b89e574af215 Cython-0.23.beta1.zip Changelog follows below. Since we're in beta phase now, new features should not generally be added for the release any more, but we can make exceptions for currently existing pull requests if you can provide a good motivation. If you think we forgot something, sending a timely reply to this email or reviving an existing mailing list thread or pull request is a good idea. Note that the coverage analysis support is still incomplete as the plugin API of the coverage.py tool itself is still subject to modifications. Have fun, Stefan Changelog: ========== Features added -------------- * PEP 492 (async/await) was implemented. See https://www.python.org/dev/peps/pep-0492/ * PEP 448 (Additional Unpacking Generalizations) was implemented. See https://www.python.org/dev/peps/pep-0448/ * Support for coverage.py 4.0+ can be enabled by adding the plugin "Cython.Coverage" to the ".coveragerc" config file. * Annotated HTML source pages can integrate (XML) coverage reports. * Tracing is supported in ``nogil`` functions/sections and module init code. * When generators are used in a Cython module and the module imports the modules "inspect" and/or "asyncio", Cython enables interoperability by patching these modules during the import to recognise Cython's internal generator and coroutine types. This can be disabled by C compiling the module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0" * When generators or coroutines are used in a Cython module, their types are registered with the ``Generator`` and ``Coroutine`` ABCs in the ``collections`` or ``collections.abc`` stdlib module at import time to enable interoperability with code that needs to detect and process Python generators/coroutines. These ABCs were added in CPython 3.5 and are available for older Python versions through the ``backports_abc`` module on PyPI. See https://bugs.python.org/issue24018 * Adding/subtracting/dividing/modulus and equality comparisons with constant Python floats and small integers are faster. * Binary and/or/xor/rshift operations with small constant Python integers are faster. * When called on generator expressions, the builtins ``all()``, ``any()``, ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()`` avoid the generator iteration overhead by inlining a part of their functionality into the for-loop. * Keyword argument dicts are no longer copied on function entry when they are not being used or only passed through to other function calls (e.g. in wrapper functions). * The ``PyTypeObject`` declaration in ``cpython.object`` was extended. * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure Python mode. * Const iterators were added to the provided C++ STL declarations. * ``NULL`` is allowed as default argument when embedding signatures. This fixes ticket 843. * When compiling with ``--embed``, the internal module name is changed to ``__main__`` to allow arbitrary program names, including those that would be invalid for modules. Note that this prevents reuse of the generated C code as an importable module. * External C++ classes that overload the assignment operator can be used. Patch by Ian Henriksen. Bugs fixed ---------- * Calling "yield from" from Python on a Cython generator that returned a value triggered a crash in CPython. This is now being worked around. See https://bugs.python.org/issue23996 * Language level 3 did not enable true division (a.k.a. float division) for integer operands. * Functions with fused argument types that included a generic 'object' fallback could end up using that fallback also for other explicitly listed object types. * Relative cimports could accidentally fall back to trying an absolute cimport on failure. * The result of calling a C struct constructor no longer requires an intermediate assignment when coercing to a Python dict. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * ``cpdef void`` methods are now permitted. * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code in C++. Patch by David Vierra, original patch by Michael Enßlin. * Buffer index calculations using index variables with small C integer types could overflow for large buffer sizes. Original patch by David Vierra. * C unions use a saner way to coerce from and to Python dicts. * When compiling a module ``foo.pyx``, the directories in ``sys.path`` are no longer searched when looking for ``foo.pxd``. Patch by Jeroen Demeyer. * Memory leaks in the embedding main function were fixed. Original patch by Michael Enßlin. * Some complex Python expressions could fail to compile inside of finally clauses. Other changes ------------- * Changed mangling scheme in header files generated by ``cdef api`` declarations.

2 2

Fwd: [BUG] Cython failing performance when Numpy is not installed
by Alex S. July 21, 2015

July 21, 2015

Hello, being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here. The issue was described here: http://trac.cython.org/ticket/847 but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code: > cdef type ndarray > try: > import numpy > ndarray = numpy.ndarray > except (ImportError, AttributeError, TypeError): > ndarray = None (Compiler/FusedNode.py:468). When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance.

2 2

Problem with declaring a map with an enum type.
by Brian McAndrews July 18, 2015

July 18, 2015

I have a C enum type that I'm attempting to put in a C++ map: Here's what I got: colors.h: enum Color { GREEN,BLUE,RED }; val.pyx: from libcpp.string cimport string from libcpp.map cimport map cdef extern from "colors.h": cpdef enum _Color "Color": GREEN,BLUE,RED cdef class TestMe: cdef public map[string, _Color] vals I get a compile error (VSC++ 2012) VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color' Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast) The offending statement is this: __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;} In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function. The cast from PyInt_AsLong(...) to an enum Color is not specified when using maps. If I don't put it in a map (just as a member variable), the cast is performed. Has anyone seen this before? Thanks, Brian This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies.

2 1

Handling of cascaded assignment
by Ian Henriksen July 15, 2015

July 15, 2015

I came across this while working on allowing proper overloading of the assignment operator. The problem is, essentially, that Cython does not follow the C/C++ ordering for cascaded assignment, even when dealing with objects at the C/C++ level. For example, calling the test function from the following Cython file and running the following C++ file should both give the same output: def test(): cdef: long long a = (2LL << 36) - 257 int b char c b = c = a print a, b, c #include <iostream> int main(){ long long a = (2LL << 36) - 257; int b; char c; b = c = a; std::cout << a << std::endl; std::cout << b << std::endl; std::cout << (int)c << std::endl; } The Cython version prints 137438953215 -257 -1 and the C++ version prints 137438953215 -1 -1 To mimic C/C++, it should go through the arguments from right to left and perform each assignment operation in that order, i.e. c = a b = c As it stands, the current code does: b = a c = a This does appear to follow what Python does. For example, the following gives the same output as the Cython version: import numpy as np a = np.array([(2 << 36) - 257], np.int64) b = np.empty(1, np.int32) c = np.empty(1, np.int8) b[:] = c[:] = a print a[0], b[0], c[0] Is this a deliberate design decision? If this weren't already included in the language, I'd be inclined to not allow cascaded assignment to C objects at all since it can give such surprising results. On the other hand I don't see any particularly clean way to deal with this incompatibility between C/C++ and Python. Thanks! -Ian Henriksen

2 1

Multidimensional indexing of C++ objects
by Ian Henriksen July 15, 2015

July 15, 2015

Hi everyone, I'm a GSOC student working to make a Cython API for DyND. DyND <https://github.com/libdynd/libdynd> is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead. Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this: a(irange(), 1).vals() = 0; Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of: 1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it. 2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now: # declared as a method in a pxd file: void assign "vals() = "(int value) except + # used in a pyx file to perform assignment to a slice of an array a: a(irange(), 1).assign(0) For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces. I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class. The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators. Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator(). Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions. Thanks! -Ian Henriksen

3 6

Regression: cdef append() no longer works
by Jeroen Demeyer July 14, 2015

July 14, 2015

Code like the following no longer works correctly: cdef class Foo: def __init__(self): self.append(1) cdef append(self, x): return x It seems that Cython now always assumes that append() is a special method: it incorrectly uses __Pyx_PyObject_Append. PS: it's not really clear if bug reports like these should go on this mailing list or on Trac (my impression is that Cython's Trac is not really used). What's the recommended procedure?

3 2