Mailman 3 August 2015 - cython-devel

Non-type template parameters
by Ian Henriksen Oct. 1, 2015

Oct. 1, 2015

Hi all, I've spent a little time working on adding support for non-type template parameters. In doing this, it has been very easy to add support for doing something like the following: cdef extern from "add_const.hpp" nogil: int myfunc1[i](int) def test(): print myfunc1[2](a) The downside of this is that it does not let the Cython compiler distinguish between different kinds of template parameters. Stricter checking could be added using a syntax like this: cdef extern from "add_const.hpp" nogil: int myfunc1[int i](int) def test(): print myfunc1[2](a) The downsides here are that the syntax doesn't really match the existing template syntax. It will also complicate the Cython codebase since we'll have to go to greater lengths to allow or disallow all the different special cases for templates. Which version would be preferable? On a similar note, for variadic templates, would we prefer something like cdef extern from "my_variadic.hpp" nogil: T myfunc2[T,...](T, ...) or something like: cdef extern from "my_variadic.hpp" nogil: T myfunc2[T, Types...](T, Types... args) Again, the latter syntax is more explicit, but it will require much more complicated code in Cython. It also doesn't match the existing syntax very well. The former syntax matches the existing syntax for templates better, but will make it hard for Cython to raise errors early on in compilation. I'd greatly appreciate any input on the best syntax for either use-case. Regards, -Ian Henriksen

3 6

DEF converts byte strings to unicode
by Jakub Wilk Sept. 12, 2015

Sept. 12, 2015

DEF silently converts byte strings to Unicode strings: $ cat testcase.pyx DEF ABYTE = b'a' print type(ABYTE) $ python --version Python 2.7.10 $ python -c 'import pyximport as p; p.install(); import testcase' <type 'unicode'> Tested with Cython 0.23.1. -- Jakub Wilk

2 5

overflow bug?
by Mark Florisson Sept. 5, 2015

Sept. 5, 2015

Hi, I think this might be a bug (python 3.3 + cython 0.22.1): def f(term=b"12345"): val = int('987278186585') # The below line does not work, because it treats 1 as a constant integer # in the C code (32 bit on my machine). Using 1L does work however. val -= 1 << (len(term) * 8) return val print(f()) This works in pure-python, but Cython generates '1 << __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform). Cheers, Mark

3 2

Compiler crash in MarkOverflowingArithmetic
by Jakub Wilk Sept. 1, 2015

Sept. 1, 2015

Tested with Cython 0.23.1 on i386. You will probably need a bigger number to trigger it on a 64-bit machine. $ cython test.pyx Error compiling Cython file: ------------------------------------------------------------ ... DEF X = 0x811C9DC5 cdef void f(): cdef unsigned int x = X ^ ------------------------------------------------------------ test.pyx:3:9: Compiler crash in MarkOverflowingArithmetic ModuleNode.body = StatListNode(test.pyx:2:5) StatListNode.stats[0] = CFuncDefNode(test.pyx:2:5, args = [...]/0, modifiers = [...]/0, visibility = 'private') CFuncDefNode.body = StatListNode(test.pyx:3:9) StatListNode.stats[0] = SingleAssignmentNode(test.pyx:3:9, first = True) Compiler crash traceback from this point on: File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Visitor.py", line 183, in _visit return handler_method(obj) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/TypeInference.py", line 327, in visit_SingleAssignmentNode self.visit_assignment(node.lhs, node.rhs) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/TypeInference.py", line 321, in visit_assignment and Utils.long_literal(rhs.value)): File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 318, in long_literal value = str_to_number(value) File "/usr/local/lib/python2.7/dist-packages/Cython/Utils.py", line 312, in str_to_number value = int(value, 0) ValueError: invalid literal for int() with base 0: '2166136261L' -- Jakub Wilk

2 1

Python extensions for Python 3.5 - useful info...
by Fernando Perez Aug. 26, 2015

Aug. 26, 2015

Just an FYI for the upcoming Python release, a very detailed post from Steve Dower, the Microsoft developer who is now in charge of the Windows releases for Python, on how the build process will change in 3.5 regarding extensions: http://stevedower.id.au/blog/building-for-python-3-5/ Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

Cython 0.23.1 released
by Stefan Behnel Aug. 22, 2015

Aug. 22, 2015

Hi everyone, based on the feedback for the recent 0.23 release, here is a bug-fix-only release that repairs several problems and regressions. https://pypi.python.org/pypi/Cython/0.23.1 Complete changelog follows below. Upgrading is generally recommended. You can get the signed release from here: http://cython.org/ http://cython.org/release/Cython-0.23.1.tar.gz http://cython.org/release/Cython-0.23.1.zip SHA1 sums: 2ff0f863d3b996d2265d0bf06e567e5dd23d004d Cython-0.23.1.tar.gz 42943dd9b1355015ec5d630419f93a91f4f118cb Cython-0.23.1.zip Have fun, Stefan 0.23.1 (2015-08-22) =================== Bugs fixed ---------- * Invalid C code for generators. This fixes ticket 858. * Invalid C code for some builtin methods. This fixes ticket 856. * Invalid C code for unused local buffer variables. This fixes ticket 154. * Test failures on 32bit systems. This fixes ticket 857. * Code that uses "from xyz import *" and global C struct/union/array variables could fail to compile due to missing helper functions. This fixes ticket 851. * Misnamed PEP 492 coroutine property ``cr_yieldfrom`` renamed to ``cr_await`` to match CPython. * Missing deallocation code for C++ object attributes in certain extension class hierarchies. * Crash when async coroutine was not awaited. * Compiler crash on ``yield`` in signature annotations and default argument values. Both are forbidden now. * Compiler crash on certain constructs in ``finally`` clauses. * Cython failed to build when CPython's pgen is installed.

1 0

gcc error when compiling cython
by Thomas Holenstein Aug. 22, 2015

Aug. 22, 2015

Hey all, recently, I had a gcc compile time error with cython. The work around is obvious, as the problem happens when a cdef-ed numpy array in a function is unused. Nevertheless, I feel it's useful to report the problem I had. The following is a test case ---------------------------------------- test.pyx import numpy as np cimport numpy as np def f(np.ndarray[dtype=np.int_t, ndim=2] arg1): cdef np.ndarray[dtype=np.int_t, ndim=2] arr return 3 ----------------------------------------- setup.py from distutils.core import setup from Cython.Build import cythonize setup( ext_modules = cythonize("test.pyx"), ) ---------------------------------------------------------- This gives me the error error: ‘__pyx_pybuffernd_arr’ undeclared Full shell output: ~/tmp/ctest > python setup.py build_ext --inplace running build_ext building 'test' extension x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c test.c -o build/temp.linux-x86_64-2.7/test.o In file included from /usr/include/python2.7/numpy/ndarraytypes.h:1761:0, from /usr/include/python2.7/numpy/ndarrayobject.h:17, from /usr/include/python2.7/numpy/arrayobject.h:4, from test.c:250: /usr/include/python2.7/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^ test.c: In function ‘__pyx_pf_4test_f’: test.c:1240:30: error: ‘__pyx_pybuffernd_arr’ undeclared (first use in this function) __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer); ^ test.c:1240:30: note: each undeclared identifier is reported only once for each function it appears in In file included from /usr/include/python2.7/numpy/ndarrayobject.h:26:0, from /usr/include/python2.7/numpy/arrayobject.h:4, from test.c:250: test.c: At top level: /usr/include/python2.7/numpy/__multiarray_api.h:1629:1: warning: ‘_import_array’ defined but not used [-Wunused-function] _import_array(void) ^ In file included from /usr/include/python2.7/numpy/ufuncobject.h:327:0, from test.c:251: /usr/include/python2.7/numpy/__ufunc_api.h:241:1: warning: ‘_import_umath’ defined but not used [-Wunused-function] _import_umath(void) ^ error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

2 1

Cython 0.23: long + int == int
by Jakub Wilk Aug. 20, 2015

Aug. 20, 2015

$ cython --version Cython version 0.23 $ python2.6 --version Python 2.6.8 $ cat testlong.pyx def f(x): return 0L + x $ python2.6 -c 'import pyximport as p; p.install(); import testlong; print type(testlong.f(42))' <type 'int'> I added a long to an int, so I'd expect the result to be a long. This works correctly in Python 2.7: $ python2.7 -c 'import pyximport as p; p.install(); import testlong; print type(testlong.f(42))' <type 'long'> It also worked correctly in Cython 0.22.1 + Python 2.6. -- Jakub Wilk

2 2

Cython 0.23 released
by Stefan Behnel Aug. 19, 2015

Aug. 19, 2015

Hi everyone, given the positive feedback on the previous beta releases, I'm happy to announce the final release of Cython 0.23. There were only minor changes and fixes since the second beta. https://pypi.python.org/pypi/Cython/0.23 This is a major feature release. Highlights include, in addition to several important bug fixes: * PEP 492 (async/await) and PEP 448 (star-unpacking generalizations) * inlined generator expressions * support for the C++ assignment and bool() operators * support for coverage analysis Complete changelog follows below. Upgrading is generally recommended. You can get the signed release from here: http://cython.org/ http://cython.org/release/Cython-0.23.tar.gz http://cython.org/release/Cython-0.23.zip SHA1 sums: 5a49123b39d89a20fb88bdc7886210c39cdff695 Cython-0.23.tar.gz 9bec6c6df861be49ed608a0ab688b67bba543abe Cython-0.23.zip Have fun, Stefan 0.23 (2015-08-08) ================= Features added -------------- * PEP 492 (async/await) was implemented. See https://www.python.org/dev/peps/pep-0492/ * PEP 448 (Additional Unpacking Generalizations) was implemented. See https://www.python.org/dev/peps/pep-0448/ * Support for coverage.py 4.0+ can be enabled by adding the plugin "Cython.Coverage" to the ".coveragerc" config file. * Annotated HTML source pages can integrate (XML) coverage reports. * Tracing is supported in ``nogil`` functions/sections and module init code. * When generators are used in a Cython module and the module imports the modules "inspect" and/or "asyncio", Cython enables interoperability by patching these modules during the import to recognise Cython's internal generator and coroutine types. This can be disabled by C compiling the module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0" * When generators or coroutines are used in a Cython module, their types are registered with the ``Generator`` and ``Coroutine`` ABCs in the ``collections`` or ``collections.abc`` stdlib module at import time to enable interoperability with code that needs to detect and process Python generators/coroutines. These ABCs were added in CPython 3.5 and are available for older Python versions through the ``backports_abc`` module on PyPI. See https://bugs.python.org/issue24018 * Adding/subtracting/dividing/modulus and equality comparisons with constant Python floats and small integers are faster. * Binary and/or/xor/rshift operations with small constant Python integers are faster. * When called on generator expressions, the builtins ``all()``, ``any()``, ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()`` avoid the generator iteration overhead by inlining a part of their functionality into the for-loop. * Keyword argument dicts are no longer copied on function entry when they are not being used or only passed through to other function calls (e.g. in wrapper functions). * The ``PyTypeObject`` declaration in ``cpython.object`` was extended. * The builtin ``type`` type is now declared as PyTypeObject in source, allowing for extern functions taking type parameters to have the correct C signatures. Note that this might break code that uses ``type`` just for passing around Python types in typed variables. Removing the type declaration provides a backwards compatible fix. * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure Python mode. * Const iterators were added to the provided C++ STL declarations. * Smart pointers were added to the provided C++ STL declarations. Patch by Daniel Filonik. * ``NULL`` is allowed as default argument when embedding signatures. This fixes ticket 843. * When compiling with ``--embed``, the internal module name is changed to ``__main__`` to allow arbitrary program names, including those that would be invalid for modules. Note that this prevents reuse of the generated C code as an importable module. * External C++ classes that overload the assignment operator can be used. Patch by Ian Henriksen. * Support operator bool() for C++ classes so they can be used in if-statements. Bugs fixed ---------- * Calling "yield from" from Python on a Cython generator that returned a value triggered a crash in CPython. This is now being worked around. See https://bugs.python.org/issue23996 * Language level 3 did not enable true division (a.k.a. float division) for integer operands. * Functions with fused argument types that included a generic 'object' fallback could end up using that fallback also for other explicitly listed object types. * Relative cimports could accidentally fall back to trying an absolute cimport on failure. * The result of calling a C struct constructor no longer requires an intermediate assignment when coercing to a Python dict. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * ``cpdef void`` methods are now permitted. * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code in C++. Patch by David Vierra, original patch by Michael Enßlin. * Buffer index calculations using index variables with small C integer types could overflow for large buffer sizes. Original patch by David Vierra. * C unions use a saner way to coerce from and to Python dicts. * When compiling a module ``foo.pyx``, the directories in ``sys.path`` are no longer searched when looking for ``foo.pxd``. Patch by Jeroen Demeyer. * Memory leaks in the embedding main function were fixed. Original patch by Michael Enßlin. * Some complex Python expressions could fail to compile inside of finally clauses. * Unprefixed 'str' literals were not supported as C varargs arguments. * Fixed type errors in conversion enum types to/from Python. Note that this imposes stricter correctness requirements on enum declarations. Other changes ------------- * Changed mangling scheme in header files generated by ``cdef api`` declarations. * Installation under CPython 3.3+ no longer requires a pass of the 2to3 tool. This also makes it possible to run Cython in Python 3.3+ from a source checkout without installing it first. Patch by Petr Viktorin. * ``jedi-typer.py`` (in ``Tools/``) was extended and renamed to ``jedityper.py`` (to make it importable) and now works with and requires Jedi 0.9. Patch by Tzer-jen Wei.

2 7

Cython fails to build if pgen is available
by Dominic Sacré Aug. 18, 2015

Aug. 18, 2015

Hi, I'm getting the following error trying to build Cython 0.23: Cython/Parser/Grammar: No such file or directory Traceback (most recent call last): File "setup.py", line 289, in <module> compile_cython_modules(cython_profile, cython_compile_more, cython_with_refnanny) File "setup.py", line 135, in compile_cython_modules os.path.join(parser_dir, 'graminit.c'), File "/data/master/build/tmp/sysroots/x86_64-linux/usr/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/data/master/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/pgen', 'Cython/Parser/Grammar', 'Cython/Parser/graminit.h', 'Cython/Parser/graminit.c']' returned non-zero exit status 1 The toolchain is based on OpenEmbedded, and unlike most distribution packages, its Python installation actually contains pgen. Where would that Cython/Parser/Grammar file come from? Alternatively, is there a clean way (without patching setup.py) to skip compiling the grammar? Cheers, Dominic

2 1