Mailman 3 October 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Strange bug? with np.int64_t
by Patrick Snape Nov. 3, 2014

Nov. 3, 2014

System information: Ubuntu 14.04 x64 gcc: Version 4.8.2 (Ubuntu 4.8.2-19ubuntu1), Target: x86_64-linux-gnu To reproduce: # distutils: language = c++ import numpy as np cimport numpy as np cimport cython cdef test_np_int64(const np.int64_t test_var): cdef np.int64_t error = test_var / 2 This fails to compile with the error: In function ‘const __pyx_t_5numpy_int64_t __Pyx_div___pyx_t_5numpy_int64_t__const__(__pyx_t_5numpy_int64_t, __pyx_t_5numpy_int64_t)’: error: assignment of read-only variable ‘q’ q -= ((r != 0) & ((r ^ b) < 0)); Using just a regular int type: # distutils: language = c++ import numpy as np cimport numpy as np cimport cython cdef test_int(const int test_var): cdef int no_error = test_var / 2 Works as expected. Fixing this is as simple as removing the const that is being complained about. I assume this is a bug? I can't see why I wouldn't be allowed to perform that operation? Regards, Patrick

2 1

[PATCH] explain how to compile C++ extensions up to Cython 0.21
by Daniele Nicolodi Oct. 31, 2014

Oct. 31, 2014

--- docs/src/userguide/wrapping_CPlusPlus.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/src/userguide/wrapping_CPlusPlus.rst b/docs/src/userguide/wrapping_CPlusPlus.rst index 59c89a1..5119b63 100644 --- a/docs/src/userguide/wrapping_CPlusPlus.rst +++ b/docs/src/userguide/wrapping_CPlusPlus.rst @@ -141,6 +141,21 @@ Note that the ``language`` option has no effect on user provided Extension objects that are passed into ``cythonize()``. It is only used for modules found by file name (as in the example above). +The ``cythonize()`` function in Cython versions up to 0.21 does not +recognize the ``language`` option and it needs to be specified as an +option to an :class:`Extension` that describes your extension and that +is then handled by ``cythonize()`` as follows:: + + from distutils.core import setup, Extension + from Cython.Build import cythonize + + setup(ext_modules = cythonize(Extension( + "rect", # the extesion name + sources=["rect.pyx", "Rectangle.cpp"], # the Cython source and + # additional C++ source files + language="c++", # generate and compile C++ code + ))) + The options can also be passed directly from the source file, which is often preferable (and overrides any global option). Starting with version 0.17, Cython also allows to pass external source files into the -- 2.1.1

3 5

Automatic conversion with fixed-size C arrays
by Kurt Smith Oct. 25, 2014

Oct. 25, 2014

Hi devs, Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. It seems like the following struct should be convertible safely: cdef struct state_t: int i, j, k float x[3] float v[3] Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well: def auto_convert(list ll): cdef int arr[10] = ll # ... return arr If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this. Thoughts? Kurt

4 17

aritmetic with arrays in Cython
by Ian Henriksen Oct. 24, 2014

Oct. 24, 2014

Hi all, I noticed that some time ago there was a pull request ( https://github.com/cython/cython/pull/144) open that was trying to implement basic arithmetic operations with arrays. This seems to have also been proposed in CEP 518 ( https://github.com/cython/cython/wiki/enhancements-simd). What is the status on this? This is a feature I'd really like to see in Cython. I'd be happy to help with it, though I don't currently know enough about the internals of Cython to do it without some guidance. What would be the best way to do this? I've had three ideas thus far that might allow something like this to work, but I'd really like to get some second opinions before I get too far with any of them. 1. Make these array operations available as a part of Cython's memory view objects. 2. Write a Cython wrapper around a NumPy-like C++ library. https://github.com/ContinuumIO/libdynd seems to be a good candidate, though I'm not as familiar with it. (This doesn't strike me as something that would be a part of the main Cython project, but it could be a good way to make this sort of functionality available. It seems like a good way to avoid a lot of duplicate work though.) 3. Write a Fortran backend for Cython that uses Fortran's arrays wherever memory views are used. (This could be a bit crazy, but with the iso_c_binding module, it might be possible. It could also allow better optimizations for arrays to be applied by the compiler.) What would be the best way to go about this? Thanks! -Ian Henriksen

5 20

wrong c file
by Shang Yu Oct. 19, 2014

Oct. 19, 2014

Hi dear all, I'm trying cyhotn(0.21) with following code (hello.pyx) cdef void cfun(): print "hello world" if __name__ == "__main__": cfun() the generated c file has following line static void "__hello" __pyx_f_5hello_cfun(void) { which can't compiled by c compiler , what's wrong with this ? Many thanks !!!

1 1

Cython 0.21.1 released
by Stefan Behnel Oct. 18, 2014

Oct. 18, 2014

Hi everyone, I uploaded Cython 0.21.1. It adds a few minor new features and some bug fixes and corrections. Downloads: https://pypi.python.org/pypi/Cython/0.21.1 http://cython.org/release/Cython-0.21.1.tar.gz http://cython.org/release/Cython-0.21.1.zip Changelog: https://github.com/cython/cython/blob/862b7869ac95e786475fcf8ac216a7c0c3378… Documentation: http://docs.cython.org/ Stefan Features added -------------- * New ``cythonize`` option ``-a`` to generate the annotated HTML source view. * Missing C-API declarations in ``cpython.unicode`` were added. * Passing ``language='c++'`` into cythonize() globally enables C++ mode for all modules that were not passed as Extension objects (i.e. only source files and file patterns). * ``Py_hash_t`` is a known type (used in CPython for hash values). * ``PySlice_*()`` C-API functions are available from the ``cpython.slice`` module. * Allow arrays of C++ classes. Bugs fixed ---------- * Reference leak for non-simple Python expressions in boolean and/or expressions. * To fix a name collision and to reflect availability on host platforms, standard C declarations [ clock(), time(), struct tm and tm* functions ] were moved from posix/time.pxd to a new libc/time.pxd. Patch by Charles Blake. * Rerunning unmodified modules in IPython's cython support failed. Patch by Matthias Bussonier. * Casting C++ ``std::string`` to Python byte strings failed when auto-decoding was enabled. * Fatal exceptions in global module init code could lead to crashes if the already created module was used later on (e.g. through a stale reference in sys.modules or elsewhere). * ``cythonize.py`` script was not installed on MS-Windows. Other changes ------------- * Compilation no longer fails hard when unknown compilation options are passed. Instead, it raises a warning and ignores them (as it did silently before 0.21). This will be changed back to an error in a future release.

1 0

Release candidate for Cython 0.21.1
by Stefan Behnel Oct. 17, 2014

Oct. 17, 2014

Hi everyone, I uploaded a candidate for a 0.21.1 bug fix release. http://cython.org/release/Cython-0.21.1pre.tar.gz It adds a few minor new features and some bug fixes and corrections. Please give it a quick try, especially if you reported problems that this release should fix. I had to manually select changes from the master branch, so I hope I didn't forget anything. Stefan Features added -------------- * New ``cythonize`` option ``-a`` to generate the annotated HTML source view. * Missing C-API declarations in ``cpython.unicode`` were added. * Passing ``language='c++'`` into cythonize() globally enables C++ mode for all modules that were not passed as Extension objects (i.e. only source files and file patterns). * ``Py_hash_t`` is a known type (used in CPython for hash values). * ``PySlice_*()`` C-API functions are available from the ``cpython.slice`` module. * Allow arrays of C++ classes. Bugs fixed ---------- * Reference leak for non-simple Python expressions in boolean and/or expressions. * To fix a name collision and to reflect availability on host platforms, standard C declarations [ clock(), time(), struct tm and tm* functions ] were moved from posix/time.pxd to a new libc/time.pxd. Patch by Charles Blake. * Rerunning unmodified modules in IPython's cython support failed. Patch by Matthias Bussonier. * Casting C++ ``std::string`` to Python byte strings failed when auto-decoding was enabled. * Fatal exceptions in global module init code could lead to crashes if the already created module was used later on (e.g. through a stale reference in sys.modules or elsewhere). Other changes ------------- * Compilation no longer fails hard when unknown compilation options are passed. Instead, it raises a warning and ignores them (as it did silently before 0.21). This will be changed back to an error in a future release.

2 1

Travis-ci builds of OSX wheels
by Matthew Brett Oct. 16, 2014

Oct. 16, 2014

Hi, We just got scikit-image OSX wheel builds more or less fully automated on travis-ci : https://travis-ci.org/scikit-image/scikit-image-wheels https://github.com/scikit-image/scikit-image-wheels Any interest in getting the same thing working for Cython? Cheers, Matthew

2 8

Cython 0.21 released
by Stefan Behnel Oct. 12, 2014

Oct. 12, 2014

Hi all, on behalf of the Cython dev team, I'm pleased to announce the release of Cython 0.21, a major feature release. Thanks everyone who contributed code, documentation improvements, test feedback, bug reports and/or otherwise helpful insights for this release. I copied the complete changelog of this release below, but here are the major highlights, in addition to the many bug fixes and general improvements in this release. * Legacy code and support for old CPython versions was removed, baselines are now Python 2.6 and 3.2. * cdef functions finally support inner Python functions with closures. * C enums can now be declared as "cpdef" to publish them to Python space. * Python method calls, boolean "and/or" operators and cascaded assignments were optimised to streamline calls and type coercions. * The annotated HTML code output is syntax highlighted and beautified. Have fun, Stefan Downloads: https://pypi.python.org/pypi/Cython/0.21 http://cython.org/release/Cython-0.21.tar.gz http://cython.org/release/Cython-0.21.zip Signatures: http://cython.org/release/Cython-0.21.tar.gz.asc http://cython.org/release/Cython-0.21.zip.asc Changelog: https://github.com/cython/cython/blob/c67e895414aac90dfe9f789530143cff5b2ec… Documentation: http://docs.cython.org/ 0.21 (2014-09-10) ================= Features added -------------- * C (cdef) functions allow inner Python functions. * Enums can now be declared as cpdef to export their values to the module's Python namespace. Cpdef enums in pxd files export their values to their own module, iff it exists. * Allow @staticmethod decorator to declare static cdef methods. This is especially useful for declaring "constructors" for cdef classes that can take non-Python arguments. * Taking a ``char*`` from a temporary Python string object is safer in more cases and can be done inside of non-trivial expressions, including arguments of a function call. A compile time error is raised only when such a pointer is assigned to a variable and would thus exceed the lifetime of the string itself. * Generators have new properties ``__name__`` and ``__qualname__`` that provide the plain/qualified name of the generator function (following CPython 3.5). See http://bugs.python.org/issue21205 * The ``inline`` function modifier is available as a decorator ``(a)cython.inline`` in pure mode. * When cygdb is run in a virtualenv, it enables the same virtualenv inside of the debugger. Patch by Marc Abramowitz. * PEP 465: dedicated infix operator for matrix multiplication (A @ B). * HTML output of annotated code uses Pygments for code highlighting and generally received a major overhaul by Matthias Bussonier. * IPython magic support is now available directly from Cython with the command "%load_ext cython". Cython code can directly be executed in a cell when marked with "%%cython". Code analysis is available with "%%cython -a". Patch by Martín Gaitán. * Simple support for declaring Python object types in Python signature annotations. Currently requires setting the compiler directive ``annotation_typing=True``. * New directive ``use_switch`` (defaults to True) to optionally disable the optimization of chained if statement to C switch statements. * Defines dynamic_cast et al. in ``libcpp.cast`` and C++ heap data structure operations in ``libcpp.algorithm``. * Shipped header declarations in ``posix.*`` were extended to cover more of the POSIX API. Patches by Lars Buitinck and Mark Peek. Optimizations ------------- * Simple calls to C implemented Python functions/methods are faster. This also speeds up many operations on builtins that Cython cannot otherwise optimise. * The "and"/"or" operators try to avoid unnecessary coercions of their arguments. They now evaluate the truth value of each argument independently and only coerce the final result of the whole expression to the target type (e.g. the type on the left side of an assignment). This also avoids reference counting overhead for Python values during evaluation and generally improves the code flow in the generated C code. * The Python expression "2 ** N" is optimised into bit shifting. See http://bugs.python.org/issue21420 * Cascaded assignments (a = b = ...) try to minimise the number of type coercions. * Calls to ``slice()`` are translated to a straight C-API call. Bugs fixed ---------- * Crash when assigning memory views from ternary conditional expressions. * Nested C++ templates could lead to unseparated ">>" characters being generated into the C++ declarations, which older C++ compilers could not parse. * Sending SIGINT (Ctrl-C) during parallel cythonize() builds could hang the child processes. * No longer ignore local setup.cfg files for distutils in pyximport. Patch by Martin Teichmann. * Taking a ``char*`` from an indexed Python string generated unsafe reference counting code. * Set literals now create all of their items before trying to add them to the set, following the behaviour in CPython. This makes a difference in the rare case that the item creation has side effects and some items are not hashable (or if hashing them has side effects, too). * Cython no longer generates the cross product of C functions for code that uses memory views of fused types in function signatures (e.g. ``cdef func(floating[:] a, floating[:] b)``). This is considered the expected behaviour by most users and was previously inconsistent with other structured types like C arrays. Code that really wants all type combinations can create the same fused memoryview type under different names and use those in the signature to make it clear which types are independent. * Names that were unknown at compile time were looked up as builtins at runtime but not as global module names. Trying both lookups helps with globals() manipulation. * Fixed stl container conversion for typedef element types. * ``obj.pop(x)`` truncated large C integer values of x to ``Py_ssize_t``. * ``__init__.pyc`` is recognised as marking a package directory (in addition to .py, .pyx and .pxd). * Syntax highlighting in ``cython-mode.el`` for Emacs no longer incorrectly highlights keywords found as part of longer names. * Correctly handle ``from cython.submodule cimport name``. * Fix infinite recursion when using super with cpdef methods. * No-args ``dir()`` was not guaranteed to return a sorted list. Other changes ------------- * The header line in the generated C files no longer contains the timestamp but only the Cython version that wrote it. This was changed to make builds more reproducible. * Removed support for CPython 2.4, 2.5 and 3.1. * The licensing implications on the generated code were clarified to avoid legal constraints for users.

3 9