Mailman 3 September 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Automatic conversion with fixed-size C arrays
by Kurt Smith Oct. 25, 2014

Oct. 25, 2014

Hi devs, Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. It seems like the following struct should be convertible safely: cdef struct state_t: int i, j, k float x[3] float v[3] Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well: def auto_convert(list ll): cdef int arr[10] = ll # ... return arr If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this. Thoughts? Kurt

4 17

aritmetic with arrays in Cython
by Ian Henriksen Oct. 24, 2014

Oct. 24, 2014

Hi all, I noticed that some time ago there was a pull request ( https://github.com/cython/cython/pull/144) open that was trying to implement basic arithmetic operations with arrays. This seems to have also been proposed in CEP 518 ( https://github.com/cython/cython/wiki/enhancements-simd). What is the status on this? This is a feature I'd really like to see in Cython. I'd be happy to help with it, though I don't currently know enough about the internals of Cython to do it without some guidance. What would be the best way to do this? I've had three ideas thus far that might allow something like this to work, but I'd really like to get some second opinions before I get too far with any of them. 1. Make these array operations available as a part of Cython's memory view objects. 2. Write a Cython wrapper around a NumPy-like C++ library. https://github.com/ContinuumIO/libdynd seems to be a good candidate, though I'm not as familiar with it. (This doesn't strike me as something that would be a part of the main Cython project, but it could be a good way to make this sort of functionality available. It seems like a good way to avoid a lot of duplicate work though.) 3. Write a Fortran backend for Cython that uses Fortran's arrays wherever memory views are used. (This could be a bit crazy, but with the iso_c_binding module, it might be possible. It could also allow better optimizations for arrays to be applied by the compiler.) What would be the best way to go about this? Thanks! -Ian Henriksen

5 20

Travis-ci builds of OSX wheels
by Matthew Brett Oct. 16, 2014

Oct. 16, 2014

Hi, We just got scikit-image OSX wheel builds more or less fully automated on travis-ci : https://travis-ci.org/scikit-image/scikit-image-wheels https://github.com/scikit-image/scikit-image-wheels Any interest in getting the same thing working for Cython? Cheers, Matthew

2 8

Cython 0.21 released
by Stefan Behnel Oct. 12, 2014

Oct. 12, 2014

Hi all, on behalf of the Cython dev team, I'm pleased to announce the release of Cython 0.21, a major feature release. Thanks everyone who contributed code, documentation improvements, test feedback, bug reports and/or otherwise helpful insights for this release. I copied the complete changelog of this release below, but here are the major highlights, in addition to the many bug fixes and general improvements in this release. * Legacy code and support for old CPython versions was removed, baselines are now Python 2.6 and 3.2. * cdef functions finally support inner Python functions with closures. * C enums can now be declared as "cpdef" to publish them to Python space. * Python method calls, boolean "and/or" operators and cascaded assignments were optimised to streamline calls and type coercions. * The annotated HTML code output is syntax highlighted and beautified. Have fun, Stefan Downloads: https://pypi.python.org/pypi/Cython/0.21 http://cython.org/release/Cython-0.21.tar.gz http://cython.org/release/Cython-0.21.zip Signatures: http://cython.org/release/Cython-0.21.tar.gz.asc http://cython.org/release/Cython-0.21.zip.asc Changelog: https://github.com/cython/cython/blob/c67e895414aac90dfe9f789530143cff5b2ec… Documentation: http://docs.cython.org/ 0.21 (2014-09-10) ================= Features added -------------- * C (cdef) functions allow inner Python functions. * Enums can now be declared as cpdef to export their values to the module's Python namespace. Cpdef enums in pxd files export their values to their own module, iff it exists. * Allow @staticmethod decorator to declare static cdef methods. This is especially useful for declaring "constructors" for cdef classes that can take non-Python arguments. * Taking a ``char*`` from a temporary Python string object is safer in more cases and can be done inside of non-trivial expressions, including arguments of a function call. A compile time error is raised only when such a pointer is assigned to a variable and would thus exceed the lifetime of the string itself. * Generators have new properties ``__name__`` and ``__qualname__`` that provide the plain/qualified name of the generator function (following CPython 3.5). See http://bugs.python.org/issue21205 * The ``inline`` function modifier is available as a decorator ``(a)cython.inline`` in pure mode. * When cygdb is run in a virtualenv, it enables the same virtualenv inside of the debugger. Patch by Marc Abramowitz. * PEP 465: dedicated infix operator for matrix multiplication (A @ B). * HTML output of annotated code uses Pygments for code highlighting and generally received a major overhaul by Matthias Bussonier. * IPython magic support is now available directly from Cython with the command "%load_ext cython". Cython code can directly be executed in a cell when marked with "%%cython". Code analysis is available with "%%cython -a". Patch by Martín Gaitán. * Simple support for declaring Python object types in Python signature annotations. Currently requires setting the compiler directive ``annotation_typing=True``. * New directive ``use_switch`` (defaults to True) to optionally disable the optimization of chained if statement to C switch statements. * Defines dynamic_cast et al. in ``libcpp.cast`` and C++ heap data structure operations in ``libcpp.algorithm``. * Shipped header declarations in ``posix.*`` were extended to cover more of the POSIX API. Patches by Lars Buitinck and Mark Peek. Optimizations ------------- * Simple calls to C implemented Python functions/methods are faster. This also speeds up many operations on builtins that Cython cannot otherwise optimise. * The "and"/"or" operators try to avoid unnecessary coercions of their arguments. They now evaluate the truth value of each argument independently and only coerce the final result of the whole expression to the target type (e.g. the type on the left side of an assignment). This also avoids reference counting overhead for Python values during evaluation and generally improves the code flow in the generated C code. * The Python expression "2 ** N" is optimised into bit shifting. See http://bugs.python.org/issue21420 * Cascaded assignments (a = b = ...) try to minimise the number of type coercions. * Calls to ``slice()`` are translated to a straight C-API call. Bugs fixed ---------- * Crash when assigning memory views from ternary conditional expressions. * Nested C++ templates could lead to unseparated ">>" characters being generated into the C++ declarations, which older C++ compilers could not parse. * Sending SIGINT (Ctrl-C) during parallel cythonize() builds could hang the child processes. * No longer ignore local setup.cfg files for distutils in pyximport. Patch by Martin Teichmann. * Taking a ``char*`` from an indexed Python string generated unsafe reference counting code. * Set literals now create all of their items before trying to add them to the set, following the behaviour in CPython. This makes a difference in the rare case that the item creation has side effects and some items are not hashable (or if hashing them has side effects, too). * Cython no longer generates the cross product of C functions for code that uses memory views of fused types in function signatures (e.g. ``cdef func(floating[:] a, floating[:] b)``). This is considered the expected behaviour by most users and was previously inconsistent with other structured types like C arrays. Code that really wants all type combinations can create the same fused memoryview type under different names and use those in the signature to make it clear which types are independent. * Names that were unknown at compile time were looked up as builtins at runtime but not as global module names. Trying both lookups helps with globals() manipulation. * Fixed stl container conversion for typedef element types. * ``obj.pop(x)`` truncated large C integer values of x to ``Py_ssize_t``. * ``__init__.pyc`` is recognised as marking a package directory (in addition to .py, .pyx and .pxd). * Syntax highlighting in ``cython-mode.el`` for Emacs no longer incorrectly highlights keywords found as part of longer names. * Correctly handle ``from cython.submodule cimport name``. * Fix infinite recursion when using super with cpdef methods. * No-args ``dir()`` was not guaranteed to return a sorted list. Other changes ------------- * The header line in the generated C files no longer contains the timestamp but only the Cython version that wrote it. This was changed to make builds more reproducible. * Removed support for CPython 2.4, 2.5 and 3.1. * The licensing implications on the generated code were clarified to avoid legal constraints for users.

3 9

Variable shadows type name in posix/time.pxd
by Joel Nothman Sept. 23, 2014

Sept. 23, 2014

a struct <https://github.com/cython/cython/blob/master/Cython/Includes/posix/time.pxd…> and a double <https://github.com/cython/cython/blob/master/Cython/Includes/posix/time.pxd…> are defined with the name 'timezone' in posix/time.pxd. Hence from posix.time cimport timeval, timezone, gettimeofday cdef timeval tv cdef timezone tz gettimeofday(&tv, &tz) fails with 'timezone' is not a type identifier It would be nice if there were a test for duplicate names in pxd files. Thanks for the awesome tool, Joel

2 1

Re: [Cython] finding the Cython version
by Stefan Behnel Sept. 11, 2014

Sept. 11, 2014

Alex K schrieb am 11.09.2014 um 14:28: > I am a user that can not install tables 3.0.0 with cython 0.21. Tables can > be installed with cython up to 0.20.2. Can you please consider to add > Cython.Compiler.Main.Version back so that users of old tables could still > use latest cython? Normally, users should not have to use (or have) Cython themselves when they want to install a Cython implemented package. Instead, the package should ship the generated C sources and do a normal extension module build by default, without running Cython at all. That avoids a build dependency and makes sure that users really get exactly what the code maintainers wanted to release and what they have properly tested before doing so. That being said, we try to keep things working for users, unless there's an actual reason to break something. I've definitely seen way worse compatibility hacks than an unused import. https://github.com/cython/cython/commit/43342ab90704f5f85073354428848504816… Stefan

1 0

finding the Cython version (was: Cython 0.21 released)
by Stefan Behnel Sept. 11, 2014

Sept. 11, 2014

Alex K schrieb am 11.09.2014 um 13:49: > tables 3.0.0 (maybe later version too) fails to install with `pip install > tables==3.0.0` due to changes in naming. PyTables's build script expects > Cython.Compiler.Main.Version, and gives an error when it doesn't find it. > Cython 0.21 provides Cython.Compiler.Main.version. The right way to check for the Cython version is Cython.__version__ as with most Python packages. For backwards compatibility with old Cython versions that didn't have it yet, you can use Cython.Compiler.Version.version Relying on a re-exported import like "Cython.Compiler.Main.Version" is like asking for "os.sys" to work because the "os" module happens to do "import sys". Stefan

2 1

Re: [Cython] [cython-users] Re: Cython 0.21 release candidate
by Stefan Behnel Sept. 9, 2014

Sept. 9, 2014

Ivan Smirnov schrieb am 09.09.2014 um 11:32: > Hi Stefan, a minor detail: might make sense to mention addition of the new > posix headers (stat.pxd etc) to the changelog? Agreed, that's worth mentioning. https://github.com/cython/cython/commit/05466cbe6c177f647de2f2f75fd00904a85… Thanks! Stefan

1 0

Cython 0.21 release candidate
by Stefan Behnel Sept. 8, 2014

Sept. 8, 2014

Hi all, I uploaded a release candidate for 0.21. There were a couple of more changes and cleanups since the last beta, so please give it another quick round of testing. http://www.cython.org/release/Cython-0.21rc1.tar.gz http://www.cython.org/release/Cython-0.21rc1.tar.gz.asc Complete changelog: https://github.com/cython/cython/blob/8337b8aec2ecbb4480d551e12b1faf9b21ba2… Stefan

1 0