Mailman 3 April 2017 - cython-devel

0.25.2b0 with 32bit fixes: (sometimes) Interrupted system call in builtin_subtype_methods_T653
by Yaroslav Halchenko July 6, 2017

July 6, 2017

A fresh one I ran into while rebuilding with fixes for 32bit (thanks BTW! sorry for not prompt followup/gratitude) ====================================================================== ERROR: runTest (__main__.CythonRunTestCase) compiling (cpp) and running builtin_subtype_methods_T653 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 998, in run self.run_tests(result, ext_so_path) File "runtests.py", line 1013, in run_tests self.run_doctests(self.module, result, ext_so_path) File "runtests.py", line 1023, in run_doctests run_forked_test(result, run_test, self.shortDescription(), self.fork) File "runtests.py", line 1076, in run_forked_test cid, result_code = os.waitpid(child_id, 0) OSError: [Errno 4] Interrupted system call ---------------------------------------------------------------------- Ran 581 tests in 188.225s FAILED (errors=1) it is not 100% reproducible. Rebuilding the package went smooth without this hiccup. Just wanted to let you know might be related that the tests were ran as set -e; for P in 2.7 3.5; do \ PYTHONPATH=`/bin/ls -d /build/cython-0.25.2~b0/build/lib.*-$P` \ /usr/bin/python$P runtests.py --shard_count=16 \ --no-refnanny -v -v --exclude="(parallel|Debugger|annotate_html)" --work-dir=build/work-dir 2>&1; \ done so testing in parallel to speed up the build of the package. This one was the only failure I have ran into but it built fine before (but there could have been some changes in the core packages since then) -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

2 2

Add a Pythran backend for Numpy operation
by Adrien Guinet May 6, 2017

May 6, 2017

Hello everyone! I've been working for quite some time on the usage of Pythran as a backend for the Numpy operations that Cython can generate. The associated PR on github can be found here: https://github.com/cython/cython/pull/1607. This work has been sponsored by the OpenDreamKit project (https://github.com/OpenDreamKit/OpenDreamKit/). First of all, the Pythran project (https://github.com/serge-sans-paille/pythran) is a (subset of) Python to C++ compiler, that aims at optimizing "scientific" Python code. It also provides a full C++ implementation of a major set of the Numpy API. Some of the advantage of this implementation is that it supports expression templates and SIMD instructions (partially thanks to Boost.SIMD [1]). One of the limitation of the current Numpy support of Cython is that it relies on the original Numpy Python module for a lot of computations. The overall idea is to replace these calls by the Numpy implementation provided within the Pythran project. I'll discuss in this mail the various choices that have been made, why and some implementation details. Then we'll also show some benchmark to see the potential improvements, which is the point of all this in the end :) Pythran limitations ------------------- The Pythran Numpy implementation has some limitations: * array "views" are not supported. That means that arrays must be stored in contiguous memory. Fortran and C-style format are supported. * the endianness of the integers must be the same that the one of the targeted architecture (note that Cython has the same limitation) That's why we did two things: * the usage of the Pythran backend needs to be explicitly asked by the user by providing the --np-pythran flag to the Cython compiler, or by using the "np_pythran" flag to the cythonize call (for distutils) * in function arguments, Numpy buffers are replaced by fused types to be able to fall back in case of unsupported buffers. More on this below. Implementation choices and details within Cython ------------------------------------------------ a) PythranExpr We defined a new type in PyrexTypes.py, which defines a Pythran buffer or expression. A Pythran expression is associated to a Pythran expression template, whose C++ type can be something like "decltype(a+b)". We thus compose every expression/function call like this, which allows us to use Pythran's expression template mechanism. We also choose to let the C++ compiler deduced the final type of every expression, and emit errors if something goes wrong. This choice allows not to have to rewrite in Python all the (potentially implicit) conversion rules that can apply in a C/C++ program, which could be error prone. The disadvantage is that it may generate not that trivial error messages for the end-user. b) Fused types for function arguments As Pythran has some limitations about the Numpy buffers it can support, we chose to replace Numpy buffer arguments by a fused type that can be either a Pythran buffer or the original Numpy buffer. The decision is made to use one type or another according to the limitations described above. This allows a fallback to the original Cython implementation in case of an unsupported buffer type. Tests ----- A flag has been added to the runtests.py script. If provided with a path to a Pythran installation, it will run the C++ tests in "Pythran" mode. This allows to reuse the whole test suite of Cython. Benchmark --------- The whole idea of this is to get better performances. Here is a simple benchmark of what this mode can achieve, using this cython code: def sqrt_sum(numpy.ndarray[numpy.float_t, ndim=1] a, numpy.ndarray[numpy.float_t, ndim=1] b): return numpy.sqrt(numpy.sqrt(a*a+b*b)) On my computer (Core i7-6700HQ), this gives there results, with an array of 100000000 32-bit floats as input: - for the classical Cython version: 960ms - for the Cython version using the Pythran backend: 761ms - for the Cython version using the Pythran backend using SIMD instructions: 243ms which makes a speedup of ~3.9x using SIMD instructions. Documentation ------------- I put an example of how to use this with distutils in the documentation. It could be put elsewhere if needed, or formatted differently. I'd be happy to discuss the various choices made here, and the implementation details. Thanks everyone! [1]: https://github.com/NumScale/boost.simd

2 1

Automatically add include_dirs for externs (#1654)
by Jeroen Demeyer April 12, 2017

April 12, 2017

Dear Cython developers, as part of a big effort to make SageMath less monolithic and make it part of a larger ecosystem, we are splitting off certain modules from SageMath as separate packages. cysignals was an early example of this. Given that we use Cython a lot, it is really important for us that inter-package C and Cython interfaces work well. One blocking issue is https://github.com/cython/cython/pull/1654 That PR improves the way how "cdef extern from" files are found: it will automatically add a disutils include_dirs argument to the Extension if it's needed to find to extern file. It is needed whenever you have a package which installs .pxd and .h files in site-packages. Cheers, Jeroen.

2 1

Treatment of right-hand side after assignment
by Clemens Hofreither April 10, 2017

April 10, 2017

Hey all, (I hope this list isn't dead.) I'm reading up on the Cython codebase because I'm trying to implement a small feature (more on that later). There's one thing that really confuses me currently, and that's the distinction between generate_post_assignment_code and generate_disposal_code. From the comments, it appears that exactly one of these should be called, and which one it is depends on whether a reference from the rhs was absorbed into the lhs. So much for the theory, but in practice it seems to work differently. I'm looking at NameNode.generate_assignment_code() and IndexNode.generate_assignment_code(), which should do similar things (generate code for an assignment) except that the lhs has different structure. Yet they treat the cleanup of the rhs completely differently. IndexNode always calls rhs.generate_disposal_code. NameNode almost always calls rhs.generate_post_assignment_code. Can anyone shed light on why this cleanup is handled differently depending on the target of the assignment? Thanks, Clemens

2 3