Mailman 3 February 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

line tracing/profiling code objects
by Syam Gadde March 27, 2014

March 27, 2014

Hi all, I tried using line tracing in conjunction with line_profiler/kernprof to attempt to see if line-based profiling works in Cython. I think it's pretty close. But I think the problem is that line_profiler is getting a different PyCodeObject when it wraps the function and when it actually gets the trace call. It adds the initial code object to a map, and later when it gets the trace call, decides that the trace call is not something it needs to pay attention to because the new code object that it gets is not the same as the original one. The first code object is created at function declaration by __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to a variable __pyx_k_codeobj_NNN. The second code object is created, essentially, during the first entry into the function (in __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting __pyx_frame_code to the initial code object before calling TraceCall() would fix this. Is this easy to do? I'd do it myself, but I'd need help figuring out how to get the name of the appropriate __pyx_k_codeobj_NNN variable from within FuncDefNode.generate_function_definitions(), which calls put_trace_call(). Thanks for your help... -syam

2 2

0.20.2
by Stefan Behnel March 21, 2014

March 21, 2014

Hi, there are a couple of nice additions and fixes in the current master that could go out as a 0.20.2 some time soon. I suggest waiting another couple of days for more bug reports to come in and then just send it out. I'd like to also fix the buffer formatting bug by then, looks like I got a hold of it. Stefan

2 2

ANN: XDress v0.4
by Anthony Scopatz Feb. 27, 2014

Feb. 27, 2014

Hello All, I am *extremely *pleased to be able to announce the version 0.4 release of xdress. This version contains much anticipated full support for Clang as a parser! This is almost entirely due to the efforts of Geoffrey Irving. Please thank him the next time you get a chance :) This release also contains a lot of other goodies that you can read about in the release notes below. Happy Generating! Anthony XDress 0.4 Release Notes<http://xdress.org/previous/0.4_release_notes.html#xdress-0-4-release-notes> XDress is a numpy-aware automatic wrapper generator for C/C++ written in pure Python. Currently, xdress may generate Python bindings (via Cython) for C++ classes, functions, and certain variable types. It also contains idiomatic wrappers for C++ standard library containers (sets, vectors, maps). In the future, other tools and bindings will be supported. The main enabling feature of xdress is a dynamic type system that was designed with the purpose of API generation in mind. Release highlights: - Clang support! All kudos to Geoffrey Irving! - NumPy dtypes may be created independently of C++ STL vectors - A complete test suite refactor - Arbitrary source code locations - Global run control files - A plethora of useful bug fixes This version of xdress is *not* 100% backwards compatible with previous versions of xdress. We apologize in the name of progress. It represents ans impressive 245 files changed, 44917 aggregate line insertions (+), and 7893 deletions (-). Please visit the website for more information: http://xdress.org/ Ask questions on the mailing list: https://groups.google.com/forum/#!forum/xdress Download the code from GitHub: http://github.com/xdress/xdress XDress is free & open source (BSD 2-clause license) and requires Python 2.7+, NumPy 1.5+, Cython 0.19+, and optionally Clang, GCC-XML, pycparser, dOxygen, or lxml. New Features<http://xdress.org/previous/0.4_release_notes.html#new-features> Clang Support<http://xdress.org/previous/0.4_release_notes.html#clang-support> Through the herculean efforts of Geoffrey Irving xdress finally has full, first-class Clang/LLVM support! This is major advancement as it allows xdress to wrap more modern versions of C++ than GCC-XML can handle. Because of deficiencies in the existing libclang and Python bindings it was necessary for us to fork libclang for xdress in the short term. We hope to integrate these changes upstream. Clang versions 3.2 - 3.4 are supported. Independent NumPy Dtypes<http://xdress.org/previous/0.4_release_notes.html#independent-numpy-dtypes> In previous versions of xdress, to create a dtype of type T the user needed to declare the desire for a wrapper of an STL vector of type T. These two desires have now been separated. It is now possible to create a dtype via the dtypes run control parameter. STL vectors are still wrapped via dtypes. See the dtypes module for more information. Shiny New Test Suite<http://xdress.org/previous/0.4_release_notes.html#shiny-new-test-suite> The xdress test suite has been completely revamped to include both unit and integration tests which are run for all available parsers. The integration tests are accomplished though two fake projects - cproj and cppproj - on which the xdress CLI is run. These tests are now fully platform independent, unlike the previous BASH-based test suite. Source Paths<http://xdress.org/previous/0.4_release_notes.html#source-paths> Source file paths are now given by either their absolute or relative path. This allows source code to be located anywhere on the user's file system and enable the wrapping of dependencies or externally supplied libraries as needed. The run control parametersourcedir has been deprecated. Global Run Control Files<http://xdress.org/previous/0.4_release_notes.html#global-run-control-files> It is sometimes useful to be able to set system-wide run control parameters. XDress will now search the following files in order of increasing precedence. - $HOME/.xdressrc - $HOME/.xdressrc.py - $HOME/.config/xdressrc - $HOME/.config/xdressrc.py $HOME is the user's home directory. Settings in the project run control file take precedence over the values here. Major Bug Fixes<http://xdress.org/previous/0.4_release_notes.html#major-bug-fixes> - Debug file now always written when in debug mode. - STL sets of custom types now allowed. - Template parameters now allowed to be enum values. - Allow classes with no default constructor. Join in the Fun!<http://xdress.org/previous/0.4_release_notes.html#join-in-the-fun> If you are interested in using xdress on your project (and need help), contributing back to xdress, starting up a development team, or writing your own code generation plugin tool, please let us know. Participation is very welcome! Authors <http://xdress.org/previous/0.4_release_notes.html#authors> - Anthony Scopatz <http://scopatz.com/> - Geoffrey Irving * - James Casbon * - Kevin Tew * - Spencer Lyon - John Wiggins - Matt McCormick - Brad Buran - Chris Harris * - Gerald Dalley * - Micky Latowicki * - Mike C. Fletcher * - Robert Schwarz * An * indicates a first time contributor. Links <http://xdress.org/previous/0.4_release_notes.html#links> 1. Homepage - http://xdress.org/ 2. Mailing List - https://groups.google.com/forum/#!forum/xdress 3. GitHub Organization - https://github.com/xdress

1 0

Invalid C++ with Python and C++ iterators (not compiling CLang)
by Gael Varoquaux Feb. 21, 2014

Feb. 21, 2014

Hi, I'd like to enquire on the status of the problem raised last year: https://mail.python.org/pipermail/cython-devel/2013-April/003629.html To summarize, the following Cython leads to code that doesn't compile with Clang and is probably invalid C++: from libcpp.vector cimport vector as libcpp_vector from cython.operator cimport dereference as deref, preincrement as inc cdef class TestClass: cdef libcpp_vector[float] inst def __iter__(self): it = self.inst.begin() while it != self.inst.end(): yield deref(it) inc(it) This create the following C++ code: p->__pyx_v_it.std::vector<float>::iterator::~iterator(); which is invalid, but happens to work on GCC and MSVC. However, Clang support is starting to become very important, as it is the default compiler on MacOSX. The original reporter suggested a valid C++ code (see original email), which wasn't very convenient as it required a typedef. Any progress on the issue or suggestion to work around? Cheers, Gaël

2 1

Cython 0.20.1 released
by Robert Bradshaw Feb. 14, 2014

Feb. 14, 2014

I just pushed the 0.20.1 bugfix release. The changelist can be found at https://github.com/cython/cython/blob/0.20.x/CHANGES.rst , but most notable is the first entry which corrects a bug optimizing literal sequence multiplication. - Robert

3 3

Bug: Memoryview of struct with adjacent string fields does not detect string boundaries
by Joshua Adelman Feb. 8, 2014

Feb. 8, 2014

This discussion was initially started on the cython user google group, but I wanted to move the issue over to the dev list, as per the suggestion on cython_trac. Given a numpy recarray containing two or more fixed-length string fields, if those string fields are adjacent to one another cython does not properly detect the boundary between the string fields. A concise test case demonstrating the problem is: ```cython cimport numpy as np cdef packed struct tstruct: np.float32_t a np.int16_t b char[6] c char[4] d def test_struct(tstruct[:] x): pass ``` We then define some data on the python side: ```python import numpy as np a = np.recarray(3, dtype=[('a', np.float32), ('b', np.int16), ('c', '|S6'), ('d', '|S4')]) a[0] = (1.1, 1, 'abcde', 'fgh') a[1] = (2.1, 2, 'ijklm', 'nop') a[2] = (3.1, 3, 'qrstu', 'vwx') test_struct(a) ``` This results in the error: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-12-ac01118a36a7> in <module>() ----> 1 test_struct(a) ValueError: Expected a dimension of size 6, got 10 If we swap the order of the fields in the recarray and `tstruct` to (a,c,b,d) so that there is a numerical field between the string fields, then the function can parse the memory view correctly. The relevant line of code that catches the incorrect value of `enc_count` is: https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L468 ``` if (ctx->enc_count != ctx->head->field->type->arraysize[0]) { PyErr_Format(PyExc_ValueError, "Expected a dimension of size %zu, got %zu", ctx->head->field->type->arraysize[0], ctx->enc_count); return -1; } ``` My naive guess is that there is something going on in: https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L738 since that appears to be the only place where `enc_count` is being incremented. That would seem like the place where a boundary between two string fields might not be properly handled (the comment in the line above "Continue pooling same type" is suggestive. I'll cross-post this on the cython trac once I have access and will then submit a pull request on Github of a test case once I have the trac issue number. Josh

2 1

Bugfix release 0.20.1 release candidate
by Robert Bradshaw Feb. 8, 2014

Feb. 8, 2014

Tarball up at http://cython.org/release/Cython-0.20.1rc1.tar.gz This is a bugfix only release, primarily to address the sequence literal multiplication bug, but the set of improvements since the last release looked relatively safe so I've pulled in everything from master (if that turns out to be a problem we'll release with just this big fix). Let us know if it breaks anything; I hope to get this out soon. - Robert

1 0

CPython ticket on allowing non-ASCII names for extension modules
by Stefan Behnel Feb. 5, 2014

Feb. 5, 2014

There's a ticket about allowing extension modules to have non-ASCII names. http://bugs.python.org/issue20485 Stefan

1 0