Mailman 3 July 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

help getting NumPy into a 32-bit Jenkins build environment
by Stefan Behnel Aug. 3, 2012

Aug. 3, 2012

Hi, I've been trying to set up a 32bit build environment on Jenkins. Most of it works nicely by just switching to "gcc -m32", but NumPy fails to build with that setup due to the lack of dependencies. It actually finds them when looking for their header files in the wrong (64bit) places, and then fails to build when it can't find the expected 32bit libraries. Is there a way to tell NumPy that I really don't care about BLAS, LAPACK and all of its friends and just want bare arrays? Or is there an easy way to install 32 bit versions of those dependencies on sage.math? I'd be fine with having them built and installed into a local directory (namely, in standard directory layout under /home/scoder/jenkins/Tools/libs32/). Given that all current 32bit problems are in NumPy related tests, it wouldn't be all that helpful to exclude NumPy from the test runs... Stefan

2 2

Re: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
by mark florisson Aug. 3, 2012

Aug. 3, 2012

On 27 July 2012 23:30, Bradley Froehle <brad.froehle(a)gmail.com> wrote: > Thanks to the work of Yaroslav Halchenko, there is an experimental Debian > package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html > > However, the builds are showing a lot of test failures on non-amd64 sytems. > See https://buildd.debian.org/status/package.php?p=cython&suite=experimental > Many of the errors seem related to assumptions made about the sizes of > various integers (int vs. long vs long long). Ideally we'll be able to > clean up these failures before the final release. > > -Brad Thanks, I think it's mostly the tests that are wrong. I'll try to get it fixed on linux 32 bit.

5 14

Cython 0.17 beta 1 released
by Stefan Behnel July 26, 2012

July 26, 2012

Hello everyone, on behalf of the Cython project team, I'm proud to announce the release of our first beta of Cython 0.17. This is another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. Download: http://cython.org/release/Cython-0.17.beta1.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-libra… http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/tutorial/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 I'm expecting at least one release candidate to follow on this beta version, and a final release in early August. Please give this beta release as much testing as you can, so that we can quickly advance towards the final release. Have fun, Stefan

3 11

Bug, or changed array assignment in 0.17beta1?
by Mike Zaletel July 26, 2012

July 26, 2012

Hello All, The exact behavior of array assignment was never entirely clear to me, but I am certain the following behavior did not occur in 0.16: --------bug.pyx---------- def foo(): cdef int i cdef int* p1 = [4, 4] cdef int* p2 = [5, 5] print "p1:", for i in range(2): print p1[i], print "\np2:", for i in range(2): print p2[i], ----------------------------- which in Cython 0.17beta1 gives me >>>import bug >>>bug.foo() p1: 5 5 p2: 5 5 while in Cython 0.16 I get >>>import bug >>>bug.foo() p1: 4 4 p2: 5 5 Has the syntax of array assignment changed, or is this a bug? If the former, how do you assign to an array literal? Thanks for your all your efforts, Mike

5 8

Type inference and C++
by Vitja Makarov July 24, 2012

July 24, 2012

Here is part of original testcase: from cython.operator cimport dereference as d from cython.operator cimport preincrement as incr from libcpp.vector cimport vector def reverse_iteration_test(L): v = new vector[int]() for a in L: v.push_back(a) it = v.rbegin() while it != v.rend(): a = d(it) incr(it) print(a) It doesn't work with local type inference enabled since `a` in for-loop is infered as pyobject and in while loop it's inferred as reverse_iterator. Then I tried to comment out for loop and compile it with upstream/master, same error here: note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *' note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator' note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator' Error compiling Cython file: ------------------------------------------------------------ ... v = new vector[int]() #for a in L: # v.push_back(a) it = v.rbegin() while it != v.rend(): a = d(it) ^ ------------------------------------------------------------ cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator' Error compiling Cython file: ------------------------------------------------------------ ... # v.push_back(a) it = v.rbegin() while it != v.rend(): a = d(it) incr(it) print(a) ^ ------------------------------------------------------------ cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object I think it's not correct to infer `a` as reverse_iterator because it's not an iterator it's vector's item. -- vitja.

3 9

failing memory view tests in Python 2.4/5
by Stefan Behnel July 23, 2012

July 23, 2012

Hi, I've fixed some of the errors that current block the tests. The remaining failures in the memory view tests exist in both Py2.4 and 2.5, which hints at a problem with the buffer support in numpy.pxd. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND… The memslice test failure is annoying. It looks like slice() fails to properly deep-copy in Py2.[45]. I don't know if we can do anything about this, seems more like a CPython bug. Deep copying is used to split function definitions for fused types and I guess slice objects are used in combination with the memory slices. So both don't currently work together in Py2.[45]. I'd like to release a 0.17 alpha soon, so please get these sorted out in one way or another ASAP. I'm also fine with declaring these features unusable in older Python releases if we can't fix them, but it would obviously be better to get them working. Stefan

2 15

code generated for arguments of type "type"
by Stefan Behnel July 23, 2012

July 23, 2012

Hi, the following pull request tries to fix the fact that we currently generate arguments of type "PyObject*" when someone spells out "type" for them. https://github.com/cython/cython/pull/126 That's mostly a problem in CPython's C-API, but also in some NumPy functions. The correct way to translate this would be as a "PyTypeObject*", not a bare "PyObject*". Does anyone see a general problem with changing this in the compiler directly? I'm not sure that it won't break code, for example. We'll at least have to cast the arguments on the way in as well. Stefan

1 1

planning for 0.17
by Stefan Behnel July 22, 2012

July 22, 2012

Hi, I'd like to get an idea of what's still open for 0.17. Mark mentioned some open memoryview issues on his list and I know that there are still issues with PyPy, some of which could get fixed in a reasonable time frame. Also, Jenkins isn't all that happy yet. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/ What's the current state of the master branch for everyone? Anything that you're working on and/or that you think should go in but isn't yet? I would like to see 0.17 released some time next month, if possible. I don't currently see any real blockers, so that might be doable. The release notes look ok so far, but the bug tracker list is really short in comparison. Please add to both as you see fit. http://wiki.cython.org/ReleaseNotes-0.17 http://trac.cython.org/cython_trac/query?status=closed&group=component&orde… Stefan

3 11

Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
by Stefan Behnel July 19, 2012

July 19, 2012

[moving this to cython-devel as it's getting technical] Robert Bradshaw, 28.06.2012 21:46: > On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote: >> currently, when I write "new CppClass()" in Cython, it generates a straight >> call to the "new" operator. It doesn't do any error handling. And the >> current documentation doesn't even mention this case. >> >> Is there a "standard" way to handle this? It seems that C++ has different >> ways to deal with failures here but raises an exception by default. Would >> you declare the constructor(s) with an "except +MemoryError"? Is there a >> reason Cython shouldn't be doing this automatically (if nothing else was >> declared) ? > > I think it certainly makes sense to declare the default constructor as > "except +" (and std::bad_alloc should become MemoryError), Right. The code in the constructor can raise other exceptions that must also be handled properly. An explicit "except +" will handle that. > but whether > to implicitly annotate declared constructors is less clear, especially > as there's no way to un-annotate them. I agree, but sadly, it's the default behaviour that is wrong. I'm sure we made lots of users run into this trap already. I fixed the documentation for now, but the bottom line is that we require users to take care of proper declarations themselves. Otherwise, the code that we generate is incorrect, although it's 100% certain that an allocation error can occur, even if the constructor code doesn't raise any exceptions itself. Apparently, changing the behaviour of the "new" operator requires a special annotation "std::nothrow", which then returns NULL on allocation failures. You can pass that from Cython by hacking up a cname, e.g. Rectangle "(std::nothrow) Rectangle" (int w, int h) I'm sure there are users out there who figured this out (I mean, I did...) and use it in their code, so I agree that this isn't easy to handle because Cython simply wouldn't know what the actual error behaviour is for a given constructor and how to correctly detect an error. This problem applies only to heap allocation in that form. However, stack allocation and the new exttype field allocation suffer from similar problems when the default constructor raises an exception. Exttype fields are a particularly nasty case because the user has no control over the allocation. A C++ exception in the C++ class constructor would terminate the exttype constructor unexpectedly and thus leak resources (in the best case - no idea how CPython reacts if you throw a C++ exception through its type instantiation code). Similarly, a C++ exception in the constructor of a stack allocated object would then originate from the function entry code and potentially hit the Python function wrapper etc. Again, potentially leaking resources or worse. To me, this sounds like we should do something about it. At least for the implicit calls to the default constructor, we should generate "except +" code automatically because there is no other way to handle them safely. Stefan

5 33