Mailman 3 September 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

array expressions
by mark florisson Oct. 28, 2012

Oct. 28, 2012

Hey, Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well. This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really. Mark

5 39

Fwd: [cython-users] can pointers be stored in Python's dict or list?
by Robert Bradshaw Oct. 11, 2012

Oct. 11, 2012

Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw <robertwb(a)gmail.com> Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield <list(a)qtrac.plus.com> Cc: cython-users(a)googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list(a)qtrac.plus.com> wrote: > Hi Robert, > > On Fri, 14 Sep 2012 10:45:06 -0700 > Robert Bradshaw <robertwb(a)gmail.com> wrote: >> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list(a)qtrac.plus.com> >> wrote: >> > Hi, >> > >> > (1) >> > >> > I am using Cython to provide a Python wrapper to a C library. >> > The C library has a function which gives a handle (a pointer) which is >> > then used to call other functions. >> > >> > I want to be able to store a dictionary of name-pointer pairs so that I >> > can resuse handles. >> > >> > This doesn't seem to work with dict or list because of the >> > Cannot convert 'Thing*' to Python object >> > error. >> > >> > Right now I have a static array and a dictionary: >> > >> > cdef Thing* pointers[30] >> > indexForThingName = {} >> > >> > but I'd really like to be able to store an arbitrary number of >> > pointers, ideally in a dict. >> > >> > Is this possible? >> >> Sounds like you want >> http://docs.python.org/release/3.1.5/c-api/capsule.html > > But that means I have to drop down to C. I want to stay in Python and > Cython. You're already "in C" if you're dealing with pointers. > I was hoping that Cython offered some kind of wrapper type, > something that at the Python level would look like: > > class Pointer(object): > def __init__(self, void *pointer): > cdef void *self.pointer = pointer # Doesn't work > > that could be put in a dict or list. Then I could do things like this: > > Pointer p = <void*>c_func_that_returns_a_pointer() > mylist.append(p) > ... > Foo *q = <Foo*>mylist.pop() How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = <void*>c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish). > Oh well, I'll see if I can use libcpp's map... > >> > (2) >> > >> > Also, I tried doing >> > >> > from libc.stdio cimport printf >> > >> > and then in my code: >> > >> > printf("%p", pointer) >> > >> > but although it compiled without error, nothing was printed. >> >> Did it perhaps not flush? You also might want to try adding a newline. > > I changed it to printf("%p\n", pointer) and now I get > Cannot convert 'void *' to Python object Are you still cimporting printf? - Robert

4 5

Cython 0.17.1 released
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Hi all, I just released Cython 0.17.1. This is a bugfix-only release for the latest stable 0.17 release series. http://pypi.python.org/pypi/Cython/0.17.1 http://wiki.cython.org/ReleaseNotes-0.17.1 Two bugs were fixed in this release: * A reference leak was fixed in the new dict iteration code when the loop target was not a plain variable but an unpacked tuple. * Memory views did not handle the special case of a NULL buffer strides value, as allowed by PEP3118. Have fun, Stefan

2 2

ref-leak in 0.17
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Hi, there is a reference leak in the new dict iteration code in 0.17: http://trac.cython.org/cython_trac/ticket/790 I'll see if I can fix it and then release a pure bugfix 0.17.1 right afterwards. Please take a look at your own patch queues to see if there are any other important bugs that you would like to see fixed in that release. Stefan

3 8

Cython 0.17 final released
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Hello everyone, on behalf of the Cython project team, I'm proud to announce the final release of Cython 0.17. This is another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. It is also the first final release of an implementation of PEP 380 (generator delegation), before it will eventually appear in CPython 3.3. Download: http://cython.org/release/Cython-0.17.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-libra… http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/userguide/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 Have fun, Stefan

5 12

Re: [Cython] [cython-users] cython error return values and pure python mode
by Stefan Behnel Sept. 26, 2012

Sept. 26, 2012

Luca Dall'Olio, 26.09.2012 10:24: > I just converted some cython code using the "@cython.returns(a_type)" > annotation and it works like a charm, thanks for that! :-) It was mostly an oversight that it hasn't been available for so long. The pure Python mode is still incomplete and gradually improving, mostly because the core developers do not use it much themselves and rarely notice what's missing. So it's always worth a) bringing up issues (as you did here, thanks!) and b) coming up with proposals and patches to make it more widely usable. We warmly welcome any contributions. > I was wondering how to convert in pure python mode the error return value > "except" clause ( > http://docs.cython.org/src/userguide/language_basics.html#error-return-valu…) > : > > cdef int spam() except -1: > ... > > @cython.cfunc > @cython.returns(cython.int) > def spam(): > ... > > Is there any way to specify that -1 is the error return value in pure > python mode ? No, not as for as I can tell. I think it would be worth adding that to the 'returns' annotation, e.g. @cython.returns(cython.int, on_error="-1") for "except -1". Similarly: @cython.returns(cython.int, on_error="-1", check=True) for "except? -1" and @cython.returns(cython.int, on_error="+") @cython.returns(cython.void, on_error="*") for "except +" and "except *" respectively. OTOH, maybe "except *" could actually be the default in pure Python mode? Stefan

1 0

Auto-pickle progress?
by Ian Bell Sept. 25, 2012

Sept. 25, 2012

There is a proposal for auto-pickling of Cython extension types: http://wiki.cython.org/enhancements/pickling I was wondering whether there has been any progress on this front? Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python. Regards, Ian

3 4

cpdef, nogil and void return (was: [cython-users] Use the function abs in a cpdef function with nogil)
by Stefan Behnel Sept. 22, 2012

Sept. 22, 2012

[CC-ing cython-devel here as this is a design issue] Joaquin Cuenca Abela, 21.09.2012 19:32: > On Fri, Sep 21, 2012 at 6:12 PM, Chris Barker wrote: >> On Fri, Sep 21, 2012 at 2:48 AM, Joaquin Cuenca Abela wrote: >>> I don't set a return type, it gives the error: >>> >>> cpdef undo_filter_sub(int filter_unit, unsigned char[:] scanline, >>> ^ >>> ------------------------------------------------------------ >>> cpngfilters.pyx:10:6: Function with Python return type cannot be declared nogil >> >> cpdef means build both a pyton function and a C function -- C >> functions need a return type. Cython defaulats to a generic Python >> object as a return type, which could cause issue with nogil, so it >> won't let you do that -- so try declareing a void return type: >> >> cpdef void undo_filter_sub(int filter_unit, unsigned char[:] scanline, >> >> I suspect that will give you None for the Python version, but I'm just >> guessing here. >> >> If that doesn't work, use int or something, and ignore the return value. > > nope, with void I was getting "Cannot convert 'void' to Python > object". What I did is put int and return 0, maybe this should be > added to the docs. I think we should support "cpdef void func()" and just let the Python wrapper return None automatically. That's the expected Python equivalent of returning nothing. Shouldn't be hard to implement. cpdef functions that are also declared nogil are somewhat unusual, but I don't see them being outright wrong. Basically, it would allow you to call them from Cython code with or without holding the GIL as well as directly from Python code (that always holds the GIL). Might be handy for avoid code redundancy in some cases. Patches welcome, as always. Note, however, that declaring the return type as void will make exception handling more difficult. That's ok for a nogil function, which cannot raise exceptions, but it's worth remembering for normal functions. Stefan

2 1

read-only memory views
by Christoph Groth Sept. 19, 2012

Sept. 19, 2012

Hello, I have written a python extension module, tinyarray, (to be made public soon) which implements an important subset of numpy optimized for _small_ arrays. Tinyarrays are immutable, which makes them usable as dictionary keys. Some important operations, i.e. creation of a small array from a tuple or the scalar product of two small arrays are 10 to 45 times faster than with numpy. This can really boost the performance of programs which use small numpy arrays. (I know that numpy is meant to be used for large arrays, but there are legitimate uses of small arrays as well.) It would be great if Cython could support tinyarrays. Right now, it doesn't. See http://article.gmane.org/gmane.comp.python.cython.user/7329 (As a temporary workaround, I have to made the buffers writable. But this breaks strict immutability.) Would fixing this be a lot of work? Cheers, Christoph

3 5