Mailman 3 October 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

porting mpi4py to PyPy runtime
by Lisandro Dalcin Nov. 6, 2012

Nov. 6, 2012

After successfully porting mpi4py to PyPy 1.9, I want to share a few issues I've encountered. The most serious onesare on PyPy's side, but Cython do require a cuple of very minor fixes. 1) Cython uses _PyLong_AsByteArray() to convert integers of unknown sizeof larger than long long. I had to cinclude the following code to get the C sources compile: static int _PyLong_AsByteArray(PyLongObject* v, unsigned char* bytes, size_t n, int little_endian, int is_signed) { PyErr_SetString(PyExc_RuntimeError, "PyPy: _PyLong_AsByteArray() not available"); return -1; } 2) PyPy's implementation of PyBuffer_FillInfo() has a bug, it ignores the "readonly" parameter instead of generating an error if readonly=1 and PyBUF_WRITABLE is on. I had to to cinclude the following code to monkeypatch it: static int PyBuffer_FillInfo_PyPy(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int flags) { if (view == NULL) return 0; if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && (readonly == 1)) { PyErr_SetString(PyExc_BufferError, "Object is not writable."); return -1; } if (PyBuffer_FillInfo(view, obj, buf, len, readonly, flags) < 0) return -1; view->readonly = readonly; return 0; } #define PyBuffer_FillInfo PyBuffer_FillInfo_PyPy 3) In PyPy, PyCode_GetNumFree() accepts a PyObject*, not PyCodeObject*. I've monkeypatched as follow: #define PyCode_GetNumFree(o) PyCode_GetNumFree((PyObject *)(o)) 4) mpi4py uses @classmethod decorator for many methods in cdef classes. I really do not know what's going on, but method lookups do not return the classmethod, but the original "unwrapped" method. I've found a way to trick it, but again, I have no idea why it is working. Basically, I have to execute hasattr() (in Python code, not Cython code) on every class method: The hack reduces to adding the following code at the very end of Cython code (note that I'm using "exec"): if PYPY: exec """ def _pypy_setup(): for klass in ( Status, Datatype, Request, Message, Op, Group, Info, Errhandler, Comm, Win, File, ): for name in klass.__dict__: meth = klass.__dict__[name] if (isinstance(meth, classmethod) or isinstance(meth, staticmethod)): hasattr(klass, name) _pypy_setup() del _pypy_setup """ I think (1) and (3) can be trivially handled in Cython, (3) is not an issue for Cython as PyBuffer_FillInfo() is not directly used. And about (4), we really need help from PyPy folks, they implement PyType_Modified() as a non-op (https://bitbucket.org/pypy/pypy/src/release-1.9/pypy/module/cpyext/typeobje…) but that is not playing well with Cython. Below, the relevant pieces of code in mpi4py containing all the code I've described previously. https://code.google.com/p/mpi4py/source/browse/src/atimport.h#324 https://code.google.com/p/mpi4py/source/browse/src/MPI/MPI.pyx#280 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 3

provision to name cython binaries
by Neal Becker Oct. 30, 2012

Oct. 30, 2012

Does cython build have a provision to name the binaries, with a suffix? This would make it easier to build co-existing versions for python2 and 3. For example: setup.py --suffix=3 to produce .../bin/cython3

2 1

array expressions
by mark florisson Oct. 28, 2012

Oct. 28, 2012

Hey, Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well. This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really. Mark

5 39

Fwd: Re: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers)
by Dag Sverre Seljebotn Oct. 27, 2012

Oct. 27, 2012

-------- Original Message -------- Subject: Re: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) Date: Sun, 28 Oct 2012 07:53:42 +1100 From: Tim Delaney <timothy.c.delaney(a)gmail.com> To: python-dev(a)python.org On 28 October 2012 07:40, Mark Shannon <mark(a)hotpy.org <mailto:mark@hotpy.org>> wrote: I suspect that stating and loading the .pyc files is responsible for most of the overhead. PyRun starts up quite a lot faster thanks to embedding all the modules in the executable: http://www.egenix.com/__products/python/PyRun/ <http://www.egenix.com/products/python/PyRun/> Freezing all the core modules into the executable should reduce start up time. That suggests a test to me that the Cython guys might be interested in (or may well have performed in the past).Â How much of the stdlib could be compiled with Cython and used during the startup process? How much of an effect would it have on startup times and these benchmarks if Cython-compiled extensions were used? I'm thinking here of elimination of .pyc interpretation and execution (stat calls would be similar, probably slightly higher). To be clear - I'm *not* suggesting Cython become part of the required build toolchain. But *if* the Cython-compiled extensions prove to be significantly faster I'm thinking maybe it could become a semi-supported option (e.g. a HOWTO with the caveat "it worked on this particular system"). Tim Delaney

1 0

mypy
by mark florisson Oct. 18, 2012

Oct. 18, 2012

Has anyone looked at this yet? http://www.mypy-lang.org/index.html

5 4

Fwd: [cython-users] can pointers be stored in Python's dict or list?
by Robert Bradshaw Oct. 11, 2012

Oct. 11, 2012

Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that <void*>object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw <robertwb(a)gmail.com> Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield <list(a)qtrac.plus.com> Cc: cython-users(a)googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield <list(a)qtrac.plus.com> wrote: > Hi Robert, > > On Fri, 14 Sep 2012 10:45:06 -0700 > Robert Bradshaw <robertwb(a)gmail.com> wrote: >> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <list(a)qtrac.plus.com> >> wrote: >> > Hi, >> > >> > (1) >> > >> > I am using Cython to provide a Python wrapper to a C library. >> > The C library has a function which gives a handle (a pointer) which is >> > then used to call other functions. >> > >> > I want to be able to store a dictionary of name-pointer pairs so that I >> > can resuse handles. >> > >> > This doesn't seem to work with dict or list because of the >> > Cannot convert 'Thing*' to Python object >> > error. >> > >> > Right now I have a static array and a dictionary: >> > >> > cdef Thing* pointers[30] >> > indexForThingName = {} >> > >> > but I'd really like to be able to store an arbitrary number of >> > pointers, ideally in a dict. >> > >> > Is this possible? >> >> Sounds like you want >> http://docs.python.org/release/3.1.5/c-api/capsule.html > > But that means I have to drop down to C. I want to stay in Python and > Cython. You're already "in C" if you're dealing with pointers. > I was hoping that Cython offered some kind of wrapper type, > something that at the Python level would look like: > > class Pointer(object): > def __init__(self, void *pointer): > cdef void *self.pointer = pointer # Doesn't work > > that could be put in a dict or list. Then I could do things like this: > > Pointer p = <void*>c_func_that_returns_a_pointer() > mylist.append(p) > ... > Foo *q = <Foo*>mylist.pop() How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = <void*>c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish). > Oh well, I'll see if I can use libcpp's map... > >> > (2) >> > >> > Also, I tried doing >> > >> > from libc.stdio cimport printf >> > >> > and then in my code: >> > >> > printf("%p", pointer) >> > >> > but although it compiled without error, nothing was printed. >> >> Did it perhaps not flush? You also might want to try adding a newline. > > I changed it to printf("%p\n", pointer) and now I get > Cannot convert 'void *' to Python object Are you still cimporting printf? - Robert

4 5

Translating docs into Japanese
by Yasushi Masuda Oct. 7, 2012

Oct. 7, 2012

Hello list, Recently I've worked on translating cython documents into Japanese, and just finished for quickstart/tutorial. It was mainly for me and my colleagues, but I think it would be useful for all cython users speaking Japanese, so I'd like to publish it via my company's unofficial site: http://omake.accense.com/static/doc-ja/cython/ I suppose that docs are permissible for translation, redistribution and publish, but if you authors of original docs have troubles, please let me know... Thanks for great library (and, of course, documents)! -- Yasushi Masuda http://ymasuda.jp/ whosaysni at twitter/gmail

2 2

Re: [Cython] [cython-users] beginner question: passing std::vector by reference
by Stefan Behnel Oct. 7, 2012

Oct. 7, 2012

Robert Bradshaw, 03.10.2012 06:55: > On Tue, Oct 2, 2012 at 12:38 AM, peter aberline wrote: >> Please excuse my basic question. I'm trying to pass a reference to a >> std::vector to a function and I'm getting a cpp compilaton error. Here's a >> minimum example of what I'm doing: >> >> test.py: >> >> import sys >> >> def vectortest(a, idx): >> print ('a[idx] is: ' + str(a[idx])) >> a[idx] += 1 >> print ('a[idx] is: ' + str(a[idx])) >> return a >> >> def run(): >> lst = range(1, 5) >> lstNew = vectortest(lst, 1) >> >> print ('lstNew[1] is :' + str(lstNew[1])) >> >> test.pxd >> import cython >> from libcpp.vector cimport vector >> >> import sys >> >> if sys.version_info < (3,): >> range = xrange >> >> cpdef vector[int] vectortest(vector[int] a, int idx) # <-- This works fine >> ('a' is pass by value?) >> >> #cpdef vector[int] vectortest(vector[int]& a, int idx) #<-- This generates >> an 'uninitialised reference' cpp compilation error ('a' is pass by >> reference?') >> >> @cython.locals(lst = vector[int]) >> cpdef run() >> >> >> Compilation error: >> >> $/test.cpp: In function ‘PyObject* __pyx_pw_4perf_1vectortest(PyObject*, >> PyObject*, PyObject*)’: >> $/test.cpp:640:21: error: ‘__pyx_v_a’ declared as reference but not >> initialized > > Looks like this was a bug introduced In 0.17.1 and already fixed in master. In that case, the fix (whatever it was) should also go into the 0.17 branch so that we can release within a suitable time frame. Stefan

1 0

import relative cimport
by Christoph Groth Oct. 2, 2012

Oct. 2, 2012

Hello, I believe that relative cimports would be useful in Cython. (For a motivation, see my recent posting on cython-user http://thread.gmane.org/gmane.comp.python.cython.user/7401) So I tried to assess how much work it would be to implement that feature, but that's a tough job as I do not have a good overview of Cython's considerable code base. Would allowing relative cimports be a rather small thing comparable to commit 599c34053 which introduced relative imports (http://codereview.appspot.com/1734044/patch/2001/3001)? Or are there any issues that require serious refactoring? Christoph

3 4