Mailman 3 July 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Memory views: dereferencing pointer does break strict-aliasing rules
by Yury V. Zaytsev Oct. 11, 2013

Oct. 11, 2013

Hi, The simplest possible program using memory views compiles with a large number of warnings for me, even for a rather outdated version of gcc: def hello(int [:] a): print(a, "world") If I translate it with the latest released version of Cython like this: cython cpp.pyx cython --cplus cpp.pyx and compile like this: gcc -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.c -o cpp.o g++ -O3 -march=native -Wall -fPIC -I/opt/ActivePython-2.7/include/python2.7 -c ./cpp.cpp -o cpp.o I get lots of warnings (see attached). It doesn't seem to be related to C++ as such, but rather it seems that the memory views code indeed somehow violates strict-aliasing rules. I'm not sure of how severe it is, but the documentation seems to suggest that this might even lead to incorrect results. Can this possibly be fixed in Cython and how important is that? Shall I create a bug report on the Trac? Is my only resort to test whether the compiler supports -fno-strict-aliasing and use that? Thanks! -- Sincerely yours, Yury V. Zaytsev

4 6

patch for #655
by Felix Salfelder Aug. 5, 2013

Aug. 5, 2013

Hi there. i am preparing a patch for #655 (write dependency makefiles) at [1]. I think it's best to use the -M, -F and -D short options for this functionality. please comment, improve and/or merge. regards felix [1] http://trac.sagemath.org/sage_trac/ticket/14728

3 15

Surprising behaviour wrt. generated tp_clear and tp_dealloc functions
by Torsten Landschoff Aug. 1, 2013

Aug. 1, 2013

Hi Cython-Developers, I am using Cython to generate a trivial wrapper for a small subset of the (already small) interface of libp11 (see https://github.com/OpenSC/libp11 for information about libp11). I know a bit about Python extension modules and just wanted to avoid writing all that boiler plate code and decided to given Cython a try. The wrapper was done in a day and no big deal, but lately I got random segmentation faults using it. After a day of debugging I found the cause in my use of the __dealloc__ special method. You may now call me stupid, because it is all in the docs: /You need to be careful what you do in a //__dealloc__()//method. By the time your //__dealloc__()//method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don't call any other methods of the object or do anything which might cause the object to be resurrected. It's best if you stick to just deallocating C data.// / http://docs.cython.org/src/userguide/special_methods.html?highlight=__deall… But this did not give me the crucial hint: Currently, any references to Python objects *may* be invalid at the time dealloc is called. This is because Cython generates a tp_clear function for each extension type that calls Py_CLEAR for each object reference. What I would really like to have is a possibility to exempt object attributes from being cleared in tp_clear, by adding a modifier to the cdef: *Exempt all attributes:* cdef noclear class Slots: cdef Context context *Exempt a single attribute:* cdef class Slots: cdef noclear Context context This is probably enough of explanation for the Cython experts, but I would still like to explain what happened in my case and a work around for illustration purposes and in the hope, that this will help somebody else at some time. Background When using the libp11 API, the client must create a Context object which is used for most operations. To query the list of slots where a card (token) could be inserted, there is a PKCS11_enumerate_slots function which returns a newly allocated buffer which contains all slots. Unfortunately this means that the resulting Slot objects can not be deallocated individually, only the whole buffer can be released using PKCS11_release_all_slots. This is completely unpythonic which is why I created a special Slots extension type that manages the storage of the Slot instances. When all Slot instances drop their reference to the Slots instance, the refcount of the Slots instance drops to zero and Python will release it automatically. The Slots type then calls the PKCS11_release_all_slots function to release the storage. This is modelled inside the __dealloc__ method for Slots. Unfortunately, a pointer to the context is required for that call. Therefore Slots also keeps a reference to Context object to keep it alive long enough and to identify the underlying C object. But: At the time when __dealloc__ is called, the tp_clear function already has cleared the context reference. Interestingly, it does not use Py_CLEAR as mandated in the Python documentation (see http://docs.python.org/2/c-api/typeobj.html?highlight=py_clear#PyTypeObject… and http://docs.python.org/3.3/c-api/typeobj.html?highlight=py_clear#PyTypeObje…) but instead redirects all Python objects to Py_NONE, which partially hides the problem. Somehow, this is not deterministic and tp_clear may have been called or not before tp_dealloc, therefore the random crashes. Even more funny: tp_clear seems to get only called if the Slots instance is involved in a reference cycle. Example Code I attached an example Cython project. If you build and install it with Cython 0.19, the following Python code will work: from fakep11 import * def works(): ctx = Context() my_slots = ctx.slots() while my_slots: assert ctx.usage_counter == 1 my_slots.pop() assert ctx.usage_counter == 0 But the following code will crash when the garbage collector runs: def crashes(): ctx = Context() slots = ctx.slots() a = {"slot": slots[0]} b = {"slot": slots[1], "cycle": a} a["cycle"] = b This is because a and b refer to each other, creating a cycle, and both refer to a slot, pulling the Slots instance into the object graph of the cycle. For this reason, tp_clear will be called for slots before tp_dealloc is called, leaving tp_dealloc with invalid data. Conclusion 1. As somebody who wrote Python extension modules manually before, I fell into this trap because I never implemented tp_clear. Reference cycles were a non-issue to me, since the wrapped objects could not refer to Python objects. Interestingly, SWIG does not seem to have any support for calling tp_clear it seems. 2. IMHO tp_clear should really use Py_CLEAR which would at least make this case a solid segfault. 3. Generating tp_clear functions is a feature that can do some harm, there I propose to allow to disable it. Work around For my current project I worked around this problem by managing the Python references manually by declaring them as PyObject* as in this diff: diff -r f2afc4865bbc fakep11.pyx --- a/fakep11.pyx Sat Apr 20 23:43:00 2013 +0200 +++ b/fakep11.pyx Sun Apr 21 23:50:10 2013 +0200 @@ -1,5 +1,10 @@ cimport cython +cdef extern from "Python.h": + ctypedef struct PyObject + void Py_INCREF(PyObject *) + void Py_DECREF(PyObject *) + cdef extern from "fakep11c.h": ctypedef struct fakep11_context: @@ -35,17 +40,19 @@ @cython.internal cdef class Slots: - cdef Context context + cdef PyObject *context cdef unsigned int nslots cdef fakep11_slot *slots def __cinit__(self, Context context): - self.context = context + self.context = <PyObject*> context + Py_INCREF(self.context) self.slots = fakep11_enumerate(context.imp, &self.nslots) def __dealloc__(self): print "Slots dealloc" - fakep11_release_all(self.context.imp, self.slots, self.nslots) + fakep11_release_all((<Context>self.context).imp, self.slots, self.nslots) + Py_DECREF(self.context) def as_list(self): l = [None] * self.nslots That way I have full control over the reference counting and can make sure that context is a valid reference when __dealloc__ is called. Please find my fake libp11 wrapper example code attached, as well as a small patch for the documentation documenting the current status. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff@dynamore.de http://www.dynamore.de DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz

3 13

Windows Debug build improvement
by Wolfgang July 26, 2013

July 26, 2013

Hi, I tried to submit a improvement for the Windows build but the tracker is not accessible without a login. On Windows if someone does a Debug build of an extension the flag _DEBUG is set and so the Python Interpreter sets Py_DEBUG and for all extension modules "_d" is appended to load the debug version of a module. This is not really practical because then all modules and the Python Interpreter must be build in Debug mode. For some modules this is even not possible for Windows. :-( To do a debug build for a Cython generated extension with a normal Python Interpreter (none Debug) I have to patch the pyconfig.h file and undef _DEBUG or I must patch the generated c file from Cython to undef _DEBUG before pyconfig.h or Python.h is included. (and enable it afterwards) Is it possible to add a flag to Cython to generate code that does this ? Something like described in Boost.Python: http://hepunx.rl.ac.uk/BFROOT/dist/releases/26.0.0/boost/libs/python/doc/bu… It is enough to have a new Preprocessor Flag, if set, then surround the Python.h inclusion with a disabled _DEBUG. My workarround is to disable it before pyconfig.h (Python.h) include: #ifdef _DEBUG #undef _DEBUG #define _RESTORE #endif and enable it afterwards #ifdef _RESTORE #define _DEBUG #undef _RESTORE #endif Regards, Wolfgang

4 4

Windows Debug build improvement
by Wolfgang July 25, 2013

July 25, 2013

Hi, thank you. But I don't use setup.py/distutils for the build. I use cython as a code generator and include the generated files into a Visual Studio project. But that's no problem, VS has the ability of custom build steps. A simple build with distutils is not possible. Because I use not the VC version used to build the python executable. The external library has other dependencies. Also the project uses over 15 other Python libraries and none of them is available as a Windows debug version with "_d" extension. Basically this is a flaw in Python itself. For a Linux build of Python there is the "Py_DEBUG" flag used as indicator, set during config. For Windows this is not explicit, it is set if "_DEBUG" is set. And I think this implicit setting is not good. (because this flag is widely used elsewhere) But I have to workaround it, even if Python 3.4 or later changes it I still use Python 2.7. Boost Python does this in its configuration, see documentation: --- copied from boost documentation On unix-variant platforms, the debugging versions of Python's data structures will only be used if the symbol Py_DEBUG is defined. On many windows compilers, when extension modules are built with the preprocessor symbol _DEBUG, Python defaults to force linking with a special debugging version of the Python DLL. Since that symbol is very commonly used even when Python is not present, Boost.Python temporarily undefines _DEBUG when Python.h is #included from boost/python/detail/wrap_python.hpp - unless BOOST_DEBUG_PYTHON is defined. The upshot is that if you want “python debugging”and you aren't using Boost.Build, you should make sure BOOST_DEBUG_PYTHON is defined, or python debugging will be suppressed. --- Regards, Wolfgang > Message: 1 > Date: Wed, 24 Jul 2013 21:15:49 +0600 > From: "Nikita Nemkin" <nikita(a)nemkin.ru> > To: "Core developer mailing list of the Cython compiler" > <cython-devel(a)python.org> > Subject: Re: [Cython] Windows Debug build improvement > Message-ID: <op.w0qfcnj8yabthb@juga> > Content-Type: text/plain; charset=windows-1251; format=flowed; > delsp=yes > > On Thu, 18 Jul 2013 19:24:21 +0600, Wolfgang <tds333(a)gmail.com> wrote: > >> Hi, >> >> I tried to submit a improvement for the Windows build but the tracker is >> not >> accessible without a login. >> >> On Windows if someone does a Debug build of an extension the flag _DEBUG >> is >> set and so the Python Interpreter sets Py_DEBUG and for all extension >> modules >> "_d" is appended to load the debug version of a module. >> This is not really practical because then all modules and the Python >> Interpreter must be build in Debug mode. For some modules this is even >> not >> possible for Windows. :-( > > To debug my extensions on Windows (in Visual Studio), I just add > the appropriate compiler flags: > > extension = Extension( > ... > extra_compile_args=['/Zi', '/Od'], # generate PDB, disable > optimization > extra_link_args=['/DEBUG']) # preserve debug info > > Add to that symbol files for the Python release you are using > ("program database" links on this page > http://www.python.org/getit/releases/2.7.5/) > and you will have a comfortable debugging environment. > > > Best regards, > Nikita Nemkin > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > cython-devel mailing list > cython-devel(a)python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > ------------------------------ > > End of cython-devel Digest, Vol 30, Issue 17 > ******************************************** >

1 0

[Fwd: [cython-users] Conditional compilation of Cython code / preprocessor / external conditions]
by Yury V. Zaytsev July 22, 2013

July 22, 2013

Hi, After a bit more research, I found two options that I initially overlooked: 1) (Apparently undocumented) provisions to provide external attributes in cython_compile_time_env 2) Generation of the PXI file from the external build system I don't like (2) very much, so I've got an idea to extend (1) with the ability to pass -D options to cython compiler, like: cython -DHAVE_LIBFOO=1 Will any such pull request be accepted? -- Sincerely yours, Yury V. Zaytsev -------- Forwarded Message -------- From: Yury V. Zaytsev <yury(a)shurup.com> Reply-to: cython-users(a)googlegroups.com To: cython-users(a)googlegroups.com Subject: [cython-users] Conditional compilation of Cython code / preprocessor / external conditions Date: Mon, 22 Jul 2013 16:17:58 +0200 Hi folks, Is it possible to conditionally compile Cython code depending on some externally defined value? >From reading the documentation, I've got an impression that presently Cython doesn't have any kind of preprocessor, and I hope that I'm just missing something here: http://docs.cython.org/src/userguide/language_basics.html My use case is as follows: The application that I'm wrapping links against a number of libraries, and some might be unavailable at compile time, so the corresponding parts of the Cython code shouldn't be compiled either. During the build, a preprocessor constant is defined, such as HAVE_LIBFOO that I can get by including config.h or pass via the autotools build system to the external tools. However, I can't figure out how to use that from Cython... So far, I was able to come up with an idea of replicating the API of the missing libraries with dummies on the C++ level, such that the Cython code compiles either way. Or else, I can make a poor man's preprocessor based on sed and just strip parts marked for conditional compilation from the PYX files. Both solutions look really disgusting and unmaintainable. I would greatly appreciate better ideas! To me it sounds like a rather common problem, so somebody must have come up with a solution already... Thanks, -- Sincerely yours, Yury V. Zaytsev -- --- You received this message because you are subscribed to the Google Groups "cython-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe(a)googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

1 0

Cython compiler directives: c_string_encoding
by Hannes Röst July 19, 2013

July 19, 2013

Dear mailing list I am trying to compile a program with Cython using the compiler directives and I am running into some trouble. Specifically, I am trying to port a Cython 0.18 program to Cython 0.19 and due to changes how char * and Python str are handled, I need to set the c_string_encoding directive. Unfortunately, this fails for my project and also in the following testcase when I try to do it locally: cimport cython cdef class TestClass: def foo(self): with cython.c_string_encoding("ascii"): return If I replace the directive with "with cython.boundscheck(True):" the program compiles fine. Also adding "#cython: c_string_encoding=ascii" as the first line of the file works fine. However, adding a decorator '@cython.c_string_encoding("ascii")' to foo also crashes the compiler. Finally, compiling with "-X c_string_encoding=ascii" also works. I was following the documentation provided here http://docs.cython.org/src/reference/compilation.html . The error message that I get is attached at the end. Did I do something wrong or can somebody point me in the right direction? Best regards Hannes Roest Error message: inimalcase.pyx:6:13: Compiler crash in InterpretCompilerDirectives ModuleNode.body = StatListNode(minimalcase.pyx:1:0) StatListNode.stats[1] = StatListNode(minimalcase.pyx:3:5) StatListNode.stats[0] = CClassDefNode(minimalcase.pyx:3:5, as_name = u'TestClass', class_name = u'TestClass', module_name = '', visibility = 'private') CClassDefNode.body = StatListNode(minimalcase.pyx:5:4) StatListNode.stats[0] = DefNode(minimalcase.pyx:5:4, modifiers = [...]/0, name = u'foo', num_required_args = 1, py_wrapper_required = True, reqd_kw_flags_cname = '0') DefNode.body = StatListNode(minimalcase.pyx:6:13) StatListNode.stats[0] = WithStatNode(minimalcase.pyx:6:13) Compiler crash traceback from this point on: File "/home/hr/lib/Cython-0.19.1/Cython/Compiler/Visitor.py", line 168, in _visit return handler_method(obj) File "/home/hr/lib/Cython-0.19.1/Cython/Compiler/ParseTreeTransforms.py", line 1002, in visit_WithStatNode for directive in self.try_to_parse_directives(node.manager) or []: File "/home/hr/lib/Cython-0.19.1/Cython/Compiler/ParseTreeTransforms.py", line 861, in try_to_parse_directives directives.append(self.try_to_parse_directive(optname, args, kwds, node.function.pos)) File "/home/hr/lib/Cython-0.19.1/Cython/Compiler/ParseTreeTransforms.py", line 913, in try_to_parse_directive assert False AssertionError:

2 2

typo in buffers.pxd?
by Lev Givon July 18, 2013

July 18, 2013

The PyBuffer_FillInfo() function definition in the latest revision of Cython/Includes/cpython/buffer.pxd appears to be missing a parameter. Shouldn't it be something like the following? int PyBuffer_FillInfo(Py_buffer *view, object obj, void *buf, Py_ssize_t len, int readonly, int flags) except -1 -- Lev Givon http://www.columbia.edu/~lev/ http://lebedov.github.com/

1 0

Bug: C++ strings defined in a PXD file are empty
by Yury V. Zaytsev July 17, 2013

July 17, 2013

Hi, Please find a minimal reproducer attached: In [1]: import pxdbug This should show 'bad value': This should show 'good value': good value Is this a real bug or I'm missing something fundamental about the PXD files? If it's the former, shall I create an issue on the Trac or GitHub? Thanks! -- Sincerely yours, Yury V. Zaytsev

2 1