Mailman 3 May 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Surprising behaviour wrt. generated tp_clear and tp_dealloc functions
by Torsten Landschoff Aug. 1, 2013

Aug. 1, 2013

Hi Cython-Developers, I am using Cython to generate a trivial wrapper for a small subset of the (already small) interface of libp11 (see https://github.com/OpenSC/libp11 for information about libp11). I know a bit about Python extension modules and just wanted to avoid writing all that boiler plate code and decided to given Cython a try. The wrapper was done in a day and no big deal, but lately I got random segmentation faults using it. After a day of debugging I found the cause in my use of the __dealloc__ special method. You may now call me stupid, because it is all in the docs: /You need to be careful what you do in a //__dealloc__()//method. By the time your //__dealloc__()//method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don't call any other methods of the object or do anything which might cause the object to be resurrected. It's best if you stick to just deallocating C data.// / http://docs.cython.org/src/userguide/special_methods.html?highlight=__deall… But this did not give me the crucial hint: Currently, any references to Python objects *may* be invalid at the time dealloc is called. This is because Cython generates a tp_clear function for each extension type that calls Py_CLEAR for each object reference. What I would really like to have is a possibility to exempt object attributes from being cleared in tp_clear, by adding a modifier to the cdef: *Exempt all attributes:* cdef noclear class Slots: cdef Context context *Exempt a single attribute:* cdef class Slots: cdef noclear Context context This is probably enough of explanation for the Cython experts, but I would still like to explain what happened in my case and a work around for illustration purposes and in the hope, that this will help somebody else at some time. Background When using the libp11 API, the client must create a Context object which is used for most operations. To query the list of slots where a card (token) could be inserted, there is a PKCS11_enumerate_slots function which returns a newly allocated buffer which contains all slots. Unfortunately this means that the resulting Slot objects can not be deallocated individually, only the whole buffer can be released using PKCS11_release_all_slots. This is completely unpythonic which is why I created a special Slots extension type that manages the storage of the Slot instances. When all Slot instances drop their reference to the Slots instance, the refcount of the Slots instance drops to zero and Python will release it automatically. The Slots type then calls the PKCS11_release_all_slots function to release the storage. This is modelled inside the __dealloc__ method for Slots. Unfortunately, a pointer to the context is required for that call. Therefore Slots also keeps a reference to Context object to keep it alive long enough and to identify the underlying C object. But: At the time when __dealloc__ is called, the tp_clear function already has cleared the context reference. Interestingly, it does not use Py_CLEAR as mandated in the Python documentation (see http://docs.python.org/2/c-api/typeobj.html?highlight=py_clear#PyTypeObject… and http://docs.python.org/3.3/c-api/typeobj.html?highlight=py_clear#PyTypeObje…) but instead redirects all Python objects to Py_NONE, which partially hides the problem. Somehow, this is not deterministic and tp_clear may have been called or not before tp_dealloc, therefore the random crashes. Even more funny: tp_clear seems to get only called if the Slots instance is involved in a reference cycle. Example Code I attached an example Cython project. If you build and install it with Cython 0.19, the following Python code will work: from fakep11 import * def works(): ctx = Context() my_slots = ctx.slots() while my_slots: assert ctx.usage_counter == 1 my_slots.pop() assert ctx.usage_counter == 0 But the following code will crash when the garbage collector runs: def crashes(): ctx = Context() slots = ctx.slots() a = {"slot": slots[0]} b = {"slot": slots[1], "cycle": a} a["cycle"] = b This is because a and b refer to each other, creating a cycle, and both refer to a slot, pulling the Slots instance into the object graph of the cycle. For this reason, tp_clear will be called for slots before tp_dealloc is called, leaving tp_dealloc with invalid data. Conclusion 1. As somebody who wrote Python extension modules manually before, I fell into this trap because I never implemented tp_clear. Reference cycles were a non-issue to me, since the wrapped objects could not refer to Python objects. Interestingly, SWIG does not seem to have any support for calling tp_clear it seems. 2. IMHO tp_clear should really use Py_CLEAR which would at least make this case a solid segfault. 3. Generating tp_clear functions is a feature that can do some harm, there I propose to allow to disable it. Work around For my current project I worked around this problem by managing the Python references manually by declaring them as PyObject* as in this diff: diff -r f2afc4865bbc fakep11.pyx --- a/fakep11.pyx Sat Apr 20 23:43:00 2013 +0200 +++ b/fakep11.pyx Sun Apr 21 23:50:10 2013 +0200 @@ -1,5 +1,10 @@ cimport cython +cdef extern from "Python.h": + ctypedef struct PyObject + void Py_INCREF(PyObject *) + void Py_DECREF(PyObject *) + cdef extern from "fakep11c.h": ctypedef struct fakep11_context: @@ -35,17 +40,19 @@ @cython.internal cdef class Slots: - cdef Context context + cdef PyObject *context cdef unsigned int nslots cdef fakep11_slot *slots def __cinit__(self, Context context): - self.context = context + self.context = <PyObject*> context + Py_INCREF(self.context) self.slots = fakep11_enumerate(context.imp, &self.nslots) def __dealloc__(self): print "Slots dealloc" - fakep11_release_all(self.context.imp, self.slots, self.nslots) + fakep11_release_all((<Context>self.context).imp, self.slots, self.nslots) + Py_DECREF(self.context) def as_list(self): l = [None] * self.nslots That way I have full control over the reference counting and can make sure that context is a valid reference when __dealloc__ is called. Please find my fake libp11 wrapper example code attached, as well as a small patch for the documentation documenting the current status. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff@dynamore.de http://www.dynamore.de DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz

3 13

CF based type inference
by Vitja Makarov June 2, 2013

June 2, 2013

Hi! Recently I've started work on new type inference engine. Now it's almost ready and I want to discuss it. It works like this: first infer type for each assignment then for whole entry. It has some advantages over previous algorithm: - it handles assignment cycles, see test_swap() for example - it can infer type using info about assignments on the whole entry: a = 1 b = a a = "str" a is python object and b is integer. Here are testcases that show some new cases it can solve: https://github.com/vitek/cython /blob/_type_inference_new/tests/run/type_inference_new.pyx Here is branch for it https://github.com/vitek/cython /tree/_type_inference_new -- vitja.

4 9

Core dump when Python is built without debug symbols
by Marin Atanasov Nikolov May 30, 2013

May 30, 2013

Hello, First, apologies if this is not exactly the right place to post this, but I have tried already cython-users@ and my post is still pending somewhere for approval, so I've decided to give it a shot here. I'm working on Cython wrappers for a C library and everything was going smooth on my dev machine, until I've decided to install the wrappers on another machine. The only difference between these machines is that my dev machine has Python built with debug symbols and the other machine does not have debug symbols in Python. The issue I'm having and was trying to solve for the past few days is that if I run my Cython wrappers on a machine where Python does not have debug symbols it core dumps ugly. If I run the same code on another machine with Python + debug symbols then everything is running perfect. I've done some debugging and tracing as well, and noticed that the core dump happens at PyTuple_Size() when I have Python *without* debug symbols. Running Python + debug symbols shows that the interpreter takes a different path and does not core dump (does not even call PyTuple_Size() for some reason) Anyway, I've uploaded the backtrace and some gdb tracing in here: * http://users.unix-heaven.org/~dnaeon/cython-wrappers/wrappers-trace.txt The code that I have you can find in here: * http://users.unix-heaven.org/~dnaeon/cython-wrappers/src/ What really puzzles me is why it core dumps when Python does not have debug symbols and why it takes a different path (and not core dumping) when Python is built with debug symbols. I'd say that if it core dumps because of a NULL pointer or something similar, then having Python with or without debug symbols should not make a difference, but it does for some reason. I've been working on this for the past few days and I'm out of ideas now. Any help/hints/feedback is much appreciated. Thanks and regards, Marin -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/

2 4

Exception check optimization
by Nikita Nemkin May 30, 2013

May 30, 2013

Hi, I wonder why is __pyx_filename (in exception check blocks) tracked dynamically? AFAIK it's impossible to split function body between multiple files (include only works at the top level), which makes filename a compile time constant for any given function. If the above is correct, __pyx_filename variable can be eliminated, saving at least 5 bytes per exception check (more on x64). On a related note, why is c_lineno enabled by default? So far I have only found one pretty rare case when it is useful - debugging autogenerated code in module init. I suspect that most people don't bother turning it off and get suboptimal code. Best regards, Nikita Nemkin

3 3

Compiler crash on unmatched kwarg
by Emmanuel Gil Peyrot May 30, 2013

May 30, 2013

Hi, When the name of a kwarg in the definition and the declaration of a function doesn’t match, the compiler crash instead of returning a meaningful error. I have joined a testcase, it is reproductible at least on 0.19.1 and latest git. -- Emmanuel Gil Peyrot XMPP: <linkmauve(a)linkmauve.fr> OpenPGP: 24B1D609

1 0

Demos is not build
by Alexander Belchenko May 29, 2013

May 29, 2013

I'm trying to build Demos as suggested in USAGE.txt and got error from linker: cheese.obj : error LNK2019: unresolved external symbol _find_cheeses referenced in function ___pyx_pf_6cheese_find C:\work\Python\Cython\Demos\cheese.pyd : fatal error LNK1120: 1 unresolved externals The same problem with released 0.19.1 and latest version from github. My system is Windows 7 64 bit with Python 2.7.5 32-bit installed on it.

1 0

Old style dependency tracking
by Nikita Nemkin May 28, 2013

May 28, 2013

Hi, I wonder if it's time to to remove old dependency tracking code? I mean this: http://trac.cython.org/cython_trac/ticket/379, CompilationOptions.recursive, .dep file generation etc. Since it is broken for a long time already, there is no problem with backwards (in)compatibility. And cythonize already provides equivalent functionality. Best regards, Nikita Nemkin

2 1

How should be C-clobal deletion handled?
by Vitja Makarov May 26, 2013

May 26, 2013

Hi! Recently I've found that the following code causes segmentation fault: cdef object f del f print f So the question is: how should that work? global objects are implicitly initialized to None and no CF and no cf analysis is performed for it. So I see three options here: 1. prohibit cglobal deletion 2. set it back to None 3. check for a null value at every reference and assignment -- vitja.

5 7

Compiler crash fix
by Nikita Nemkin May 24, 2013

May 24, 2013

Hi, Someone please apply this patch (too simple for pull request): diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py index 0406fad..640d002 100755 --- a/Cython/Compiler/ExprNodes.py +++ b/Cython/Compiler/ExprNodes.py @@ -4993,7 +4993,7 @@ class AttributeNode(ExprNode): # creates a corresponding NameNode and returns it, otherwise # returns None. type = self.obj.analyse_as_extension_type(env) - if type: + if type and type.scope: entry = type.scope.lookup_here(self.attribute) if entry and entry.is_cmethod: if type.is_builtin_type: It fixes CompilerCrash (None does not have "lookup_here" method) that I have observed on two occasions: 1) cdef class Name; cimport Name; Name.attr 2) from X cimport Name; Name.attr # cimport_from_pyx is active, Name is a class with errors Makes me wonder if ErrorScope should be introduced to avoid None scope checks. Best regards, Nikita Nemkin

3 2