Mailman 3 June 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

patch for #655
by Felix Salfelder Aug. 5, 2013

Aug. 5, 2013

Hi there. i am preparing a patch for #655 (write dependency makefiles) at [1]. I think it's best to use the -M, -F and -D short options for this functionality. please comment, improve and/or merge. regards felix [1] http://trac.sagemath.org/sage_trac/ticket/14728

3 15

Surprising behaviour wrt. generated tp_clear and tp_dealloc functions
by Torsten Landschoff Aug. 1, 2013

Aug. 1, 2013

Hi Cython-Developers, I am using Cython to generate a trivial wrapper for a small subset of the (already small) interface of libp11 (see https://github.com/OpenSC/libp11 for information about libp11). I know a bit about Python extension modules and just wanted to avoid writing all that boiler plate code and decided to given Cython a try. The wrapper was done in a day and no big deal, but lately I got random segmentation faults using it. After a day of debugging I found the cause in my use of the __dealloc__ special method. You may now call me stupid, because it is all in the docs: /You need to be careful what you do in a //__dealloc__()//method. By the time your //__dealloc__()//method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don't call any other methods of the object or do anything which might cause the object to be resurrected. It's best if you stick to just deallocating C data.// / http://docs.cython.org/src/userguide/special_methods.html?highlight=__deall… But this did not give me the crucial hint: Currently, any references to Python objects *may* be invalid at the time dealloc is called. This is because Cython generates a tp_clear function for each extension type that calls Py_CLEAR for each object reference. What I would really like to have is a possibility to exempt object attributes from being cleared in tp_clear, by adding a modifier to the cdef: *Exempt all attributes:* cdef noclear class Slots: cdef Context context *Exempt a single attribute:* cdef class Slots: cdef noclear Context context This is probably enough of explanation for the Cython experts, but I would still like to explain what happened in my case and a work around for illustration purposes and in the hope, that this will help somebody else at some time. Background When using the libp11 API, the client must create a Context object which is used for most operations. To query the list of slots where a card (token) could be inserted, there is a PKCS11_enumerate_slots function which returns a newly allocated buffer which contains all slots. Unfortunately this means that the resulting Slot objects can not be deallocated individually, only the whole buffer can be released using PKCS11_release_all_slots. This is completely unpythonic which is why I created a special Slots extension type that manages the storage of the Slot instances. When all Slot instances drop their reference to the Slots instance, the refcount of the Slots instance drops to zero and Python will release it automatically. The Slots type then calls the PKCS11_release_all_slots function to release the storage. This is modelled inside the __dealloc__ method for Slots. Unfortunately, a pointer to the context is required for that call. Therefore Slots also keeps a reference to Context object to keep it alive long enough and to identify the underlying C object. But: At the time when __dealloc__ is called, the tp_clear function already has cleared the context reference. Interestingly, it does not use Py_CLEAR as mandated in the Python documentation (see http://docs.python.org/2/c-api/typeobj.html?highlight=py_clear#PyTypeObject… and http://docs.python.org/3.3/c-api/typeobj.html?highlight=py_clear#PyTypeObje…) but instead redirects all Python objects to Py_NONE, which partially hides the problem. Somehow, this is not deterministic and tp_clear may have been called or not before tp_dealloc, therefore the random crashes. Even more funny: tp_clear seems to get only called if the Slots instance is involved in a reference cycle. Example Code I attached an example Cython project. If you build and install it with Cython 0.19, the following Python code will work: from fakep11 import * def works(): ctx = Context() my_slots = ctx.slots() while my_slots: assert ctx.usage_counter == 1 my_slots.pop() assert ctx.usage_counter == 0 But the following code will crash when the garbage collector runs: def crashes(): ctx = Context() slots = ctx.slots() a = {"slot": slots[0]} b = {"slot": slots[1], "cycle": a} a["cycle"] = b This is because a and b refer to each other, creating a cycle, and both refer to a slot, pulling the Slots instance into the object graph of the cycle. For this reason, tp_clear will be called for slots before tp_dealloc is called, leaving tp_dealloc with invalid data. Conclusion 1. As somebody who wrote Python extension modules manually before, I fell into this trap because I never implemented tp_clear. Reference cycles were a non-issue to me, since the wrapped objects could not refer to Python objects. Interestingly, SWIG does not seem to have any support for calling tp_clear it seems. 2. IMHO tp_clear should really use Py_CLEAR which would at least make this case a solid segfault. 3. Generating tp_clear functions is a feature that can do some harm, there I propose to allow to disable it. Work around For my current project I worked around this problem by managing the Python references manually by declaring them as PyObject* as in this diff: diff -r f2afc4865bbc fakep11.pyx --- a/fakep11.pyx Sat Apr 20 23:43:00 2013 +0200 +++ b/fakep11.pyx Sun Apr 21 23:50:10 2013 +0200 @@ -1,5 +1,10 @@ cimport cython +cdef extern from "Python.h": + ctypedef struct PyObject + void Py_INCREF(PyObject *) + void Py_DECREF(PyObject *) + cdef extern from "fakep11c.h": ctypedef struct fakep11_context: @@ -35,17 +40,19 @@ @cython.internal cdef class Slots: - cdef Context context + cdef PyObject *context cdef unsigned int nslots cdef fakep11_slot *slots def __cinit__(self, Context context): - self.context = context + self.context = <PyObject*> context + Py_INCREF(self.context) self.slots = fakep11_enumerate(context.imp, &self.nslots) def __dealloc__(self): print "Slots dealloc" - fakep11_release_all(self.context.imp, self.slots, self.nslots) + fakep11_release_all((<Context>self.context).imp, self.slots, self.nslots) + Py_DECREF(self.context) def as_list(self): l = [None] * self.nslots That way I have full control over the reference counting and can make sure that context is a valid reference when __dealloc__ is called. Please find my fake libp11 wrapper example code attached, as well as a small patch for the documentation documenting the current status. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff@dynamore.de http://www.dynamore.de DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz

3 13

Python type assignment errors (was: [cython-users] How to prevent garbage collecting an object?)
by Stefan Behnel June 24, 2013

June 24, 2013

Robert Bradshaw, 25.06.2013 05:15: > On Mon, Jun 24, 2013 at 3:01 PM, Zak wrote: >> I have tried a few things, and I always get >> the error at run-time. Cython is certainly doing type checking, but if the >> expected type (dict) and the actual type (Python integer) are both Python >> types and not C types, it seems the error always appears at run-time, never >> at compile-time. For instance, this compiles fine: >> >> cdef dict first_func(int x): >> return x >> >> cdef int second_func(int x): >> cdef dict y >> y = first_func(x) >> return 5 >> >> The code above causes a run-time error, but I feel that in an ideal world it >> should be a compile-time error. > > Yes, it should be. Agreed. Cython has inherited this behaviour from Pyrex which originally only knew "object", and we didn't do much about it since. There are rather fuzzy limits to this, though. For example, this would be stupid but legal wrt. language semantics: cdef dict func(): return None cdef list x = func() So, the only case that we can really handle is when we know the typed value originated from a C type that cannot coerce to None nor to the expected type, i.e. essentially this case: cdef int someting = 5 cdef dict x = something whereas only slightly more involved code would end up passing through the analysis, at least for now: cdef int something = 5 cdef dict x = something + 999 # unknown integer size => object I.e., this can only be handled during coercion of C types to known (builtin) Python object types, not during assignment - although we do have the may_be_none() predicate for nodes, and although there's still Vitja's pending inference rewrite which I didn't have time to look into any recently. Both can be used to push the fuzzy border a bit further into the right direction. Stefan

1 0

ANN: XDress v0.2
by Anthony Scopatz June 24, 2013

June 24, 2013

Hello All, I am pleased to announce the latest version of xdress, in preparation for SciPy 2013. For more information please visit the website: http://xdress.org Be Well Anthony ======================== XDress 0.2 Release Notes ======================== XDress is an automatic wrapper generator for C/C++ written in pure Python. Currently, xdress may generate Python bindings (via Cython) for C++ classes, functions, and certain variable types. It also contains idiomatic wrappers for C++ standard library containers (sets, vectors, maps). In the future, other tools and bindings will be supported. The main enabling feature of xdress is a dynamic type system that was designed with the purpose of API generation in mind. Release highlights: - First class support for C via pycparser. - Python 3 support, by popular demand! - A plethora of awesome type system updates, including: - type matching - lambda-valued converters - function pointer support - Easy to use and implement plugin architecture. Please visit the website for more information: http://xdress.org/ Ask questions on the mailing list: xdress(a)googlegroups.com Download the code from GitHub: http://github.com/scopatz/xdress XDress is free & open source (BSD 2-clause license) and requires Python 2.7, NumPy 1.5+, Cython 0.19+, and optionally GCC-XML, pycparser, and lxml. New Features ============ First Class C Support --------------------- Wrapping C code is now fully handled through with the optional pycparser. This means that you don't have to worry about whether or not the GCC-XML parser will work on your particular bit of code. Furthermore, C structs and their members are now handled idiomatically. (C++ structs are actually treated as C++ classes, which means that their are allowed to have constructors and other C++ concepts not present in C.) Python 3 Support ---------------- The entire code base is built and tested under Python 3.3. This uses the single code base model, as most development takes pace in Python 2.7. The Cython code that is generated may be used by both Python 2 & 3. Type System ----------- The type system has been expanded and hardened to handle additional use cases, largely motivated by the desire for realistic C support. A new function pointer (``fucntion_pointer``) refinement type has been added. When converting from C to Python, a new function object is created that wraps the underlying call. For converting from Python to C, a virtual table of callback functions is constructed that have the same signature of the pointer but hold a reference to a Python function. The size of the table and thus how many callbacks you can have before overwriting previous ones, is set by the ``max_callbacks`` key in the extra dictionary in class descriptions. This defaults to 8. A new enum refinement type now also comes stock. These may be exposed to Python in the ``rc.variables`` list. The type system now also comes with basic type matching tools. There is a new ``TypeMatcher`` class, a ``matches()`` function, and a singleton ``MatchAny`` that may be used for determining whether a type adheres to a pattern. The TypeMatcher class itself is immutable and hashable and therefore may be used anywhere other type elements (tuples, str, int) may be used including as dict keys! This is helpful for specifying conversions for large groups of types. Finally, the type system conversion dictionaries now accept callable objects as values. This was put in to handle templated types where the number of argument is not in general know beforehand, e.g. enums and function pointers. The values must be callable with only a single argument -- the type itself. For example, ``lambda t: rtnvalue`` is valid. Plugins! -------- XDress is a suite of tools written on top of a type system. Thus the entire core has been refactored to implement a very nimble and easy plugin system. The plugin mechanism enables external projects with their own code generators to easily hook into the xdress tools. Additionally, this allows users to decide at run time which plugins they want to use. Mini-FAQ ======== * Why not use an existing solution (eg, SWIG)? Their type systems don't support run-time, user provided refinement types, and thus are unsuited for verification & validation use cases that often arise in computational science. Furthermore, they tend to not handle C++ dependent types well (i.e. vector<T> does not come back as a np.view(..., dtype=T)). * Why GCC-XML and not Clang's AST? I tried using Clang's AST (and the remnants of a broken visitor class remain in the code base). However, the official Clang AST Python bindings lack support for template argument types. This is a really big deal. Other C++ ASTs may be supported in the future -- including Clang's. * I run xdress and it creates these files, now what?! It is your job to integrate the files created by xdress into your build system. Join in the Fun! ================ If you are interested in using xdress on your project (and need help), contributing back to xdress, starting up a development team, or writing your own code generation plugin tool on top of the type system and autodescriber, please let us know. Participation is very welcome! Authors ======= - `Anthony Scopatz <http://scopatz.com/>`_ - Spencer Lyon * - Gerald Dalley * - Alexander Eisenhuth * An * indicates a first time contributor. Links ===== 1. Homepage - http://xdress.org/ 2. Mailing List - xdress(a)googlegroups.com 3. GitHub Organization - https://github.com/xdress 4. Pycparser - https://pypi.python.org/pypi/pycparser

1 0

Conditional cast and builtin types
by Nikita Nemkin June 15, 2013

June 15, 2013

Hi, I have just discovered that the following operations do not perform any type checking ("?" is ignored): <unicode?>obj, <str?>obj, <bytes?>obj, <list?>obj, <dict?>obj. Can someone please confirm this as a bug? Best regards, Nikita Nemkin

3 2

Time to close relative import ticket on Trac?
by W. Trevor King June 15, 2013

June 15, 2013

Browsing around today I noticed that the relative import ticket is still open [1], even though support for relative imports landed in 0.15 [2]. There's also CEP 307 (Generators and PEP 342 Coroutines) [1], which is still listed as “planning, undecided” but also seems to be fixed in 0.15. The RTD page is also frozen at 0.15pre [3]. Cheers, Trevor [1]: cython-devel(a)python.org [2]: http://wiki.cython.org/ReleaseNotes-0.15 [3]: https://cython.readthedocs.org/en/latest/ -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

1 0

Funny idea: interpreted def functions
by Nikita Nemkin June 11, 2013

June 11, 2013

Hi, Pure Python functions rarely benefit from compilation. I thought it would be interesting to add an "interpreted" directive (global, module, class, function level + automatic heuristic) that will instruct Cython to compile def functions into _bytecode_ and store that bytecode in the binary. Together with module bundling and embed/freeze it could make a neat deployment solution. (I have no plans to implement this.) Best regards, Nikita Nemkin

2 2

AC�LIO JO�O CARLOS MOREIRA DE CARVALHO AMORIM NETO
by Michael Klein June 11, 2013

June 11, 2013

GOULART JUCIMARA VICENTE DOS SANTOS, DAMIANA PEREIRA DE OLIVERIA, ANA ANGELICA PEREIRA ALVES, PAULO DE SIQUEIRA SILVA. PAULO DE SIQUEIRA SILVA. AMAURI CLIFE, ELEUTÉRIO LEAL, LEAL CORLETTO, CLIFE ACÍLIO LEAL. ELEUTÉRIO GOULART, VICTOR DI MELLO, JAVERT MONTEIRO, NELSON DANTAS, DANIEL DE OLIVEIRA, RICARDO CORTE REAL, FLÁVIO SILVINO, LUIS GUSTAVO, BABU SANTANA, SERAFIM GONZALEZ, HARILDO DEDA, MARCOS FROTA, CARLOS VERGUEIRO, PAULO GIARDINI, ELIEZER MOTTA, JULIANO CAZARRÉ, ANDERSON LAU ANTUNES, ACILINO. AFONSO CLÁUDIO, INDAIABIRA, FERNANDO PEDROZA IGARATINGA. AMAURI ANTUNES, ACILINO. AFONSO CLÁUDIO, INDAIABIRA, FERNANDO PEDROZA JOÃO CARLOS MOREIRA DE CARVALHO OURO VERDE DE MINAS PAULO DE SIQUEIRA SILVA BALDOMERO GOULART, CLIFE LEAL.

1 0

Feature proposal: conditional cast with None
by Nikita Nemkin June 7, 2013

June 7, 2013

Hi, Currently, conditional casts like <Name?>obj do not allow None values. I have found it useful to allow None sometimes. The syntax could be one of: * <Name or None?>obj * <Name? or None>obj * <Name??>obj Use case (without this feature): tn_obj = self.any.non_trivial.nullable.expression() cdef TypeName tn if tn_obj is not None: tn = <TypeName?>tn_obj ... Use case (with this feature): cdef TypeName tn = <TypeName or None?>self.any.non_trivial.nullable.expression() if tn is not None: ... As you can see, without this feature, two local variables pointing to the same object are required. This creates unnecessary confusion. Implementation is trivial, all that is necessary is to pass "notnone=False" flag from parser to TypecastNode to PyTypeTestNode. What do you think? Best regards, Nikita Nemkin

2 2