Mailman 3 April 2013 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Surprising behaviour wrt. generated tp_clear and tp_dealloc functions
by Torsten Landschoff Aug. 1, 2013

Aug. 1, 2013

Hi Cython-Developers, I am using Cython to generate a trivial wrapper for a small subset of the (already small) interface of libp11 (see https://github.com/OpenSC/libp11 for information about libp11). I know a bit about Python extension modules and just wanted to avoid writing all that boiler plate code and decided to given Cython a try. The wrapper was done in a day and no big deal, but lately I got random segmentation faults using it. After a day of debugging I found the cause in my use of the __dealloc__ special method. You may now call me stupid, because it is all in the docs: /You need to be careful what you do in a //__dealloc__()//method. By the time your //__dealloc__()//method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don't call any other methods of the object or do anything which might cause the object to be resurrected. It's best if you stick to just deallocating C data.// / http://docs.cython.org/src/userguide/special_methods.html?highlight=__deall… But this did not give me the crucial hint: Currently, any references to Python objects *may* be invalid at the time dealloc is called. This is because Cython generates a tp_clear function for each extension type that calls Py_CLEAR for each object reference. What I would really like to have is a possibility to exempt object attributes from being cleared in tp_clear, by adding a modifier to the cdef: *Exempt all attributes:* cdef noclear class Slots: cdef Context context *Exempt a single attribute:* cdef class Slots: cdef noclear Context context This is probably enough of explanation for the Cython experts, but I would still like to explain what happened in my case and a work around for illustration purposes and in the hope, that this will help somebody else at some time. Background When using the libp11 API, the client must create a Context object which is used for most operations. To query the list of slots where a card (token) could be inserted, there is a PKCS11_enumerate_slots function which returns a newly allocated buffer which contains all slots. Unfortunately this means that the resulting Slot objects can not be deallocated individually, only the whole buffer can be released using PKCS11_release_all_slots. This is completely unpythonic which is why I created a special Slots extension type that manages the storage of the Slot instances. When all Slot instances drop their reference to the Slots instance, the refcount of the Slots instance drops to zero and Python will release it automatically. The Slots type then calls the PKCS11_release_all_slots function to release the storage. This is modelled inside the __dealloc__ method for Slots. Unfortunately, a pointer to the context is required for that call. Therefore Slots also keeps a reference to Context object to keep it alive long enough and to identify the underlying C object. But: At the time when __dealloc__ is called, the tp_clear function already has cleared the context reference. Interestingly, it does not use Py_CLEAR as mandated in the Python documentation (see http://docs.python.org/2/c-api/typeobj.html?highlight=py_clear#PyTypeObject… and http://docs.python.org/3.3/c-api/typeobj.html?highlight=py_clear#PyTypeObje…) but instead redirects all Python objects to Py_NONE, which partially hides the problem. Somehow, this is not deterministic and tp_clear may have been called or not before tp_dealloc, therefore the random crashes. Even more funny: tp_clear seems to get only called if the Slots instance is involved in a reference cycle. Example Code I attached an example Cython project. If you build and install it with Cython 0.19, the following Python code will work: from fakep11 import * def works(): ctx = Context() my_slots = ctx.slots() while my_slots: assert ctx.usage_counter == 1 my_slots.pop() assert ctx.usage_counter == 0 But the following code will crash when the garbage collector runs: def crashes(): ctx = Context() slots = ctx.slots() a = {"slot": slots[0]} b = {"slot": slots[1], "cycle": a} a["cycle"] = b This is because a and b refer to each other, creating a cycle, and both refer to a slot, pulling the Slots instance into the object graph of the cycle. For this reason, tp_clear will be called for slots before tp_dealloc is called, leaving tp_dealloc with invalid data. Conclusion 1. As somebody who wrote Python extension modules manually before, I fell into this trap because I never implemented tp_clear. Reference cycles were a non-issue to me, since the wrapped objects could not refer to Python objects. Interestingly, SWIG does not seem to have any support for calling tp_clear it seems. 2. IMHO tp_clear should really use Py_CLEAR which would at least make this case a solid segfault. 3. Generating tp_clear functions is a feature that can do some harm, there I propose to allow to disable it. Work around For my current project I worked around this problem by managing the Python references manually by declaring them as PyObject* as in this diff: diff -r f2afc4865bbc fakep11.pyx --- a/fakep11.pyx Sat Apr 20 23:43:00 2013 +0200 +++ b/fakep11.pyx Sun Apr 21 23:50:10 2013 +0200 @@ -1,5 +1,10 @@ cimport cython +cdef extern from "Python.h": + ctypedef struct PyObject + void Py_INCREF(PyObject *) + void Py_DECREF(PyObject *) + cdef extern from "fakep11c.h": ctypedef struct fakep11_context: @@ -35,17 +40,19 @@ @cython.internal cdef class Slots: - cdef Context context + cdef PyObject *context cdef unsigned int nslots cdef fakep11_slot *slots def __cinit__(self, Context context): - self.context = context + self.context = <PyObject*> context + Py_INCREF(self.context) self.slots = fakep11_enumerate(context.imp, &self.nslots) def __dealloc__(self): print "Slots dealloc" - fakep11_release_all(self.context.imp, self.slots, self.nslots) + fakep11_release_all((<Context>self.context).imp, self.slots, self.nslots) + Py_DECREF(self.context) def as_list(self): l = [None] * self.nslots That way I have full control over the reference counting and can make sure that context is a valid reference when __dealloc__ is called. Please find my fake libp11 wrapper example code attached, as well as a small patch for the documentation documenting the current status. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff@dynamore.de http://www.dynamore.de DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz

3 13

Re: [Cython] [cython-users] Re: Cython Typed Memoryviews versus Cython+Numpy Speed ups
by Stefan Behnel May 17, 2013

May 17, 2013

Robert Bradshaw, 17.04.2013 06:02: > On Tue, Apr 16, 2013 at 6:46 PM, pythonOmetrist >> Attached HTML seems mostly white, and not clear where to go from here. Nogil >> seems to be one option. > > Just a brief glance at the code, you have a lot of n_k_w[x][y] which > is vastly more expensive than n_k_w[x, y]. Is that always equivalent? If so, we might want to transform it internally. Stefan

2 1

Generated C++ code incompatible with Clang
by Hannes Röst May 7, 2013

May 7, 2013

Dear mailing list In the process of developing a rather large Cython project where we wrap the scientific OpenMS library (http://sourceforge.net/projects/open-ms/) using Cython, we have come across an issue with the C++ code that is generated by Cython. The issue is that Cython generates C++ that is not compatible with the Clang compiler. I hope this is the correct mailing list to report this issue since it seems to be a bug in Cython and not in Clang (after some discussion we have concluded that Clang is most likely correct in rejecting the generated code). Since we do not have access to the issue tracker, we hoped posting our issue here would clear up matters. The issue occurs with the following minimal testcase: from libcpp.vector cimport vector as libcpp_vector from cython.operator cimport dereference as deref, preincrement as inc cdef class TestClass: cdef libcpp_vector[float] inst def __iter__(self): it = self.inst.begin() while it != self.inst.end(): yield deref(it) inc(it) When compiled with Cython to C++, it generates C++ that cannot be compiled with Clang (however it seems that gcc and MSVS accept the code). It seems that the same issue with Clang was already discussed here: https://bugzilla.mozilla.org/show_bug.cgi?id=623303 and the conclusion was that Clang is correct in reporting the code as erroneous. In the above case, the invalid C++ code that gets generated is: p->__pyx_v_it.std::vector<float>::iterator::~iterator(); The correct code for the above case would probably be (this is just a suggestion, it compiles with gcc and clang on our machines and our tests run through with it): typedef std::vector<float>::iterator _it; p->__pyx_v_it.~_it(); We have tested it with the 0.18 release as well as the newest 0.19b2 build from github (58131b68dc033fc7ca269d875a2aab2b4e9646a2) and the results were the same. Thanks, Hannes Röst -- Hannes Röst Institute of Molecular Systems Biology ETH Zürich Hönggerberg HPT C75 Wolfgang-Pauli Strasse 16 CH-8093 Zürich (Switzerland) phone: 0041 44 633 3945 fax: 0041 44 633 10 51 e-mail: roest(a)imsb.biol.ethz.ch

2 3

Re: [Cython] [cython] Autodoc improvements (#216)
by Stefan Behnel May 1, 2013

May 1, 2013

Nikita Nemkin, 18.04.2013 07:07: > Three improvements are included in this patch that I have found helpful when using Sphinx autodoc to document my project: > > 1) Default values preserve and display binary expressions and attribute access. For example, the following function will have its signature displayed as is: > > cpdef func(int flags=MyModule.FLAG1 | MyModule.FLAG2) > > This only affects expressions not reducible at compilation time, i.e. `2+2` will appear as `4`. > > 2) Public/readonly cdef attributes can have docstrings attached, using the [PEP 258 attribute docstring syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings). Example: > > cdef class X: > cdef public attr > """attr docstring""" > > When ``embedsignature=True`` Cython already adds a signature docstring to the generated property (assuming the example above). This patch allows to add an actual docstring too. > > Dosctrings on private cdef attributes will give a warning ("docstring ignored"). > Other forms and uses of attribute docstrings, as described in the PEP 258, are not supported at all (SyntaxError), because there is no standard introspection mechanism designed for them. ([PEP 224](http://www.python.org/dev/peps/pep-0224/) describing such mechanism had been rejected.) > > 3) `bint` appears as `bool` (instead of `int`) in autogenerated signatures, as I suggested on the mailing list. > > Each change is contained in its own commit in case you want to review them separately. > > https://github.com/cython/cython/pull/216 I like 1) and 3), but I'm not sure about 2). Worth noting that PEP 258 was rejected, although definitely for other reasons than this feature. I remember discussing this topic before on the mailing list, but don't remember if we reached a conclusion. I faintly recall that we preferred making the property explicit as soon as there's more to it than a plain 1:1 mapping to the cdef attribute. The above syntax is not wrong, but it also doesn't feel right to me. For example, it's impossible to draw a distinction between the above and something like cdef class X: cdef public attr """ quotes used for commenting out code here. """ And users are unlikely to notice that they just dropped some internals into the docstring of a property they didn't even know they were implementing. IMHO, the user intention is way too unclear in this syntactic construct. Stefan

3 10

Biggest Fake Conference in Computer Science
by eliswilson＠hushmail.com April 30, 2013

April 30, 2013

Biggest Fake Conference in Computer Science We are researchers from different parts of the world and conducted a study on the world’s biggest bogus computer science conference WORLDCOMP http://sites.google.com/site/worlddump1 organized by Prof. Hamid Arabnia from University of Georgia, USA. We submitted a fake paper to WORLDCOMP 2011 and again (the same paper with a modified title) to WORLDCOMP 2012. This paper had numerous fundamental mistakes. Sample statements from that paper include: (1). Binary logic is fuzzy logic and vice versa (2). Pascal developed fuzzy logic (3). Object oriented languages do not exhibit any polymorphism or inheritance (4). TCP and IP are synonyms and are part of OSI model (5). Distributed systems deal with only one computer (6). Laptop is an example for a super computer (7). Operating system is an example for computer hardware Also, our paper did not express any conceptual meaning. However, it was accepted both the times without any modifications (and without any reviews) and we were invited to submit the final paper and a payment of $500+ fee to present the paper. We decided to use the fee for better purposes than making Prof. Hamid Arabnia richer. After that, we received few reminders from WORLDCOMP to pay the fee but we never responded. This fake paper is different from the two fake papers already published (see https://sites.google.com/site/worlddump4 for details) in WORLDCOMP. We MUST say that you should look at the above website if you have any thoughts of participating in WORLDCOMP. DBLP and other indexing agencies have stopped indexing WORLDCOMP’s proceedings since 2011 due to its fakeness. See http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for of one of the conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of http://sites.google.com/site/dumpconf for comments from well-known researchers about WORLDCOMP. The status of your WORLDCOMP papers can be changed from scientific to other (i.e., junk or non-technical) at any time. Better not to have a paper than having it in WORLDCOMP and spoil the resume and peace of mind forever! Our study revealed that WORLDCOMP is money making business, using University of Georgia mask, for Prof. Hamid Arabnia. He is throwing out a small chunk of that money (around 20 dollars per paper published in WORLDCOMP’s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) who publicizes WORLDCOMP and also defends it at various forums, using fake/anonymous names. The puppet uses fake names and defames other conferences to divert traffic to WORLDCOMP. He also makes anonymous phone calls and threatens the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). That is, the puppet does all his best to get a maximum number of papers published at WORLDCOMP to get more money into his (and Prof. Hamid Arabnia’s) pockets. Prof. Hamid Arabnia makes a lot of tricks. For example, he appeared in a newspaper to fool the public, claiming him a victim of cyber-attack (see Item 8 in Section 5 of above website). Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has refused to provide the venue for WORLDCOMP’13 because of the fears of their image being tarnished due to WORLDCOMP’s fraudulent activities. That is why WORLDCOMP’13 is taking place at a different resort. WORLDCOMP will not be held after 2013. The draft paper submission deadline is over but still there are no committee members, no reviewers, and there is no conference Chairman. The only contact details available on WORLDCOMP’s website is just an email address! We ask Prof. Hamid Arabnia to publish all reviews for all the papers (after blocking identifiable details) since 2000 conference. Reveal the names and affiliations of all the reviewers (for each year) and how many papers each reviewer had reviewed on average. We also ask him to look at the Open Challenge (Section 6) at https://sites.google.com/site/moneycomp1 and respond if he has any professional values. Sorry for posting to multiple lists. Spreading the word is the only way to stop this bogus conference. Please forward this message to other mailing lists and people. We are shocked with Prof. Hamid Arabnia and his puppet’s activities at http://worldcomp-fake-bogus.blogspot.com Search Google using the keyword worldcomp fake for additional links.

1 0

Re: [Cython] -O3 causes MinGW to segfault python
by David Hirschfeld April 29, 2013

April 29, 2013

Forwarded because attachments were too large. The source files and generated .c file can now be viewed at: https://gist.github.com/dhirschfeld/5480711 Is this a bug or is the recommendation to not build with -O3? > > Build log, segfault and version info below. > Input files and generated .c file attached. > > C:\temp> python setup.py build_ext --inplace --force > running build_ext > cythoning test_o3.pyx to test_o3.c > building 'test_o3' extension > C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall > -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC > -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o > test_o3.c: In function '__Pyx_RaiseArgtupleInvalid': > test_o3.c:1002:18: warning: unknown conversion type character 'z' in > format [-Wformat] > test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *', > but argument 5 has type 'Py_ssize_t' [-Wformat] > test_o3.c:1002:18: warning: unknown conversion type character 'z' in > format [-Wformat] > test_o3.c:1002:18: warning: too many arguments for format > [-Wformat-extra-args] > writing build\temp.win32-2.7\Release\test_o3.def > C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s > build\temp.win32-2.7\Release\test_o3.o > build\temp.win32-2.7\Release\test_o3.def > -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 > -lmsvcr90 -o > C:\temp\test_o3.pyd > > C:\temp> python > Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] > on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import Cython; Cython.__version__ > '0.19' > >>> import faulthandler; faulthandler.enable() > >>> from test_o3 import f > >>> f(1., 2., 3., 4.) > Fatal Python error: Segmentation fault > > Current thread 0x00001c38: > File "<stdin>", line 1 in <module> > > C:\temp> gcc --version > gcc.exe (tdm-1) 4.7.1 > Copyright (C) 2012 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > Thanks, > Dave > >

3 2

A little bugtracker cleanup
by Nikita Nemkin April 28, 2013

April 28, 2013

Hi, While browsing Cython's bugtracker I've found a few issues that should be closed as fixed: http://trac.cython.org/cython_trac/ticket/42 http://trac.cython.org/cython_trac/ticket/94 http://trac.cython.org/cython_trac/ticket/113 http://trac.cython.org/cython_trac/ticket/246 http://trac.cython.org/cython_trac/ticket/358 Best regards, Nikita Nemkin

2 3

Fwd: Re: Version of Cython packages in Ubuntu PPA
by Stefan Behnel April 26, 2013

April 26, 2013

[forwarding] -------- Original-Message -------- Subject: Re: Version of Cython packages in Ubuntu PPA Date: Fri, 26 Apr 2013 07:24:48 +0000 From: Thomas Pietrowski <thopiekar...(a)gmail.com> Hey Stefan, I wished it would work. Talked about that with the maintainers of the launchpad.net site. There is a option using tags as version number, but as we are importing code from github this feature does not work :-( So as the version changes it needs to be changed in the recipes: https://code.launchpad.net/~thopiekar/+recipe/cython-py2+3-daily https://code.launchpad.net/~thopiekar/+recipe/cython-py2-daily You can also use the packaging files for doing that, but I decided to overwrite it in the recipes, which might me easier and faster for you changing it online. Fell free to ask me, if any questions are still left :-) Regards, Thomas Am 26.04.2013 08:34 schrieb Stefan Behnel: > Hi, > > the packages in the Ubuntu PPA still have 0.18.0 in their name, although > they actually build the current master, which is at 0.19. > > Is there a way to adapt that automatically? > > Stefan >

1 0

Version of Cython packages in Ubuntu PPA
by Stefan Behnel April 26, 2013

April 26, 2013

Hi, the packages in the Ubuntu PPA still have 0.18.0 in their name, although they actually build the current master, which is at 0.19. Is there a way to adapt that automatically? Stefan

1 0