Mailman 3 March 2011 - cython-devel

Re: [Cython] adding support for __dict__ in extension types
by Lisandro Dalcin Sept. 26, 2011

Sept. 26, 2011

Bringing up this old post... On 21 June 2010 15:41, Robert Bradshaw <robertwb(a)math.washington.edu> wrote: > On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: > >> If we special case a __dict__ attribute in extension types, i.e: >> >> cdef class Foo: >> cdef dict __dict__ >> >> and fill type->tp_dictoffset, then we can support __dict__ in >> extension types. >> >> What do you think? > > Sounds like a good idea to me. Note that we check tp_dictoffset for > fast dispatching for cpdef methods (which would be correct as a dict > lookup *would* be needed if __dict__ is available). > I still have this patch lying around in my disk. I remember Stefan had some objections. For example, when the user ask for __dict__, a new dict is unconditionally created (in CPython, type dict are allocated on-demand). I propose to get this patch pushed now, and optimize later (however, I really don't know how to safely implement this optimization). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 4

Re: [Cython] Control flow graph
by Vitja Makarov April 14, 2011

April 14, 2011

2011/2/23 Vitja Makarov <vitja.makarov(a)gmail.com>: > 2011/2/23 Vitja Makarov <vitja.makarov(a)gmail.com>: >> 2011/2/22 Stefan Behnel <stefan_ml(a)behnel.de>: >>> Vitja Makarov, 20.02.2011 18:23: >>>> >>>> 2011/2/16 Vitja Makarov: >>>>> >>>>> Hmm... both python and codespeaks in the thread >>> >>> Yes, we should keep it to cython-devel only. Sorry for mixing it up. >>> >>> >>>>> Here is my commit it's mostly broken now but anyway >>>>> >>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb1998… >>> >>> Flow control support is large enough to merit its own module. Not sure how >>> 'smart' git is here, but you can always keep the history by explicitly >>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated >>> sections from both files. >>> >> >> Ok. >> > > Oops there is no copy command in git. > > >>> You are duplicating some code from the type inferencer. We might want to >>> clean that up at some point. However, given that flow control analysis will >>> allow us to improve the type inferencer, I think it's best to keep this code >>> in the FCA part. >>> >> >> Yes, I think it could replace MarkAssignments transform later. >> Unreachable code could be delete there too. >> >>> >>>> I've update stuff: >>>> - algo for finding definitions >>>> - warnings for uninitialized and may be uninitialised use >>>> - few test cases >>> >>> That looks very nice so far. Any idea how well it scales? >>> >> >> "Usually iterative algorithm takes no more then 5 iterations" >> >> For ExprNodes.py max number is 15 while avg is about 3 >> >> About execution time: >> >> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325, >> ~10% slow down. >> -O flag could be introduced but I don't think that's a good idea. >> >> Should later try to execute cython compiled code. >> >>> >>>> Trying to compile ParseTreeTransforms.py I've found this for example: >>>> >>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable >>>> 'template' may be used uninitialized >>>> >>>> def create_Property(self, entry): >>>> if entry.visibility == 'public': >>>> if entry.type.is_pyobject: >>>> template = self.basic_pyobject_property >>>> else: >>>> template = self.basic_property >>>> elif entry.visibility == 'readonly': >>>> template = self.basic_property_ro >>>> property = template.substitute({ >>>> u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, >>>> >>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"), >>>> attribute=entry.name), >>>> }, pos=entry.pos).stats[0] >>> >>> Ok, I guess that code generally works, but it's better to get rid of the >>> code smell. >>> >> >> Might be used warning should be disabled by default, because algorithm >> isn't smart enough: >> >> a = 1 >> if (a): b = 1 >> if (a): print b >> >> See also: >> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized… >> I've updated things: - Now it passes all the tests (some tests are run with remove_unreachable=False) - Control flow graph support all the control statements (hope so) - Analyse PyClassDefNode (now only inner, need some more work) - Now we have the following optional warnings: 1. uninitialized variable use [default on] 2. possibly uninitialized use [-Wextra] 3. unreachable code [default on] (this needs rework, unreachable code detection could be better handled inside CreateControlFlowGraph) 4. unused variable [-Wextra] 5. unused argument [-X warn.unused_arg=True] 6. unused result [-X warn_unused_result=True] - Optional dot graph output: $ cython foo.pyx -X control_flow.dot_output=foo.dot $ dot -Tpng -o foo.png foo.dot -- vitja.

2 3

Interest in contributing to the project
by Arthur de Souza Ribeiro April 5, 2011

April 5, 2011

Hello everybody, My name is Arthur de Souza Ribeiro and I'm a fourth-year student of Computer Science in Federal University of Campina Grande, Brazil. I'm a python programmer and have knowledge of other languages too, like Java, C, C++, Qt, Grails and ActionScript (used in Flex framework of Adobe). I saw Cython project and got really interested in contributing to it. By the way, I saw that the project is trying to participate of GSoC under Python Software Foundation umbrella. I know the student application period have already started, but, I'd really enjoy to participate of GSoC 2011 as a Cython's student. Until day 8 I could work really hard to show you that I can be selected as a GSoC student for Cython. I looked for an Ideas Page of the project but didn't find it, Is there any idea that you have to submit a project in GSoC? If possible, please tell me things that I can start doing to help the project. Best Regards Arthur

3 7

Cannot profile nogil function. Error or Warn?
by Lisandro Dalcin April 1, 2011

April 1, 2011

Error compiling Cython file: ------------------------------------------------------------ ... cdef int PyMPE_Raise(int ierr) except -1 with gil: __Pyx_Raise(RuntimeError, "MPE logging error [code: %d]" % ierr, NULL) return 0 cdef inline int CHKERR(int ierr) nogil except -1: ^ ------------------------------------------------------------ /home/dalcinl/Devel/mpi4py-dev/src/MPE/helpers.pxi:22:5: Cannot profile nogil function. Do we REALLY want this to be an error? Why not just a warning? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 1

unspecified return type for extern C++ functions
by Lisandro Dalcin March 30, 2011

March 30, 2011

In a previous thread, Robert asked about issuing a warning in the case of unspecified return type for extern C++ functions. I'm definitely +1 for it... Moreover, I would extend it for ANY extern C function. IMHO, this is a case for "explicit is better than implicit". -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

How to improve the performance when doing string/unicode replace and search?
by Yunfan Jiang March 30, 2011

March 30, 2011

hi, i used to ask some string process question here, and found a bug, it seems you guys fix the bug but not use it and this time , my problem is about the performance, i need to wrote a filter which search sorts of keywords in the target string , and stop if matched, this act require unicode input/output , so i wrote a trie like module to done it, it works ,but i found its too slower than using regex module so could you guys give some tips on string process performance? -- ME = { "name": [ "jyf", "yunfan", "wuxian" ], "im": { "gtalk": "jyf1987(a)gmail.com", "msn": "geek42(a)live.cn" }, "job": "python engineer", "site": "http://hi.baidu.com/jyf1987", "interested": { "tech": [ "linux", "python", "lua", "php", "html5", "c", "nosql"], "history": ["chinese history", "global history"], "SF": [ "hard SF", "Thought experiment" ], "music": [ "New Age", "Chinese old theme", "Electronic music", "Strange Music :}"] } }

2 2

cygdb gdb script problem
by Vitja Makarov March 29, 2011

March 29, 2011

Running cygdb with Cython installed in the system leads to the following problem: vitja@vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named Debugger /tmp/tmp1ZvOf9:3: Error in sourced command file: Error while executing Python code. (gdb) So may be it's better to explicitly specify correct path to Cython package in gdb script? Small example patch is attached. -- vitja.

2 3

Rewriting/compiling parts of CPython's stdlib in Cython
by Stefan Behnel March 29, 2011

March 29, 2011

Hi, there seems to be quite some interest in a project to get parts of CPython and specifically its stdlib rewritten in Cython. I've copied the latest python-dev mail below. The relevant part of the thread is here: http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 In short, we have strong supporters, but Guido has understandable doubts against a new (and quite large) dependency and potential semantic deviations. But there seem to be cases where slight changes would be acceptable that Cython compiled modules might introduce, such as emitting different exception messages, changing Python classes into extension classes, or even preventing monkey patching in modules that are backed by C modules anyway. It would be helpful to get support from the side of external distributors that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If they agreed to test the Cython generated stdlib modules in their distributions, we could get user feedback that would allow python-dev to take a well founded decision. Do we have any volunteers for trying this out? Both on the side of distributors and implementors? At the current state of affairs, the implementation could still be financed by a Python backed GSoC project, although it would be cool if more users could just step up and simply try to compile and optimise stdlib modules (preferably without major changes to the code). It's certainly a great way to show off your Cython skills :). I gave it a try with difflib and it turned out to be quite easy. http://blog.behnel.de/index.php?p=155 Reimplementing existing C modules in Cython might, however, be more interesting for python-dev, but also be a larger undertaking. So a GSoC might be worth running on that. Note that the latest Cython release does not have generator support yet, and Vitja's branch on github is not very stable. We will try to get it up to speed and merged during the workshop next week, at which point it will make more sense to get this project started than right now. Stefan Guido van Rossum, 22.03.2011 00:04: > On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: >> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>> Guido van Rossum, 21.03.2011 03:46: >>>> Thanks for the clarifications. I now have a much better understanding >>>> of what Cython is. But I'm not sold. For one, your attitude about >>>> strict language compatibility worries me when it comes to the stdlib. >>> >>> Not sure what you mean exactly. Given our large user base, we do worry a >>> lot about things like backwards compatibility, for example. >>> >>> If you are referring to compatibility with Python, I don't think anyone >>> in the project really targets Cython as a a drop-in replacement for a >>> Python runtime. We aim to compile Python code, yes, and there's a >>> hand-wavy idea in the back of our head that we may want a plain Python >>> compatibility mode at some point that will disable several important >>> optimisations. >> >> I think that's the attitude Guido worries about: if you don't have the >> desire to provide 100% Python compatibility under all circumstances >> (i.e. including if someone passes parameters of "incorrect" types), >> then there is very little chance that we would replace a Python module >> with a Cython-compiled one. >> >> The only exception would be cases where the Python semantics is murky >> (e.g. where Jython or so actually behaves differently for the same >> Python code, and still claims language conformance). E.g. the exact >> message on a TypeError might change when compiling with Cython, >> but the cases in which you get a TypeError must not change. > > One other significant use case is the situation where we have an > optional replacement module written in C (e.g. heapqmodule.c vs. > heapq.py). There are usually many semantic differences between the C > and pure-python module that we don't care about (e.g. monkeypatching > won't work). > > The size of Cython as a dependency and its development speed are still > problems though. In general for the core I don't think we want the > repo to contain generated code that can only be regenerated using a > 3rd party dependency. (True, we have a few generated files, e.g. > configure; but in that case the generator -- autoconf -- is a > standard installed tool on Linux and is used by most open source > projects.) > > Still, I think it would be great if someone tried something like this > for a specific stdlib module and came back with a story about the > experience, rather than having a theoretical discussion about possible > pros and cons.

8 25

Re: [Cython] cython crash: default parameters in methods of template cppclass
by Robert Bradshaw March 29, 2011

March 29, 2011

On Tue, Mar 22, 2011 at 7:21 AM, Simon Anders <anders(a)embl.de> wrote: > Hi, > > I found a little bug in Cython 0.14.1. > > The following causes the Cython compiler to throw an exception: > > ---8<--- > cdef extern from "foo.h": > cdef cppclass foo[ T ]: > bar( int b = 0 ) > > cdef foo[ int ] a > a.bar( 1 ) > ---8<--- > > The exception is "AttributeError: 'CFuncType' object has no attribute > 'op_arg_struct'", thrown in line 3044, in generate_result_code. Full stack > trace attached. > > The file compiles without crash if one either > - removes the last line, i.e. the call to the class's method, or > - removes the template argument, i.e., the "[ T ]" in line 2 and > the "[ int ]" in line 5. > > Hence, the issue only appears when calling methods of _templated_ classes, > which have a _default_ value for their argument. Default arguments in C++ are completely different from default arguments in Cython, this is our bug for not catching that. You have to declare this as an overloaded method cdef extern from "foo.h": cdef cppclass foo[ T ]: bar() bar(int) Btw, unless it returns an object, you probably want to declare the return type of your methods. Perhaps we should make this a warning for extern declarations? - Robert

1 0

Bug in cython-mode.el under GNU Emacs 23.2.1
by Rafe Kettler March 25, 2011

March 25, 2011

I've just tried out cython-mode.el and I was a bit disappointed when it didn't work right off the bat. I added the following to my .emacs: (setq load-path (cons "~/lib/emacs-plugins" load-path)) (require 'cython-mode) this initially gave me the following error when I restarted emacs: Warning (initialization): An error occurred while loading `/home/rafe/.emacs': File error: Cannot open load file, python-mode To ensure normal operation, you should investigate and remove the cause of the error in your initialization file. Start Emacs with the `--debug-init' option to view a complete error backtrace. So, seeing that python-mode wasn't around, I changed line 3 of cython-mode.el from (require 'python-mode) to (require 'python) and everything works now. I got this mode from Github this morning and it is the latest version (you can see it at https://github.com/cython/cython/blob/master/Tools/cython-mode.el). I'm not an emacs lisp expert, so I'm not sure if there's something I'm missing here or if I somehow loaded the package correctly. Anyway, I thought you guys should know that I had a problem with the file but managed to get a fix. If my fix needs to be applied, I'll patch it on Github. This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. Rafe

4 3