Mailman 3 June 2011 - cython-devel

PEP 380 approved: syntax for generator delegation
by Stefan Behnel March 6, 2012

March 6, 2012

Hi, PEP 380 ("yield from") recently went through the approval gate. http://www.python.org/dev/peps/pep-0380/ The implementation in CPython isn't rounded up yet and lots of tests are missing from the test suite. But it's agreed that it'll make it into 3.3, and we should try to support it as well. It's a really nice feature. Stefan

2 3

Re: [Cython] adding support for __dict__ in extension types
by Lisandro Dalcin Sept. 26, 2011

Sept. 26, 2011

Bringing up this old post... On 21 June 2010 15:41, Robert Bradshaw <robertwb(a)math.washington.edu> wrote: > On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: > >> If we special case a __dict__ attribute in extension types, i.e: >> >> cdef class Foo: >> cdef dict __dict__ >> >> and fill type->tp_dictoffset, then we can support __dict__ in >> extension types. >> >> What do you think? > > Sounds like a good idea to me. Note that we check tp_dictoffset for > fast dispatching for cpdef methods (which would be correct as a dict > lookup *would* be needed if __dict__ is available). > I still have this patch lying around in my disk. I remember Stefan had some objections. For example, when the user ask for __dict__, a new dict is unconditionally created (in CPython, type dict are allocated on-demand). I propose to get this patch pushed now, and optimize later (however, I really don't know how to safely implement this optimization). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 4

strange bug in parallel assignment
by Vitja Makarov June 22, 2011

June 22, 2011

Hi! Looking at hudson pyregr output I've found this issue: (a, b), (c,) = (1, 2), {1: 2} Traceback (most recent call last): File "cython.py", line 17, in <module> main(command_line = 1) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 821, in main result = compile(sources, options) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 796, in compile return compile_multiple(source, options) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 768, in compile_multiple result = run_pipeline(source, options) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 632, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 251, in run_pipeline data = phase(data) File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line 173, in generate_pyx_code module_node.process_implementation(options, result) File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py", line 73, in process_implementation self.generate_c_code(env, options, result) File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py", line 310, in generate_c_code self.generate_module_init_func(modules[:-1], env, globalstate['init_module']) File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py", line 1924, in generate_module_init_func self.body.generate_execution_code(code) File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line 347, in generate_execution_code stat.generate_execution_code(code) File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line 3811, in generate_execution_code stat.generate_assignment_code(code) File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line 3709, in generate_assignment_code self.lhs.generate_assignment_code(self.rhs, code) File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py", line 4036, in generate_assignment_code self.generate_parallel_assignment_code(rhs, code) File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py", line 4088, in generate_parallel_assignment_code self.generate_generic_parallel_unpacking_code(code, rhs) File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py", line 4137, in generate_generic_parallel_unpacking_code item.result(), File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py", line 2047, in result print self.pos, self.type, self.temp_cname AttributeError: 'PyTempNode' object has no attribute 'temp_cname' -- vitja.

1 1

Bindings performance issue
by Vitja Makarov June 21, 2011

June 21, 2011

Hi! Is bindings performance issue valuable? $ cat bindbench.pyx def wo_bindings(): pass def outer(): def inner(): pass return inner with_bindings = outer() $ python >>> import timeit >>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000) [6.169871807098389] >>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000) [4.609416961669922] PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times slower for CPython interpreter execution. As CPython has some optimizations for CFunctions and PyCFunctions. Does it make sense for us? Or we can easily switch to bindings? -- vitja.

4 26

Some bugs found while testing cython on django
by Vitja Makarov June 21, 2011

June 21, 2011

Hi! Trying to compile django I've found some problems: 1. It seems that cython currently doesn't support tuples inside args definition: def foo((a, b), c): Currently this gives missing argument name and crash inside CreateControlFlow graph cython -X binding=True django/contrib/gis/forms/fields.py -o django/contrib/gis/forms/fields.c 2. Concatenating unicode and str isn't supported: I guess that this is mostly django problem Error compiling Cython file: ------------------------------------------------------------ ... default_error_messages = { 'no_geom' : _(u'No geometry value provided.'), 'invalid_geom' : _(u'Invalid geometry value.'), 'invalid_geom_type' : _(u'Invalid geometry type.'), 'transform_error' : _(u'An error occurred when transforming the geometry ' 'to the SRID of the geometry form field.'), ^ ------------------------------------------------------------ django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals of different types, expected u'', got '' 3.Reraise not inside except clause, here is simple example def foo(): print 'catched' raise try: raise IndexError except IndexError: foo() -- vitja.

4 6

Test runner changes for Python/PyPy backend
by Stefan Behnel June 17, 2011

June 17, 2011

Hi, Romain wants to start writing tests for his Python/ctypes backend branch, so we need to extend the test runner. We'll eventually want to merge his branch back into normal mainline, so I suggest actually just running the complete test suite in a new, separate configuration, just like we do with C and C++ now. So we'd have a switch "--no-ctypes" that disables these tests, and (I assume) a new test class (inheriting from the normal compile test case) that properly runs them in plain CPython. If the implementation becomes PyPy specific, "--no-pypy" will make more sense, but it currently appears to target ctypes in general. I'm also considering making the backend selection a positive opt-in, i.e. switch from "--no-cpp" and friends to "--backends=c,cpp,ctypes". That would have avoided the future configuration change in all existing Jenkins test jobs (which will then need to disable the new backend tests). Does that fit everyone's expectations? There's one problem I see: the ctypes backend will likely have most optimisations disabled, so many (most?) of the test tree assertions will make no sense. I think it's best to just keep that out of the backend specific pipeline completely for now. We have the C/C++ test modes for that. And, obviously, switching from duplicating to triplicating the test cases will substantially increase the time for a complete test run, although a lot of the running time is currently spent in the C compiler more than the actual test code runs. Stefan

2 1

Possible bug related to multiple assignment
by Andrew Collette June 16, 2011

June 16, 2011

Hi, I ran into some odd behavior when working on my cython-based project (h5py). The following cython code snippet is the culprit ("priv" is a function argument of type void**): cdef conv_size_t *sizes priv[0] = sizes = <conv_size_t*>malloc(sizeof(conv_size_t)) gets turned into this (with Cython 0.14.1): (__pyx_v_priv[0]) = ((__pyx_t_4h5py_5_conv_conv_size_t *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t)))); __pyx_v_sizes = ((__pyx_t_4h5py_5_conv_conv_size_t *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t)))); which leads to much head-scratching when one initializes "sizes", and then later tries to recover that value via "priv". Interestingly, Cython 0.13 correctly initializes both priv[0] and sizes to the same value: __pyx_t_5 = ((__pyx_t_4h5py_5_conv_conv_size_t *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t)))); (__pyx_v_priv[0]) = __pyx_t_5; __pyx_v_sizes = __pyx_t_5; I wasn't able to find anything on this in the list archives or issue tracker. Right now I'm working around it by splitting the statement into multiple lines: sizes = <conv_size_t*>malloc(sizeof(conv_size_t)) priv[0] = sizes Andrew

2 2

Re: [Cython] setup.py refusing to run cython
by Lisandro Dalcin June 15, 2011

June 15, 2011

On 14 June 2011 16:20, Nathaniel Smith <njs(a)pobox.com> wrote: > Hello Cython folks, > > This message (see below) is the second report I've gotten of a very > strange build problem with a cython module. I'm just using the > standard 'from Cython.Distutils import build_ext', 'cmdclass = > {"build_ext": build_ext}' incantation, but for some reason setup.py is > attempting to just run gcc directly without calling cython first to > generate the .c file. > > You can see the setup.py here: > https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikit… > > Anyone have any ideas? > Your setup.py is mixing setuptools and Cython.Distutils. Both are monkeypatching distutils, and in general that's a very bad idea. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

4 3

local variable handling in generators
by Stefan Behnel June 12, 2011

June 12, 2011

Hi, I've been looking at the nqueens benchmark for a while, and I think it's actually not that a bad benchmark for generators. http://hg.python.org/benchmarks/file/tip/performance/bm_nqueens.py A better implementation only for Py2.7/Py3 is here: https://github.com/cython/cython/blob/master/Demos/benchmarks/nqueens.py Cython currently runs the first implementation about as fast as Py3.3: https://sage.math.washington.edu:8091/hudson/job/cython-devel-pybenchmarks-… and the second one more than 3x as fast: https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-cy… However, I think there's still some space for improvements, and local variables are part of that. For generator functions that do non-trivial things between yields, I think that local variables will quickly become a bottleneck. Currently, they are always closure fields, so any access to them will use a pointer indirection to a foreign struct, originally passed in as an argument to the function. Given that generators often do Python object manipulation through C-API calls, any such call will basically require the C compiler to assume that all values in the closure may have changed, thus disabling any optimisations for them. The same applies to many other object related operations or pointer operations (even DECREF!), as the C compiler cannot know that the generator function owns the closure during its lifetime exclusively. I think it would be worth changing the current implementation to use local C variables for local Cython variables in the generator, and to copy the values back into/from the closure around yields. I'd even let local Python references start off as NULL when the generator is created, given that Vitek's branch can eliminate None initialisations now. I started looking into this a bit, but it's not a quick change. The main problem I see is the current code duplication between the generator body node and the DefNode. It would be better if both could share more code. Basically, if local variables become truly local, the only differences will be that they come from call arguments in one case and from the closure in the other, and that generators have the additional jump-to-yield entry step. Everything else could hopefully be identical. Once again, this goes hand in hand with the still pending DefNode refactoring... Stefan

4 20

Constant folding
by Vitja Makarov June 11, 2011

June 11, 2011

Hi! When CF is there it is possible to improve constant folding. Some NameNode references could be easily replaced by appropriate ConstNode. Could that help GCC to produce faster code? -- vitja.

2 1