Mailman 3 December 2011 - cython-devel

PEP 380 approved: syntax for generator delegation
by Stefan Behnel March 6, 2012

March 6, 2012

Hi, PEP 380 ("yield from") recently went through the approval gate. http://www.python.org/dev/peps/pep-0380/ The implementation in CPython isn't rounded up yet and lots of tests are missing from the test suite. But it's agreed that it'll make it into 3.3, and we should try to support it as well. It's a really nice feature. Stefan

2 3

Cannot assign type 'set<size_t> &' to 'set<size_t>'
by Dimitri Tcaciuc Jan. 16, 2012

Jan. 16, 2012

Hello everyone, Here's a small test case I'm trying to compile. I'm trying to pass a STL set reference to a method in a template class. x.pyx: from libcpp.set cimport set as cpp_set cdef extern from "x.hh": cdef cppclass Foo [T]: Foo() void set_x(cpp_set[size_t] & x) cpdef func(): cdef Foo[int] foo cdef cpp_set[size_t] x cdef cpp_set[size_t] & xref = x foo.set_x(xref) x.hh: #include <set> template <typename T> struct Foo { void set_x(const std::set<size_t> & x) { /* do nothing */ } }; To compile, bash $ cython --cplus x.pyx Which results in foo.set_x(xref) ^ ------------------------------------------------------------ x.pyx:15:18: Cannot assign type 'set<size_t> &' to 'set<size_t>' However, if I remove the template parameter from Foo, everything works. y.pyx: from libcpp.set cimport set as cpp_set cdef extern from "y.hh": cdef cppclass Foo: Foo() void set_x(cpp_set[size_t] & x) cpdef func(): cdef Foo foo cdef cpp_set[size_t] x cdef cpp_set[size_t] & xref = x foo.set_x(xref) y.hh: #include <set> struct Foo { void set_x(const std::set<size_t> & x) { /* do nothing */ } }; >From what I can tell, the CppClassType instance the CReferenceType is pointing to has the correct name "set<size_t>", however it's a different class instance. The particular failing expression is in `ExprNode.coerce_to` if not (str(src.type) == str(dst_type) or dst_type.assignable_from(src_type)) I wish I could suggest a patch, but unfortunately I'm a complete newbie to Cython internals. Perhaps someone could give a few pointers as to what should be done to fix this? Thanks, Dimitri

2 2

Generators & closure optimization
by Vitja Makarov Dec. 27, 2011

Dec. 27, 2011

Some time ago we were talking about generators optimization by copying local variables from closure into local scope. Now I think that should be a good idea to implement this for both generators and regular closure functions. So local var will be used for reference and assignment should be made to both copies. Of course there are some variables that shouldn't be copied: non-local vars, arrays, C++ classes and structures. Also it may be a good idea to move outer scope pointer into local variable. So I'm wondering what is a good test to measure actual speedup? -- vitja.

2 5

Re: [Cython] [cython-users] memoryviews & parameter passing
by mark florisson Dec. 20, 2011

Dec. 20, 2011

On 20 December 2011 18:57, Dirk Rothe <thecere(a)gmail.com> wrote: > Hello Cython-Devs, > > I'v thought I check out the memoryview syntax from cython-trunk to > refactor some tight loops on numpy arrays into smaller functions. But > either I'm doing something wrong or the call-overhead (of dostuff() ) > is still very large. Am I missing something? > > @cython.boundscheck(False) > cdef inline int dostuff(np.int_t[:] data, int i, int j) nogil: > return data[j] + i + j > > @cython.boundscheck(False) > def test(): > cdef np.int_t[:, :] data = np.zeros((3000, 20000), dtype=np.int) > cdef int i, j > with nogil: > for i in range(3000): > for j in range(20000): > # try to be as fast > data[i, j] = dostuff(data[i], i, j) > # as direct array access > #~ data[i, j] = data[i, j] + i + j > > thnx, dirk The performance difference is indeed quite large. There are several problems with the implementation of slices: 1) the overhead of PyThread_acquire_lock() is quite large, we should resort to an atomic approach 2) the slices support up to 32 dimensions by default (configurable as compiler option). This is a lot of memory to copy around all the time. I think a default of 8 would be more sensible and the compiler option should be documented well (who uses 32 dimensions anyway?) 3) the slice function has a generic approach and could be somewhat faster if the slice is direct and strided Addressing these problems by tweaking the generated code brings it down from ~16 seconds to ~2.4 seconds. The direct indexing approach without function call takes ~0.35 seconds. Slicing will never be as fast, so if you'd really want to write that code you'd move the data[i] call to the outer loop, as in: for i in range(3000): dataslice = data[i] for j in range(...): ... Now Cython could do that optimization itself as the 'data' slice does not change in the inner loop, but it doesn't. But at least it should not be more than 10 times slower (so this will be worked on). @cython-dev How should atomic operations be supported? Should this use something like http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html , or something like libatomic? Or should we "just" implement a garbage collector for pure-Cython level stuff (like memoryview slices), thereby avoiding the need to acquisition count?

1 0

Bug report: building an extension with --pyrex-gdb fails
by Lars Buitinck Dec. 20, 2011

Dec. 20, 2011

Hello all, I was trying to build a C++ extension with debugging support as described in http://docs.cython.org/src/userguide/debugging.html, but I got an error from the Cython compiler: lars@schoothond:~/src/sortedcollections$ python setup.py build_ext --inplace --pyrex-gdb running build_ext cythoning sortedcollections/set.pyx to sortedcollections/set.cpp Traceback (most recent call last): File "setup.py", line 9, in <module> language="c++")] File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 125, in run _build_ext.build_ext.run(self) File "/usr/lib/python2.7/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 132, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/usr/local/lib/python2.7/dist-packages/Cython/Distutils/build_ext.py", line 275, in cython_sources full_module_name=module_name) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 625, in compile return compile_single(source, options, full_module_name) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 570, in compile_single return run_pipeline(source, options, full_module_name) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Main.py", line 462, in run_pipeline err, enddata = Pipeline.run_pipeline(pipeline, source) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/Pipeline.py", line 313, in run_pipeline data = phase(data) File "Visitor.py", line 276, in Cython.Compiler.Visitor.CythonTransform.__call__ (Cython/Compiler/Visitor.c:5173) File "Visitor.py", line 259, in Cython.Compiler.Visitor.VisitorTransform.__call__ (Cython/Compiler/Visitor.c:4917) File "Visitor.py", line 165, in Cython.Compiler.Visitor.TreeVisitor._visit (Cython/Compiler/Visitor.c:3401) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/ParseTreeTransforms.py", line 2613, in visit_ModuleNode self.visit_FuncDefNode(nested_funcdef) File "/usr/local/lib/python2.7/dist-packages/Cython/Compiler/ParseTreeTransforms.py", line 2660, in visit_FuncDefNode self.tb.start('Function', attrs=attrs) File "/usr/local/lib/python2.7/dist-packages/Cython/Debugger/DebugWriter.py", line 52, in start self.tb.start(name, attrs or {}) File "saxparser.pxi", line 398, in lxml.etree.TreeBuilder.start (src/lxml/lxml.etree.c:83668) File "saxparser.pxi", line 430, in lxml.etree.TreeBuilder._handleSaxStart (src/lxml/lxml.etree.c:84030) File "apihelpers.pxi", line 224, in lxml.etree._makeSubElement (src/lxml/lxml.etree.c:12831) File "apihelpers.pxi", line 219, in lxml.etree._makeSubElement (src/lxml/lxml.etree.c:12756) File "apihelpers.pxi", line 299, in lxml.etree._initNodeAttributes (src/lxml/lxml.etree.c:13615) File "apihelpers.pxi", line 1364, in lxml.etree._utf8 (src/lxml/lxml.etree.c:22190) TypeError: Argument must be bytes or unicode, got 'NoneType' My setup.py is : from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {'build_ext': build_ext}, ext_modules = [Extension("sortedcollections.set", ["sortedcollections/set.pyx"], language="c++")] ) Importing Extension from Cython.Distutils.extension and adding pyrex_gdb=True to the Extension gives the same error. Regards, -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam

2 5

Merging of the ctypes backend branch
by Romain Guillebert Dec. 18, 2011

Dec. 18, 2011

Hi everyone I rebased the ctypes backend branch to the last cython commit, and I wondered how the branch should be merged with the main cython repository. I see 3 options : a) upload the branch without merging it b) merge the branch but not run the test suite on ctypes by default or c) merge the branch and run the test suite. I think it's better to do c) but the test results shouldn't mix with the results of the other backends. What do you think ? Romain

2 1

Distributing Windows binary using OpenMP / cython.parallel
by Wes McKinney Dec. 13, 2011

Dec. 13, 2011

I'm interested in using the Cython OpenMP extensions in pandas for various calculations, but I'm concerned about cross-platform issues, especially distributing built binaries of the extensions to Windows users. Is there a clean way to bundle the relevant OpenMP DLLs in distutils? thanks, Wes

3 3

Re: [Cython] [cython-users] malloc vs PyMem_malloc
by mark florisson Dec. 11, 2011

Dec. 11, 2011

On 10 December 2011 22:00, Stefan Behnel <stefan_ml(a)behnel.de> wrote: > mark florisson, 10.12.2011 21:44: >> >> On 10 December 2011 20:39, mark florisson wrote: >> >>> On 10 December 2011 19:16, Robert Bradshaw wrote: >>>> >>>> On this note, a useful pattern is >>>> >>>> try: >>>> x = malloc(...) >>>> finally: >>>> free(x) >>>> >>>> It could be nice to encapsulate this in a context manager. > > > Absolutely. > > > >>> I think I'd prefer variable-sized arrays that would always get >>> deallocated on exit of the function > > > Why? A context manager is much clearer That is highly subjective, I think it would be harder to read and introduce more code blocks and nesting. > and gives users total control over > the lifetime of the memory. > Yes, but very often you don't need it. And if Cython would support declarations in blocks you'd get it for free. Supporting that (disregarding the difficulties of that) would also be helpful in identifying the scope and privatization rules in parallel blocks. The thing is that a context manager would be very Cython-specific, whereas most people are already familiar with arrays of variable size from C or Java. Lets compare the following statements and decide which is more aesthetically pleasing: cdef int array1[m] cdef double array2[n] vs cdef int *array1 cdef double *arrays2 with cython.malloc(sizeof(int) * m), cython.malloc(sizeof(double) * n) as array1, array2: ... > >>> (which could be implemented as C99 >>> variable sized arrays, with alloca or with malloc, depending on the >>> size of the array and the availability of the respective >>> functionalities). > > > That could still be done for a context manager, just like we do with > gil/nogil blocks today. > Sure (it was more of an observation than an argument). > >> That wouldn't tackle every use case, such as for instance mallocing >> stuff in a parallel section (until we get declarations in blocks!), >> but special cases can still just malloc and use try blocks, as >> demonstrated. > > > I would consider the usage of memory over the whole lifetime of a function > the special case, not the other way round. Yes, but the point is not where to deallocate the memory, the point is that you very often don't care. You need it somewhere in the function, and deallocation on return is fine (or, "at the end of the block"). Analogously, you don't 'del' your variables once you have stopped using them. I also gave this functionality some thought for memoryviews, e.g. cdef int[:m, :n] myslice # this gets you a view on a cython.array of size m * n > Stefan

2 1