Mailman 3 October 2011 - cython-devel

PEP 380 approved: syntax for generator delegation
by Stefan Behnel March 6, 2012

March 6, 2012

Hi, PEP 380 ("yield from") recently went through the approval gate. http://www.python.org/dev/peps/pep-0380/ The implementation in CPython isn't rounded up yet and lots of tests are missing from the test suite. But it's agreed that it'll make it into 3.3, and we should try to support it as well. It's a really nice feature. Stefan

2 3

relative_path_in_code_position_comments
by Brett Calcott Nov. 15, 2011

Nov. 15, 2011

Hi all, The path in cython's error output is insensitive to the cwd where cython in invoked. It seems to be based on this variable in Cython/Compiler/Main.py: relative_path_in_code_position_comments = True, If I understand it right, the path output is relative to module it is in, instead of relative to the cwd. This plays havoc with my editor (Vim), as it attempts to try and load the file that caused the error based on parsing the output. I've just changed this variable to False in my install, and it works fine. Is there a reason for this? Am I missing some easy way around this? Thanks, Brett

3 2

Cython 0.16
by Robert Bradshaw Nov. 4, 2011

Nov. 4, 2011

With Mark's fused types and memory views going in, I think it's about time for a new release. Thoughts? Anyone want to volunteer to take up the process? - Robert

8 32

Acquisition counted cdef classes
by mark florisson Oct. 26, 2011

Oct. 26, 2011

Hey, This is in response to http://groups.google.com/group/cython-users/browse_thread/thread/bcbc5fe0e3… and http://trac.cython.org/cython_trac/ticket/498 , and some of the previous discussion on cython.parallel. Basically I think we should have something more powerful than 'cdef borrowed CdefClass obj', something that also doesn't rely on new syntax. What if we support acquisition counting for every instance of a cdef class? In Python and Cython GIL mode you use reference counting, and in Cython nogil mode and for structs attributes, array dtypes etc you use acquisition counting. This allows you to pass around cdef objects without the GIL and use their nogil methods. If the acquisition count is greater than 1, the acquisition count owns a reference to the object. If it reaches 0 you discard your owned reference (you can simply acquire the GIL if you don't have it) and when you increment from zero you obtain it. Perhaps something like libatomic could be used to efficiently implement this. The advantages are: 1) allow users to pass around cdef typed objects in nogil mode 2) allow cdef typed objects in as struct attributes or array elements 3) make it easy to implement things like memoryviews (already done but would have been a lot easier), cython.parallel.async/future objects, cython.parallel.mutex objects and possibly other things in the future We should then allow a syntax like with mycdefobject: ... to lock the object in GIL or nogil mode (like java's 'synchronized'). For objects that already have __enter__ and __exit__ you could support something like 'with cython.synchronized(mycdefobject): ...' instead. Or perhaps you should always require cython.synchronized (or cython.parallel.synchronized). In addition to nogil methods a user may provide special cdef nogil methods, i.e. cdef int __len__(self) nogil: ... which would provide a Cython as well as a Python implementation for the function (with automatic cpdef behaviour), so you could use it in both contexts. There are two options for assignment semantics to a struct attribute or array element: - decref the old value (this implies always initializing the pointers to NULL first) - don't decref the old value (the user has to manually use 'del') I think 1) is more definitely consistent with how everything else works. All of this functionality should also get a sane C API (to be provided by cython.h). You'd get a Cy_INCREF(obj, have_gil)/Cy_DECREF() etc. Every class using this functionality is a subclass of CythonObject (that contains a PyObject + an acquisition count + a lock). Perhaps if the user is subclassing something other than object we could allow the user to specify custom __cython_(un)lock__ and __cython_acquisition_count__ methods and fields. Now, building on top of this functionality, Cython could provide built-in nogil-compatible types, like lists, dicts and maybe tuples (as a start). These will by default not lock for operations to allow e.g. one thread to iterate over the list and another thread to index it without lock contention and other general overhead. If one thread is somehow changing the size of the list, or writing to indices that another thread is reading from/writing to, the results will of course be undefined unless the user synchronizes on the object. So it would be the user's responsibility. The acquisition counting itself will always be thread-safe (i.e., it will be atomic if possible, otherwise it will lock). It's probably best to not enable this functionality by default as it would be more expensive to instantiate objects, but it could be supported through a cdef class decorator and a general directive. Of course one may still use non-cdef borrowed objects, by simply casting to a PyObject *. Thoughts? Mark

5 25

Buffer interface to boolean arrays with cast=True on Python 2.5 failing
by Wes McKinney Oct. 24, 2011

Oct. 24, 2011

I've been using ndarray[uint8_t, cast=True] bool_arr to work with dtype=bool arrays in Cython lately. When testing using Python 2.5 / NumPy 1.6.1 on Windows, I'm getting "unknown dtype code in numpy.pxd (0)". Everything works fine with Python 2.6/2.7 and NumPy 1.6.1. This is with Cython 0.15.1. Any advice or do I have to (very unhappily) work around this? thanks, Wes

2 5

Compiler crash at parsing stage
by Vitja Makarov Oct. 23, 2011

Oct. 23, 2011

Hi! This simple code crashes compiler: lambda i=1: i """ File "/home/vitja/work/cython-vitek-git/Cython/Compiler/Parsing.py", line 122, in p_test return p_lambdef(s) File "/home/vitja/work/cython-vitek-git/Cython/Compiler/Parsing.py", line 102, in p_lambdef s, terminator=':', annotated=False) File "/home/vitja/work/cython-vitek-git/Cython/Compiler/Parsing.py", line 2741, in p_varargslist annotated = annotated) File "/home/vitja/work/cython-vitek-git/Cython/Compiler/Parsing.py", line 2388, in p_c_arg_list annotated = annotated)) File "/home/vitja/work/cython-vitek-git/Cython/Compiler/Parsing.py", line 2435, in p_c_arg_decl print s.level AttributeError: 'PyrexScanner' object has no attribute 'level' """ I'm not sure what's the best way to fix this. -- vitja.

2 2

What's wrong with py3k pyregr tests?
by Vitja Makarov Oct. 23, 2011

Oct. 23, 2011

I tried to run pyregr tests on my localhost and it doesn't sigsegv. Perhaps I should try compiled version of Cython. Btw, I've implemented noargs super and now I want to see how does it affect py3k-pyregr test results. -- vitja.

2 5

ImportError: DLL load failed: The specified module could not be found.
by Adrian Martínez Vargas Oct. 23, 2011

Oct. 23, 2011

Hi cython list, I am having problems to distribute a python module for windows (written with cython). It compile ok with mingw (installed with but when I import the module in python I get this error ImportError: DLL load failed: The specified module could not be found. I guess that some where there is an error linking the DLL. Bout how to solve it to create a nice distribution (.exe) for my package? Thanks Adrian

4 5

cython.parallel tasks, single, master, critical, barriers
by mark florisson Oct. 21, 2011

Oct. 21, 2011

Hey, So far people have been enthusiastic about the cython.parallel features, I think we should introduce some new features. I propose the following, assume parallel has been imported from cython: with parallel.master(): this is executed in the master thread in a parallel (non-prange) section with parallel.single(): same as master, except any thread may do the execution An optional keyword argument 'nowait' specifies whether there will be a barrier at the end. The default is to wait. with parallel.task(): create a task to be executed by some thread in the team once a thread takes up the task it shall only be executed by that thread and no other thread (so the task will be tied to the thread) C variables will be firstprivate Python objects will be shared parallel.taskwait() # wait on any direct descendent tasks to finish with parallel.critical(): this section of code is mutually exclusive with other critical sections optional keyword argument 'name' specifies a name for the critical section, which means all sections with that name will exclude each other, but not critical sections with different names Note: all threads that encounter the section will execute it, just not at the same time with parallel.barrier(): all threads wait until everyone has reached the barrier either no one or everyone should encounter the barrier shared variables are flushed Unfortunately, gcc again manages to horribly break master and single constructs in loops (versions 4.2 throughout 4.6), so I suppose I'll first file a bug report. Other (better) compilers like Portland (and I'm sure Intel) work fine. I suppose a warning in the documentation will suffice there. If we at some point implement vector/SIMD operations we could also try out the Fortran openmp workshare construct. What do you guys think? Mark

4 36

Cython-ctypes branch
by Stefan Behnel Oct. 19, 2011

Oct. 19, 2011

Hi Romain, I know your branch isn't "ready" in the sense that it's useful for the real world, but I'd like to find a way to get it merged, and to find a time frame for that. Otherwise, it will just bit-rot, which is certainly not what anyone wants. How would you judge your availability for this in the near future? I hope you're interested. :) For those who didn't follow the project, the branch lives here: https://github.com/hardshooter/CythonCTypesBackend The first thing that (IMHO, let's see if the others agree) needs to happen is that you should try to rebase it on the latest master branch. There were changes in the meantime that will not make this go clean. For example, the pipeline code was factored out of Main.py into a separate module Pipeline.py, so you will have to migrate your pipeline changes manually. That shouldn't be too hard, though, and it's the only major conflict that I currently anticipate. There's a test runner change in the master branch that will allow you to select the tested backends with a positive list, i.e. as in runtests.py --backends=c,cpp You'd want to add the ctypes backend here. The "--no-cpp" etc. set of switches become very unwieldy as new backends are added. You will also notice that Cython gained a couple of new features and syntax since you started, specifically fused types, an extended array syntax for memoryviews and parallel OpenMP loops. I'm not sure how (or even if) they will translate to the Python backend. I think all of them will need a dedicated implementation in some way, which is very unfortunate. But I don't think that has to bother us for the moment. I recreated the Jenkins build and test jobs for your branch: https://sage.math.washington.edu:8091/hudson/view/dev-romain/ There's currently a unit test failure in the build job that keeps me from trying the subsequent test runs. It looks trivial, though, so if you could push a fix, I can make sure the build and test jobs work as expected. That will give us an idea about the current status of your code. I also noticed that the ctypes_configure script is not Py3 clean, so we can't currently test your code on that platform. 2to3 may be able to do the job, but the package needs fixing upstream. Stefan

4 5