[Cython] OpenMP thread private variable not recognized (bug report + discussion)

[Leon Bottou]

The attached cython program uses an extension class to represent a unit of 
work. The with parallel block temporarily gets the gil to allocate the object, 
then release the gil and performs the task with a for i in prange(...) 
statement. My expectation was to have w recognized as a thread private 
variable.

    with nogil, parallel():
        with gil: 
            w = Worker(n)                 # should be thread private
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()          # should be reduction
            w = None                      # is this needed?


Cythonize (0.20.2) works without error but produces an incorrect C file.

hello.c: In function ‘__pyx_pf_5hello_run’:
hello.c:2193:42: error: expected identifier before ‘)’ token
hello.c: At top level:

The erroneous line is:

#pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5, 
__pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2) 
private(__pyx_filename, __pyx_lineno, __pyx_clineno) 
shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value, 
__pyx_parallel_exc_tb)

where you can see that the first private() clause has no argument. The 
variable __pyx_v_w is not declared as private either as I would expect.

I believe that the problem comes from line 7720 in Cython/Compiler/Node.py

        if self.privates:
            privates = [e.cname for e in self.privates
                                    if not e.type.is_pyobject]
            code.put('private(%s)' % ', '.join(privates))

And I further believe that the clause "if not e.type.is_pyobject" has been 
added because nothing would decrements the reference count of the thread 
private worker object when leaving the parallel block. 

My quick fix would be to remove this clause and make sure that my program 
contains the line "w = None" before leaving the thread. But I realize that 
this is not sufficient for you.

Note that the temporary python objects generated by the call to the Worker 
construction are correctly recognized as thread private and their reference 
count is correctly decremented when they are no longer needed.  The problem 
here is the clash between the python scoping rules and the semantics of thread 
private variables. This is one of these cases where I would have liked to be 
able to write

    with nogil, parallel():
        with gil: 
            cdef w = Worker(n)            # block-scoped cdef
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()

with an understanding that the scope of the cdef variable is limited to the 
block where the cdef appears. But when you try this, cython tells you that 
cdefs are not legal there. 


-------------- next part --------------
# -*- Python -*-

import numpy as np
cimport numpy as np
from cython.parallel import parallel, prange


cdef class Worker:
    cdef double[::1] v
    def __init__(self, int n):
        self.v = np.random.randn(n)
    cdef double run(self) nogil:
        cdef int i
        cdef int n = self.v.shape[0]
        cdef double s = 0
        for i in range(0,n): 
            s += self.v[i]
        return s / n

def run(int n, int m):
    cdef Worker w
    cdef double r
    cdef int i
    with nogil, parallel():
        with gil: 
            w = Worker(n)
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()
            w = None
    return r / m

-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 349 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140810/932f28a5/attachment.py>