[pypy-dev] Calling into PyPy from multi-threaded C++

[René Dudfield]

Hi,

here is a basic function... without the whole module stuff around it.
I'd need to do the rest of the module another time.

PyObject* do_long_run (PyObject* self) {
    int i, a, b, c;
    a = 2;

    Py_BEGIN_ALLOW_THREADS;
    for(i = 0; i < 2000000000; i++) {
        b = i + a;
        c = a + i;
    }
    a = c - b;
    Py_END_ALLOW_THREADS;

    Py_RETURN_NONE;
}


// ...
    { "do_long_run", (PyCFunction) do_long_run , METH_NOARGS, "runs
some cpu intensive code whilst releasing the GIL"},
// ...



You could test it by calling that function in separate threads.  With
another thread printing something from python... like in the python
code below.



from threading import Thread
# this assumes the _long_runner is the C module with the do_long_run
function in it.
import _long_runner

class PythonPrinter(Thread):
    def run(self):
        for x in range(1000000):
            print (x)

class CLongRunner(Thread):
    def run(self):
        _long_runner.do_long_run()


threads = [PythonPrinter(), CLongRunner()]
for t in threads:
    t.start()
for t in threads:
    t.join()


That is untested, and typed without compiling/running... but it should
work, and should give you some insight on how the threading works.

Maybe you can throw the c function into an existing module for easier
testing... otherwise maybe I can write it in a few days.

cya.



On Thu, Jan 6, 2011 at 2:13 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Hi,
>
> 2011/1/6 René Dudfield <renesd at gmail.com>:
>> Hi,
>>
>> are the thread GIL releasing functions going to be supported in pypy
>> with CPyExt?
>>
>> To allow this usage:
>>
>> PyThreadState *_save;
>> _save = PyEval_SaveThread();
>> // ...Do some blocking I/O operation, or CPU intensive operation that
>> does not use the python api...
>> PyEval_RestoreThread(_save);
>>
>> Or using the macros...
>>
>> Py_BEGIN_ALLOW_THREADS;
>> // ...Do some blocking I/O operation, or CPU intensive operation that
>> does not use the python api...
>> Py_END_ALLOW_THREADS;
>>
>> Is it not possible or too hard, or is it just not implemented yet?
>
> Frankly, I even don't know.
> At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile.
>
> OTOH it seems that every call from the pypy interpreter to C releases the GIL,
> and that conversely, every call from to a Python API function will
> grab the GIL for the duration of the call (yes, even Py_INCREF)
> cpyext is maybe already thread-safe, only awfully slow...
> But this is just a guess, from looking at the source code.
>
> I'd be grateful if you could write a custom module and do some tests
> in this area,
> with and without the Py_BEGIN_ALLOW_THREADS calls.
>
> Cheers,
>
> --
> Amaury Forgeot d'Arc
>