[Cython] [cython-users] Re: Segfault in PyThread_release_lock
Dag Sverre Seljebotn
d.s.seljebotn at astro.uio.no
Mon Aug 15 11:54:10 CEST 2011
On 08/15/2011 11:42 AM, mark florisson wrote:
> On 15 August 2011 02:22, Nikolaus Rath<Nikolaus at rath.org> wrote:
>> mark florisson<markflorisson88 at gmail.com> writes:
>>> On 14 August 2011 17:50, Nikolaus Rath<Nikolaus at rath.org> wrote:
>>>> mark florisson<markflorisson88 at gmail.com> writes:
>>>>> On 12 August 2011 20:23, Nikolaus Rath<Nikolaus at rath.org> wrote:
>>>>>> The following segfault is completely incomprehensible to me:
>>>>>> (gdb) bt full
>>>>>> #0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
>>>>>> No locals.
>>>>>> #1 0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>>>>>> thelock = 0x0
>>>>>> status = 0
>>>>>> error = 0
>>>>>> #2 0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
>>>>>> No locals.
>>>>>> #3 0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>>>>>> tstate = 0x925da0
>>>>>> #4 0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>>>>>> at ../Python/pystate.c:651
>>>>>> tcur = 0x925da0
>>>>>> __PRETTY_FUNCTION__ = "PyGILState_Release"
>>>>>> Why am I getting an error so deep in the python internals?
>>>>> The error is deep because Python needs to do certain things. So what
>>>>> it's doing here is destructing your thread state, which means that was
>>>>> the last call to PyGILState_Ensure(). This means your function was
>>>>> called from another thread than a Python thread. It looks to me like
>>>>> the GIL is not initialized (__pyx_gilstate_save =
>>>>> PyGILState_UNLOCKED). Are you sure you have called
>>>> No, I have not. But isn't Cython supposed to do this automatically when
>>>> entering the function if I define the function as "with gil"?
>>> No, it acquires the GIL, but it does not attempt to initialize it.
>>> Seeing how many people have issues with it, I'm thinking that perhaps
>>> we should if the module contains with gil functions, or with gil
>>> blocks in nogil functions. If you're using cython.parallel and with
>>> gil blocks, the GIL will be initialized at module-load time. Note that
>>> initializing the GIL slows down serially executed Python code, so
>>> that's why Cython doesn't initialize it unconditionally. It will have
>>> to be called at module-load time, as it should be called in the main
>>> python thread (or from any other python thread started by the
>>> thread(ing) module).
>> Oh, wow. That's a tricky one. And it explains why I have the segfault
>> only in a reduced testcase, but not in the full blown application (the
>> application probably creates a Python thread before calling into Cython
>> code). I banged my head against the wall for quite some time already
>> because of that.
>> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
>> to figure that out myself.
>> I wouldn't mind if Cython inserted the call automatically, but a big
>> warning in
>> (under the "acquiring the GIL") might already be enough.
>> »Time flies like an arrow, fruit flies like a Banana.«
>> PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
> Definitely, I'll do that one of these days. I do think it's rather
> common to have with gil functions called as a callback from another
> (non-python) thread. I'm not so sure about with gil blocks in nogil
> @Cython-dev: Do we merely want to update the docs, or do we want to
> initialize the GIL for either case, or only for the with gil
> functions? I'm not entirely sure what the overhead is for
> single-threaded code, but I'd say we need to initialize it for both
I'm not sure if I like magic related to having used "with gil" on a
function signature -- it's a bit too magic and obscure.
I think we should have another directive for this as well as the #ifdef
macro (that is, the directive controls the default value of the C
macro), and document that a bit better.
As for the default value of that, here's two proposals:
Alt. 1) Always initialize the GIL by default.
Alt. 2) Issue a warning whenever a with-gil function is present unless
the directive is explicitly set (one way or the other). Perhaps use the
same warning if "with nogil" is used as well.
I'm in favour of 1), since it would save a lot of frustration among
users, since alt 2) is a rather minor case only costing a little bit of
performance (?), and since in many of the cases where the GIL should not
be initialized, the multithreading support is somewhat likely to turned
off at the CPython level anyway during compilation (?).
More information about the cython-devel