BEGIN_ALLOW_THREADS

Hello Is there a reason the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS don't allow an argument specifying what variable to save the state to? I needed this myself so I wrote the following: #ifdef WITH_THREAD # define MY_BEGIN_ALLOW_THREADS(st) \ { st = PyEval_SaveThread(); } # define MY_END_ALLOW_THREADS(st) \ { PyEval_RestoreThread(st); st = NULL; } #else # define MY_BEGIN_ALLOW_THREADS(st) # define MY_END_ALLOW_THREADS(st) { st = NULL; } #endif It works just fine but has one drawback: Whenever Py_BEGIN_ALLOW_THREADS changes, I have to change my macros too. Wouldn't it be reasonable to supply two sets of macros, one that allows exactly this, and one that does what Py_BEGIN_ALLOW_THREADS currently does. Martin Sjögren -- Martin Sjögren martin@strakt.com ICQ : 41245059 Phone: +46 (0)31 405242 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html

On Mon, Jul 23, 2001 at 02:21:57PM +0200, Martin Sjögren wrote:
If you're wondering why I'm replying to my own age-old-mail, it is because I noticed the following snippet in python-announce-list: """ Martin Sjögren wondered why the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros don't take an argument specifying a variable to save the thread state to. He said he needed this, but didn't explain why. """ Looking back I see that, no, I did not motivate that one bit :) Here's the scoop: One of the functions (say foo()) in my extension module calls another function (real_foo()) which might block since it does creepy things like I/O. Being a nice fella I surrounded the call with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS so that other threads could jump in and do stuff. The problem is that real_foo() might call a previously defined callback (written in C) which in turn has to call a Python callback (what good is an extension module if you have to write your callbacks in C?), but when I try to PyEval_CallObject() it - KABOOM! Segfault somewhere in the interpreter (on PyFrame_New() or something like that). After a lot of thinking (and a lot of help from python-list) I figured out that the state I push away using Py_BEGIN_ALLOW_THREADS must be retrieved again before the callback is called, or PyFrame_New() won't like me. Problem is that that state variable is declared in Py_BEGIN_ALLOW_THREADS and there's no way to reach it (and it also includes a naughty { with the corresponding } in the Py_END_ALLOW_THREADS so it won't work to use them anyway). My solution: I wrote the two macros above. This works for me, but obviously I have to update my macros when the "real" macros in the Python distribution change. Okay, that's a more thorough explanation, I hope everybody are happy :) Martin -- Martin Sjögren martin@strakt.com ICQ : 41245059 Phone: +46 (0)31 405242 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html

Instead of asking us to add more macros, I recommend that you use the lower-level C API that is used by the macros. There's a bunch of ways to save and restore the state. The macros are just shortcuts (and a handy way to make all the code disappear when threads don't exist). --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, Aug 08, 2001 at 10:40:11AM -0400, Guido van Rossum wrote: [snip]
Well, that's exactly what I'm doing, isn't it? Martin -- Martin Sjögren martin@strakt.com ICQ : 41245059 Phone: +46 (0)31 405242 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html

[Guido]
[Martin Sjögren]
Well, that's exactly what I'm doing, isn't it?
You are using macros, yes <wink>. What Guido is telling you is that the set of macros we already have is adequate for everything we do, so it would just bloat our code base with stuff we never ever test if we were add additional stuff we never use. He's also telling you that you can write your macros in such a way that you don't have to change them across Python releases, and it was *that* complaint you seemed keenest about. That one you can solve on your own.

On Mon, Jul 23, 2001 at 02:21:57PM +0200, Martin Sjögren wrote:
If you're wondering why I'm replying to my own age-old-mail, it is because I noticed the following snippet in python-announce-list: """ Martin Sjögren wondered why the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros don't take an argument specifying a variable to save the thread state to. He said he needed this, but didn't explain why. """ Looking back I see that, no, I did not motivate that one bit :) Here's the scoop: One of the functions (say foo()) in my extension module calls another function (real_foo()) which might block since it does creepy things like I/O. Being a nice fella I surrounded the call with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS so that other threads could jump in and do stuff. The problem is that real_foo() might call a previously defined callback (written in C) which in turn has to call a Python callback (what good is an extension module if you have to write your callbacks in C?), but when I try to PyEval_CallObject() it - KABOOM! Segfault somewhere in the interpreter (on PyFrame_New() or something like that). After a lot of thinking (and a lot of help from python-list) I figured out that the state I push away using Py_BEGIN_ALLOW_THREADS must be retrieved again before the callback is called, or PyFrame_New() won't like me. Problem is that that state variable is declared in Py_BEGIN_ALLOW_THREADS and there's no way to reach it (and it also includes a naughty { with the corresponding } in the Py_END_ALLOW_THREADS so it won't work to use them anyway). My solution: I wrote the two macros above. This works for me, but obviously I have to update my macros when the "real" macros in the Python distribution change. Okay, that's a more thorough explanation, I hope everybody are happy :) Martin -- Martin Sjögren martin@strakt.com ICQ : 41245059 Phone: +46 (0)31 405242 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html

Instead of asking us to add more macros, I recommend that you use the lower-level C API that is used by the macros. There's a bunch of ways to save and restore the state. The macros are just shortcuts (and a handy way to make all the code disappear when threads don't exist). --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, Aug 08, 2001 at 10:40:11AM -0400, Guido van Rossum wrote: [snip]
Well, that's exactly what I'm doing, isn't it? Martin -- Martin Sjögren martin@strakt.com ICQ : 41245059 Phone: +46 (0)31 405242 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html

[Guido]
[Martin Sjögren]
Well, that's exactly what I'm doing, isn't it?
You are using macros, yes <wink>. What Guido is telling you is that the set of macros we already have is adequate for everything we do, so it would just bloat our code base with stuff we never ever test if we were add additional stuff we never use. He's also telling you that you can write your macros in such a way that you don't have to change them across Python releases, and it was *that* complaint you seemed keenest about. That one you can solve on your own.
participants (3)
-
Guido van Rossum
-
Martin Sj�gren
-
Tim Peters